Building a model to read deal emails

Three iterations, from NLP to a multi-model service that finally reads deal emails the way an analyst does, with its eyes.

Key takeaways

This blog covers three rebuilds of the system that reads dataroom notifications, and how the sharpest version yet earns its accuracy by handing the model less to do.

• Rules memorize; models understand. When other people control your input format, pattern-matching is maintenance debt with no payoff date. Version one broke every time a dataroom changed its layout.
• A text-only model reads with its eyes closed. An email is a rendered document, so we render it and let the model look. Tables, columns and hierarchy that text-flattening destroys come straight back.
• Restraint beats capability. Every task we pulled out of the model made the service more accurate, cheaper and easier to trust. Deterministic code does deterministic work, which makes hallucinated links structurally impossible.
• It survives a bad provider day. An in-house client fails across vendors, clouds and regions, so “the LLM provider is down” stopped being a failure mode. Well under a cent per email, with results in seconds.

“Every day, financial datarooms fire off email notifications announcing deal events. Buried inside is exactly what a deal team needs. Miss it in the boilerplate, and a deadline slips.”

Introduction

Every day, financial datarooms fire off email notifications announcing deal events: a new deal added, documents uploaded, something changed. Buried inside is exactly what a deal team needs: the deal name, where documents live, who to contact, and the occasional note that actually matters (“commitments due Friday, 5pm ET”). Miss it in the boilerplate, and a deadline slips.

The Octus platform needs that information in structured form, immediately and at scale, so downstream systems can automatically fetch and index deal documents the moment they land. The catch: every dataroom renders its notifications as its own HTML — different layouts, different markup, redesigned without warning — with the two sentences that matter buried in walls of boilerplate.

We’ve rebuilt this system three times over the past few years. The version running today is the most accurate, and it gets there by asking the model to do less.

Model 1 (NLP): rules, patterns and a parser for every template

The original model was built on classical NLP and information extraction: regex, entity matching and hand-tuned heuristics layered with business logic for every dataroom template. Fast, free and deterministic when it worked, but it had a fatal flaw: it memorized the shape of emails instead of understanding their content.

Any layout redesign, new dataroom or minor phrasing tweak could break it silently, leaving room for the kind of accuracy drift that’s hard to catch. Every change on the dataroom side, whether a new template or a redesigned layout, meant rerunning our NLP analysis, re-deriving the patterns that drove extraction and adjusting heuristics accordingly, with real overhead and lag before the model caught up.

Model 2: the LLM pivot, robust but not quite there

To escape pattern-matching, we pivoted to a text-only LLM. As with all model usage at Octus, this runs within our SOC 2-compliant, AI-governed infrastructure: providers never retain or train on our data. The structural improvement was immediate: The LLM read content instead of relying on layout. Template changes and new datarooms were handled instantly, largely eliminating the accuracy drift Model 1 had left us exposed to, and closing the overhead gap of constant model updates.

However, a text-only LLM processing complex HTML falls short in key areas:

• Structural collapse: Flattening nested tables and multi-column layouts into text turns documents into word soup.
• Boilerplate blindness: It struggled to differentiate crucial deal notes from thick legalese.
• URL hallucination: Models routinely corrupted or confidently invented long, complex links.
• Cost and latency: Multiple calls for different extraction tasks drove up token usage and latency.

Ultimately, asking a model to understand a highly visual document with its eyes closed left real accuracy on the table.

Model 3: the multi-model approach, reading with eyes open

The current system is a rebuilt pipeline around one core insight: an email is a rendered document that happens to travel as HTML. A human analyst doesn’t read the markup; they look at the email. Now, so does the model.

Here’s the shape of it (with the specific machinery left deliberately vague):

• See it and read it. Every email is rendered in a headless browser into an image, exactly as a recipient would see it. The pipeline then gives the model both the parsed text and the visual rendering, simultaneously. Tables, columns, groupings, visual hierarchy: the structure that text-flattening destroys comes back. Document lists that Model 2 skipped are now simply… read off the page. And if the visual path ever fails, the system degrades gracefully to text-only rather than falling over.
• More than one model, each doing what it’s best at. Instead of betting everything on a single model, the pipeline orchestrates multiple models in tiers: a fast, inexpensive model handles the standard flow, with escalation paths for the cases that need more firepower. Different strengths, one pipeline. (Which models and how they hand off is part of the sauce.)
• Let deterministic code do deterministic work. The biggest philosophical shift from Model 2: we stopped asking the LLM to do things software does perfectly. Identifying which dataroom sent the email? Deterministic matching against reference data, no model needed, no model wanted. And the radioactive-URL problem is gone entirely, because the model never touches a real URL anymore: links are shielded from the model during extraction and re-attached programmatically afterward. Hallucinated links are now structurally impossible rather than merely unlikely.
• One shot, heavily cached. Classification and extraction collapsed into a single consolidated call, with the static instruction set cached so each email pays only for what’s unique about it. Fewer round-trips, lower latency and a token bill that dropped instead of growing.
• Show your work. The model now explains its classification reasoning, which gets logged alongside every extraction, so when something looks off, we can audit why it decided what it decided instead of staring at a black box. Every input email is also archived, continuously growing the regression corpus we test the next iteration against.

System evolution comparison

The safety net: one client, many clouds

None of this matters if the pipeline falls over the moment a provider has a bad day, and every provider has bad days. Rate limits, regional capacity crunches, transient 5xx errors, the occasional full-blown outage. A system processing deal flow can’t shrug and drop emails until things recover.

So under everything sits a piece of infrastructure we’re quietly proudest of: a unified LLM client, built in-house, that every model call in the pipeline goes through. It abstracts multiple frontier-model vendors across multiple clouds behind a single interface, which means the extraction model doesn’t know or care which provider is answering. What it buys us:

• Tiered failover, across vendors. Every stage has a primary and a secondary model, and they don’t have to share a provider or cloud. If the primary fails, the call re-routes transparently, and an outage becomes a latency blip rather than an incident.
• Retries that understand the error. Transient failures get retried with backoff; permanent ones (a malformed request will never succeed) don’t get retried at all.
• Regional failover. If a region runs out of capacity, the client rotates to another and re-establishes its cache there, so performance doesn’t dip after the switch.
• Built-in observability. Every call logs cost, latency and model identity. When we say the pipeline costs well under a cent per email, we get that number from a query.

The pipeline’s multi-model orchestration and this failover layer compose: the pipeline decides which model class a task deserves, and the client guarantees that decision survives contact with the real world. Between the two, “the LLM provider is down” has stopped being a failure mode we think about.

Implementation insight: JSON shielding

To execute link shielding securely, our deterministic code identifies URL patterns in the HTML DOM and replaces them with strict placeholders before passing text to the LLM. Reassembly looks like this:

Did it work?

On our benchmark data set, Model 3 processed every email without a single fatal failure. Extraction quality on the hard cases (dense document tables, ambiguous notifications, boilerplate-heavy bodies) is the best we’ve measured across all three versions, with a meaningful improvement of roughly 20% over Model 1. It also let us expand scope: newer email categories and datarooms that earlier versions couldn’t reliably handle are now in the pipeline. And it costs well under a cent per email, with results in seconds.

More telling than the benchmark: the failure modes we spent a year fighting in Model 2 are mostly gone as categories. Layout complexity is handled by vision. Link hallucination is architecturally impossible. Classification edge cases get reasoned about visibly instead of pattern-matched.

What we learned

• Rules memorize; models understand. If your input format is controlled by other people, pattern-matching is a maintenance debt with no payoff date.
• A text-only LLM reads with its eyes closed. If the source material is visual (and email absolutely is), multimodal is the difference between “usually right” and “right.”
• Restraint beats capability. Every task we moved out of the model made our overall AI service more accurate, cheaper and easier to trust. The model’s job is judgment. Knowing when to use it, and when not to, is as much the craft as building it in the first place.

The result is a service that reads a deal email the way a good analyst does: looks at it, understands what kind of email it is, pulls out exactly what matters, ignores the boilerplate and never, ever invents a link.

Across the series

Three problems, three teams, one pattern. Covenant similarity search reads a bond offering against the whole library. The extraction pipeline turns millions of raw pages into a clean index. This service reads deal emails the way an analyst does. None of them was won by reaching for the biggest model available.

Each earned its accuracy from the same moves. Structure the problem so the model only does what it is uniquely good at: the document’s own table of contents, a per-document pipeline, the rendered email. Encode what the experts know, from the 12 covenants that drive comparability to the discrepancy tiers that decide what to double-check. Then decide, deliberately, what the model is for and build everything else around that decision.

In every case the output can be traced. A covenant ranking explains which sections matched and how they were weighted. A document’s journey through the pipeline is observable end to end. An email’s classification is logged with the reasoning behind it. For work that feeds credit decisions, traceability is the requirement.

Every one of these systems asks the model for judgment, not labor. Knowing exactly where that line sits — and building around it — is the advantage.

This AI extraction service is part of Octus’ AI infrastructure and platform. The prompts, model orchestration details and grounding mechanics are our own, but the lessons are free.

More in the AI & Data Science Series

Blog Post

Aug 13, 2026

Finding the comparable: How Octus built covenant similarity search

Read More

Blog Post

Aug 21, 2026

Inside the document extraction pipeline behind CreditAI

Read More