Finding the comparable: How Octus built covenant similarity search

Key takeaways

This blog reviews how the Octus Data Science team trained a model to: 1) read a new bond offering the way a covenant analyst does; 2) then scour the entire debt documents library for its closest comparables in minutes.

• The comparable is the whole game: Before an analyst can say anything useful about a new covenant package, they need to know what it looks like. That answer used to live in memory and reading time. Neither scales with a library that grows every year.
• Two answers, minutes after the OM lands: A ranked list of comparable deals scored 0 to 100, plus a covenant-by-covenant breakdown that points straight at the provisions negotiated away from the market.
• Reading the document was the hard part: We used each offering’s own table of contents as a blueprint to isolate and structure the covenant text, because the scoring is only as good as what you feed it.
• Analyst judgment is wired into the math: Our covenant team named the 12 sections that actually drive comparability, and every ranking traces back to which sections matched and how they were weighted. Ask the model why, and it answers, with every ranking fully traceable.

_{Diagram titled Building covenant similarity search. Shows a pipeline: an incoming offering memorandum goes through document parsing with deterministic NLP, reaching about 95% extraction accuracy, then scoring and filtering that compares deals by note type, legal regime, and vintage. This produces a ranked comparables list with similarity scores and a covenant-level matrix scoring each deal against subsections, highlighting gaps between overall and section-level scores. Key covenants carry extra weight in the scoring, parallelized across worker processes. The system rests on a pre-structured debt documents library and analyst-set, revised covenant weights, tuned against an issuer-recurrence test.}

Introduction

A new high-yield bond comes to market. Before an analyst can say anything useful about its covenant package, they need context: which deals has this been drafted from? What’s the closest thing we’ve seen? Where do this issuer’s protections sit relative to comparable offerings?

Answering that has always meant recall and reading. An analyst thinks of a few deals that feel similar, pulls the offering memoranda, and works through them. It’s a good instinct applied to an impossible surface area. The Description of Notes section alone routinely runs over a hundred pages, and our debt documents library grows every year. The set of possible comparables expands faster than anyone can hold in their head.

So we built a model to do the recall, and to do it exhaustively: For every incoming offering memorandum, score it against every comparable document in the library and rank what comes back.

What it produces

Within minutes of an offering memorandum (OM) landing, an analyst gets two things.

1. A ranked list of comparable deals, each with a similarity score from 0 to 100, alongside metadata: Issuer, sponsor, counsel, auditor, launch date. A score of 100 would mean the two covenant packages are identical in language, terminology and provisions; 0 would mean they share nothing. Scores above 80 have become our team’s working rule of thumb for “very similar.” Because the metadata travels with the scores, an analyst can slice the list: show me the closest deals from this sponsor, or from this counsel.
2. A covenant-level breakdown. This is the part analysts use most. It’s a matrix: Every covenant subsection in the new deal, scored against the matching subsection in each comparable document. A deal might score 85 overall while its restricted payments covenant scores 42. That gap is the finding. It tells an analyst exactly where to start reading, and it surfaces the provisions that were negotiated away from the market rather than leaving them buried in a document that looks conventional in aggregate.

Notably, what we deliberately did not build was sentence-level document comparison. Our covenant analyst team already has redlining software they know and trust. They already had a diffing tool. What they lacked was a way to know which document to redline against.

Teaching a machine how to read an OM

Scoring is the easy part. The hard part is turning a several-hundred-page PDF into something structured enough to compare, and that is where the natural language processing work sits. It draws on language modeling to recognize how legal documents are constructed, information extraction to pull the relevant provisions out of them, and information retrieval to match and rank what’s been extracted.

The model uses the document’s own table of contents as its schema. It locates the table of contents, reads off the section inventory and uses it to find where the Description of Notes section begins in the body and, just as importantly, where the section after it begins, which marks the end. Everything between those two points is the covenant text we care about.

That sounds simple and mostly isn’t. Legal drafting is full of traps for a naive parser: A phrase like “as defined in the Description of Notes” looks identical to a section heading if you’re matching on text alone, and US offering memoranda frequently print their table of contents in two columns, which scrambles the reading order and lands the model on the wrong section boundary.  It reads surrounding context to discount cross-references and detects and unpicks the two-column layout. Once the section is isolated it gets split again into covenant subsections, working from typography and language rather than a fixed list. Iterating on these rules took subsection identification from roughly 80% accuracy to roughly 95%. The result is that every document in the library becomes a structured object, covenant heading mapped to covenant text, and that is the unit everything downstream compares.

Deterministic language processing is the right tool for this. The document declares its own structure in its table of contents, so the model is locating something already there rather than inferring it. That takes seconds per document, carries no per-document inference cost, and returns an identical answer every time it runs. Run it again and the result does not move, which is exactly what an auditable pipeline needs.

Comparing like with like

An offering memorandum often describes more than one set of notes. Senior notes and senior secured notes are typically described in separate sections; sometimes euro and dollar tranches, or fixed and floating rate notes, are split out the same way.

Scoring those together would be misleading. A senior secured note’s covenant package isn’t a meaningful comparable for an unsecured one. So the model detects each note type in the incoming document and produces its own ranked comparable set for each. Senior secured notes are scored only against other senior secured notes; senior notes against senior notes. Where the model genuinely can’t determine the type, the document is compared against both, rather than being silently misfiled.

The comparison pool is scoped in other ways too. We restrict it to documents drafted under comparable legal regimes, since drafting conventions differ enough across jurisdictions that the comparison stops meaning much. We restrict it by vintage, so the market conventions being compared are roughly contemporaneous. And where both a preliminary and a final OM exist for the same deal, only the final one is used.

Scoring, and where the expertise matters

We tested a range of similarity algorithms before settling on one. Several of the more conventional choices failed in instructive ways. Cosine similarity, for instance, clustered nearly every document above 90, which is useless for ranking; the effect comes from normalizing documents of wildly different lengths, which is exactly the situation when one covenant package runs twice as long as another. Choosing between the remaining candidates came down to testing them across different samples of the library rather than to any theoretical property, which meant we first needed a way to judge what a right answer looks like.

The scoring itself runs section by section. Every covenant section in the incoming deal is compared directly against its counterpart in the comparison document, producing a similarity score for each pairing. Those individual scores are then aggregated into a single document-level score, with the high-interest covenants carrying proportionally more of the result than the rest.

Here, s_w aggregates the scores across the high-interest covenants the two documents share, s_u aggregates everything else, and a is greater than b. Only covenants present in both documents contribute to the weighted term, so a provision absent from one doesn’t distort the comparison.

That yardstick was the harder problem, because there is no ground truth set of “documents that are similar” to tune against. So we used a proxy our covenant analyst team suggested: Issuers that come to market repeatedly tend to reuse their own documentation, so if the model is working, another offering from the same issuer should surface near the top. Testing against issuers with multiple offerings, Altice France, Ardagh and United, gave us something concrete to iterate against. We then generated results under a range of weightings until the rankings satisfied both that test and the team’s own assessment of what a good comparable looks like. The weights, in other words, weren’t chosen by intuition; they were tuned until the model agreed with the analysts.

That same yardstick is what settled the choice of similarity measure, and it remains how we sanity-check the model as the library grows.

Underneath all of it, the domain expertise is explicit, and that’s the point. Our covenant analyst team reviewed an exhaustive list of over one hundred covenant subsection headings and their drafting variations, and flagged 12 standardized titles as the ones that genuinely drive comparability, among them limitations on indebtedness and restricted payments. Those are the sections that carry the extra weight.

Every input here is a list a human wrote and can revise. When an analyst asks why a document ranked first, there’s an answer: Which sections matched, how strongly and how they were weighted. For work that feeds financial and legal analysis, traceability is the requirement, and every ranking here meets it.

Making it run

For each note type in an incoming document, scored against a library of N comparable documents across k covenant sections present in the incoming document, the model performs N x k similarity evaluations, one for every covenant section in the new deal against its counterpart in every comparable document, each one over passages of dense legal text.

At current library size that runs to tens of thousands of evaluations for a single incoming document. And the profile compounds: Per-deal cost grows linearly with the library, so cumulative work across a year of issuance scales with the square of it. This is a workload that gets harder precisely because the product is succeeding.

Two things keep it tractable. The comparison is parallelized, with the library partitioned across worker processes, which roughly halved runtime. And the covenant-level grid is surfaced only for documents that clear the similarity threshold. Analysts get the section-by-section detail where it’s informative, rather than a matrix spanning the entire library.

“This is what our AI framework looks like in practice. Every weight in the scoring model traces back to a decision a covenant analyst made, not a default a model assumed. When a document ranks first, there's an answer for why: which sections matched, how strongly, and who decided that mattered. That's not a constraint on the system. It's the design.”

Learn more about the Octus AI Framework

What we learned

The instinct with a problem like this is to reach for the most powerful available technique. What actually worked was a carefully chosen similarity measure, applied to text that had been carefully and correctly structured, weighted by priorities that domain experts set explicitly.

Most of the NLP work went into reading the documents properly. Most of the accuracy came from analysts telling us which covenants matter. And the result is a system whose every output can be traced back to the language on the page: An analyst can question any ranking and get a real answer instead of staring at a black box.