Solving the ASR Accuracy Gap in Due Diligence

Motivation

Call recordings are Junior’s primary source of data. Clients use these calls (and the research outputs built from them) to guide commercial due diligence and important investment decisions. Data quality is therefore critical. Today’s state-of-the-art transcription systems still fall short of the accuracy these workflows require. Errors can introduce false claims, poison downstream research, and lose client trust.

We benchmarked major ASR models, transcription providers, and consumer transcription apps on a collection of real CDD calls. These calls are especially difficult: they contain domain-specific acronyms, terminology, and company names. For each human-verified transcript, we label key entities and facts, then use LLM-based evaluation to determine whether those items were transcribed correctly in the machine-generated transcript.

Across this benchmark, leading systems scored only 50–75% on these entity and key fact accuracy.

LLM-based correction agents can substantially improve flawed outputs post-transcription. Given enough context, they can reason about phonetic similarity and resolve likely misspellings. But their effectiveness varies significantly across languages and accents, and some transcription errors are fundamentally unrecoverable.

For example:

“We should check with agencies” → “We should check ACs”

Here, “ACs” refers to associate consultants. The incorrect output is still a plausible English phrase, so a downstream model may have little reliable signal to recover the correct meaning.

“We experimented with malware” → “We experimented with Milvus”

Milvus is a high performance vector database (used by Junior). Even if we identify that malware doesn’t fit the context, it may be difficult to recover the correct entity reliably without the original audio.

Parameter maxxing

The highest-ranked models on the Hugging Face ASR leaderboard typically contain 1–10B parameters. For example, Qwen3-ASR-1.7B, ranked #2 on Hugging Face’s private benchmark, has 2B parameters in total, including a 300M-parameter audio encoder.

Our intuition was that models of this size do not contain enough world or domain knowledge to reliably recognize less-common entities. When they do not recognize an entity, they are more likely to output a phonetically similar word that fits the sentence.

We tested this across models of different sizes. The difference was obvious. Take Turbopuffer, a vector database startup used by Cursor, Notion, and Linear:

• Qwen/Qwen3.5-4B (4B params) suggested it could be a puffer-jacket company or possibly a Pokemon.
• Qwen3.5-397B-A17B (403B params) correctly identified it as a vector database company and gave detailed background on it.

The takeaway: if a smaller LLM cannot identify a reasonably well-known company when asked directly, it is unlikely to recover the right spelling while decoding noisy audio.

Context maxxing

Many ASR models fall short through an additional mechanism: poor context windows.

OpenAI’s Whisper, for example, transcribes audio in 30-second chunks and only sees a few hundred text tokens from the preceding chunk. It’s essentially blind to the broader conversation and cannot leverage it to resolve ambiguity.

This causes problems like:

• A speaker mentions a muffled company name near the beginning of a call, then says it clearly later. Whisper transcribes each mention independently, often producing several different spellings of the same name.
• The call introduction provides useful industry context, but that context is no longer available when the model later encounters unfamiliar acronyms or terminology.

Modern LLMs can work with context windows of up to 1M tokens. Qwen’s audio encoder produces roughly 45,000 tokens per hour of audio, well within those limits. Leveraging larger context windows should therefore be feasible for transcription.

Implementation

We used large multimodal models to tackle both problems: their world knowledge helps with unfamiliar names, and their longer context helps resolve ambiguous speech using clues from the rest of the call.

That improved accuracy, but introduced a risk. Hallucinations were rare, yet dangerous: a plausible sentence that was never spoken could make its way into a client’s research. We iterated on the harness and added guardrails that detect likely hallucinations and fall back to other models when needed.

With those changes, entity accuracy on our benchmark rose from 75% to 97%.

The remaining 3% are the hard cases. This is an area of active research that we’re working on daily, and we’ll share more as the approach matures.

If you are interested in working on solving problems such as this example, we’re hiring.

As a bonus, the same approach improved diarization. By using both audio cues and conversation context to identify speaker changes, we saw substantial gains over a traditional pipeline of speech detection, voice embeddings, and clustering.