Models sound fluent. The question is whether the transcript holds up.
Two benchmarks: what a model heard, and when it heard it. Every model runs on the same audio under the same scoring, and every number on the boards is one we measured.
- 2
- Published benchmark tracks
- 23
- Speech models on the boards
- 12
- Providers compared
- 46
- Metrics reported across both boards
Two tracks: the words, and their timing.
Transcription is solved on clean audio. Nobody calls from clean audio.
PERIT-STT scores models on real support recordings — mobile lines, crosstalk, accents and the domain vocabulary that decides whether a transcript is usable downstream.
- 1qwen3-asr-flash-2026-02-101.53 1.33–1.76
- 2universal-3-5-pro1.55 1.36–1.78
- 3mai-transcribe-1.51.57 1.37–1.80
- 4transcribe-11.59 1.39–1.83
- 5qwen3-asr-1.7b1.60 1.40–1.84
The right word, a quarter-second late, is still the wrong caption.
PERIT-ALIGN scores the word-level timestamps a speech model returns — where each word starts and where it ends — against reference timings, so captions, redaction and audio search land on the right moment.
- 1Perit Golden Data Fine-tuned Model97.85
- 2transcribe-192.33
- 3voxtral-mini-transcribe85.04
- 4grok-stt-1.078.04
- 5qwen3-asr-0.6b77.41
A model drops a digit in an account number or mishears a drug name, the transcript still reads well, and the error only surfaces three systems downstream. So the transcription track reports what kind of error a model makes, not just how many.
A transcript can be word-perfect and still put every caption a quarter-second late, stop a redaction short of the account number, or send a search hit to the wrong second of the call. The alignment track scores where each word starts and ends, not only what it says.
Three ways speech usually gets measured, and why we do it differently.
None of these are wrong. They answer a different question from the one a team ships against.
An agent is put through thousands of generated calls and scored by another model.
Cheap and repeatable, and it inherits whatever the generator thinks a hard call is. We put models on recordings of calls that actually happened, and on live calls with people trained to derail them.
Listeners rank clips they have no stake in — naturalness, warmth, which voice they preferred.
That measures whether a voice is pleasant, not whether the account got unlocked. Our graders worked the desk the call came from, and score against a written rubric.
Clean studio audio, one speaker, scripted sentences, published years ago and long since trained on.
Held-out support audio: mobile lines, crosstalk, room noise, and the vocabulary that decides whether the transaction goes through.
How a run is produced, and the notes written along the way.
Method
Six rules every benchmark follows: held-out audio, two-pass references, runs not a run, a human control on every board, operator-written rubrics, provenance on every item.
What we publish, and what we will notField notes
Short notes on the decisions that change what a score means, written while building the benchmarks rather than after.
4 notes · method and positionPut a model on the board.
Pick a track. We run your model on the same audio, under the same scoring as every model already on the board, and publish it next to them.