Method

How a run is produced, and what we will not do to make it look better.

Six rules apply to every benchmark on this site. They are the reason a number here is worth arguing with.

Evaluate your model

The six rules

Every benchmark on this site follows these.

01Held-out audio stays held out
The scored set is never published. A 60-clip development split per sector is open so anyone can calibrate against the same conventions without seeing the test.
02Two passes, then adjudication
Every reference transcript and every rubric score is produced twice, independently. A senior reviewer resolves disagreements and the resolution is recorded with the item.
03Runs, not a run
A single pass hides how much a score moves. Intervals are bootstrapped at 95% and printed next to the headline number wherever a run produces them.
04A human control on every board
Someone who does that job takes the same set, blind and timed, at market rate. It is a working standard, not a ceiling — the human line carries its own error bar.
05Rubrics come from operators
Each criterion is written by a practitioner and then rewritten until two strangers grading the same call land on the same score.
06Provenance travels with the artifact
Annotator, reviewer and adjudicator are attached to every item, so a disputed score can be traced to the people who set it.

Disclosure

What we publish, and what we will not.

The line is drawn so the boards stay useful without the test set leaking.

We publish

Leaderboards with confidence intervals, per sector and overall
The development split, and the conventions the references follow
The metric definitions, including what counts as an entity error
Notes on what failed and why, when a run is complete

We do not

The held-out audio, or transcripts of it
Contributor and reviewer identities
Customer-commissioned sets, ever — those belong to the customer
Scores for a named model we have not run ourselves

How a run is executed

Two tracks, one discipline.

The same rules, applied to what a model heard and to when it heard it.

PERIT-STT

Running speech-to-text.

Scored on word error rate — errors per hundred words against a double-passed human transcript, adjudicated by a third reviewer on disagreement.

01Audio is held out and never published. A 60-clip development split per sector is open for calibration.
02Reference transcripts are written by two annotators independently; a senior reviewer adjudicates every disagreement.
03Each clip is run 3 times per model. Reported intervals are bootstrapped at 95%.
04Human control is a working transcriptionist from the same sector, timed and paid at market rate.

See the speech-to-text boards

PERIT-ALIGN

Running word alignment.

Scored on timing accuracy @100 ms — share of words whose start and end both land within 100 ms of the reference timing.

01Every model gets the same audio. Speech-to-text models return word timestamps with their transcript; forced aligners place a known transcript in time.
02Returned words are matched to the reference words. Timing is scored on matched words, and coverage reports how many matched.
03Each boundary is compared on its own — start against start, end against end — and reported at 20, 50, 100 and 200 ms.
04Ranked by timing accuracy at 100 ms; ties are broken by mean boundary error.
05Latency is measured under concurrent load and is indicative only.

See the word alignment boards

The protocol comes first.

It is published in full so the format can be argued with before any score is read, not after it flatters someone. Held-out sets stay held out; the development split is open to every entrant.

Evaluate your model Request the development split