Position14 Aug 2026
Why the score is task success, not latencyLatency is the easiest speech number to publish and the easiest to game. It belongs in the report, not at the top of it.
Read the noteWritten while building the benchmarks, not after. Run findings land here once the held-out sets close.
Latency is the easiest speech number to publish and the easiest to game. It belongs in the report, not at the top of it.
Read the noteIf two reviewers grading the same call disagree, the criterion is broken — not the reviewer.
Read the noteText benchmarks leak by being quoted. Audio leaks by being reused — the same speaker, the same room, the same script.
Read the notePublishing a human line at 100 makes the benchmark useless the moment a model gets close.
Read the note