PERIT-S2S puts voice models into live calls with scripted callers who interrupt, change their mind and give the account number one digit at a time. Scored on whether the caller's problem was actually solved.
The caller's stated goal was met, verified against a rubric written by an operator from that sector.
End of caller speech to first audible token of the reply.
Interrupted mid-answer, the model stops, listens and picks up the new intent.
Each model’s mean across all 10 sectors. Open a sector below for the run behind it.
| Model | Task success (%) | Median turn latency | Barge-in recovery |
|---|---|---|---|
Human control · sector agentHuman control | 92.4 ± 1.7 | 605ms | 95.8% |
Frontier realtime · lab AFrontier | 61.5 ± 1.6 | 751ms | 67.8% |
Frontier realtime · lab BFrontier | 56.9 ± 1.6 | 884ms | 61.7% |
Cascaded stack (ASR → LLM → TTS)Specialist | 51.8 ± 1.5 | 1444ms | 44.7% |
Frontier multimodal · lab CFrontier | 51.0 ± 1.4 | 1006ms | 54.3% |
Open weights · 8B speechOpen weights | 29.7 ± 1.5 | 1150ms | 33.5% |
A refund request in retail and a claim in insurance share a shape. They do not share vocabulary, background noise, or what counts as a wrong answer.
Plan changes, SIM swaps and outage triage, mostly over mobile lines that drop mid-sentence.
Card disputes, KYC checks and payment failures, where a misheard digit is the whole error.
First-notice-of-loss calls taken from roadsides and hospital corridors, rarely from a quiet room.
Appointment intake, triage lines and medication questions, dense with drug names and dosages.
Order status, returns and refunds, with catalogue names that no general model has seen.
Rebookings during disruption — airport noise, urgency, and three changes in one turn.
Driver and dispatch calls from vehicles, warehouses and doorsteps. Half of it is hands-free.
Admissions, fees and course guidance, often with a parent and a student on the same line.
Benefits, permits and civic helplines, where callers read reference numbers aloud, badly.
Technicians on rooftops and in basements, talking through a fix with a machine running.