PERIT-S2S · Retail & e-commerce

Order status, returns and refunds, with catalogue names that no general model has seen.

Audio comes from order support, returns desks, seller helplines. Scored on task success, with a working retail & e-commerce operator as the control.

1,340
Clips in the target held-out set
92h
Audio, never published
8
Languages in the set
1
Working retail & e-commerce operator as control

Leaderboard

Illustrative figures. Public runs begin when the held-out sets close.
ModelTask success (%)Median turn latencyBarge-in recovery
Human control · sector agentHuman control
91.1 ± 2.1
528ms94.5%
Frontier realtime · lab AFrontier
67.9 ± 2.0
763ms63.6%
Cascaded stack (ASR → LLM → TTS)Specialist
53.7 ± 2.1
1473ms39.6%
Frontier multimodal · lab CFrontier
52.6 ± 1.5
1002ms48.6%
Frontier realtime · lab BFrontier
52.0 ± 1.4
764ms67.3%
Open weights · 8B speechOpen weights
26.5 ± 2.3
1177ms28.4%
Task families

What the set actually asks for.

  • Turn-taking & barge-in
  • Task completion
  • Instruction adherence
  • Disambiguation & repair
  • Escalation & refusal
  • Tone under pressure
Method & limits

How a run on this set is produced.

  1. 01Callers are scripted personas played by trained speakers, not synthesised — including the ones who talk over the agent.
  2. 02Every call is scored against the rubric by two reviewers; a third adjudicates splits.
  3. 038 calls per persona per model. Reported intervals are bootstrapped at 95%.
  4. 04Human control is a licensed agent from that sector handling the same personas blind.

Limits, stated up front: 8 languages only, single-channel audio, and no cross-sector transfer. The control is one operator per sector, so the human line carries its own error bar — it is a working standard, not a ceiling.