Static eval sets cannot interrupt. Our environments are live speech-to-speech calls with trained speakers playing callers who lose the thread, change their mind and read the account number one digit at a time — scored on whether the problem was actually solved.
The scenario families, signals and connection spec below are fixed and running. Teams shipping voice agents connect their model over the same interface, get a per-turn trace with audio on every call, and argue with the scoring — which is how the rubrics keep getting sharper. If your traffic looks like one of these sectors, the environment is ready for it; if it does not, a new sector takes three to four weeks.
Terminals, browsers and repos are cheap to instantiate. A voice environment needs a person on the other end of the line — which is why the shelf is empty, and why we are the ones filling it.
The environments market grew up around agents that type. Terminals, browsers and repos are cheap to instantiate and free to run a thousand times. A voice environment needs a person on the other end of the line, so almost nobody builds them.
Not in step forty of a plan. It talks over the caller, misses the correction, or answers the question that was abandoned two turns ago. None of that is reachable from a text transcript of a successful call.
The bottleneck is recruiting, training and paying people to play a caller who interrupts on cue, in the right accent, from the right room. That is the operation we already run for recording and grading.
Each sector ships with the same six families, tuned to its own vocabulary and policy. You weight the mix.
Latency is easy to publish and easy to game. We report it, but the environment is scored on task success first — and every score comes back as a trace you can argue with.
Did the caller's stated goal get met? Scored against a rubric written by an operator from that sector.
End of caller speech to first audible token. Reported as median and p95, not average.
On interruption: stop within 300ms, then act on the new intent rather than the old one.
When the model mishears, does it ask for the one missing token or restart the whole exchange?
Policy held under pressure — no promises the business does not make.
Rated by the same operators who take these calls. Neutral under pressure counts; cheerful does not.
caller: no no not that card — the other one, ending [overlap] four seven
Every failed call comes back with its audio attached. A score you cannot listen to is a number you cannot act on.
See our speech benchmarksA websocket in, audio frames out. If your model speaks, it can be scored — no SDK to adopt.
Ten sectors, six scenario families each. Weight them the way your traffic actually looks.
Personas are played by trained speakers, including the ones who talk over your agent.
Per-turn transcript, latency trace, rubric hits and misses, and the audio of every failed call.
Pick the sector and the scenario mix, connect over the websocket, and the first traces are back the same day. A new sector takes three to four weeks.