Say “yeah” while another person is speaking and you have created a surprisingly difficult machine-learning problem.
Maybe the word means, “I am listening, please continue.” Maybe it means, “Yes, exactly, and I want to add something.” Or maybe it means, “No, stop, I disagree.” The first is a backchannel. The second is a cooperative interruption. The third is a competitive interruption. At the instant the sound begins, all three can look nearly identical.
A human conversation partner rarely needs the labels. They hear the pitch, the force of the onset, the unfinished sentence, the direction of the argument, and the tiny history between two people. Somehow, usually without noticing, they decide whether to continue speaking or surrender the floor.
A voice agent must make the same decision, except it has to turn that vague social judgment into a timestamp.
That is the problem behind TurnBench, a new benchmark from Sesame AI and collaborators for evaluating turn-taking in spoken dialogue. It is easy to describe it as a benchmark for end-of-turn and interruption detection. The more interesting reading is this: after years of making machines sound human, we have reached the harder problem of teaching them when they are socially allowed to use that voice.
The 200 millisecond magic trick
Human conversation moves too quickly to be purely reactive. The typical gap between turns is around 200 milliseconds, yet planning even a short spoken response takes well over 600 milliseconds. If people waited to hear a speaker finish, recognized the ending, planned an answer, and only then opened their mouth, ordinary conversation would contain long, mechanical silences.
It does not. We start preparing while the other person is still talking.
This is an old insight. In 1974, Sacks, Schegloff, and Jefferson described conversation as a locally managed system in which speakers continuously negotiate the floor. They introduced the idea of a transition-relevance place, or TRP: a point where a speaker change would be structurally legitimate. A TRP is not simply silence. Syntax may be complete, intonation may sound final, and the current action, a question, story, complaint, or instruction, may finally invite a response.
TurnBench makes the prediction argument unusually concrete. In its human conversations, the incoming speaker begins a median 281 milliseconds before the outgoing speaker has completely finished. Remove floor-taking interruptions and the median is still 151 milliseconds early. Humans are not waiting for a clean boundary. They are forecasting one.
For decades, voice systems often used a simpler rule. Run voice activity detection, notice that speech became silence, wait perhaps 500 or 800 milliseconds, then declare the turn complete. This works until a person says, “The address is…” and pauses to remember it. Silence is acoustically obvious. Whether the silence yields the floor is not.
The same mistake appears in reverse during interruptions. Detecting listener speech is easy. Deciding whether an “mm-hm” is support or a bid for control is a prediction about intent. Prosody helps, words help, syntax helps, and context helps. Earlier work found systematic prosodic cues for backchannels, while studies of turn exchange identified additive acoustic, prosodic, and lexico-syntactic cues. None is a magic flag. The evidence accumulates.
TurnBench changes the object being measured
TurnBench’s real contribution is not raw scale. Switchboard contains roughly 260 hours of telephone conversation and Fisher contains far more. TurnBench contains about 30 hours: 154 two-person English conversations, recorded from 106 voice actors in 53 pairs. Its advantage is that every dialogue has isolated audio channels and three independent annotations under a common, conversation-analysis-grounded scheme. It also releases a separate 104-hour hand-labelled training set.
The annotators begin with 17 fine-grained behaviours, then map them into seven canonical categories. A turn means somebody holds the floor. An interruption means the listener enters mid-turn and successfully takes it. A backchannel acknowledges the speaker without claiming the floor. Crucially, the taxonomy also preserves a non-floor-taking interruption: somebody tried to enter, but the other speaker did not yield.
That last label exposes the causal difficulty beautifully. When Speaker B begins saying “but”, the model cannot yet know whether B will take the floor. If A yields, the event becomes a successful interruption. If A continues and B retreats, it becomes a failed attempt. The final label depends partly on the future, but a real-time model must act before that future exists. TurnBench therefore excludes non-floor-taking attempts from the interruption negatives. Punishing a model for firing there would punish it for not seeing the future.
The corpus also refuses to pretend that there is one distribution called conversation. It covers six interaction styles: casual, task-oriented, instructional, collaborative, argumentative, and narrative. Participants receive a style and a starting context, but are not ordered to interrupt or backchannel at particular rates. The behaviour is allowed to emerge.
This turns conversational style into an experimental variable. Argumentative speech in the corpus has 3.09 backchannels per minute and 2.48 interruptions per minute. Casual speech has 5.20 backchannels and 2.05 interruptions per minute. Collaborative dialogue has the most overlap. Instructional dialogue has longer turns and relatively little overlap. The difficulty of turn-taking changes even when the language remains English and the microphones remain identical.
From this common annotation, TurnBench constructs two tasks. End-of-turn detection asks whether a speaker has truly yielded; the hard negatives are mid-turn pauses. Interruption detection asks whether a listener has taken the floor; the hard negatives are backchannels and non-content sounds. The symmetry is the point:
listener speech ≠ floor takeover
A benchmark for decisions, not beautiful probability curves
Most benchmarks let a model emit scores and allow the evaluator to search for a convenient threshold later. TurnBench asks for committed timestamps. At 32.41 seconds, did the system behave as though the turn had ended? That operating point is what a deployed agent eventually needs.
For a gold event at time t, the scorer searches for a prediction in:
It reports recall, false-positive rate, and signed latency:
A negative latency means anticipation. A positive one means the system waited. Baseline thresholds are selected on the development set for the highest recall under a 0.10 false-positive budget, frozen, and then evaluated on the held-out test set. The leaderboard permits up to 0.15 FPR before placing an entry below qualifying systems.
The results form a clean frontier rather than producing one magical winner.
| System | EOT recall | EOT FPR | Median latency |
|---|---|---|---|
| RMS energy VAD | 0.718 | 0.632 | −117 ms |
| OpenAI Server VAD | 0.955 | 0.525 | 282 ms |
| Voice Activity Projection | 0.845 | 0.055 | 368 ms |
| Gemini 3.1 Live | 0.657 | 0.022 | 1,234 ms |
The RMS detector appears superhuman at first. Its median end-of-turn decision arrives 117 milliseconds early. It achieves that speed by firing on the wrong thing constantly: its FPR is 0.632. OpenAI Server VAD reaches 0.955 recall at 282 milliseconds, but its 0.525 FPR places it in the same overeager corner.
Gemini 3.1 Live sits at the other extreme. Its EOT FPR is only 0.022, but recall falls to 0.657 and median latency grows to 1.234 seconds. It avoids barging in partly by being cautious. This is not a direct comparison of internal detector speed: Gemini exposes no EOT head, so TurnBench infers commitment from the onset of generated speech. Its latency therefore includes response generation and network delay.
The strongest qualifying compromise is Voice Activity Projection, or VAP. It reaches 0.845 EOT recall at 0.055 FPR and 368 milliseconds. On interruption detection it reaches 0.945 recall at 0.107 FPR, but takes 994 milliseconds to commit. By then, a human exchange has moved an eternity.
VAP winning is conceptually satisfying. It is not merely a better detector of current speech. It predicts the joint future voice activity of both speakers over a two-second window. Its target is closer to:
than to P(speech at t). In other words, the best baseline is the one explicitly trained to imagine what the conversation will look like next.
Why casual conversation is the nightmare mode
The obvious guess is that argumentative dialogue should be hardest. People compete, overlap, and forcefully take the floor. TurnBench instead finds that interruption false positives are highest in casual conversation. Every evaluated model has a higher interruption FPR on casual dialogue than on argumentative dialogue.
The reason is not difficult language. It is dense social signalling. Casual conversations contain “yeah”, “right”, “oh”, laughter, continuers, and small noises whose job is to keep the other person talking. Machines repeatedly interpret evidence of listening as a demand to speak.
This matters beyond one leaderboard. Talking Turns had already shown that audio foundation models struggle to understand, predict, and perform turn-taking, while Full-Duplex-Bench tested pause handling, backchannels, smooth transitions, and interruptions through interactive scenarios. TurnBench does not invent the field. Its sharper move is to derive both endpoint and interruption evaluation from one linguistically grounded annotation layer, then vary the interaction style itself.
That also hints at the next failure. TurnBench is English-only, studio-recorded, and dyadic. Real agents encounter echo, packet loss, televisions, traffic, several people in one room, and timing norms that vary across languages. Cross-linguistic work finds a broad human preference for minimizing gaps and overlap, but also meaningful variation in response timing. A model calibrated to one conversational culture can feel sluggish in another and aggressive in a third.
The benchmark is useful precisely because it is incomplete
TurnBench should not be read as a universal naturalness score. Its FPR is event-anchored: the scorer examines designated hard negatives such as mid-turn pauses and backchannels. A bizarre prediction elsewhere on the timeline can be invisible. Thus an FPR of 0.05 means that five percent of the scored negative sites triggered a false decision, not that the system produces only five percent false alarms in general.
Causality is also affirmed rather than technically enforced. Submitters must timestamp a decision using the latest audio it depended on, including any lookahead, but the external scorer cannot inspect a private inference pipeline. Majority consensus improves label reliability, yet events without agreement disappear. The paper itself notes that disagreement carries signal. Ambiguity is not always annotation noise; sometimes ambiguity is the phenomenon.
Finally, the benchmark cannot yet score full-duplex models on interruption in the same clean way. A model that speaks and listens simultaneously changes the acoustic scene it is being evaluated inside. Solving that requires evaluating interaction, not merely replaying two human channels.
These caveats do not weaken the paper’s central result. They locate it. TurnBench is a carefully designed probe for two specific social decisions, not a complete simulator of conversation.
The agent must begin thinking before it is its turn
The larger lesson is that turn-taking is a partially observable control problem. At every moment, the agent chooses among actions such as listen, backchannel, take the floor, hold it, or yield. It observes audio, words, prosody, timing, and dialogue history, but the true intention of the other speaker is hidden.
Waiting provides evidence. Waiting also creates awkwardness. The policy is balancing something like:
VAD minimizes the last term and pays heavily on the first. Conservative semantic systems reverse the bargain. Humans appear to use enough syntax, prosody, semantics, shared context, and social expectation to act before certainty exists.
This points toward anticipation rather than faster reaction. A 2026 paper called Endpoint Anticipation predicts an end of turn as far as 2.56 seconds ahead and speculatively starts the downstream language-model and text-to-speech pipeline. If the user continues, the partial computation is discarded. If the boundary arrives, the system has already begun its answer. Integrated into the Unmute pipeline, the method reduced average latency by 505 milliseconds, from 1,195 to 690 milliseconds, at a reported 28.4 percent redundant-computation cost.
That architecture is strangely human. Do not wait for the other person to finish, then begin thinking. Predict where they are going, prepare quietly, and commit only when the conversational floor becomes yours.
We have spent years making AI voices smoother, warmer, and more expressive. TurnBench shows why that is not enough. A perfect voice that cuts you off still feels artificial. A brilliant answer delivered a second too late still feels like a database returning a result.
The next leap in voice AI may not come from teaching the machine a better voice. It may come from teaching it what one tiny “yeah” means before the word has even finished.
References and further reading
- F. Jiang et al., “TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue,” 2026. arXiv:2608.25218
- Sesame AI, “TurnBench: measuring the timing of conversation,” 2026. Sesame AI
- H. Sacks, E. A. Schegloff, and G. Jefferson, “A Simplest Systematics for the Organization of Turn-Taking for Conversation,” Language, 1974. DOI
- S. C. Levinson and F. Torreira, “Timing in turn-taking and its implications for processing models of language,” Frontiers in Psychology, 2015. DOI
- N. Ward and W. Tsukahara, “Prosodic features which cue back-channel responses in English and Japanese,” Journal of Pragmatics, 2000. DOI
- A. Gravano and J. Hirschberg, “Turn-taking cues in task-oriented dialogue,” Computer Speech & Language, 2011. DOI
- E. Ekstedt and G. Skantze, “Voice Activity Projection: Self-supervised Learning of Turn-taking Events,” Interspeech, 2022. DOI
- S. Arora et al., “Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics,” ICLR, 2025. arXiv:2503.01174
- G.-T. Lin et al., “Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities,” ASRU, 2025. arXiv:2503.04721
- T. Stivers et al., “Universals and cultural variation in turn-taking in conversation,” PNAS, 2009. DOI
- S. Udupa, S. Watanabe, P. Schwarz, and J. Cernocky, “Endpoint Anticipation for Low-Latency Spoken Dialogue,” 2026. arXiv:2606.13450
- Sesame AI Labs, “TurnBench scorer and submission protocol,” 2026. GitHub
