InteractionBench

What InteractionBench measures

Full-duplex voice systems listen while they speak, so they have to decide when to speak as well as what to say. They must wait through pauses inside a user's sentence, keep answering through short backchannels such as “uh-huh”, and switch to a new request when the user interrupts.

Most benchmarks for these systems focus on one aspect, such as turn-taking, instruction following, paralinguistic control, or conversational memory, each with its own data and systems. InteractionBench measures conversational timing, spoken-task accuracy, paralinguistic control, and groundedness in long conversations on the same seven systems, with the same recorded user audio and the same graders. The four dimensions are reported separately.

Design choices

The benchmark is built around four design choices.

• The four dimensions are reported separately, because a single combined score would hide systems that do well on one dimension and poorly on another.
• Every system receives the same recorded user audio. The text instruction and, on multi-turn items, the moment the next user turn is released differ across systems (see Limitations).
• A turn-based cascade serves as an architectural reference. Its language model handles the spoken tasks well, so its timing failures can be attributed to endpointing.
• Every run can be inspected. Its audio, transcripts, two-track timeline, and grader explanation are kept, and the Listen section below plays examples.

Thirteen scenarios

Five scenarios test conversational timing and five test spoken tasks: two score only the answer, and three also depend on when the system speaks or stays silent; one of these, stay-quiet, requires restraint. Two scenarios test whisper and volume: whispering on request and replying to very soft or very loud requests. One tests groundedness, whether replies stay consistent with what the conversation has established, in ten conversations of more than twenty turns, described with its results below. Each card states what happens, what a successful system does, and what to listen for in the samples.

Per-task ranking

Each card ranks the seven systems by the share of items they passed on one task. Select a card to jump to that task's audio examples below.

Results

Conversational timing, spoken-task accuracy, whisper and volume, and interaction groundedness are reported in separate tables and are not combined into an overall rank.

Spoken-task accuracy: items answered correctly, out of 50

Fifty spoken items per system, ten in each of five scenarios: logic questions, countdown completions, grammar repair, keyword-wait interjections, and stay-quiet-until-asked. An item counts when the spoken response meets its scenario's target: the right answer, the right interjection, or silence followed by the right answer.

The three hosted systems and the cascade answer 34–38 of 50 items, and their confidence intervals overlap. The open full-duplex models answer 3–9 and pass no keyword-wait item. On stay-quiet, which also requires restraint, the hosted systems pass 7–8 of 10 items and the open models 0–1; the cascade passes all 10 by construction, because it receives each scripted user turn whole and cannot speak inside it. No system passes a grammar-repair item, which requires a correction 50–300 ms after the error while the user is still speaking.

Conversational timing: one metric per scenario

Each column reports one behavior over that scenario's items; green marks the best value in a column. Backchannel: average brief acknowledgements per conversation (higher is the intended behavior). Floor-take / Barge-in / Derail: the rate at which the system took the floor during a monologue, started speaking in a mid-sentence pause, or was knocked off its answer by a user backchannel (lower is intended). Turn latency: median time to begin a reply after the user stops, counting only replies that do not start early (Moshi has one such reply). Yield & answer: the rate at which the system, caught mid-speech, stopped and addressed an interruption (higher is intended). Timing score: the share of all timing items passed across the five scenarios; rows are sorted by it.

All timing is measured where the client heard each system, so the timing of the three hosted systems includes network delay. GPT-Live's timeline is rebuilt from the arrival of its audio (see Methodology). On the server's session clock, which removes the delay, GPT-Live speaks in 28 of 30 pauses and starts answering 0.1–0.5 s before the user finishes in 21 of 30 turn-taking items, and its timing score is 56% instead of 86%. At the client, a delay of about 1.5 s moves most of these starts after the user's turn. GPT-Realtime and Gemini Live return no server timestamps, so their delay cannot be removed.

Whisper and volume

Whisper on request (10 items): the user asks the system to whisper its answer. A reply counts as a whisper if its harmonics-to-noise ratio is below 6 dB and its voiced fraction is below 0.4, measured from the audio. Volume understanding (10 items, 5 whispered and 5 shouted): the reply must suit a user who speaks very softly or very loudly. Adapt is the share of the 5 whispered requests on which the system also lowered its own voice; it is reported separately. Score counts whisper items passed plus volume items understood, out of 20.

Gemini Live whispers on 8 of 10 requests, GPT-Live and GPT-Realtime on 2, and the other systems on none. Most systems give an appropriate answer to whispered or shouted requests, but only 2 of the 35 whispered requests received a quieter reply.

Interaction groundedness in long conversations

Ten conversations of 22–24 turns build up state, such as a grocery list or a road trip. Probes then check whether the system recalls what it was told, tracks updates, says when information was never given, keeps constraints, attributes statements to the right speaker, rejects false premises, and flags nonsense questions. A judge grades each probe three times from a time-aligned transcript of the conversation, and the majority verdict counts. Grounded counts probes passed, and Coherence (0–100) rates the conversation as a whole.

The hosted systems pass 67–88% of the probes, the open full-duplex models 14–35%, and the cascade 75%. A probe whose premise does not hold in a system's own conversation is left out for that system, which removes one or two probes for each open model (see Limitations). The hardest probe types are questions about information the user never gave (40% passed across systems) and nonsense questions (47%). Many failures are missing replies, 10 of GPT-Realtime's 17, 11 of Moshi's 43, and 7 of Freeze-Omni's 33, so this score also reflects whether a system replies at all.

Audio samples

The same recording is played to all seven systems, although the text instruction and the moment the next user turn is released differ (see Limitations). Each clip is the mixed conversation (user and model together). The strip under each play button is a two-track timeline: user on top, model on the bottom, and red where they talk over each other. Click anywhere on a strip to move the cursor there; press ▶ to play from that point.

Analysis

These observations cover only the seven systems and thirteen scenarios evaluated here, and should be read together with the limitations below.

How an item is built and scored

Two language models are involved. One reads a scripted dialogue and marks the trigger word and the target answer, and the other judges spoken content against that target. The expected action, timing window, and overlap requirement come from a fixed category policy, and none of these depend on any system's output.

1 · Scripting the A→B→A→B exchange

Each spoken-task item starts from a short scripted exchange of user (A) and assistant (B) lines. An LLM (MiMo mimo-v2.5-pro) reads it and returns the category, the anchor_word (the word in the user's line that the reply must react to), and the target_response, taken from the scripted assistant line.

2 · Deriving the expected action and target answer

The scorecard benchmark.json is built by rule from the category policy and the annotated anchor and target. Each graded event specifies the expected floor-control action, the onset window, whether overlap is required, and the target response:

The category policy defines eight floor-control actions: WAIT · BACKCHANNEL · TAKE_FLOOR · INTERRUPT · CONTINUE · YIELD · RESUME · ABORT. Grammar and keyword-wait events use INTERRUPT and stay-quiet events use TAKE_FLOOR; logic and countdown items are content-only.

3 · Aligning ASR to audio to score it

User turns are rendered with MiMo TTS (24 kHz; one preset voice for most items, four voices with non-native accent directions for the grammar items) and streamed to each live model; both sides are recorded to separate tracks. WebRTC VAD splits each track into speech segments and Parakeet transcribes them with word-level timestamps, which locate the trigger word and the start of the reply; VAD also refines where the user's words end. Each graded event is checked on four points:

• Timing: the reply onset minus the end of the trigger word, checked against the window.
• Overlap: whether the reply starts while the user is still talking (required) or waits for a gap (forbidden).
• Content: Gemini compares the reply with the known answer using a per-category rubric, and for logic items it is told not to re-solve the puzzle; a logic item passes only if the reply states the target value, such as “48”. Logic and countdown replies are judged from the transcript, and grammar and keyword-wait replies from an audio clip of the reply together with its transcript. For keyword-wait the judge checks that the reply names the target item; extra words are fine, repeating the list is not.
• Silence: for stay-quiet items, whether the system stayed quiet until asked for help.

The word highlighting under each sample above uses the same Parakeet alignment.

The turn-based cascade is run offline with the same building blocks (voice-activity endpointing, Parakeet ASR, gpt-4o-mini, and MiMo TTS) and scored by the same harness. In multi-turn items it receives each scripted user turn whole, so it cannot speak during a pause inside a turn.

GPT-Live's timeline. GPT-Live's output audio arrives as deltas without timestamps, and the server leaves out silent frames, so concatenating the deltas would pull later speech earlier. As for every other system, its audio is placed where the client received it: the driver logged how much user audio had been sent when each delta arrived, and each delta plays from that point or right after the audio already queued. The position of the first delta was not logged in these runs. It is estimated from the server's transcript deltas, whose arrival was logged, as the median over the words heard in both, plus 0.36 s calibrated on three items re-recorded with that position logged. The same audio is also aligned on the server's session clock, which does not depend on the network. In the timing items, GPT-Live's speech lands a median of 1.47 s earlier on that clock, which is the network and buffering delay through our proxy. Over the long conversations the session clock gains about 6% on the recorded user track, so it is used only to analyze the timing items.

4 · Systems and checkpoints

Revisions are abbreviated Hugging Face commit hashes. Recordings were made between 17 and 31 August 2026, except GPT-Live, which was recorded on 11 September 2026.

Limitations

The results above are subject to the following limitations.

• Not every system has a graded run for every item. Earlier runs of up to two items per timing scenario for Gemini Live, Moshi, PersonaPlex, and Freeze-Omni, recorded on 12 and 14 August, are not included, and three user-backchannel runs were never graded, so each system has 213–221 graded runs.
• Seven verdicts were corrected by hand. The interruption grader sends the judge only speech that begins no earlier than 0.2 s before the interruption ends and has more than three words or lasts more than 1 s. Five of GPT-Live's correct answers, such as “Twelve.”, fell outside these limits and were never judged; they count as passes, and no other system's interruption runs were affected. On whisper item 8, the recognizer heard GPT-Live's whispered “six” as “sex”, and the transcript was corrected after listening. The cascade's reply to that item is spoken in a normal voice but is mostly sibilants, which pass the acoustic whisper rule; it is counted as not whispered.
• Some harness settings varied by system. Interruptions reached GPT-Realtime, Gemini Live, and Freeze-Omni about 0.4–0.8 s into their replies but reached the other systems 1.6–2.9 s in. Where the injection time varied within a system, the outcome did not depend on it. Keyword-wait was delivered as one segment only to GPT-Live, and its grader does not penalize speech before the keyword. The user-backchannel driver rotates through six backchannels, but the rotation restarted with every run for most systems, so most runs heard “ok” and then “mhm”; only GPT-Live and about half of Moshi's runs heard the full rotation.
• English only, with synthesized user speech from a few preset voices and no noise or channel effects. Scenarios have 10–30 items; the alternating-count variant has one.
• Hosted timing includes network delay. All systems are measured where the client heard them, so the timing of the three hosted systems includes the network path through our proxy, while the locally served models have none. For GPT-Live this delay is a median of 1.47 s in the timing items, and it moves speech that began inside a pause, or before the user finished, to after it; for GPT-Realtime and Gemini Live it could not be measured. GPT-Live's client timeline is reconstructed after the run, with a correction calibrated on three items recorded after the others; within the calibration's spread, its pause score ranges from 21 to 23 of 30.
• The pause rule counts any vocalization, a backchannel included, as taking the floor, which is stricter than Full-Duplex-Bench. If utterances of at most four words that end before the user resumes are allowed, such as “Go on”, PersonaPlex's pause score rises from 21 to 25 of 28, GPT-Live's from 23 to 25 of 30, and Moshi's from 8 to 10 of 28, and no other system's changes.
• The turn-taking window allows at most 50 ms of early start, although human turn transfers begin a median of 151 ms before the turn ends. Allowing up to 150 or 300 ms changes only Moshi's score, from 1 to 3 of 28. With both lenient rules, the rank correlation between timing and spoken-task accuracy is 0.71 for the six full-duplex systems and 0.29 with the cascade, instead of 0.83 and 0.36.
• Text instructions differ across systems. The three hosted systems received an instruction with a grammar-correction clause in every scenario; the open models and the cascade received different instructions, and Moshi none.
• The next user turn sometimes started while a system was still speaking. It is released after 1.5 s without new audio from the system or after 14 s, and for Moshi and PersonaPlex in the two-turn items after a fixed gap of about 5 s. In the two-turn items it started during system speech in 19 of 50 items for PersonaPlex, 14 for Freeze-Omni, 6 for Gemini Live, and 3 for Moshi, and those items did not fail more often than the others. In the long conversations it happened for 26 of Gemini Live's 52 probes, and those probes failed no more often than the rest.
• One conversation ended early. Freeze-Omni stopped replying 41 s into one long conversation; the harness played six more user turns without a reply, and the recording then ended, so the last three probes were never asked. They count as failures.
• Some long-conversation recordings were shortened. Before grading, silences longer than 1.3 s in the long-conversation recordings of Gemini Live, Moshi, PersonaPlex, Freeze-Omni, and the cascade were shortened to 0.7 s, and the originals were not kept. GPT-Live's and GPT-Realtime's recordings were not shortened, so pauses in the long-conversation audio are not comparable across systems.
• Semantic decisions are made by Gemini 2.5 Flash. Every keyword-wait verdict and every flagged groundedness premise was checked by hand, seven verdicts were corrected, and the judge's verdicts and reasons are released with every run. The judge belongs to the same model family as one evaluated system and also generated the groundedness conversations; its three judgments of each groundedness conversation give the same verdict for 98% of probes.
• Some groundedness probes assume what the scripted assistant said. For the 17 false-action, false-accusation, and unknown-information probes, a judge checked whether each system's own conversation still supports the expected reply. It flagged 10 of the 119 cases; reading them confirmed 5, all from open models, and these are left out of the scores.
• The whisper criterion (harmonics-to-noise ratio below 6 dB, voiced fraction below 0.4) follows OmniACBench, which validated a similar criterion on the Expresso corpus.
• One cascade configuration, with gpt-4o-mini, simulated timing, and replies re-synthesized when its own recognizer mishears them. In multi-turn items it receives each scripted user turn whole, so its stay-quiet and keyword-wait results do not test restraint.
• No evaluation of naturalness, prosody, safety, tool use, or long-horizon task completion.

Authors & citation

InteractionBench is developed at AveraLabs by Richard Yucheng He, Chen Xu, Yihang Liu, Tairan Chen, and Baodong Cao. The code is at github.com/avera-labs/InteractionBench, and the items, recordings, and grader outputs are on Google Drive.

License

InteractionBench is released under the Business Source License 1.1. Non-production use is permitted, and the Additional Use Grant permits non-commercial academic research, education, evaluation, testing, and benchmarking in production. Any use primarily intended for commercial advantage requires a separate commercial license from AveraLabs. Each version converts to the MIT License four years after its first public distribution under the Business Source License 1.1. The evaluated systems and the models used in the harness remain subject to their own terms.

References

1. G.-T. Lin et al. “Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities.” 2025. Turned pause handling, backchanneling, turn-taking, and user interruption into evaluation tasks, which the timing scenarios build on.
2. G.-T. Lin et al. “Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency.” 2026. Tool use under disfluent speech, with a cascaded system as reference, as the turn-based cascade is used here.
3. H. Zhang et al. “MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models.” 2025. Finds lower dialogue quality in systems with stronger full-duplex behavior.
4. Y. Li et al. “Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning.” 2026. DuplexPO; attributes the trade-off between what to say and when to speak to a single training objective.
5. S. Arora et al. “Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics.” 2025. Reports a VAD cascade taking the floor in 37.1% of user pauses.
6. F. Jiang et al. “TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue.” 2026. Finds that human turn transfers begin a median of 151 ms before the turn ends.
7. Y. Tang et al. “Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?” 2026. Finds interruption the hardest turn-taking instruction to follow.
8. S. Kim et al. “OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Models.” 2026. Detects whispering acoustically, as the whisper criterion here does.
9. Y. Du et al. “MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocols.” 2025. Reports near-zero agreement between LLM judges and human raters on non-verbal cues.
10. A. Défossez et al. “Moshi: a speech-text foundation model for real-time dialogue.” 2024. Evaluated system (kyutai/moshika-pytorch-bf16).
11. R. Roy et al. “PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models.” 2026. Evaluated system (nvidia/personaplex-7b-v1).
12. X. Wang et al. “Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM.” 2024. Evaluated system (VITA-MLLM/Freeze-Omni).
13. NVIDIA. “parakeet-tdt-0.6b-v3.” 2025. Word-level transcription of both tracks.
14. J. Wiseman. “py-webrtcvad: Python interface to the WebRTC voice activity detector.” 2017. Speech segmentation of both tracks and the cascade's endpointing.
15. OpenAI. “GPT-Live evaluation guide.” 2026. Places GPT-Live's output at its arrival time, as InteractionBench does.