An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue

Chen, Tairan; He, Richard Yucheng; Liu, Yihang; Cao, Baodong; Xu, Chen

Full-duplex behavior is usually obtained by training on hundreds of thousands to millions of hours of speech. That is costly, rewrites the language model that supplies the system’s knowledge, and still leaves turn-taking either implicit or decided coarsely rather than word by word.

Awaken takes a frozen, general-purpose multimodal LLM, Qwen3-Omni, and turns it into a full-duplex dialogue system that decides turn-taking word by word: it backchannels while you talk, yields when you cut in, and takes the floor back.

On Full-Duplex-Bench and InteractionBench, the system is competitive with open-source full-duplex systems built through large-scale speech training. Among them, it is the least likely to take the floor when the user merely pauses, backchannels most often, and leads in spoken-task accuracy, groundedness, and coherence over long conversations.

▶Samples

Listen to Awaken

Three conversations with Awaken. Each sample is one continuous stereo recording, the user on the left channel and the agent on the right, so headphones let you hear both sides at once.

The strip in each player shows who is speaking when. Click it to jump.

01Method

How it works

We adapt, rather than retrain, a model that already understands speech and text. Starting from Qwen3-Omni, we keep its weights frozen and convert it into the thinking half of a full-duplex system. Only four things are trained: low-rank adapters, seventeen new token embeddings, an audio adapter that reads neural-codec tokens, and a channel-fusion layer. On a 12.5 Hz frame clock, the thinker reads the user’s speech and emits one token of the agent’s text per 80 ms frame. Pauses, yields, and onsets are tokens of the same vocabulary, and an explicit finite-state machine masks illegal tokens before sampling.

Two timelines with a user lane above an agent lane. (a) The agent says 'mm-hm' and 'yeah' while the user describes a job interview, then takes the turn after the user finishes. (b) The user cuts in while the agent is answering; the agent yields within 0.2 seconds and then answers the new question. — **Figure 1.** Two behaviors on a shared timeline (user above, agent below; schematic: timings and text are illustrative). **(a)** A free-running conversation: the agent backchannels (“mm-hm”, “yeah”) while the user is still talking and takes the turn when the user finishes, with no external turn detector. **(b)** A user interruption: the agent yields the floor when the user cuts in (silence-start token, `SS`) and takes it back to answer the new request (silence-end token, `SE`).

Four stage cards leading from a frozen, turn-based Qwen3-Omni to a full-duplex thinker. Each card shows the frame-level input and output cells, the trained modules, and the KL reference model of that stage; the talker turns the thinker's tokens into speech. — **Figure 2.** Overview. A frozen, general-purpose multimodal LLM (Qwen3-Omni), which only talks in turns, becomes the thinker of a full-duplex system through four training stages. Each stage card shows what the model reads and emits (one cell per frame: user codec tokens, agent words, pause, `SE`/`SS` and `PAD` cells), which modules it trains, and which model anchors its KL term. The frozen core is never changed. The resulting thinker emits agent text and timing tokens frame by frame, and the talker turns them into speech, so that the agent listens and speaks at the same time.

The work has three contributions.

A data-efficient recipe. It converts a frozen, general-purpose multimodal LLM into a word-level full-duplex dialogue model through four training stages. The stages total about 30 GPU-hours, instead of large-scale speech training.
A finite-state treatment of turn-taking. It keeps per-frame sampling within legal moves. Pause, yield, and onset are tokens of the model’s own vocabulary, predicted at every 80 ms frame, and a state machine restricts each draw to the tokens that are legal in the current state.
A thinker–talker interface made only of discrete tokens. As a result, the two halves are trained separately and in either order.

02Evaluation

Results

We evaluate on two external benchmarks of real-time behavior, Full-Duplex-Bench (v1.0 and v1.5) and InteractionBench, and compare with the published results of open-source full-duplex systems.

It waits through pauses. Of the open-source systems, ours is the least likely to take the floor when the user merely pauses: its turn-over rate on the synthetic subset of Full-Duplex-Bench v1.0 is 0.170, against 0.584 for the next-best system. It also has the highest backchannel frequency.
It reacts fastest. After a user interruption, it stops talking and starts its response sooner than Freeze-Omni and Moshi, and its response latency is the lowest in every scenario of Full-Duplex-Bench v1.5.
It stays grounded. On InteractionBench it is the strongest open-source system on groundedness, coherence, and spoken tasks; its groundedness is 53%, against 35% for Freeze-Omni.

The full tables, the analyses and the limitations are in the paper.

Citation

@misc{chen2026llm,
  title        = {An {LLM} Is All You Need: Data-Efficient Adaptation to
                  Full-Duplex, Word-Level Spoken Dialogue},
  author       = {Tairan Chen and Richard Yucheng He and Yihang Liu and
                  Baodong Cao and Chen Xu},
  year         = {2026},
  howpublished = {\url{https://github.com/avera-labs/AwakenFullDuplex-Qwen3Omni}}
}

License

Awaken’s code is released under the Apache License 2.0. The license of the model weights is pending; see the model card. Awaken builds on Qwen3-Omni (Apache 2.0), on Moshi and its Mimi codec (weights under CC-BY 4.0), and on PersonaPlex, from which the talker’s lineage begins; each keeps its own license. See NOTICE for details.