Awaken Full duplex

Listen to Awaken

Five conversations, each highlighting a different side of full-duplex interaction. In every sample a caller talks with Awaken in real time; the caller is on the left channel and Awaken on the right.

Building on Fun-Audio-Chat

Fun-Audio-Chat combines speech understanding and generation with a shared language backbone. Its Dual-Resolution Speech Representations let the backbone operate at 5 Hz while a speech refinement head generates tokens at 25 Hz. Each backbone step therefore corresponds to 200 ms and five speech tokens. This provides an efficient foundation for jointly generating text and speech. The original architecture, capabilities, and demonstrations are described on the Fun-Audio-Chat project website.

Awaken’s streaming structure

Awaken takes a continuous user-audio stream and generates aligned text and speech. A causal audio encoder converts incoming waveform segments into features. At each 200 ms step, these features are fused with the assistant’s text and speech history and passed into a shared language backbone. The backbone predicts the next text/control token at 5 Hz and provides hidden states to a speech refinement head, which produces five speech codes per step at 25 Hz, conditioned on the current text embedding and the preceding codes. A speech decoder converts those codes into the output waveform, and the generated text and codes become history for the next step. The system prompt and streaming role prefix are prefilled once into the backbone’s attention cache. Incoming audio continues to be processed while the assistant speaks.

Adapting the incoming-audio encoder

The starting audio encoder uses bidirectional attention within its input windows. Streaming requires features to advance as audio arrives, so we replace that attention with causal, windowed attention: each encoder position can attend to the current position and a bounded history, without attending to future encoder positions.

We adapt the encoder through feature distillation. A frozen copy of the original encoder serves as the teacher, and a student initialized from its weights learns with the causal mask. Both receive the same audio; the student matches the teacher’s features using mean-squared-error loss. The adaptation setup samples real and synthesized speech at a 70:30 ratio, with noise, reverberation, and loudness augmentation applied consistently to teacher and student inputs. The attention window covers 100 preceding encoder positions, approximately two seconds per layer. This adapted encoder initializes supervised training; during streaming inference, per-layer attention caches carry its recent history forward.

Advancing text and audio together

The sequence begins with a system prompt and a streaming role prefix, followed by a start marker. These are prefilling context, rather than repeated tokens at every time step. After that prefix, incoming user features, assistant text/control tokens, and assistant speech-code groups advance on the shared clock.

The text/control stream distinguishes listening, producing words, and finishing speech. PAD represents listening with the speech head inactive. A word token starts a speech segment, whose audio stream is initialized with BOS. Further word tokens accompany the generated speech. Once the words are complete, SIL allows the speech stream to finish voicing. Returning to PAD releases the segment, paired with EOS in the audio stream; subsequent idle audio groups use audio PAD. Text PAD and audio PAD belong to different token vocabularies.

The diagram shows the recorded text-token sequence “oh no.” during ongoing user speech. It does not imply that every word occupies exactly one frame or that a response has a fixed duration. Generated text and speech history feed subsequent steps, alongside newly arriving user audio. A user can therefore continue speaking through a brief acknowledgment or interrupt an ongoing response without waiting for a separate listening phase.

Relative to the Fun-Audio-Chat foundation, which also offers a full-duplex mode, the changes described here are the causally adapted incoming-audio encoder and this continuous streaming sequence with explicit listening, voicing-tail, and release states.

Composing data for continuous interaction

The streaming structure gives the model a way to represent continuous conversation. The training data must then demonstrate how to use that structure.

We organize our data by the contribution each source makes to the training mixture. The selected training pool contains approximately 8,454 conversation hours and 4,337 agent-audio hours, distributed across the three source categories below. Conversation hours measure the full shared timeline of each composed record, including pauses and overlap. Agent hours measure the annotated assistant-audio segments after interruption cutoffs, including pauses within those segments and excluding listening-only spans.

Shares are computed within each hours column. These totals describe the selected composed training pool before training-window truncation and replay; validation is excluded, and composed variants count as separate records. They are not counts of unique original recordings or voiced-only speech. Values are rounded.

Both SFT stages mix all three categories, with interaction-data replay in the second stage.

These categories describe the source material. Turn-taking and interruption behavior are taught primarily through how we compose utterances into conversations. A dialogue's timing is an explicit part of the training example.

Selecting conversational content

Podcast-style and conversational dialogues offer expressive delivery and natural timing, but not every exchange provides a useful target for an assistant. In our experiments, an emphasis on loosely structured casual conversation could weaken response accuracy and relevance, with more unsupported assertions and occasional incoherent replies. We therefore apply a content filter to this part of the corpus, favoring exchanges with substantive information, explanations, explicit reasoning, or problem-solving.

The filter scores cues in scene descriptions, speaker descriptions, and the opening transcript. It favors informational and reasoning content while reducing the share of exchanges dominated by small talk, banter, or context-dependent storytelling. This is a heuristic selection signal, rather than a factual-correctness check. It helps us retain conversational delivery while choosing more useful response targets.

From utterances to timed conversations

We place user and assistant utterances on a shared timeline, preserving the correspondence between the incoming waveform, assistant text, and assistant speech tokens. Composition recipes specify the spacing between utterances, the timing of speaker changes, and the response to a user interruption.

For a speaker change, the floor-transfer offset (FTO) is the next speaker’s onset minus the previous speaker’s voice end:

Positive FTO creates a gap; negative FTO creates overlap. We sample it over −0.6 to +1.0 seconds, using a truncated split-normal distribution centered at zero: 60% of draws select the positive side and 40% the negative side, with scale parameters 0.3715 s and 0.25 s, respectively. When the assistant utterance involved is at most two seconds long, a negative draw is resampled from the positive side. Consecutive utterances by the same speaker instead receive a uniform 0.3–0.6 s gap after the previous channel segment ends.

For a user interruption, let t_a be the assistant utterance’s start and D its original voice duration before interruption. We sample an interruption position p uniformly over 0.4–0.8 of that duration, placing the user’s onset at t_a + pD. Thus, the interruption begins 40–80% through the original assistant utterance, rather than at a fixed number of seconds.

The interruption reaction delay δ is the designed interval from that user onset to the assistant’s stop: assistant stop = user onset + δ. Its proposal distribution assigns 20% probability to exactly 0 s; the remaining 80% is sampled from a normal distribution with location 0.4 s and scale 0.5 s, truncated to (0, 2] s.

We accept an interruption placement only when pD + δ ≤ D − 2 s, so it cuts off at least two seconds of the original utterance. Position and delay are resampled jointly up to 32 times; if no placement satisfies the constraint, that candidate uses ordinary turn-taking. Consequently, accepted positions and delays are conditioned on utterance length and this constraint: the final examples need not retain the proposal’s uniform position distribution or exact 20% zero-delay share. Placements are represented on the shared 200 ms frame grid, with same-channel overlap guards. The interrupted assistant target ends at the composed stop.

Each composed record stores its event placements and timing choices. The data loader reconstructs the conversation from this record, making its timing reproducible. We keep related variants together when constructing train and validation splits.

Giving backchannels an initial foothold

A backchannel is a brief listener response, such as “yeah” or “mm-hmm,” that can occur while the user continues speaking. It should acknowledge the user without becoming a competing substantive turn.

We include synthesized conversations containing this behavior in the supervised mixture. They give the model initial examples of when a brief response can occur, how it fits into overlapping speech, and how the assistant can return to listening before giving a substantive reply.

These examples are part of the SFT mixture and prepare a starting policy for the later reinforcement learning stage, where conversational windows provide contexts in which the policy samples and improves its own behavior.

The training recipe

Our selected recipe combines two supervised stages with a subsequent backchannel-focused GRPO stage.

Stage 1: learning the conversation representation

We begin supervised training from Fun-Audio-Chat with the adapted incoming-audio encoder. The model learns the new streaming input pattern and the aligned text and speech targets in composed conversations.

The objective covers both response content and conversational transitions. Long listening spans occupy many positions, while the decisions to start, stop producing words, and release speech are relatively sparse. We therefore weight the output classes and explicitly supervise the important transition positions, allowing those decisions to contribute meaningfully to training.

In the selected run, stage 1 uses 64,000 composed records and an effective batch size of 64 for 1,000 optimizer steps. Training windows contain up to 900 frames, or 180 seconds. The initial learning rates are 1e-5 for the language backbone, 6e-5 for the speech refinement head, and 1e-5 for the incoming-audio encoder, with cosine decay.

Stage 2: continued adaptation with interaction-data replay

Before stage 2, we blend the compatible language-backbone and speech-head parameters equally between the stage-1 model and the base checkpoint. We retain the stage-1 audio encoder. This initialization is part of our selected approach to balancing the pretrained capabilities with continued conversational adaptation.

Stage 2 trains on the remaining composed records and replays the designated interaction-data sources from stage 1. This provides further conversational coverage while continuing to expose the model to the explicit interaction examples.

The stage-2 mixture contains 100,132 records, including 6,780 replayed records. We use the same 180-second window limit and effective batch size of 64. Learning rates decay from 5e-6 to 1e-6 for the language backbone, 5e-5 to 1e-5 for the speech head, and 1e-6 to 1e-7 for the incoming-audio encoder.

Both supervised stages use FP32 model weights with BF16 autocast. We track text, speech, and transition metrics alongside conversational and capability evaluations.

GRPO: developing brief listener responses

The SFT checkpoint establishes the conversational foundation, but its measured backchannel frequency remains low. We use Group Relative Policy Optimization (GRPO) to develop this behavior through sampled interactions.

Learning interaction decisions without unnecessarily changing knowledge

We want to improve when the assistant speaks and yields while preserving the knowledge and response capabilities developed through pretraining and supervised fine-tuning. Our recipe combines action-space projection with targeted token-space updates. Each reward signal is assigned only to the frames directly responsible for the behavior being evaluated. This focuses the update on the relevant decisions and limits changes to the model’s output distribution elsewhere.

In ordinary token space, the policy update uses the probability of the exact sampled vocabulary token—for example, “yeah.” A timing reward applied to that probability can also reinforce the particular wording, even when the scorer has not established that the words are correct. In the projected action space, we instead add together token probabilities according to the interaction decision they represent:

For speech activity, we compare W + S versus P: activate or maintain the speech stream, versus listen or release. For text production, we compare W versus S + P: produce another content token, versus stop producing words. The model still generates ordinary text/control tokens; projection changes the probabilities used for credit assignment, without adding a separate action head.

Projection alone leaves room for shortcuts. A model rewarded for brevity can learn to stop an intended sentence after a few words, satisfying a length budget while leaving an incomplete utterance. We therefore also retain token-space credit for backchannel attempts, alongside explicit checks at the word-ending and speech-release positions. The goal is a brief, complete listener response, rather than a longer reply that has simply been cut short. These checks remain behavioral heuristics, not a semantic correctness guarantee.

Backchannel contexts: when to acknowledge, and where to assign credit

We construct backchannel contexts from conversational windows in which the user holds the floor for at least five seconds. The model receives the preceding context and generates while the recorded user audio continues. The selected setup mixes opening-turn contexts with contexts later in a conversation, targeting half of the BC draws from each. User-span duration buckets provide coverage across shorter and longer speaking spans.

The scorer identifies assistant attempts within the user-speaking span and evaluates their timing, spacing, brevity, text ending, and release. In the selected run, a backchannel is limited to four words and less than two seconds of speech. Word count is distinct from the number of tokenizer tokens. Completion checks use sentence-ending punctuation and the transition through the voicing tail; an unfinished fragment or a direct word-to-release cutoff does not receive the same credit as a completed acknowledgment.

We apply each score at the position responsible for the behavior:

For token-space credit, shorter valid responses receive stronger scores. Longer attempts lose that credit, and sufficiently long attempts receive negative scores. Invalid short attempts have their positive text credit withheld. A moving expression-frequency rule also removes token-space credit from overused backchannel forms, reducing the incentive to repeat a single acknowledgment everywhere.

Each event is compared across the group of sampled continuations, or across pooled attempts for the corresponding backchannel signal. Its group-relative advantage is assigned only to the credited positions above. We do not distribute one undifferentiated backchannel score across every token in the continuation. Where a BC context includes the subsequent user-turn ending, we also evaluate the reply that follows.

Reply contexts: retaining substantive responses

Backchannel training must not teach the assistant to treat every opportunity to speak as a brief acknowledgment. We therefore include reply contexts where a substantive user turn ends and a longer assistant response is appropriate.

We select user-turn endings with a reference assistant response of more than six words nearby, excluding short user backchannels and cases where another substantive user turn immediately begins. The rollout starts from an assistant-idle position before the user finishes, with the conversation history available. It continues through the reply, giving the policy an opportunity to answer rather than only acknowledge. Reply-context rewards are applied only in projected action space.

Reply scoring focuses on two decisions:

The length score is negative at or below 75% of the reference word count, rises to its maximum at 90%, and stays flat above that. Replies still open at the rollout boundary are excluded from that length comparison. If the user begins a new substantive turn, credit is masked from that point onward. Reference length is a behavioral proxy: the scorer does not verify the answer’s meaning, so we continue to assess accuracy, groundedness, and knowledge separately.

Frame-level examples of credit assignment

The aligned examples below make the two reward spaces explicit. For the acknowledgment “Right.”, the onset frame receives both a projected speech-activity advantage, based on the combined probability of speaking, and a separate exact-token form advantage, based on the probability of the sampled word. For a reply that stops too short, the length signal lands on the first non-content frame and changes the projected stop-versus-continue decision there, rather than touching the preceding words.

Selected optimization settings

For each context, we sample 16 continuations and use clipped group-relative updates. Both projected decisions and generated words are regularized toward the fixed SFT reference. We use weaker constraints where new backchannel behavior is being learned and stronger constraints on the surrounding responses.

The class and word regularizers use sampled KL estimators. The BC word weight applies to identified attempt words; the BC action weight applies to the eligible span, including listening positions where the model decides whether to respond.

We update the top six language-backbone blocks, the final normalization, and the text output head, using FP32 optimizer master weights. Text embeddings, lower backbone blocks, incoming-audio encoder, fusion component, and speech head remain frozen. Audio is generated during rollout, but audio-code probabilities do not receive a direct GRPO policy-gradient loss. The policy at round 160 is the candidate discussed below.

Evaluation: listening, following, and knowing

A full-duplex model can learn to react while the user speaks and lose the thread, or keep its knowledge and never react. We measure three things side by side: listener behavior, following the conversation, and knowledge and reasoning.

We compare against our half-duplex starting point Fun-Audio-Chat-8B, its official full-duplex variant Fun-Audio-Chat-Duplex-8B (reported scores only), the open full-duplex models Moshi, Freeze-Omni, and PersonaPlex, and the hosted GPT-Realtime. External results come from the public InteractionBench results and the Fun-Audio-Chat technical report; we run Awaken and Fun-Audio-Chat-8B ourselves on the same items and pass rules.

Listener behavior: backchannels

Following the conversation: groundedness

Knowledge and reasoning

In the report’s own speech-output results, moving Fun-Audio-Chat to full duplex costs 14–24 points on Llama Questions, TriviaQA, and Web Questions. Awaken matches or beats the official duplex model and the other open full-duplex models on knowledge, and solves 90% of the logic puzzles against 30–40% for them. On spoken OpenBookQA, GRPO and SFT score within noise of each other (75.6 vs 76.2): the interaction training leaves reasoning intact.

Related work

DuplexPO and ASPIRin also separate when to speak from what to say during reinforcement learning, to avoid the reasoning and content losses that timing-focused training can cause. Our GRPO stage uses ASPIRin-style action projection, with event-specific credit and separate word-token regularization.

Conclusion

Careful data composition and staged training let a full-duplex model listen actively without giving up its intelligence. Awaken backchannels competitively with dedicated full-duplex models while largely keeping its grounding, knowledge, and reasoning. Listener-response variety and reply consistency are next.

References

1. Fun-Audio-Chat Technical Report. arXiv:2512.20156, 2025.
2. CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training. arXiv:2505.17589, 2025.
3. Moshi: a speech-text foundation model for real-time dialogue. arXiv:2410.00037, 2024.
4. Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM. arXiv:2411.00774, 2024.
5. NVIDIA PersonaPlex: Natural Conversational AI With Any Role and Voice. NVIDIA ADLR.
6. Introducing gpt-realtime. OpenAI, 2025.
7. InteractionBench. Avera Labs.
8. VoiceBench: Benchmarking LLM-Based Voice Assistants. arXiv:2410.17196, 2024.
9. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (OpenBookQA). EMNLP 2018.
10. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. ACL 2017.
11. Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM (Llama Questions). arXiv:2305.15255, 2023.
12. Semantic Parsing on Freebase from Question-Answer Pairs (WebQuestions). EMNLP 2013.
13. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO). arXiv:2402.03300, 2024.
14. Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning (DuplexPO). arXiv:2607.07148, 2026.
15. ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Models. arXiv:2604.10065, 2026.

Citation

@misc{awaken2026,
  title        = {Awaken: Efficient Full-Duplex Adaptation with Capability Retention},
  author       = {Yihang Liu and Richard Yucheng He and Tairan Chen and Baodong Cao and Chen Xu},
  year         = {2026},
  howpublished = {\url{https://github.com/avera-labs/AwakenFullDuplex}}
}

License

Awaken’s code and model weights are released under the Apache License 2.0, the same license as the base model Fun-Audio-Chat-8B. Awaken builds on Fun-Audio-Chat and CosyVoice (Apache 2.0) and the Qwen3 text backbone (Apache 2.0); its audio encoder follows Whisper’s encoder (MIT). See NOTICE for details.