---
title: "Privacy-Preserving End-to-End Full-Duplex Speech Dialogue Models"
authors: ["Nikita", "Tao", "Jiajun", "Yingke", "Tristan", "Tianxiang", "Simon", "Kong Aik", "Eng Siong", "Nanyang Technological University, Singapore", "A*STAR, Singapore", "Huawei Leibniz Research Center", "The Chinese University of Hong Kong", "The Hong Kong Polytechnic University"]
url: "https://arxiv.org/abs/2603.08179"
sections: 19
estimated_tokens: "6.1k"
---

## Contents
- keywords:
- 1 Introduction
- 2 Related Work
  - 2.1 E2E speech dialogue language models
  - 2.2 Speaker anonymization
  - 2.3 Privacy in neural speech and language representations
- 3 Methodology
  - 3.1 E2E full-duplex dialogue models
  - 3.2 User encoder variants
  - 3.3 Anonymization setups
    - 3.3.1 Anon-W2W: Wave-to-wave anonymization
    - 3.3.2 Anon-W2F: Wave-to-feature anonymization
  - 3.4 Hidden state extraction
  - 3.5 Setup
- 4 Results
  - 4.1 Ablation studies
- 5 Conclusion
- 6 Generative AI Use Disclosure
- References

## Abstract

Abstract End-to-end full-duplex speech models feed user audio through an always-on LLM
backbone, yet the speaker privacy implications of their hidden representations
remain unexamined.
Following the VoicePrivacy 2024 protocol with a lazy-informed
attacker, we show that the hidden states of SALM-Duplex and Moshi leak
substantial speaker identity across all transformer layers.
Layer-wise and turn-wise analyses reveal that leakage persists across all
layers, with SALM-Duplex showing stronger leakage in early layers while Moshi
leaks uniformly, and that Linkability rises sharply within the first few turns.
We propose two streaming anonymization setups using Stream-Voice-Anon: a
waveform-level front-end (Anon-W2W) and a feature-domain replacement
(Anon-W2F).
Anon-W2F raises EER by over 3.5 × \times relative to the discrete encoder
baseline (11.2% → \rightarrow 41.0%), approaching the 50% random-chance ceiling, while
Anon-W2W retains 78–93% of baseline sBERT across setups with
sub-second response latency (FRL under 0.8 s). 1 1 1 Demo page: https://anonymous-569230593.github.io/

###### keywords:

## 1 Introduction

End-to-end (E2E) full-duplex speech dialogue systems represent a fundamental shift
from turn-taking interaction to always-on, simultaneous listening and speaking.
Models such as SALM-Duplex [salmduplex2025], Moshi [defossez2024moshi],
and SyncLLM [emnlp2024synclm] route raw user audio continuously through a
decoder-only LLM backbone, computing hidden state representations at every
transformer layer throughout the entire conversation.
Unlike cascaded systems where user and agent streams are processed separately,
the LLM core of these full-duplex systems maintains a persistent internal state
over the user's speech stream, capturing their voice, speaking style, and identity.

Figure: Figure 1: Speaker privacy ($1-\mathrm{Linkability}$) vs. dialogue turn count. Only discrete encoder variants are shown; continuous encoder Linkability is omitted for clarity. Red lines: no anonymization; green lines: post-anonymization. Without anonymization, both systems drop into the low-privacy zone within a few turns; anonymization lifts privacy into the protected zone, though Moshi + W2W shows gradual degradation over dialogue length.

This creates a privacy exposure that remains largely unexplored.
Under GDPR and similar regulations, the mere presence of identifiable speaker
information in model representations constitutes a compliance risk, regardless
of whether external access is exploited, making it essential to audit and
mitigate such leakage proactively.
Probing studies on self-supervised speech models (e.g., wav2vec 2.0, HuBERT,
WavLM) have established that their hidden representations strongly encode
speaker identity [pasad2021layerwise, yang2021superb, chiu2025probing].
In the text domain, prior work has shown that LLM hidden states can leak
demographic and content-level user attributes [staab2024beyond], and
membership inference probes have exposed privacy risks in masked language
models [mireshghallah2022quantifying].
Our focus is distinct: we investigate *speaker identity* leakage, the
persistent encoding of *who is speaking*, which enables re-identification
regardless of conversational content.
Yet, to the best of our knowledge, no prior work has examined whether these always-on LLM hidden states retain sufficient speaker identity to enable re-identification.

Figure: Figure 2: Overview of the original SALM-Duplex pipeline and proposed anonymization setups. The main diagram shows the ASR-based encoder baseline: an ECAPA-TDNN probe attached to the LLM's hidden states (red dashed path) reveals substantial speaker identity leakage. The Anon-W2W inset (upper left) prepends Stream-Voice-Anon to anonymize the waveform before the unchanged ASR encoder. The Anon-W2F inset (upper right) replaces the ASR encoder with the Stream-Voice-Anon encoder (anonymization active) and fine-tunes the LLM, eliminating the redundant waveform synthesis step. (Anon-W2F is demonstrated for SALM-Duplex; the Anon-W2W setup is additionally evaluated on Moshi.)
Refer to caption: 2603.08179v1/x1.png

We answer this question empirically for two prominent E2E full-duplex systems,
SALM-Duplex and Moshi [defossez2024moshi].
Following the VoicePrivacy 2024 Challenge [voiceprivacy2024] evaluation
protocol with a lazy-informed attacker scenario, we train a speaker verification
attacker (probe) on hidden state representations extracted from each layer group.
Equal error rate (EER) serves as our primary privacy metric; we additionally
report Linkability from the legally validated evaluation
framework of [vauquier2025legally].
Our mean-pooled EER for the original SALM-Duplex (continuous encoder) is
28.5%, revealing significant speaker identity leakage, while Moshi's discrete
encoder reaches 6.4% EER (near-perfect identification).
We further propose and evaluate two streaming anonymization
setups (Figure [2](#S1.F2)) using Stream-Voice-Anon [streamvoiceanon]
to mitigate this leakage without sacrificing dialogue utility.

Our contributions are:

- •
We characterise speaker identity leakage in the hidden states of two
E2E full-duplex dialogue LLMs (SALM-Duplex and Moshi), extending probing
methodology [tenney2019bert, pasad2021layerwise] from static encoders
to always-on dialogue backbones.
- •
We provide a layer-wise and turn-length-wise analysis showing which
parts of the LLM carry the most speaker-identifying information and how
leakage accumulates over dialogue length.
- •
We propose two streaming anonymization setups: Anon-W2W applies
Stream-Voice-Anon at the waveform level (retaining the original encoder,
validated on both SALM-Duplex and Moshi), while Anon-W2F replaces the
continuous encoder with a discrete encoder and activates anonymization
at the feature level.

## 2 Related Work

### 2.1 E2E speech dialogue language models

The generative spoken language model (GSLM) [lakhotia2021generative]
established the paradigm of modelling speech directly from discrete units,
subsequently extended to two-channel spoken dialogue by
dGSLM [nguyen2023generative], the first fully E2E full-duplex dialogue
model.
SpeechGPT [zhang2023speechgpt] and SpiRit-LM [nguyen2024spiritlm]
demonstrated that a single LLM backbone can natively interleave speech and text
tokens for conversational interaction.
SyncLLM [emnlp2024synclm] introduced synchronised LLM inference to support
real-time full-duplex turn management.
The most recent systems, SALM-Duplex [salmduplex2025] and
Moshi [defossez2024moshi], achieve low-latency always-on operation through
continuous dual-stream processing.
Crucially, because all of these models process raw user speech as input,
speaker identity information may be inadvertently encoded in their LLM hidden
representations, yet none has been analysed from a privacy perspective.

### 2.2 Speaker anonymization

The VoicePrivacy Challenge [tomashenko2024voiceprivacy, voiceprivacy2024]
established standardised evaluation protocols and benchmarks for speaker
anonymization, using the EER of an ASV system as the primary privacy metric;
Vauquier et al. [vauquier2025legally] recently extended this with a legally
validated framework incorporating Linkability metrics aligned
with GDPR requirements.
Anonymization approaches include x-vector-based
methods [meyer22_slt, tomashenko2024voiceprivacy],
perturbation [kuzmin24_spsc], phonetic intermediate
representations [meyer22_interspeech], neural audio codec language
models [panariello2024nac, yao24_spsc], and disentangled speech
representations [champion22_interspeech, aloufi2020privacy].
Streaming anonymization systems, including
Stream-Voice-Anon [streamvoiceanon], TVTSyn [quamer2026tvtsyn],
DarkStream [darkstream2025], and [quamer2024streaming], extend these
methods to the real-time setting required by E2E full-duplex models; we select
Stream-Voice-Anon as our front-end owing to its competitive privacy–utility
trade-off and open-source
availability(^2^22[https://github.com/Plachtaa/StreamVoiceAnon](https://github.com/Plachtaa/StreamVoiceAnon)).
All of these approaches operate at the *waveform level*.
To the best of our knowledge, our work is the first to analyse and mitigate
speaker identity leakage *inside* a full-duplex LLM's hidden states.

**Table 1: Privacy, dialogue quality, and efficiency on the VPC2024 evaluation set. Privacy: higher EER and lower Linkability (Lnk) = stronger privacy; EER = 50% = perfect anonymization. Linkability for no-anon rows uses the ``orig'' attacker; for anonymized rows, the ``lazy-informed'' attacker [vauquier2025legally]. Quality: higher sBLEU / sBERT = better. Efficiency: RTFx = 1/RTF; FRL = First Response Latency (s); TTSR = Turn-Taking Success Rate; Int.L. = Interruption Latency (s); ISR = Interruption Success Rate. Bold = best per column; underline = second best. Shaded rows = anonymized conditions.**
| Model | User | Anonymization | Privacy | Quality | Efficiency |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Name | Encoder | Type | EER$\uparrow$ | Lnk$\downarrow$ | sBLEU$\uparrow$ | sBERT$\uparrow$ | RTFx$\uparrow$ | FRL$\downarrow$ | TTSR$\uparrow$ | Int.L.$\downarrow$ | ISR$\uparrow$ |  |  |
|  |  |  |  |  | S2T | S2S | S2T | S2S |  |  |  |  |  |
|  | discrete | – | 6.4 | 0.90 | 7.18 | 6.85 | 0.50 | 0.48 | 17 | 0.50 | 0.85 | 1.24 | 0.45 |
| Moshi | discrete | W2W | 36.9 | 0.35 | 5.20 | 5.00 | 0.39 | 0.38 | 1.6 | 0.72 | 0.72 | 1.31 | 0.38 |
|  | discrete | – | 11.2 | 0.79 | 4.18 | 3.15 | 0.45 | 0.29 | 238 | 0.44 | 0.85 | 1.03 | 0.96 |
|  | discrete | W2F | 41.0 | 0.23 | 3.57 | 1.49 | 0.39 | 0.20 | 2.5 | 0.50 | 0.68 | 1.18 | 0.93 |
|  | continuous | – | 28.5 | 0.29 | 6.91 | 6.35 | 0.59 | 0.50 | 263 | 0.68 | 0.97 | 0.60 | 0.99 |
| SALM-<br>[-2pt]Duplex | continuous | W2W | 34.6 | 0.24 | 6.46 | 5.49 | 0.55 | 0.45 | 1.7 | 0.80 | 0.94 | 0.57 | 0.98 |

### 2.3 Privacy in neural speech and language representations

Probing studies have established that self-supervised speech models encode
substantial speaker identity across their transformer layers: Pasad
et al. [pasad2021layerwise] showed speaker information peaks in the lower
and middle layers of wav2vec 2.0, and SUPERB [yang2021superb] demonstrated
that lightweight probes achieve strong speaker verification from frozen SSL
representations.
Chiu et al. [chiu2025probing] recently confirmed this at large scale across
multiple SSL architectures and speaker attributes.
The layer-wise probing methodology itself was established by Tenney
et al. [tenney2019bert] for text models.
Our focus is distinct: rather than protecting conversational content, we
investigate *speaker identity* leakage, the persistent encoding of
*who is speaking*, which enables re-identification regardless of what is
said and may additionally extend to attributes such as gender, accent, and
health status.
Nautsch et al. [nautsch2019preserving] provide a comprehensive survey of
privacy threats in speaker and speech characterisation, identifying
re-identification from audio representations as a key risk category.

We extend this line of work to E2E full-duplex dialogue LLMs, whose hidden states, unlike static SSL encoders, are computed continuously over live user speech.

## 3 Methodology

### 3.1 E2E full-duplex dialogue models

We analyse two E2E full-duplex dialogue systems.
Moshi [defossez2024moshi] uses a decoder-only Transformer backbone
with a residual-quantization (RVQ) audio codec encoder to jointly model user and
system streams in a single autoregressive pass.
SALM-Duplex [salmduplex2025] is a decoder-only LLM that encodes
user audio via an ASR-initialized continuous encoder and processes dual
user–agent audio streams frame-synchronously, maintaining persistent hidden
states over the entire conversation; we reimplement a modified version that
adopts the speech decoder architecture from [wu2025chronological] but
without the chronological thinking mechanism.
Our reimplementation yields comparable response quality; the effect of the chronological thinking mechanism on privacy is left for future work.
Both systems maintain a persistent LLM hidden state over continuous user speech,
making them candidates for speaker identity probing.

### 3.2 User encoder variants

We distinguish two encoder families by their output representation, which determines how much speaker identity reaches the LLM backbone.

Discrete encoders (Moshi and SALM-Duplex variant):
Moshi natively uses an RVQ codec encoder that produces discrete token
representations.
For SALM-Duplex we replace the continuous front-end with a discrete encoder
based on the Firefly architecture (implementation details in
Section [3.5](#S3.SS5)).
Because both discrete encoders are trained for high-fidelity speech
reconstruction, they preserve rich speaker information in their token
representations.
The SALM-Duplex discrete encoder can be operated in two modes: with
anonymization *disabled*, serving as our ablation to isolate the effect of
the encoder swap alone; or with anonymization *active* via
Stream-Voice-Anon [streamvoiceanon], constituting our Anon-W2F
setup (Section [3.3](#S3.SS3)).

Continuous encoder (original SALM-Duplex): An ASR-initialized
adapter encodes raw user audio into continuous embeddings.
Because ASR pretraining optimizes for speech content rather than speaker
characteristics, this encoder represents a good design choice for built-in
partial privacy protection, though with a limited privacy ceiling as we show
in Section [4](#S4).

### 3.3 Anonymization setups

We propose two streaming anonymization setups using Stream-Voice-Anon [streamvoiceanon].

#### 3.3.1 Anon-W2W: Wave-to-wave anonymization

Anon-W2W pairs with the continuous encoder for SALM-Duplex and with
Moshi's native discrete codec encoder.
Stream-Voice-Anon is applied as a pre-processing step that transforms the raw
user waveform into an anonymized waveform before it is fed to the dialogue
model.
However, it introduces a redundant processing step: the anonymized waveform is
first synthesized and then re-encoded by the model's original encoder.

#### 3.3.2 Anon-W2F: Wave-to-feature anonymization

Anon-W2F pairs with the discrete encoder (anonymization active).
We replace the continuous encoder front-end of SALM-Duplex with the discrete
encoder and enable Stream-Voice-Anon's anonymization module, which operates
natively on the discrete token representations.
Because the anonymization step operates in the same feature domain as the
encoder, this eliminates the redundant waveform synthesis step of Anon-W2W.
The modified model is pretrained on a similar data mixture as [wu2025chronological]
(approximately 12k hours of multi-turn dialogue and 2.7k hours of QA data),
then fine-tuned on
InstructS2S-200K(^3^33[https://huggingface.co/datasets/ICTNLP/InstructS2S-200K](https://huggingface.co/datasets/ICTNLP/InstructS2S-200K)) [fang2025llamaomni2].
This setup requires architectural modification but achieves stronger privacy
guarantees through feature-domain-native anonymization.
While the Anon-W2F principle (replacing a model's encoder with an
anonymization-capable discrete encoder) is architecturally transferable, its
effectiveness depends on the target model's tokenizer compatibility; we
demonstrate it for SALM-Duplex and leave Moshi integration for future work.

### 3.4 Hidden state extraction

We extract hidden state representations from both Moshi ($N{=}32$ transformer
layers) and SALM-Duplex ($N{=}20$) at three individual layers: early
(layer 1), mid (layer $N/2$), and late (layer $N$), as well
as a mean-pooled representation averaging across all $N$ layers
(denoted ``All'' in Table [2](#S3.T2)).
Temporal pooling is handled internally by the probe architecture.

### 3.5 Setup

Dataset. Privacy metrics are evaluated on the VoicePrivacy 2024
Challenge [voiceprivacy2024] evaluation set, derived from LibriSpeech
dev-clean and test-clean splits.
While this dataset consists of read speech rather than spontaneous conversation,
it serves as a good starting point given the standardised evaluation protocol;
evaluating on conversational corpora is an important direction for future work.
The speaker verification attacker is trained on LibriSpeech train-clean-360.
Quality and efficiency metrics are evaluated on
MtBenchEval [yan2025uro, lin2025full_v15], a multi-turn dialogue benchmark
standardised by URO-Bench.
Moshi uses open-source pretrained weights; SALM-Duplex is reimplemented
following the speech decoder architecture of [wu2025chronological].

Speaker verification attacker.
We train separate ECAPA-TDNN [desplanques20_interspeech] attackers from
scratch for SALM-Duplex and Moshi, replicating the lazy-informed attacker
scenario from VPC2024 [voiceprivacy2024]: identical hyperparameters, with
an input projection layer adapted to each model's hidden state dimension.

Privacy metrics. Following the VoicePrivacy 2024
Challenge [voiceprivacy2024] protocol, we report equal error rate (EER%)
as the primary privacy metric.
A higher EER indicates stronger privacy protection; EER $\approx 50\%$
corresponds to random-chance discrimination, i.e., complete anonymization.
To further strengthen privacy analysis, we additionally report
Linkability [vauquier2025legally].

Utility metrics. Dialogue quality is assessed using evaluation scripts
adapted from URO-Bench [yan2025uro],
Full-Duplex-Bench [lin2025full_v15], and
SALM-Duplex [salmduplex2025].
Quality metrics (sBLEU, sBERT) are reported separately for speech-to-text (S2T)
and speech-to-speech (S2S) response modes: in S2T mode the model outputs text
directly; in S2S mode the model outputs speech which is subsequently transcribed
by an ASR system.
Efficiency metrics include Real-Time Factor Speedup (RTFx), First Response
Latency (FRL), Turn-Taking Success Rate (TTSR), Interruption Latency (Int.L.),
and Interruption Success Rate (ISR), reported per system condition
(Table [1](#S2.T1)).
All RTFx and latency measurements are estimated on a single GPU.

**Table 2: Layer-wise EER (%) of the speaker verification attacker. Higher = better privacy; 50% = chance level. All = mean-pooled over all layers. Bold = best; underline = second best. Shaded rows = anonymized conditions.**
| System | Encoder | Anon. | Early | Mid | Late | All |
| --- | --- | --- | --- | --- | --- | --- |
|  | discrete | – | 7.3 | 5.6 | 6.4 | 6.4 |
| Moshi | discrete | W2W | 42.5 | 37.6 | 35.2 | 36.9 |
|  | discrete | – | 7.5 | 14.0 | 20.1 | 11.2 |
|  | discrete | W2F | 43.8 | 40.5 | 40.1 | 41.0 |
|  | continuous | – | 24.6 | 28.6 | 32.1 | 28.5 |
| SALM-<br>[-2pt]Duplex | continuous | W2W | 31.5 | 33.7 | 35.3 | 34.6 |

## 4 Results

Table [1](#S2.T1) reports privacy, response quality, and efficiency metrics.
Discrete encoders, trained for high-fidelity speech reconstruction, leak
substantially more speaker identity than the continuous encoder (Moshi: 6.4%
EER; SALM-Duplex discrete: 11.2% vs. continuous: 28.5%), confirming that
ASR pretraining acts as a privacy-positive design choice by discarding speaker
characteristics in favour of linguistic content.
Both anonymization setups substantially reduce leakage: Anon-W2W raises EER to
36.9% for Moshi (+30.5 points) and 34.6% for SALM-Duplex continuous (+6.1
points), while Anon-W2F achieves 41.0% EER, approaching the 50% chance ceiling.
Crucially, comparing Anon-W2F against the discrete-only ablation (11.2%) confirms
the gain is entirely from anonymization, not the encoder swap.
These patterns hold across both architectures, confirming speaker identity
exposure as a consistent property of E2E full-duplex LLM hidden states.
Our ECAPA-TDNN probe represents a *lower bound* on leakage; the
VoicePrivacy Attacker Challenge [vpc_attacker2025] has shown that more
sophisticated attackers can further reduce EER, suggesting that actual privacy
risk may be higher than reported here.

Anonymization introduces moderate quality degradation (sBERT S2T drops 7–22%
relative) but privacy gains consistently outweigh the cost, with EER improving
21–477% relative across setups.
We note that our text-based quality metrics do not capture speech-level
attributes such as naturalness and prosody; speech quality evaluation (e.g.,
MOS, UTMOS) remains for future work.
Adding the anonymization module reduces RTFx substantially (from 17–263$\times$
to 1.6–2.5$\times$) as the anonymizer dominates inference time, but all
conditions remain real-time viable (RTFx $>1$).
Anon-W2F is faster than Anon-W2W (RTFx 2.5 vs. 1.6–1.7) because the
feature-domain encoder avoids the redundant waveform synthesis step.
Reducing the anonymization module's computational cost is an important direction
for future work.

### 4.1 Ablation studies

Layer-wise analysis.
Table [2](#S3.T2) shows per-layer EER.
Moshi exhibits uniformly low EER across all layers (5.6–7.3%), while both
SALM-Duplex variants show decreasing leakage from early to late layers,
consistent with deeper layers progressively abstracting away speaker features.
Anonymization raises EER uniformly across all layer groups, with all anonymized
conditions reaching the 31–44% range.

Turn-length analysis.
Figure [1](#S1.F1) plots $1-\mathrm{Linkability}$ vs. turn count.
Without anonymization, privacy degrades rapidly within the first few turns.
Both anonymized systems maintain acceptable protection even after 10 turns
(SALM-Duplex + W2F: 0.90$\to$0.78; Moshi + W2W: 0.83$\to$0.67).

## 5 Conclusion

We have shown that significant speaker identity leakage is a consistent property
of E2E full-duplex architectures, demonstrated across both SALM-Duplex and
Moshi, with discrete encoders leaking substantially more than the
ASR-pretrained continuous encoder.
Waveform-level anonymization (Anon-W2W) reduces this risk across
both architectures, with the largest relative gains observed for the most
exposed systems, while feature-domain anonymization (Anon-W2F) achieves the
strongest protection, raising EER by over 3.5x and approaching chance level.
These findings motivate privacy-by-design for always-on speech AI systems.

Future work includes extending Anon-W2F to Moshi and other E2E full-duplex
architectures, studying personalization with minimal privacy threats,
improving privacy protection systems to have less impact on response quality
and latency, and evaluating privacy under stronger and more diverse attacker
models.

## 6 Generative AI Use Disclosure

Generative AI tools were used for proofreading and grammar correction with minor
changes. All research ideas, experimental design, implementation, and analysis
were conducted by the authors.

## References