# 2309.07405

FUNCODEC: A FUNDAMENTAL, REPRODUCIBLE AND INTEGRABLE OPEN-SOURCE
TOOLKIT FOR NEURAL SPEECH CODEC
Zhihao Du, Shiliang Zhang, Kai Hu, Siqi Zheng

arXiv:2309.07405v2 [cs.SD] 7 Oct 2023

Speech Lab of DAMO Academy, Alibaba Group, China
{neo.dzh,sly.zsl}@alibaba-inc.com

ABSTRACT
This paper presents FunCodec, a fundamental neural speech codec
toolkit, which is an extension of the open-source speech processing
toolkit FunASR. FunCodec provides reproducible training recipes
and inference scripts for the latest neural speech codec models,
such as SoundStream and Encodec. Thanks to the unified design
with FunASR, FunCodec can be easily integrated into downstream
tasks, such as speech recognition. Along with FunCodec, pretrained models are also provided, which can be used for academic
or generalized purposes. Based on the toolkit, we further propose the frequency-domain codec models, FreqCodec, which can
achieve comparable speech quality with much lower computation
and parameter complexity. Experimental results show that, under
the same compression ratio, FunCodec can achieve better reconstruction quality compared with other toolkits and released models.
We also demonstrate that the pre-trained models are suitable for
downstream tasks, including automatic speech recognition and
personalized text-to-speech synthesis. This toolkit is publicly available at https://github.com/alibaba-damo-academy/
FunCodec.
Index Terms— FunCodec, speech codec, FreqCodec, SoundStream, Encodec
1. INTRODUCTION
Speech codecs are designed to compress and decompress speech signals for efficient transmission and storage. They consist of an encoder, which encodes speech into a compact representation, and a
decoder to reconstruct the signal. Traditional speech codecs rely
on a carefully designed pipeline that incorporates expert knowledge
of psycho-acoustics and speech synthesis to achieve efficient coding [1, 2].
Thanks to advancements in deep learning techniques, neural
speech codecs have been introduced, demonstrating superior performance compared to traditional speech codecs. In neural codec
models, raw waveforms are fed into deep neural network-based
encoders to extract compact representations. This is followed by
a residual vector quantizer (RVQ) [3, 4] to obtain parallel token
streams. Meanwhile, a neural network-based decoder is also trained
alongside the encoder and RVQ to reconstruct the signal. Building
upon the progress in text-to-speech synthesis [5], adversarial training losses are employed to enhance reconstruction quality. There are
two popular neural codec models, SounStream [4] and Encodec [6].
While SoundStream utilizes streaming SEANet [7, 8] as its encoder
and decoder, Encodec incorporates extra LSTM [9] layers and a
Demos are available at: https://funcodec.github.io

HuggingFace
& ModelScope
SoundStream
Encodec
FreqCodec
Semantic Augment
...

Model Zoo

FunCodec

Down-stream Tasks

train.sh
finetune.sh
inference.sh
...
Codebase

ASR
PTTS
...
Application

Fig. 1. Overview of FunCodec design.
Transformer-based language model [10] to improve the sequence
modeling ability. Following this line of work, extensive efforts
have been made on reducing the bit rate [11–13]. Additionally, the
modified discrete cosine transform (MDCT) domain has also been
explored [14, 15].
Although neural speech codecs were originally proposed to
compress signals for telecommunication, they can also be used to
extract discrete speech representations in generative models. In
VALL-E [16], text and speech tokens are joined into a sequence,
and a language model is trained to estimate their probabilities. This
formulation demonstrates impressive zero-shot synthesis capability.
Moreover, neural speech codecs facilitate the modeling of speech
and text within a single framework, enabling the model to both listen
and speak [17,18]. Recently, several speech codec toolkits have been
released for telecommunications purposes [19, 20]. However, there
is still a lack of open-source toolkits that provide a reproducible and
integrable framework for developing and evaluating neural speech
codecs in the context of speech-text modeling.
To address this gap, we present FunCodec, a fundamental,
reproducible, and integrable open-source toolkit for neural speech
codecs. FunCodec provides a versatile platform enabling researchers
to build, train, and evaluate various neural speech codecs. Fig.1
shows an overview of the FunCodec design. The contributions of
FunCodec are as follows: (1) The open-source codebase provides
recipes to finetune pre-trained models or train a model from scratch.
(2) Frequency-domain codec (FreqCodec) models are proposed,
which can achieve comparable performance with less parameters
and lower computation complexity. (3) The impact of semantic information is evaluated for speech codec, which improves the speech
quality under low bit rate. (4) Pre-trained academic and generalized
models are released through Huggingface and ModelScope 1 . (5)
Inference and evaluation scripts are also provided, which support
batch mode to fully utilize the parallelism capability of GPUs.
2. RELATED WORK
In this section, we compare FunCodec with other toolkits. We have
chosen the following four toolkits available on Github:
1 https://www.modelscope.cn/models?page=1&tasks=audiocodec&type=audio

En

er
cod

cod

Table 1. Feature comparison of FunCodec and other open-source
speech codec toolkits.

er

De

Encodec EncTrain Dac. AudioDec FunCodec
Organization
Facebook Mikxox Descript Facebook Alibaba
Released models
2
1
3
3
7
Training recipe
✗
✓
✓
✓
✓
Training stages
✗
1
1
2
1
Discriminators
✗
1
3
2
4
Dist. training
✗
✗
✓
✓
✓
K-means init.
✗
✓
✗
✗
✓
Low-frame-ratio
✗
✗
✗
✗
✓
Freq. domain
✗
✗
✗
✗
✓
Semantic aug.
✗
✗
✗
✗
✓
• Encodec: facebookresearch/encodec [6],
• EncTrainer: Mikxox/EnCodec Trainer [21],
• Dac.: descriptinc/descript-audio-codec [20],
• AudioDec: facebookresearch/AudioDec [19].
Table 1 summaries the differences between FunCodec and these
toolkits. While the other toolkits offer a limited number of pretrained models, FunCodec provides seven models for both academic
and generalized purposes. This allows researchers to use them as a
baseline system and also enables general users to directly apply them
to downstream tasks. Additionally, FunCodec provides comprehensive and efficient recipes that require only a single training stage. To
enhance the speech quality, FunCodec incorporates various discriminators, including multiple scale discriminator (MSD) [3,7], multiple
period discriminator (MPD) [5], multiple short-time Fourier transformation discriminator (MSTFTD) [4], and their combinations. For
training efficiency, FunCodec supports distributed training across
multiple GPUs. Moreover, FunCodec ensures high inference efficiency by simultaneously producing token streams for all samples in
a mini-batch. Furthermore, FunCodec enables k-means initialization
for quantization codebooks, improving the code utilization [22, 23].
Based on these features, FunCodec introduces low-frame-rate models. The frequency-domain transformation and semantic augmentation are also explored in FunCodec.
3. FUNCODEC TOOLKIT
FunCodec codebase consists of two main components: a library of
neural network models and recipes for replicating the experiments.
The library part is written in python with PyTorch [24]. The recipes
are all-in-one Bash scripts written in the Kaldi-style [25].
3.1. Model architecture
The architecture of FunCodec models is depicted in Fig. 2. Given a
speech signal x, it is passed through the domain transformation module. For time-domain models like SoundStream and Encodec, the
module functions as an identity mapping. However, for frequencydomain models, two representations Xmag,ang and Xmag,pha are explored:
X = STFT(x)
Xmag,ang = log (|X|) , angle(Xi , Xr )
Xr Xi
,
Xmag,pha = log (|X|) ,
|X| |X|

(1)

RVQ
Speech

Speech

Domain
Tranformation

Domain
Inversion

Semantic Tokens
(Optional)

Fig. 2. The overall architecture of the FunCodec models.
where Xr , Xi denote the real and imaginary parts of complex spectrum, respectively. | · | represents the norm of a complex value. After
the domain transformation module, the speech is inputted into an
encoder to extract acoustic representations: Va = Encoder(X). For
time-domain models, we adopt the same SEANet architectures as
Encodec and SoundStream. In the case of frequency-domain models
(FreqCodec), the encoder details are given in Table 2. The decoder
has a mirror architecture of the encoder. More details can be found
in our released codebase. Finally, a domain inversion module is utilized to reconstruct the raw waveforms from decoder outputs.
Table 2. The encoder architecture details of FreqCodec. Sb and Cb
denote the stride and filters of EncBlock b.
Layer Name
Inputs
Kernel, Stride
Outputs
Domain Trans.
(1, t)
512, 160
(∗, T, 257)
PreConv2D
(∗, T, 257)
(7, 7), (1, 1)
(C, T, 256)
EncBlock ×B (Cb , Tb , Fb )
-, (Sb , 4)
(2Cb , Tb /Sb , Fb /4)
Conv2D 1
(Cb , Tb , Fb )
(3, 3), (1, 1)
(Cb /2, Tb , Fb )
Conv2D 2
(Cb /2, Tb , Fb )
(1, 1), (1, 1)
(Cb , Tb , Fb )
Conv2D ds
(Cb , Tb , Fb ) (2Sb , 8), (Sb , 4) (2Cb , Tb /Sb , Fb /4)
Reshape
(24 C, TB , 1)
(TB , 24 C)
LSTM
(TB , 24 C)
(24 C, 24 C)
(TB , 24 C)
4
4
OutLinear
(TB , 2 C)
(2 C, D)
(TB , D)

3.2. Semantic-augmented residual vector quantization
To obtain discrete speech tokens, we employ a residual vector quantization (RVQ) module consisting of several quantizers:
!
n−1
X
Qn = VQ Q0 −
Qi
(2)
i=1

where Qn represents the outputs of n-th vector quantizer (VQ) and
Q0 represents the input of RVQ. To improve code utilization, we initialize the VQ codebook by clustering the samples in the first minibatch with k-means. The codes are then updated using a moving
average with a decay rate of 0.99. Moreover, if a code is activated
fewer than two times in a mini-batch, it will be reassigned.
In addition to the encoder outputs, we explore three methods to
incorporate semantic information into the codec models:
fcat (Va , Vs ) = Concat(RVQ(Va ), Vs )
fadd (Va , Vs ) = RVQ(Va ) + Vs
fres (Va , Vs ) = RVQ(Va − Vs ) + Vs

(3)

where Vs denotes semantic tokens, such as frame-aligned phoneme
labels and Hubert embeddings [26]. To make a single model operate across variable bitrates, structured quantization dropout is also
implemented in FunCodec.

3.3. Adversarial training objective with multiple discriminators
The training objective consists of three components: reconstruction
loss terms, adversarial loss terms, and the RVQ commit losses. The
L1 distance between original x and reconstructed speech x̂ is minimized over time domain: Lt (x, x̂) = ||x − x̂||1 . For the frequency
domain, both L1 and L2 distances are minimized on multiple Mel
and magnitude spectra:
1 X
Lf (x, x̂) =
(||Si (x) − Si (x̂)||1 + ||Si (x) − Si (x̂)||2
|α| i∈α
(4)

Table 3. Comparison of academic models in terms of ViSQOL
scores on LibriTTS dataset. † means the model is causal.
Models

Stride 400 TKR 200 TKR 100 TKR 50 TKR
†

SoundStream
SoundStream
Encodec
FunCodec
FunCodec-2x
FunCodec-4x

320
320
320
320
640
1280

4.23
4.28
4.24
4.29
4.29
4.31

4.00
4.06
4.05
4.12
4.16
3.94

3.60
3.76
3.73
3.86
3.94
3.43

3.12
3.31
3.30
3.43
3.64
2.91

+ ||Mi (x) − Mi (x̂)||1 + ||Mi (x) − Mi (x̂)||2 )
where, Si and Mi represent the log-compressed power and Mel
spectra with a window size of 2i and a shift length of 2i /4. α is
set to [5, 6, . . . , 11]. It worth noting that the log-compressed power
spectrum loss improves speech quality in the middle and high frequencies, which is missed in other toolkits and models.
For adversarial losses, FunCodec incorporates several discriminators, including multi-scale discriminator (MSD), multi-period discriminator (MPD), multi-scale STFT-based (MSTFTD) discriminator. By providing a unified interface, FunCodec allows for various
combinations of these discriminators, resulting in a more powerful
discriminative ability. In addition, a “feature” matching loss is also
involved:


X 1
1
Ladv (x̂) = Ex̂ 
max(0, 1 − Dk,t (x̂))
K
Tk
k,t

 (5)
X 1
1
(l)
(l)
Lfeat (x, x̂) = Ex,x̂ 
Dk,t (x) − Dk,t (x̂) 
KL
Tk
1

that FunCodec also supports other commonly-used sampling rate,
including 8k and 24k, among others.
We adopt the Virtual Speech Quality Objective Listener score
(ViSQOL) [33] as the primary evaluation metric, which ranges from
1 to 5, with higher scores indicating better quality. To ensure a fair
comparison, we introduce a measurement called token ratio (TKR),
which indicates the number of tokens the model requires to represent
one second of speech at a sampling rate of 16k. Under the same
TKR, models with higher ViSQOL scores are preferred.
4.2. Training details

where V represents the inputs of RVQ, which can vary depending
on the semantic augmentation methods used. The total training objective is obtained by summing up the individual loss terms:

During the training stage, we randomly clip a continuous segment
of 3.2 seconds from an utterance, which is considered as a training
sample. Before being fed into the encoder, the segment undergoes
root-mean-square (RMS) normalization. The reconstructed output is
rescaled using inverse normalization to calculate losses. For the LibriTTS corpus, we train the models on two Tesla-V100 GPUs with
a total batch size of 32. On the other hand, the generalized models are trained on four Tesla-A100 GPUs with a total batch size of
128. Under the adversarial training framework, we update the codec
model 300,000 times. To prevent the discriminator from becoming
too dominant, we only update it when its loss exceeds that of the
codec model.
The hyperparameters λt , λf , λadv , λfeat and λcm are set to 1.0,
and 1.0, respectively. We find that this hyperparameter
1.0, 19 , 100
9
setting is suitable for speech signals, and the loss balancer [6] is also
a work in progress. For FreqCodec models, the speech segment is
first transformed into a spectrogram using STFT with a window size
of 512 and a shift size of 160.

L = λt Lt + λf Lf + λadv Ladv + λfeat Lfeat + λcm Lcm

5. EXPERIMENTAL RESULTS

k,t,l

where Dk,t represents the output of discriminator k at timestep t,
(l)
and Dk,t denotes the outputs of layer l.
The commit loss takes into account the quantization errors of the
whole RVQ module and sub-quantizers:
N

Lcm = ||V − RVQ(V )||2 +

1 X
||Qi−1 − VQi (Qi−1 )||2 (6)
N i=1

(7)

4. EXPERIMENTAL SETTINGS
4.1. Experimental condition
We conduct experimental evaluations under two conditions, one for
academic usage and the other for generalized purposes. For academic usage, we employ the commonly-used LibriTTS corpus [27]
to train and evaluate models. This corpus consists of 585 hours of
English speech. We follow the official data partition of training, validation and test sets. For generalized usage, we train the model on a
large-scale in-house dataset, which contains 27.68 million bilingual
(English and Mandarin) utterances with a total duration of approximately 25,000 hours. To assess the generalization ability of our
generalized model, we evaluate it on the test sets of multiple opensource corpora, including Librispeech [28], aishell-1 [29], aishell2 [30], Wenet [31] and Gigaspeech [32]. In this paper, we resample all utterances to a sampling rate of 16k Hz. It is worth noting

5.1. Evaluation of academic models on LibriTTS corpus
We first conduct evaluations on LibriTTS and perform architecture
selection for the generalized models. The experimental results are
presented in Table 3. From the table, it can be observed that our
reproduced SoundStream and Encodec models achieve the-state-ofart performance at different token rates, confirming the accuracy of
our implementation. The inclusion of the log-compressed power
spectrum loss consistently improves speech quality, particularly at
lower token rates. Building upon FunCodec, we propose low-framerate models with strides that are two and four times longer, resulting in FunCodec-2x and FunCodec-4x, respectively. We find that
FunCodec-2x achieves higher ViSQOL scores, while FunCodec-4x
exhibits degraded performance at lower token rates. This indicates
that reducing the frame rate by two times strikes a favorable balance between time and quantization resolution. Thus, we adopt the
FunCodec-2x settings for the generalized models.

4.0

Table 4. Comparison of FreqCodec and other time domain models
in terms of ViSQOL score on LibriTTS. Mag denotes magnitude
spectrogram. Cin represents the channel number of inputs.

3.8
3.6
3.4
3.2

Param.

Groups
(Enc,Dec)

Flops

Domain

M1
M2
M3
M4
M5
M6
M7

Time
14.85M 3.72G
1, 1
Mag,Angle 16.21M 6.39G
1, 1
Mag,Phase 16.21M 6.47G
1, 1
Mag,Phase 4.38M 1.73G
1, Cin
Mag,Phase 4.50M 2.18G
1, Cin /8
Mag,Phase 0.52M 0.34G
Cin , Cin
Mag,Phase 0.83M 1.03G Cin /4, Cin /4

3.0
2.8
2.6
LS-clean

LS-other

aishell-1

Encodec

Dspt-codec

AudioDec-V

aishell-2

Wenet

Giga

AudioDec-L

FunCodec

(a) Open-source models with lower token rate
4.2

TKR
400 100

ID

4.29
4.32
4.36
4.28
4.31
4.21
4.25

3.86
3.84
3.85
3.79
3.81
3.65
3.80

Table 5. The impact of semantic information under low token rates.

4.0
3.8
3.6
3.4

ID

Info. Combiner

TKR=100

TKR=50

M1
SM1
SM2
SM3

None
Concat.
Residual
Addition

3.86
3.87
3.93
3.92

3.43
3.69
3.60
3.59

3.2
LS-clean

LS-other

aishell-1

Encodec

Dspt-codec

AudioDec-V

aishell-2

Wenet

Giga

AudioDec-L

FunCodec

(b) Open-source models with higher token rate

Fig. 3. Comparison of open-source generalized models under (a)
lower and (b) higher token rate. LS denotes Librispeech test sets.
While Librispeech and gigaspeech are English corpora, aishell and
Wenet are Mandarin corpora.
5.2. Evaluation of generalized models on multiple corpus
To ensure compatibility with other open-source models, the test files
are resampled to match the desired sampling rate. Additionally, for
a fair comparison, the token rate is also normalized based on the
working sampling rate. Fig. 3 illustrates the results of open-source
models at different token rates, where lower and higher token rates
correspond to the generation of 100 and 200 tokens per 16k waveform samples, respectively. It was observed that higher token rates
consistently yield improved compressed quality. In comparison to
other open-source models, our model demonstrates significantly better quality for both English and Mandarin speech at the same token
rate. However, to our surprise, all models perform poorly on the
Wenet test set. This can be attributed to the fact that the Wenet corpus is recorded in complex acoustic environments and contains a
higher degree of non-speech noises.
5.3. Comparison of frequency and time domain models
Table 4 presents a comparison of different frequency and time domain models in terms of parameters, computation complexity and
quantized speech quality. Due to the distinct structures in the frequency domain, FreqCodec models demonstrate superior speech
quality at higher token rates. Notably, the representation of magnitude and normalized phase spectra, as observed in M2 and M3,
proves to be more suitable for speech signals compared to the magnitude and angle spectra. Additionally, we successfully reduce
the number of parameters and computational complexity through
the utilization of depthwise convolutions. The results of M4-M7
indicate that, by appropriately splitting the groups in the encoder
and decoder, both the model and computational complexity can be
significantly reduced without compromising quality.

Table 6. Comparison of different inputs for ASR on Librispeech.
Inputs

dev-clean dev-other test-clean test-other

fBank

2.68

7.31

3.01

7.00

Codec Embedding
with SpecAug
Codec Index
with SpecAug

4.83
3.29
6.98
4.37

13.80
9.57
19.06
12.43

4.85
3.59
7.00
4.48

14.49
9.67
19.84
12.81

5.4. The impact of semantic augmentation
In contrast to other audio signals, speech carries explicit semantic
information. As a result, we enhance our codec models by incorporating force-aligned phoneme labels. Results in Table 5 demonstrate that the inclusion of semantic tokens consistently enhances the
quality of quantized speech. Moreover, the “residual” combination
method appears to be more suitable for speech codecs. This finding
highlights the viability of decoupling semantic and acoustic information as an effective approach for reducing token rates.
5.5. Applying to down-stream tasks
Except for speech quality, we also evaluate FunCodec in downstream
tasks. Table 6 presents the results of ASR task (TKR=100). Synthesis examples of VALL-E TTS system can be found on Demo pages.
We find that quantized tokens preserve a large proportion of speech
content, leading to a low recognition error rate. Furthermore, the
codebook embedding emerges as a crucial factor for the ASR task.
By comparing the results of clean and other tests, it is observed that
codec-based discrete inputs are more sensitive to the acoustic environment than continuous fbank features.
6. SUMMARY
In this paper, we introduce the open-source toolkit, FunCodec, which
is reproducible and integrable. Along with the codebase, pre-trained
academic and generalized models are released on Huggingface and
ModelScope. Based on FunCodec, we further evaluate the frequency
domain models and augment the RVQ module with semantic information. In addition, we also validate FunCodec models in the context of down-stream tasks, including ASR and TTS.

7. REFERENCES
[1] Jean-Marc Valin, Koen Vos, and Timothy B. Terriberry, “Definition of the opus audio codec,” RFC, vol. 6716, pp. 1–326,
2012.
[2] Martin Dietz, Markus Multrus, Vaclav Eksler, and et al.,
“Overview of the EVS codec architecture,” in ICASSP, 2015,
pp. 5698–5702.
[3] Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, and
et al., “Melgan: Generative adversarial networks for conditional waveform synthesis,” in NeurIPS, 2019, pp. 14881–
14892.
[4] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan
Skoglund, and Marco Tagliasacchi, “Soundstream: An endto-end neural audio codec,” IEEE ACM Trans. Audio Speech
Lang. Process., vol. 30, pp. 495–507, 2022.

[19] Yi-Chiao Wu, Israel D. Gebru, Dejan Markovic, and Alexander
Richard, “AudioDec: An open-source streaming high-fidelity
neural audio codec,” in ICASSP, 2023.
[20] Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan
Kumar, and Kundan Kumar, “High-fidelity audio compression
with improved rvqgan,” arXiv:2306.12925, 2023.
[21] Michael Meuleman, “Encodec trainer,” https://github.com/
Mikxox/EnCodec Trainer, 2023.
[22] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu,
“Neural discrete representation learning,” in NeurIPS, 2017,
pp. 6306–6315.
[23] Yi Ren, Ming Lei, Zhiying Huang, and et al., “Prosospeech:
Enhancing prosody with quantized vector pre-training in textto-speech,” in ICASSP, 2022, pp. 7577–7581.
[24] Adam Paszke, Sam Gross, Soumith Chintala, and et al., “Automatic differentiation in pytorch,” in NeurIPS Workshop, 2017.

[5] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae, “Hifi-gan:
Generative adversarial networks for efficient and high fidelity
speech synthesis,” in NeurIPS, 2020.

[25] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, and et al.,
“The kaldi speech recognition toolkit,” in ASRU, 2011.

[6] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and
Yossi Adi,
“High fidelity neural audio compression,”
arXiv:2210.13438, 2022.

[26] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, and
et al., “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE ACM Trans.
Audio Speech Lang. Process., vol. 29, pp. 3451–3460, 2021.

[7] Marco Tagliasacchi, Yunpeng Li, Karolis Misiunas, and Dominik Roblek, “Seanet: A multi-modal speech enhancement
network,” in INTERSPEECH, 2020, pp. 1126–1130.
[8] Yunpeng Li, Marco Tagliasacchi, Oleg Rybakov, Victor Ungureanu, and Dominik Roblek, “Real-time speech frequency
bandwidth extension,” in ICASSP, 2021, pp. 691–695.
[9] Sepp Hochreiter and Jürgen Schmidhuber, “Long short-term
memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997.

[27] Heiga Zen, Viet Dang, Rob Clark, and et al.,
“Libritts: A corpus derived from librispeech for text-to-speech,”
arXiv:1904.02882, 2019.
[28] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev
Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in ICASSP, 2015, pp. 5206–5210.

[10] Ashish Vaswani, Noam Shazeer, Niki Parmar, and et al., “Attention is all you need,” in NeurIPS, 2017, pp. 5998–6008.

[29] Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng,
“Aishell-1: An open-source mandarin speech corpus and a
speech recognition baseline,” in O-COCOSDA, 2017, pp. 1–
5.

[11] Jean-Marc Valin, Jan Büthe, and Ahmed Mustafa, “Lowbitrate redundancy coding of speech using a rate-distortionoptimized variational autoencoder,” in ICASSP, 2023.

[30] Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu, “Aishell2: Transforming mandarin asr research into industrial scale,”
arXiv:1808.10583, 2018.

[12] Teerapat Jenrungrot, Michael Chinen, W. Bastiaan Kleijn, and
et al., “LMCodec: A low bitrate speech codec with causal
transformer models,” in ICASSP, 2023.

[31] Binbin Zhang, Hang Lv, Pengcheng Guo, and et al., “Wenetspeech: A 10000+ hours multi-domain mandarin corpus for
speech recognition,” in ICASSP, 2022, pp. 6182–6186.

[13] Haici Yang, Wootaek Lim, and Minje Kim, “Neural feature
predictor and discriminative residual coding for low-bitrate
speech coding,” in ICASSP, 2023.

[32] Guoguo Chen, Shuzhou Chai, Guanbo Wang, and et al., “Gigaspeech: An evolving, multi-domain asr corpus with 10,000
hours of transcribed audio,” arXiv:2106.06909, 2021.

[14] Grant Davidson, Mark Vinton, Per Ekstrand, and et al., “High
quality audio coding with mdctnet,” in ICASSP, 2023.

[33] Andrew Hines, Jan Skoglund, Anil C Kokaram, and Naomi
Harte, “Visqol: an objective speech quality model,” EURASIP
Journal on Audio, Speech, and Music Processing, vol. 2015,
no. 1, pp. 1–18, 2015.

[15] Hyungseob Lim, Jihyun Lee, Byeong Hyeon Kim, Inseon
Jang, and Hong-Goo Kang, “End-to-end neural audio coding
in the mdct domain,” in ICASSP, 2023.
[16] Chengyi Wang, Sanyuan Chen, Yu Wu, and et al., “Neural
codec language models are zero-shot text to speech synthesizers,” arXiv:2210.13438, 2023.
[17] Tianrui Wang, Long Zhou, Ziqiang Zhang, and et al., “Viola:
Unified codec language models for speech recognition, synthesis, and translation,” arXiv:2305.16107, 2023.
[18] Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung
Nguyen, and et al., “Audiopalm: A large language model that
can speak and listen,” arXiv:2306.12925, 2023.