# 2206.08317

Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive
End-to-End Speech Recognition
Zhifu Gao1 , Shiliang Zhang1 , Ian McLoughlin2 , Zhijie Yan1
1

2

Speech Lab, Alibaba Group, China
ICT Cluster, Singapore Institute of Technology, Singapore

{zhifu.gzf, sly.zsl}@alibaba-inc.com, ian.mcloughlin@singaporetech.edu.sg
AR

Transformers have recently dominated the ASR field. Although
able to yield good performance, they involve an autoregressive
(AR) decoder to generate tokens one by one, which is computationally inefficient. To speed up inference, non-autoregressive
(NAR) methods, e.g. single-step NAR, were designed, to enable parallel generation. However, due to an independence assumption within the output tokens, performance of single-step
NAR is inferior to that of AR models, especially with a largescale corpus. There are two challenges to improving singlestep NAR: Firstly to accurately predict the number of output
tokens and extract hidden variables; secondly, to enhance modeling of interdependence between output tokens. To tackle both
challenges, we propose a fast and accurate parallel transformer,
termed Paraformer. This utilizes a continuous integrate-andfire based predictor to predict the number of tokens and generate
hidden variables. A glancing language model (GLM) sampler
then generates semantic embeddings to enhance the NAR decoder’s ability to model context interdependence. Finally, we
design a strategy to generate negative samples for minimum
word error rate training to further improve performance. Experiments using the public AISHELL-1, AISHELL-2 benchmark,
and an industrial-level 20,000 hour task demonstrate that the
proposed Paraformer can attain comparable performance to the
state-of-the-art AR transformer, with more than 10x speedup.1 .
Index Terms: ASR, E2E, non-autoregressive, single step NAR,
Paraformer

1. Introduction
Over the past few years, the performance of end-to-end (E2E)
models has surpassed that of conventional hybrid systems on
automatic speech recognition (ASR) tasks. There are three
popular E2E approaches: connectionist temporal classification
(CTC) [1], recurrent neural network transducer (RNN-T) [2]
and attention based encoder-decoder (AED) [3, 4]. Of these,
AED models have dominated seq2seq modeling for ASR, due
to their superior recognition accuracy. Examples are Transformer [4] and Conformer [5]. While performance is good, the
auto-regressive (AR) decoder inside such AED models needs
to generate tokens one by one, since each token is conditioned
on all previous tokens. Consequently, the decoder is computationally inefficient, and decoding time increases linearly with
the output sequence length. To improve efficiency and accelerate inference, non-autoregressive (NAR) models have been proposed to generate output sequences in parallel [6–8].
Based on the number of iterations duration inference, NAR
models can be categorized as either iterative or single step.
1 This work was supported in part by Key R & D Projects of the
Ministry of Science and Technology (2020YFC0832500)

vanilla NAR

Paraformer

0.09
0.08

0.07
0.06

error rate

arXiv:2206.08317v3 [cs.SD] 30 Mar 2023

Abstract

0.05
0.04
0.03
0.02
0.01
0.00
insertion

deletion

substitution

error type

Figure 1: Analysis of different error types for three systems,
evaluated on the industrial 20,000 hour task.
Among the former, A-FMLM was the first attempt [9], designed
to predict masked tokens conditioned on unmasked ones over
constant iterations. Performance suffers due to the need to predefine a target token length. To address this issue, Mask-CTC
and variants proposed enhancing the decoder inputs with CTC
decodings [10–12]. Even so, these iterative NAR models require multiple iterations to obtain a competitive result, limiting
the inference speed in practice. More recently, several single step NAR models were proposed to overcome this limitation [13–17]. These generate output sequences simultaneously
by removing temporal dependency. Although single step NAR
models can significantly improve inference speed, their recognition accuracy is significantly inferior to AR models, especially
when evaluated on a large-scale corpus.
The single step NAR works mentioned above mainly focus on how to predict token numbers as well as extract hidden
variables accurately. Compared to machine translation which
predicts token number by a predictor net, it is indeed difficult
for ASR due to various factors such as the speaker’s speech
rate, silences, and noise. On the other hand, according to our
investigation, single step NAR models make a lot of substitution mistakes compared to AR models (Depicted as AR and
vanilla NAR in Fig. 1). We believe that lack of context interdependence leads to increased substitution mistakes, particularly due to the conditional independence assumption required
in single step NAR. Besides this, all of these NAR models were
explored on academic benchmarks recorded from reading scenarios. Performance has not yet been assessed on a large scale
industrial-level corpus. This paper therefore aims to improve
the single step NAR model so as that it can obtain recognition
performance on par with an AR model on a large-scale corpus.
This work proposes a fast and accurate parallel transformer
model (termed Paraformer) which addresses both challenges as
stated above. For the first, unlike previous CTC based works,
we utilize a continuous integrate-and-fire (CIF) [18] based predictor net to estimate the target number and generate the hidden

MAE

CE MWER

predictor

parallel
bi-decoder

x0.3 x0.5

Pass1:
w/o grads

Pass2:
w grads

semantic embedding

encoder

sampler
acoustic
embedding

embedding

fbanks

char tokens

Figure 2: Structure of the proposed Paraformer.
variables. For the second challenge, we design a glancing language model (GLM) based sampler module to strengthen the
NAR decoder with the ability to model token inter-dependency.
This is mainly inspired by work in neural machine translation [19]. We additionally design a strategy to include negative
samples, to improve performance by exploiting minimum word
error rate (MWER) [20] training.
We evaluate Paraformer on the public 178 hour AISHELL1 and 1000 hour AISHELL-2 benchmarks, as well as an
industrial 20,000 hour Mandarin speech recognition task.
Paraformer obtains CERs of 5.2% and 6.19% on AISHELL-1
and AISHELL-2 respectively, which not only outperforms other
recent published NAR models but is comparable to the stateof-the-art AR transformer without an external language model.
As far as we know, Paraformer is the first NAR model able to
achieve comparable recognition accuracy to an AR transformer,
and it does so with a 10x speedup on the large corpus.

2. Methods
2.1. Overview
The overall framework of the proposed Paraformer model is illustrated in Fig. 2. The architecture consists of five modules,
namely the encoder, predictor, sampler, decoder and loss function. The encoder is the same as an AR encoder, consisting of
multiple blocks of memory equipped self-attention (SAN-M)
and feed-forward networks (FFN) [21] or conformer [5]. The
predictor is used to produce the acoustic embedding and guide
the decoding. The sampler module then generates a semantic
embedding according to the acoustic embedding and char token embedding. The decoder is similar to an AR decoder except for being bidirectional. It consists of multiple blocks of
SAN-M, FFN and cross multi-head attention (MHA). Besides
the cross-entropy (CE) loss, the mean absolute error (MAE),
which guides the predictor to convergence, and MWER loss,
are combined to jointly train the system.
We denote the inputs as (X, Y), where X is the acoustic
feature of frame number T , and Y is the target label with token
number N . The encoder maps input sequence X to a sequence
of hidden representations H. These hidden representations H
0
are then fed up to the predictor to predict token number N and
produce acoustic embedding Ea . The decoder takes in acoustic
embedding Ea and hidden representation H to generate target
0
predictions Y for the first pass without backward gradients.
The sampler samples between acoustic embedding Ea and target embedding Ec to generate semantic embedding Es accord0
ing to the distance between predictions Y and target label Y.

0.3

x0.2

0.5

x0.1 x0.6

0.3

0.6

x0.3

0.4

x0.1

x0.9

0.9

x0.2

0.2

x0.6

0.6

Figure 3: An illustration of the CIF process (β is set to 1).
The decoder then takes in semantic embedding Es as well as
00
hidden representations H to generate final predictions Y for
the second pass, this time with backward gradients. Finally, the
00
predictions Y are sampled to produce negative candidates for
the MWER training, and the MAE is computed between target
0
token number N and predicted token number N . Both MWER
and MAE are jointly trained with a CE loss.
During inference, the sampler module is inactive and the
bidirectional parallel decoder directly utilizes acoustic embeddings Ea and hidden representation H to output final prediction
0
Y over only a single pass. Although the decoder is operational
in the forward direction twice during each training stage, the
computational complexity does not actually increase during inference thanks to the single step decoding process.
2.2. Predictor
The predictor consists of two convolution layers, with the output being a float weight α ranging from 0 to 1. We accumulates the weight α to predict token number. An MAE loss
P
LM AE = N − Tt=1 αt is added to guide the learning.
We introduce the mechanism of Continuous Integrate-and-Fire
(CIF) to generate acoustic embedding. CIF is a soft and monotonic alignment, which was proposed as a streaming solution for
AED models in [18]. To generate acoustic embedding Ea , CIF
accumulates the weights α and integrates hidden representations H until the accumulated weight reaches a given threshold
β, which indicates that an acoustic boundary has been reached
(an illustration of this process is shown in Fig. 3). According
to [18], the weight α is scaled by target length during training
so as to match the number of acoustic embeddings Ea with target embeddings Ec , while weight α is directly used to produce
Ea for inference. There may thus exist a mismatch between
training and inference, causing the precision of the predictor to
decay. Since the NAR model is more sensitive to predictor accuracy than a streaming model, we propose using a dynamic
threshold β instead of a predefined one to reduce the mismatch.
The dynamic threshold mechanism is formulated as:
PT
αt
β = PTt=1
(1)
d t=1 αt e
2.3. Sampler
In a vanilla single step NAR, the objective of its optimization
could be formulated as:
LNAT =

N
X

log P (yn | X; θ)

(2)

n=1

However, as noted, the conditional independence assumption
leads to inferior performance compared to an AR model. Meanwhile the glancing language model (GLM) loss is defined as:
h 00
i
0
log p yn | GLM(Y, Y ), X; θ (3)

X

LGLM =
00

0

yn ∈GLM(Y,Y )

Table 1: Comparison of ASR systems on AISHELL-1 and
AISHELL-2 tasks (CER%), without LM. AR and N AR denotes
the AR baseline with beamsearch and proposed NAR method respectively reported in each work. (? : RTF is evaluated with
batchsize of 8, † : batchsize is unreported, batchsize of the others is 1).
AISHELL-1
A-FMLM [9]
Mask CTC [22]
LASO [23]
NAT-UBD [24]
TSNAT [25]
CTC-Enhanced [12]
Improved
CASS-NAT [15]
Ours
AISHELL-2
LASO [23]
CTC-Enhanced [12]
Ours

AR / NAR
NAR
NAR
NAR
NAR
NAR
AR
NAR
AR
NAR
AR
Vanilla-NAR

dev / test
6.2 / 6.7
6.9 / 7.8
5.9 / 6.6
5.1 / 5.6
5.1 / 5.6
5.2 / 5.7
5.3 / 5.9
4.8 / 5.2
4.9 / 5.4
4.7 / 5.2
4.7 / 5.3

Paraformer

4.6 / 5.2

AR / NAR
NAR
AR
NAR
AR
Vanilla-NAR
Paraformer

test ios
6.8
6.8
7.1
6.18
6.23
6.19

RTF
0.2800
0.0500
0.0035 †
0.0081 †
0.0185
0.1703
0.0037 ?
0.2000
0.0230
0.2100
0.0168
0.0168
0.0026 ?
RTF
0.1703
0.0037 ?
0.2100
0.0168
0.0168

0

Where GLM(Y, Y ) denotes the subset of tokens selected by
the sampler module between Ec and Ea . And GLM(Y, Y 0 )
denotes the remaining unselected subset of tokens within the
target Y.
0

0

GLM(Y, Y ) = Sampler(Es | Ea , Ec , dλd(Y, Y )e)

(4)

Where λ is a sampling factor to control the sample ratio. The
0
d(Y, Y ) term is the sampling number. It will be larger when
the model is poorly trained, and should decrease along with the
training process. For this, we simply use the Hamming distance,
defined as:
N 

X
0
0
d(Y, Y ) =
yn 6= yn
(5)
n=1

To summarize, the sampler module incorporates target em0
beddings Ec by randomly substituting dλd(Y, Y )e tokens into
acoustic embedding Ea to generate semantic embedding Es .
The parallel decoder is trained to predict the target tokens
0
GLM(Y, Y 0 ) with semantic context GLM(Y, Y ), enabling the
model to learn interdependency between output tokens.
2.4. Loss Function
There are three loss functions defined, namely the CE, MAE
and MWER losses. The types are jointly trained, as follows:
Ltotal = γLCE + LM AE + LNwerr (x, y∗ )
For the MWER, it could be formulated as [20]:
LNwerr (x, y∗ ) =

X
yi ∈

Sample(x,N )

(6)

h
i
c
Pb (yi | x) W (yi , y∗ ) − W

Table 2: Evaluation of sampling ratio (CER%).
λ
Far-field
Common

0.2
14.64
8.22

0.5
14.37
8.13

0.75
14.17
7.98

1.0
14.24
8.09

1.5
14.47
8.13

There is only one output path for NAR models due to the greedy
search decoding. As noted above, we exploit the negative sampling strategy to generate multiple candidate paths by randomly
masking the top1 score token during the MWER training.

3. Experiments
3.1. Experimental Setup
We evaluated the proposed methods on the openly available AISHELL-1 (178-hours) [26], AISHELL-2 (1000-hours)
benchmarks [27], plus a 20,000 hour industrial Mandarin task.
The latter task is the same large corpus as in [21,28]. A far-field
set of about 15 hours data and a common set of about 30 hours
data are used to evaluated the performance. Other configurations could be found in [21,28,29]. Real time factor (RTF) was
used to measure the inference speed on GPU (NVIDIA Tesla
V100). Our code is available in FunASR 2 .
3.2. AISHELL-1 and AISHELL-2 task
The AISHELL-1 and AISHELL-2 evaluation results are detailed in Table 1. For fair comparison with published works,
RTF is evaluated on ESPNET [30]. No external language model
(LM) nor unsupervised pre-training is used with any of the experiments in Table 1. For the AISHELL-1 task, we firstly
trained an AR transformer as baseline, with the configuration
matching the AR baseline in [15]. The performance of the baseline is state-of-the-art among AR transformers, excluding system with large scale knowledge transfer such as [31] since we
aim for architectural improvement rather than gains from a bigger dataset. The vanilla NAR shares the same structure with our
proposed model Paraformer, but without the sampler. It can be
seen that our vanilla NAR surpasses the performance of other
recent published NAR works, e.g., improved CASS-NAT [15]
and CTC-enhanced NAR [12]. Nevertheless, its performance
is slightly inferior to the AR baseline due to the lack of context
dependency between output tokens. But when we enhance the
vanilla NAR with GLM via the sampler module in Paraformer,
we obtain comparable performance to the AR model. While
Paraformer obtains a recognition CER of 4.6% and 5.2% on
dev and test set respectively, the inference speed (RTF) is more
than 12 times faster than the AR baseline. For the AISHELL2 task, the model configuration is the same with AISHELL-1.
From Table 1, it can be seen that the performance gains are similar to those for AISHELL-1. Specifically, Paraformer achieved
a CER of 6.19% for the test ios task, with more than 12 times
faster inference speed. As far as the authors are aware, this
is the state-of-the-art performance among NAR models on both
AISHELL-1 and AISHELL-2 tasks.
3.3. Industrial 20,000 hour task
Extensive experiments were used to evaluate our proposed
approach, detailed in Table 3. Dynamic β denotes the dynamic threshold detailed in Section 2.2 while CTC refers to the
2 https://github.com/alibaba-damo-academy/FunASR

Table 3: Performance of three systems on the industrial 20,000 hour task (CER%).
Parameter
Model
Dynamic β
MWER
RTF
Far-field
Common

CTC
17.71
9.93

AR
0.067
13.76
7.75

Transformer-SAN-M (41M)
Vanilla NAR
Paraformer
w
w/o
w
w
w/o
w/o
w
0.007
0.007 0.007 0.007
16.39
14.39 14.17 14.07
9.35
8.15
7.98
7.86

DFSMN-CTC-sMBR system with LM [32]. RTF is evaluated
on OpenNMT [33].
Looking first at the models with a size of 41M, the AR baseline with attention dimension of 256 is the same as in [21]. We
can see a different phenomenon to that noted in the Sec. 3.2.
Here we find that the CER of vanilla NAR differs from that of
the AR model by a large margin. Nevertheless, vanilla NAR
still outperforms CTC, which makes a similar conditional independence assumption. When equipped with GLM, Paraformer
obtains 13.5% and 14.6% relative improvement on Far-field and
Common tasks respectively, compared to vanilla NAR. When
we further add the MWER training, the accuracy improves
slightly. More importantly, Paraformer achieves comparable
performance to the AR model (less than 2% relative loss) with
the benefit of 10x faster inference speed. We have also evaluated the dynamic threshold for CIF. From Table 3, it is evident that the dynamic threshold helps to further improve accuracy. Compared to a predefined threshold as in CIF, the dynamic
threshold reduces the mismatch between inference and training,
to extract acoustic embedding more accurately.
Evaluating on the larger model size of 63M, the phenomenon seen is similar to that noted above. Here, Paraformer
achieves 13.0% and 11.1% relative improvement on Farfield and Common task respectively, over vanilla NAR.
Again, Paraformer achieves comparable accuracy to the AR
model (less than 2.8% relative difference), again achieving
10x speedup. If we compare Paraformer-63M against AR
transformer-41M, although the Paraformer model size is larger,
its inference speed improves (RTF from 0.067 to 0.009). Hence
Paraformer-63M can achieve a 6.0% relative improvement
over the AR transformer-41M on the Far-field task, while
accelerating the inference speed by 7.4 times. This reveals
that Paraformer can achieve superior performance through increased model size, while still maintaining a faster inference
speed than AR transformer.
Finally, we evaluate the sampling factor λ in the sampler,
as shown in Table 2. As expected, the recognition accuracy improves as λ increases, due to the better context provided by the
targets. However when the sampling factor is too large, it will
cause a mismatch between training and inference, where we
decode twice with targets for training and decode once without targets for inference. Nevertheless, the performance of
Paraformer is robust to λ in a range from 0.5 to 1.0.
3.4. Discussion
From the above experiments, we note that, compared to the AR
model, the performance decay of vanilla NAR on AISHELL-1
and AISHELL-2 task is small, but is much bigger for the large
scale industrial-level corpus. Compared to academic benchmarks from a reading corpus (e.g. AISHELL-1 and -2) , the
industrial-level dataset reflects more complicated scenarios, and
thus is more reliable for evaluating NAR models. As far as we

Transformer-SAN-M-large (63M)
AR
Vanilla NAR Paraformer
w
w
w
0.094
0.009
0.009
12.57
14.86
12.93
7.55
8.67
7.71

know, this is the first work which explores NAR models on a
large-scale industrial-level corpus task.
The experiments above show that Paraformer obtains significant improvement compared to vanilla NAR by over 11%
while Paraformer performs similarly to well-trained AR transformer.
To understand why, we performed further analysis. First,
we determined the error type statistics for the AR, vanilla NAR
and Paraformer models on the 20,000 hour task, plotted in
Fig. 1. We counted the total number of errors types, namely insertion, deletion and substitution respectively on Far-field and
Common, and normalized them by the total number of target
tokens. The vertical axis of Fig. 1 is the ratio of error types.
We can see that, compared to the AR system performance, the
insertion errors in the vanilla NAR increase slightly, while deletion decreases to a small extent. This indicates that the accuracy of the predictor is superior, with the help of the dynamic
threshold. However, the substitution errors rise dramatically,
which explains the large gap in performance between them. We
believe this is caused by the conditional independence assumption within the vanilla NAR model. Compared to vanilla NAR,
the substitution errors in Paraformer decrease significantly, accounting for most of its performance improvement. We believe
the decline in substitution is because the enhanced GLM enables the NAR model to better learn inter-dependency between
output tokens. Nevertheless, compared to AR, there still exists
a small gap in the number of substitution errors, leading to the
slight difference in recognition accuracy. We think the reason is
that the beam search decoding of AR could play a strong role in
the language model compared to the glancing language model.
To eliminate this remaining performance gap, we aim to combine Paraformer with an external language model in our future
work.

4. Conclusion
This paper has proposed a single-step NAR model, Paraformer,
to improve the performance of NAR end-to-end ASR systems.
We first utilize a continuous integrate-and-fire (CIF) based predictor to predict the token number and generate hidden variables. We improved CIF with a dynamic threshold instead of a
predefined one, to reduce the mismatch between inference and
training. Then we design a glancing language model based sampler module to generate semantic embeddings to enhance the
NAR decoder’s ability to model the context interdependence.
Finally, we designed a strategy to generate negative samples in
order to perform minimum word error rate training to further
improve performance. Experiments conducted on the public
AISHELL-1 (178 hours) and AISHELL-2 (1000 hours) benchmark as well as an industrial-level 20,000 hour corpus show that
the proposed Paraformer model can achieve comparable performance to the state-of-the-art AR transformer with 10x speedup.

5. References
[1] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of
the 23rd international conference on Machine learning. ACM,
2006, pp. 369–376.
[2] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition
with deep recurrent neural networks,” in 2013 IEEE international
conference on acoustics, speech and signal processing. IEEE,
2013, pp. 6645–6649.
[3] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend
and spell: A neural network for large vocabulary conversational
speech recognition,” in 2016 IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016,
pp. 4960–4964.
[4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N.
Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”
in Advances in neural information processing systems, 2017, pp.
5998–6008.
[5] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han,
S. Wang, Z. Zhang, Y. Wu et al., “Conformer: Convolutionaugmented transformer for speech recognition,” Proc. Interspeech
2020, pp. 5036–5040, 2020.
[6] J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher, “Nonautoregressive neural machine translation,” in International Conference on Learning Representations, 2018.
[7] J. Lee, E. Mansimov, and K. Cho, “Deterministic nonautoregressive neural sequence modeling by iterative refinement,”
in EMNLP, 2018.
[8] M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer, “Maskpredict: Parallel decoding of conditional masked language models,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International
Joint Conference on Natural Language Processing (EMNLPIJCNLP), 2019, pp. 6112–6121.
[9] N. Chen, S. Watanabe, J. Villalba, and N. Dehak, “Listen and fill
in the missing letters: Non-autoregressive transformer for speech
recognition,” arXiv preprint arXiv:1911.04908, 2019.

[17] K. Deng, Z. Yang, S. Watanabe, Y. Higuchi, G. Cheng, and
P. Zhang, “Improving non-autoregressive end-to-end speech
recognition with pre-trained acoustic and language models,” arXiv
preprint arXiv:2201.10103, 2022.
[18] L. Dong and B. Xu, “CIF: Continuous integrate-and-fire for endto-end speech recognition,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP). IEEE, 2020, pp. 6079–6083.
[19] L. Qian, H. Zhou, Y. Bao, M. Wang, L. Qiu, W. Zhang, Y. Yu,
and L. Li, “Glancing transformer for non-autoregressive neural
machine translation,” arXiv preprint arXiv:2008.07905, 2020.
[20] R. Prabhavalkar, T. N. Sainath, Y. Wu, P. Nguyen, Z. Chen, C.C. Chiu, and A. Kannan, “Minimum word error rate training for
attention-based sequence-to-sequence models,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4839–4843.
[21] Z. Gao, S. Zhang, M. Lei, and I. McLoughlin, “San-m: Memory
equipped self-attention for end-to-end speech recognition,” arXiv
preprint arXiv:2006.01713, 2020.
[22] T. Wang, Y. Fujita, X. Chang, and S. Watanabe, “Streaming
end-to-end ASR based on blockwise non-autoregressive models,”
arXiv preprint arXiv:2107.09428, 2021.
[23] Y. Bai, J. Yi, J. Tao, Z. Tian, Z. Wen, and S. Zhang, “Fast end-toend speech recognition via non-autoregressive models and crossmodal knowledge transferring from bert,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp.
1897–1911, 2021.
[24] C.-F. Zhang, Y. Liu, T.-H. Zhang, S.-L. Chen, F. Chen, and
X.-C. Yin, “Non-autoregressive transformer with unified bidirectional decoder for automatic speech recognition,” arXiv preprint
arXiv:2109.06684, 2021.
[25] Z. Tian, J. Yi, J. Tao, Y. Bai, S. Zhang, Z. Wen, and X. Liu,
“Tsnat: Two-step non-autoregressvie transformer models for
speech recognition,” arXiv preprint arXiv:2104.01522, 2021.
[26] H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, “Aishell-1: An
open-source Mandarin speech corpus and a speech recognition
baseline,” in 2017 20th Conference of the Oriental Chapter of
the International Coordinating Committee on Speech Databases
and Speech I/O Systems and Assessment (O-COCOSDA). IEEE,
2017, pp. 1–5.

[10] Y. Higuchi, S. Watanabe, N. Chen, T. Ogawa, and T. Kobayashi,
“Mask ctc: Non-autoregressive end-to-end ASR with CTC and
mask predict,” 2020.

[27] J. Du, X. Na, X. Liu, and H. Bu, “Aishell-2: transforming Mandarin asr research into industrial scale,” arXiv preprint
arXiv:1808.10583, 2018.

[11] Y. Higuchi, H. Inaguma, S. Watanabe, T. Ogawa, and
T. Kobayashi, “Improved mask-CTC for non-autoregressive endto-end ASR,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
IEEE, 2021, pp. 8363–8367.

[28] Z. Gao, S. Zhang, M. Lei, and I. McLoughlin, “Universal asr:
Unifying streaming and non-streaming asr using a single encoderdecoder model,” arXiv preprint arXiv:2010.14099, 2020.

[12] X. Song, Z. Wu, Y. Huang, C. Weng, D. Su, and H. Meng, “Nonautoregressive transformer ASR with CTC-enhanced decoder input,” in ICASSP 2021-2021 IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021,
pp. 5894–5898.
[13] Z. Tian, J. Yi, J. Tao, Y. Bai, S. Zhang, and Z. Wen, “Spiketriggered non-autoregressive transformer for end-to-end speech
recognition,” 2020.
[14] R. Fan, W. Chu, P. Chang, and J. Xiao, “Cass-nat: CTC
alignment-based single step non-autoregressive transformer for
speech recognition,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP). IEEE, 2021, pp. 5889–5893.
[15] R. Fan, W. Chu, P. Chang, J. Xiao, and A. Alwan, “An improved
single step non-autoregressive transformer for automatic speech
recognition,” arXiv preprint arXiv:2106.09885, 2021.
[16] N. Chen, P. Zelasko, L. Moro-Velázquez, J. Villalba, and
N. Dehak, “Align-denoise: Single-pass non-autoregressive speech
recognition,” 2021.

[29] S. Zhang, Z. Gao, H. Luo, M. Lei, J. Gao, Z. Yan, and L. Xie,
“Streaming chunk-aware multihead attention for online end-toend speech recognition,” arXiv preprint arXiv:2006.01712, 2020.
[30] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno,
N. E. Y. Soplin, J. Heymann, M. Wiesner, N. Chen et al.,
“Espnet: End-to-end speech processing toolkit,” arXiv preprint
arXiv:1804.00015, 2018.
[31] K. Deng, S. Cao, Y. Zhang, L. Ma, G. Cheng, J. Xu,
and P. Zhang, “Improving CTC-based speech recognition via
knowledge transferring from pre-trained language models,” 2022.
[Online]. Available: https://arxiv.org/abs/2203.03582
[32] S. Zhang, M. Lei, Y. Liu, and W. Li, “Investigation of modeling
units for Mandarin speech recognition using dfsmn-ctc-smbr,” in
ICASSP 2019-2019 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 7085–
7089.
[33] G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. Rush, “OpenNMT:
Open-source toolkit for neural machine translation,” in Proceedings of ACL 2017, System Demonstrations. Vancouver, Canada:
Association for Computational Linguistics, Jul. 2017, pp. 67–72.
[Online]. Available: https://www.aclweb.org/anthology/P17-4012