SELF-ATTENTION FOR AUDIO SUPER-RESOLUTION

Nathanaël Carraz Rakotonirina

Université d’Antananarivo, Madagascar

ABSTRACT
arXiv:2108.11637v1 [cs.SD] 26 Aug 2021

Convolutions operate only locally, thus failing to model

global interactions. Self-attention is, however, able to learn
representations that capture long-range dependencies in se-
quences. We propose a network architecture for audio
super-resolution that combines convolution and self-attention.
Attention-based Feature-Wise Linear Modulation (AFiLM)
uses self-attention mechanism instead of recurrent neural net- (a) High-resolution (b) Low-resolution (c) Reconstruction
works to modulate the activations of the convolutional model.
Extensive experiments show that our model outperforms Fig. 1: Spectrograms of the high-resolution signal, the sub-
existing approaches on standard benchmarks. Moreover, it sampled low-resolution signal and the reconstructed signal.
allows for more parallelization resulting in significantly faster
training.
attention instead of recurrent neural networks to alter the acti-
Index Terms— audio super-resolution, bandwidth exten- vations of the convolutions. It allows the model to efficiently
sion, self-attention incorporate long-range information. We evaluate the model
on the VCTK [10] and Piano [11] datasets. Experiments show
that our approach outperforms previous methods.
1. INTRODUCTION

Audio super-resolution is the task of reconstructing a high- 2. RELATED WORK

resolution audio signal from a low-resolution one. The sam-
pling rate of the low-resolution signal is increased. It is also Audio super-resolution. Audio super-resolution, also known
known as time series super-resolution [1] or bandwidth exten- as bandwidth extension [2, 12], is the task of predicting
sion [2]. Since audio signals are long sequences, long-range high-resolution signal using a low-resolution one. It pre-
dependencies need to be captured in order to obtain good per- dicts the signal’s high frequencies from its low frequen-
formance. The architectures that are used to process sequen- cies. More formally, given a low-resolution audio sig-
tial inputs include convolutional neural networks (CNN) [3] nal x = (x1/R1 , ..., xR1 T /R1 ) which is sampled at a rate
and recurrent neural networks (RNN) [4, 5]. Recent audio R1 /T , we want to reconstruct a high-resolution version
super-resolution models use deep neural network (DNN) with y = (y1/R2 , ..., yR2 T /R2 ) of x with a sampling rate R2 > R1 .
dense layers [6] or one-dimensional convolutions [7]. Tempo- We note r = R2 /R1 the upscaling factor. Spectrograms are
ral Feature-Wise Linear Modulation (TFiLM) [1] uses recur- presented in Figure 1 to illustrate it.
rent neural networks to modify the activations of the convolu- Early audio super-resolution models utilize matrix fac-
tions. However, there are limitations associated with convo- torization [13, 14]. They are trained on very small datasets
lutions and RNNs due respectively to limited receptive fields due to the computational cost of factorizing matrices. Dong
and the vanishing gradient problem [8]. In contrast, self- et al. [15] use analysis dictionary learning. Learning-based
attention [9], a recent advance for sequence modeling and methods exploit Gaussian mixture models [16–18] and linear
generative modeling tasks, is able to capture long-range in- predictive coding [19]. Li et al. [6] introduce a neural net-
formation and is more parallelizable. work with dense layers. The first convolutional architecture
In this paper, we consider the use of the self-attention is proposed by Kuleshov et al. [7] to scale better with dataset
mechanism for the audio super-resolution task. The architec- size. Time-Frequency Network (TFNet) [20] works in both
ture we propose makes use of a combinations of self-attention the time and frequency domain. Wang et al. [21] builds on
and convolution. We introduce the Attention-based Feature- the WaveNet model [22]. Macartney et al. [23] use the Wave-
Wise Linear Modulation (AFiLM) layer which relies on self- U-Net [24] architecture for audio super-resolution. Some
approaches [25–29] leverage generative adversarial networks a one-dimensional version of the subpixel shuffling layer [40]
(GANs) [30]. Temporal Feature-Wise Linear Modulation is used for upscaling.
(TFiLM) [1] uses RNNs to alter the activations of the convo-
lutional layers. Our approach is based on TFiLM but we use 3.2. Attention-based Feature-Wise Linear Modulation
self-attention instead of RNNs. layer
We present a novel neural network layer that exploits the self-
Self-attention. Self-attention produces a weighted average
attention mechanism in order to capture long-range depen-
of values computed from hidden units using a similarity func-
dencies. Feature-Wise Linear Modulation (FiLM) [41] is a
tion. It has been used for sequence modeling because of
neural network component that applies an affine transforma-
its ability to capture long-range interactions [31, 32]. The
tion to feature maps conditioned on some input. A function
Transformer architecture [9] using only attention without any
called FiLM generator outputs the normalizers (γ, β) which
other architecture produces state-of-the-art results in Machine
modulate the activations of a neural network called FiLM-
Translation. Recent speech enhancement approaches [33–
ed network. In Temporal Feature-Wise Linear Modulation
35] also use self-attention. Attention augmented convolutions
(TFiLM) [1], the FiLM-ed network is a convolutional model
[36] achieve significant improvement in discriminative visual
and the FiLM generator network is an RNN. This is a self-
tasks. This combination of convolution and self-attention is
conditioned model since the inputs of the FiLM generator are
also applied to speech recognition [37]. Our approach is sim-
the activations themselves. However, RNNs are less effec-
ilar in that it captures both local and global dependencies of
tive in modeling very long sequences [8]. Besides, their se-
an audio sequence.
quential nature can be computationally limiting. On the other
hand, self-attention can represent dependencies regardless of
their distance in the input sequence.
We propose the Attention-based Feature-Wise Linear
Modulation (AFiLM) layer which makes use of self-attention
as the FiLM generator to modulate the activations of the con-
volutional model as depicted in Figure 3. Our FiLM genetator
is the Transformer’s [9] basic block. Following TFiLM, the
feature map modulation is applied on a block level. The ten-
sor of activations F ∈ RT ×C is split into B blocks resulting
in a block tensor F block ∈ RB×T /B×C . Max pooling is then
used along the second dimension to downsample F block into
a tensor F pool ∈ RB×C . The normalizers γ, β ∈ RB×C
Fig. 2: Network architecture used for audio super-resolution are then obtained by applying the FiLM generator. For each
where K = 4 as used in our implementation. It consists block b, the parameters are computed as follows:
of downsampling blocks, a bottleneck block and upsampling
blocks. (γ[b, :], β[b, :]) = TransformerBlock(F pool [b, :]) (1)

where TransformerBlock is composed of a stack of multi-

head self-attention and point-wise fully connected layers.
3. PROPOSED METHOD Residual connections and layer normalization [42] are used
in the sub-layers. After that, for each block b, the normalizers
3.1. Network architecture γb and βb modulate the activations via a feature-wise affine
transformation:
We use the following naming conventions : T and C are,
respectively, the 1D spatial dimension and the number of AFiLM(F block [b, t, c]) = γ[b, c] · F block [b, t, c] + β[b, c] (2)
channels. The network architecture, presented in Figure 2,
follows the overall architecture of [1, 7] which is composed Finally, the resulting tensor is reshaped back into its original
of K downsampling blocks, a bootleneck layer and K up- shape (T, C). Different combinations of γ and β can mod-
sampling blocks; there are symmetric residual skip connec- ulate feature maps in multiple ways. Independently of the
tions [38] between blocks. The bottleneck is the same as the length of the feature map, AFiLM can efficiently influence
downsampling blocks but with Dropout [39]. Each downsam- it taking into account the global interactions. Furthermore,
pling block k = 1, 2, ..., K contains max(26+k , 512) filters by replacing the RNNs with self-attention, the model bene-
of length min(27−k + 1, 9) with a stride of 2. Each upsam- fits from better parallelization which significantly speeds up
pling block k = 1, 2, ..., K contains max(27+(K−k+1) , 512) training. The AFiLM layer is added at the end of each down-
of length min(27−(K−k+1) + 1, 9) .In the upsampling blocks, sampling, bottleneck and upsampling block as presented in
Fig. 3: The AFiLM layer uses a Transformer block as a FiLM generator. Element-wise multiplication and addition are per-
formed to modulate the feature maps. This illustration uses T = 8, C = 4, B = 2.

Figure 2. Our implementation uses K = 4 and a Transformer Table 1: Quantitative evaluation of audio super-resolution
block that is composed of a stack of 4 layers with the number models at different upsampling rates. Left/right results are
of heads h = 8 and hidden dimension d = 2048. SNR/LSD (higher is better for SNR while lower is better for
LSD). Baseline results are those reported in [1].

4. EXPERIMENTS VCTK VCTK

Method Scale Piano
Single Multi
4.1. Datasets Bicubic 2 19.0/3.5 18.0/2.9 24.8/1.8
DNN [6] 2 19.0/3.0 17.9/2.5 24.7/2.5
The model is trained and evaluated on the VCTK [10] and Pi- CNN [7] 2 19.4/2.6 18.1/1.9 25.3/2.0
ano [11] datasets. The VCTK dataset contains speech data TFiLM [1] 2 19.5/2.5 19.8/1.8 25.4/2.0
from 109 native speakers of English with various accents. AFILM 2 19.3/2.3 20.0/1.7 25.7/1.5
Each speaker reads out about 400 diferent sentences. There is Bicubic 4 15.6/5.6 13.2/5.2 18.6/2.8
a total of 44 hours of speech data. The Piano dataset contains DNN [6] 4 15.6/4.0 13.3/3.9 18.6/3.2
10 hours of Beethoven sonatas. Both datasets are used at a CNN [7] 4 16.4/3.7 13.1/3.1 18.8/2.3
sampling rate of 16 kHz. Following previous works [1, 7], we TFiLM [1] 4 16.8/3.5 15.0/2.7 19.3/2.2
apply an order 8 Chebyshev type I low-pass filter before sub- AFILM 4 17.2/3.1 15.4/2.3 20.4/2.1
sampling the high-resolution signal. The single speaker task Bicubic 8 12.2/7.2 9.8/6.8 10.7/4.0
trains the model on the first 223 recordings of VCTK Speaker DNN [6] 8 12.3/4.7 9.8/4.6 10.7/3.5
1 which is approximately 30 minutes and tests on the last 8 CNN [7] 8 12.7/4.2 9.9/4.3 11.1/2.7
recordings. Concerning the multi speaker task, we train on TFiLM [1] 8 12.9/4.3 12.0/2.9 13.3/2.6
the first 99 VCTK speakers and test on the 8 remaining ones. AFILM 8 12.9/3.7 12.0/2.7 12.9/2.5
The Piano dataset is split into 88% training, 6% validation,
and 6% testing.
4.3. Results

4.2. Training details The metrics used to evaluate the model are the the signal to
noise ratio (SNR) and the log-spectral distance (LSD) [44].
Our model is trained for 50 epochs on patches of length 8192, These are standard metrics used in the signal processing lit-
as are existing audio super-resolution models. This ensures erature. Given a reference signal y and the corresponding ap-
a fair comparison. The low-resolution audio signals are first proximation x, the SNR is defined as
processed with bicubic upscaling before they are fed into the
model. The learning-rate is set to 3 × 10−4 . The model is ||y||22
optimized using Adam [43] with β1 = 0.9 and β2 = 0.999. SNR(x, y) = 10 log (3)
||x − y||22
Table 2: Training time evaluation. The number of seconds Generalization of the super-resolution model. We inves-
per epoch was obtained using an NVIDIA Tesla K80 GPU. tigate the model’s ability to generalize to other domains. In
order to do this, we switch from speech to music and the other
Model TFiLM AFiLM way around. The results are presented in Table 3. As seen in
Number of parameters 6.82e7 1.34e8 previous models [1,7], the super-resolved samples contain the
Seconds per epoch 370 276 high frequency details but still sound noisy. The models spe-
cialize to the specific type of audio they are trained on.
The LSD measures the reconstruction quality of individual
frequencies and is defined as 6. CONCLUSION
v
T u K  2 In this work, we introduce the use of self-attention for the
1 Xu t1
X
LSD(x, y) = X(t, k) − X̂(t, k) (4) audio super-resolution task. We present the Attention-based
T t=1 K Feature-Wise Linear Modulation (AFiLM) layer which relies
k=1
on attention instead of recurrent neural networks to alter the
where X and X̂ are the log-spectral power magnitudes of y activations of the convolutional model. The resulting model
and x defined as X = log |S|2 , where S is the short-time efficiently captures long-range temporal interactions. It out-
Fourier transform (STFT) of the signal. t and k are respec- performs all previous models and can be trained faster.
tively index frames and frequencies. In future work, we want to develop super-resolution mod-
We compare our best models with existing approaches at els that generalize well to different types of inputs. We also
upscaling ratios 2, 4 and 8. The results are presented in Ta- want to investigate perceptual-based models.
ble 1. All the baseline results are those reported in [1]. Our
contributions result in an average improvement of 0.2 dB over
7. ACKNOWLEDGEMENTS
the TFiLM in terms of SNR. Concerning the LSD metric, our
approach improves by 0.3 dB on average. This shows that our We would like to thank Bruce Basset for his helpful comments
model effectively uses the self-attention mechanism to cap- and advice.
ture long-term information in the audio signal. Our attention-
based model outperforms all previous models on the multi
speaker task. It is the most difficult task and is the one that 8. REFERENCES
benefits most from more long-term information.
[1] Sawyer Birnbaum, Volodymyr Kuleshov, Zayd Enam,
Table 3: Out-of-distribution evaluation of the model. The Pang Wei W Koh, and Stefano Ermon, “Temporal
model is trained on the VCTK Multispeaker and Piano film: Capturing long-range sequence dependencies with
datasets at scale r = 2 and tested both on the same and on feature-wise modulations.,” in Advances in Neural In-
the other dataset. Left/right results are SNR/LSD. formation Processing Systems, 2019, pp. 10287–10298.

VCTK Multi (Test) Piano (Test) [2] Per Ekstrand, “Bandwidth extension of audio signals
VCTK Multi (Train) 20.0/1.7 24.4/2.5 by spectral band replication,” in in Proceedings of the
Piano (Train) 10.2/2.6 25.7/1.5 1st IEEE Benelux Workshop on Model Based Processing
and Coding of Audio (MPCA’02. Citeseer, 2002.

[3] Yann LeCun, Bernhard Boser, John S Denker, Donnie

5. DISCUSSION Henderson, Richard E Howard, Wayne Hubbard, and
Lawrence D Jackel, “Backpropagation applied to hand-
In this section, we present further analyses and discuss some written zip code recognition,” Neural computation, vol.
important aspects of the audio super-resolution task. 1, no. 4, pp. 541–551, 1989.

[4] Jeffrey L Elman, “Finding structure in time,” Cognitive

Computational performance. In addition to capturing
science, vol. 14, no. 2, pp. 179–211, 1990.
long-range dependencies, AFiLM trains faster. This is mainly
due to the Transformer’s ability to be more parallelizable [9]. [5] Sepp Hochreiter and Jürgen Schmidhuber, “Long short-
We compare the run-time efficiency of TFiLM and AFiLM. term memory,” Neural computation, vol. 9, no. 8, pp.
As presented in Table 2, even though the number of pa- 1735–1780, 1997.
rameters of AFiLM is higher, it trains on average over 34%
faster than the TFiLM model. Unlike the RNN-based model, [6] Kehuang Li, Zhen Huang, Yong Xu, and Chin-Hui Lee,
AFiLM effiently exploits the GPU resulting in much faster “Dnn-based speech bandwidth expansion and its appli-
training. cation to adding high-frequency missing features for
 automatic speech recognition of narrowband speech,” Conference on Acoustics, Speech and Signal Processing
in Sixteenth Annual Conference of the International (ICASSP). IEEE, 2011, pp. 5100–5103.
Speech Communication Association, 2015.
[18] Kun-Youl Park and Hyung Soon Kim, “Narrowband to
[7] Volodymyr Kuleshov, S Zayd Enam, and Stefano Er- wideband conversion of speech using gmm based trans-
mon, “Audio super-resolution using neural nets,” in formation,” in 2000 IEEE international conference on
ICLR (Workshop Track), 2017. acoustics, speech, and signal processing. Proceedings
(Cat. No. 00CH37100). IEEE, 2000, vol. 3, pp. 1843–
[8] Yoshua Bengio, Patrice Simard, and Paolo Frasconi,
1846.
“Learning long-term dependencies with gradient de-
scent is difficult,” IEEE transactions on neural net- [19] Jeremy Bradbury, “Linear predictive coding,” Mc G.
works, vol. 5, no. 2, pp. 157–166, 1994. Hill, 2000.
[9] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
[20] T. Y. Lim, R. A. Yeh, Y. Xu, M. N. Do, and
Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser,
M. Hasegawa-Johnson, “Time-frequency networks for
and Illia Polosukhin, “Attention is all you need,” in Ad-
audio super-resolution,” in 2018 IEEE International
vances in neural information processing systems, 2017,
Conference on Acoustics, Speech and Signal Processing
pp. 5998–6008.
(ICASSP), 2018, pp. 646–650.
[10] Junichi Yamagishi, “English multi-speaker corpus for
cstr voice cloning toolkit,” URL http://homepages. inf. [21] Mu Wang, Zhiyong Wu, Shiyin Kang, Xixin Wu, Jia
ed. ac. uk/jyamagis/page3/page58/page58. html, 2012. Jia, Dan Su, Dong Yu, and Helen Meng, “Speech super-
resolution using parallel wavenet,” in 2018 11th Inter-
[11] Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, national Symposium on Chinese Spoken Language Pro-
Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron cessing (ISCSLP). IEEE, 2018, pp. 260–264.
Courville, and Yoshua Bengio, “Samplernn: An un-
conditional end-to-end neural audio generation model,” [22] Aaron van den Oord, Sander Dieleman, Heiga Zen,
arXiv preprint arXiv:1612.07837, 2016. Karen Simonyan, Oriol Vinyals, Alex Graves, Nal
Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu,
[12] Erik Larsen and Ronald M Aarts, Audio bandwidth ex- “Wavenet: A generative model for raw audio,” arXiv
tension: application of psychoacoustics, signal process- preprint arXiv:1609.03499, 2016.
ing and loudspeaker design, John Wiley & Sons, 2005.
[23] Daniel Stoller, Sebastian Ewert, and Simon Dixon,
[13] Dhananjay Bansal, Bhiksha Raj, and Paris Smaragdis, “Wave-u-net: A multi-scale neural network for end-
“Bandwidth expansion of narrowband speech using to-end audio source separation,” arXiv preprint
non-negative matrix factorization,” in Ninth European arXiv:1806.03185, 2018.
Conference on Speech Communication and Technology,
2005. [24] Craig Macartney and Tillman Weyde, “Improved speech
enhancement with the wave-u-net,” arXiv preprint
[14] Dawen Liang, Matthew D Hoffman, and Daniel PW El-
arXiv:1811.11307, 2018.
lis, “Beta process sparse nonnegative matrix factoriza-
tion for music.,” in ISMIR, 2013, pp. 375–380. [25] Sefik Emre Eskimez and Kazuhito Koishida, “Speech
[15] Jing Dong, Wenwu Wang, and Jonathon Chambers, super resolution generative adversarial network,” in
“Audio super-resolution using analysis dictionary learn- ICASSP 2019-2019 IEEE International Conference on
ing,” in 2015 IEEE International Conference on Digital Acoustics, Speech and Signal Processing (ICASSP).
Signal Processing (DSP). IEEE, 2015, pp. 604–608. IEEE, 2019, pp. 3717–3721.

[16] Yan Ming Cheng, Douglas O’Shaughnessy, and Paul [26] Shichao Hu, Bin Zhang, Beici Liang, Ethan Zhao, and
Mermelstein, “Statistical recovery of wideband speech Simon Lui, “Phase-aware music super-resolution us-
from narrowband speech,” IEEE Transactions on ing generative adversarial networks,” arXiv preprint
Speech and Audio Processing, vol. 2, no. 4, pp. 544– arXiv:2010.04506, 2020.
548, 1994.
[27] Sen Li, Stéphane Villette, Pravin Ramadas, and Daniel J
[17] Hannu Pulakka, Ulpu Remes, Kalle Palomäki, Mikko Sinder, “Speech bandwidth extension using genera-
Kurimo, and Paavo Alku, “Speech bandwidth extension tive adversarial networks,” in 2018 IEEE International
using gaussian mixture model-based estimation of the Conference on Acoustics, Speech and Signal Processing
highband mel spectrum,” in 2011 IEEE International (ICASSP). IEEE, 2018, pp. 5029–5033.
[28] Xinyu Li, Venkata Chebiyyam, Katrin Kirchhoff, and Proceedings of the IEEE conference on computer vision
AI Amazon, “Speech audio super-resolution for speech and pattern recognition, 2016, pp. 770–778.
recognition.,” in INTERSPEECH, 2019, pp. 3416–3420.
[39] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky,
[29] Rithesh Kumar, Kundan Kumar, Vicki Anand, Yoshua Ilya Sutskever, and Ruslan Salakhutdinov, “Dropout:
Bengio, and Aaron Courville, “Nu-gan: High reso- a simple way to prevent neural networks from overfit-
lution neural upsampling with gan,” arXiv preprint ting,” The journal of machine learning research, vol.
arXiv:2010.11362, 2020. 15, no. 1, pp. 1929–1958, 2014.
[30] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, [40] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes
Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert,
Courville, and Yoshua Bengio, “Generative adversarial and Zehan Wang, “Real-time single image and video
networks,” arXiv preprint arXiv:1406.2661, 2014. super-resolution using an efficient sub-pixel convolu-
tional neural network,” in Proceedings of the IEEE
[31] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- conference on computer vision and pattern recognition,
gio, “Neural machine translation by jointly learning to 2016, pp. 1874–1883.
align and translate,” arXiv preprint arXiv:1409.0473,
2014. [41] Ethan Perez, Florian Strub, Harm De Vries, Vincent Du-
moulin, and Aaron Courville, “Film: Visual reason-
[32] Irwan Bello, Hieu Pham, Quoc V Le, Mohammad ing with a general conditioning layer,” arXiv preprint
Norouzi, and Samy Bengio, “Neural combinatorial op- arXiv:1709.07871, 2017.
timization with reinforcement learning,” arXiv preprint
arXiv:1611.09940, 2016. [42] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E
Hinton, “Layer normalization,” arXiv preprint
[33] Yuma Koizumi, Kohei Yaiabe, Marc Delcroix, Yoshiki
arXiv:1607.06450, 2016.
Maxuxama, and Daiki Takeuchi, “Speech enhancement
using self-adaptation and multi-head self-attention,” in [43] Diederik P Kingma and Jimmy Ba, “Adam: A
ICASSP 2020-2020 IEEE International Conference on method for stochastic optimization,” arXiv preprint
Acoustics, Speech and Signal Processing (ICASSP). arXiv:1412.6980, 2014.
IEEE, 2020, pp. 181–185.
[44] Augustine Gray and John Markel, “Distance measures
[34] Xiang Hao, Changhao Shan, Yong Xu, Sining Sun, and for speech processing,” IEEE Transactions on Acous-
Lei Xie, “An attention-based neural network approach tics, Speech, and Signal Processing, vol. 24, no. 5, pp.
for single channel speech enhancement,” in ICASSP 380–391, 1976.
2019-2019 IEEE International Conference on Acous-
tics, Speech and Signal Processing (ICASSP). IEEE,
2019, pp. 6895–6899.
[35] Ritwik Giri, Umut Isik, and Arvindh Krishnaswamy,
“Attention wave-u-net for speech enhancement,” in
2019 IEEE Workshop on Applications of Signal Process-
ing to Audio and Acoustics (WASPAA). IEEE, 2019, pp.
249–253.
[36] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon
Shlens, and Quoc V Le, “Attention augmented con-
volutional networks,” in Proceedings of the IEEE In-
ternational Conference on Computer Vision, 2019, pp.
3286–3295.
[37] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki
Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang,
Zhengdong Zhang, Yonghui Wu, et al., “Conformer:
Convolution-augmented transformer for speech recog-
nition,” arXiv preprint arXiv:2005.08100, 2020.
[38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
Sun, “Deep residual learning for image recognition,” in