IEEE COMMUNICATIONS LETTERS, VOL. 27, NO.

12, DECEMBER 2023 3255

Residual Stack-Aided Hybrid CNN-LSTM-Based Automatic Modulation

Classification for Orthogonal Time-Frequency Space System
Anand Kumar , Graduate Student Member, IEEE, Manish, and Udit Satija , Senior Member, IEEE

Abstract— In this letter, for the first time, we propose an The identification of modulation techniques in signals,
automatic modulation classification (AMC) method for orthog- known as automatic modulation classification (AMC), is a
onal time-frequency space (OTFS) signal modulation using a challenging task that holds great importance in a wide array
hybrid convolutional neural network and long short-term mem-
ory (CNN-LSTM) network with a residual stack. The proposed of real-world applications, particularly those characterized by
method uses in-phase and quadrature-phase (IQ) of the received high mobility and operating at high frequencies [5]. This is
OTFS modulated signal to classify the received modulation especially relevant in domains such as next-generation wireless
accurately. We consider the six digital modulation schemes such communication systems, unmanned aerial vehicles (UAVs),
as binary phase shift keying (BPSK), quadrature PSK (QPSK), satellite communications, and Internet of Things (IoTs), where
minimum-shift keying (MSK), on-off keying (OOK), 4-amplitude
shift keying (4ASK), and 8ASK for orthogonal time-frequency rapid movement and the use of high-frequency bands introduce
space (OTFS) in the delay-Doppler domain. Results depict that unique modulation challenges [6].
the proposed method achieves a high classification performance In general, AMC methods can be categorized into three
even at a low signal-to-noise ratio (SNR). main groups: likelihood-based AMC (LB-AMC), feature-
Index Terms— Deep learning, automatic modulation classifica- based AMC (FB-AMC), and distribution test-based AMC
tion, OTFS, CNN, LSTM, residual learning. (DB-AMC). The likelihood ratios between desired received
signals and known signals are computed as part of the
I. I NTRODUCTION LB-AMC process. Based on these likelihood ratios and a pre-
determined threshold, a choice of the modulation class is made
C OMMUNICATION systems with high mobility and oper-
ating at high frequencies face a significant challenge
known as Doppler spread, which arises due to the movement of
[7]. The LB-AMC provides optimal performance, because of
its enormous complexity, it may have problems converting
the transmitter, receiver, and scatterers. In scenarios with high the theoretical system model into real-world applications.
mobility, conventional orthogonal frequency-division multi- However, the FB-AMC stands out as the best option due to
plexing (OFDM) modulation, commonly utilized in 4G, 5G, its extraordinary robustness, capacity to simplify difficult prob-
and WiFi networks, encounters limitations and difficulties [1], lems, and adaptability in processing different signals. Usually,
[2]. The performance of the OFDM waveform is negatively FB-AMC includes both feature extraction and classification
affected by a significant issue called inter-carrier interference steps. Hence, the efficiency and robustness of AMC can be
(ICI), which is further exacerbated by the notable difference in significantly improved by robust and reliable feature extraction
normalized Doppler between the highest and lowest subcarri- process and classification algorithms [8], [9].
ers. Therefore, achieving synchronization is also a significant In recent times, deep learning (DL) techniques have been
hurdle. A promising solution has emerged in the form of utilized in FB-AMC to automatically extract relevant features
a new two-dimensional (2D) modulation scheme known as from received signals. This approach has shown promising
orthogonal time-frequency space (OTFS), which has recently results in achieving accurate classification by training the
been put forward as a potential choice for high-mobility models with appropriate datasets. Hong et al. [10] employed a
communications [3]. OTFS modulation is an innovative mod- deep convolutional neural network (CNN) network to address
ulation scheme in which every transmitted symbol maintains signal modulation identification (SMI). This network featured
a nearly constant channel gain, even in challenging conditions a simple structure comprising consecutive convolutional layers
like high Doppler, large antenna arrays in massive multiple- and fully connected layers. However, despite their attempts to
input multiple-output (massive MIMO), or high-frequency classify modulations under challenging channel circumstances,
environments such as millimeter waves. Specifically, the OTFS they encountered limitations in effectively capturing correla-
modulation technique maps each information symbol to a two- tions between distinct OFDM symbols using straightforward
dimensional (2D) orthogonal basis function from a predefined architectures. In [11], the authors utilize CNNs with resid-
set [4]. ual learning to effectively tackle issues such as sub-carrier
fluctuations and fading channels. The technique demonstrates
Manuscript received 2 October 2023; accepted 23 October 2023. Date of superior performance compared to established methodolo-
publication 27 October 2023; date of current version 12 December 2023. The
associate editor coordinating the review of this letter and approving it for
gies through real-time trials utilizing a radio frequency (RF)
publication was W. Yuan. (Corresponding author: Udit Satija.) dataset. However, OFDM loses subcarrier orthogonality at
The authors are with the Department of Electrical Engineering, Indian high Doppler shifts due to mobility, rendering it unsuitable for
Institute of Technology Patna, Dayalpur Daulatpur 801106, India (e-mail:
[email protected]; [email protected]; [email protected]).
high delay and Doppler conditions whereas OTFS modulation
Digital Object Identifier 10.1109/LCOMM.2023.3328011 offers better adaptability in dynamic scenarios. Therefore,
1558-2558 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: Indian Institute of Technology - Jodhpur. Downloaded on March 25,2025 at 17:29:56 UTC from IEEE Xplore. Restrictions apply.
3256 IEEE COMMUNICATIONS LETTERS, VOL. 27, NO. 12, DECEMBER 2023

Fig. 1. Block diagram of the proposed hybrid CNN-LSTM method with a residual stack.

AMC for OTFS systems will play a crucial role in future through learned representations. Fig. 1 presents the block
wireless communication where high data rates and reliable diagram of the proposed method designed for recognizing
communication are critical. There are various studies available modulations in OTFS systems. The method involves gen-
that focus on channel state information estimation to enhance erating the modulation signal using OTFS modulation and
the performance of signal detection in OTFS systems using subsequently transforming it into an OTFS signal for further
deep learning [12], [13]. However, to the best of our knowl- analysis and classification.
edge, there is no work available in the literature on AMC for
the OTFS system. A. OTFS Modulation Procedure
In this work, we present a novel method for AMC in the The OTFS modulation scheme employs a configuration
OTFS framework. To successfully complete this assignment, where two successive 2D transforms are interconnected within
we employ CNN-LSTM with residual learning. While CNN the transmitter. The basic OTFS modulation scheme’s block
represents a simpler neural network architecture to extract diagram is shown in Fig. 1. At the transmitter, windowing
relevant features in few epochs, the inclusion of LSTM is of is used after the inverse symplectic finite Fourier transform
paramount importance due to its capacity to capture long-term (ISFFT) maps the information symbols from the delay-
dependencies which has been explored in OFDM and other Doppler (DD) domain to the time-frequency (TF) domain.
modulation classification techniques. To achieve robustness Prior to the transmission through the channel, the TF symbols
and reliability in the context of modulation classification, are further transformed to the time domain using the Heisen-
we employ a combination of CNN-LSTM with a residual berg transform [14].
stack, which is able to extract spatial and temporal features, The N × M DD grid with the information symbols x[k, l]
mitigating the effect of the gradient vanishing problem that is is given by [14]
caused by random noise introduced due to high mobility.   
We considered the set of six digital modulated baseband k l
Γ= , | k = 0, . . . , N −1, l = 0, . . . , M −1 ,
signals include binary phase shift keying (BPSK), quadrature N T M ∆f
phase shift keying (QPSK), minimum shift keying (MSK), (1)
on-off keying (OOK), 4-amplitude shift keying (4ASK), and
where N and M are the number of Doppler and delay bins,
8-amplitude shift keying (8ASK). On the basis of the baseband
respectively, and N1T and M1∆f are the bin sizes in the Doppler
IQ samples alone, our proposed approach can correctly classify
and delay domains, respectively. The DD domain symbols
the modulation format for OTFS system. The remaining sec-
x[k, l] are transferred to the symbols in the TF domain X[n, m]
tions of the letter are organized as follows: Section II outlines
using ISFFT. The TF signal can be expressed as follows,
the OTFS system model. In Section III, we elaborate on the
according to rectangular windowing
proposed neural network for AMC, encompassing the dataset
N −1 M −1
generation procedure and training methodology. Moving for- 1
x [k, l] ej2Π( N − M ) .
X X nk ml

ward, Section IV provides a comprehensive description of the X [n, m] = √ (2)

M N k=0 l=0
simulation results. Ultimately, the concluding remarks of the
proposed work are presented in the final section. Using the Heisenberg transform [15] and transmit signal
gtx (t), this TF signal is transformed into a time domain signal
x(t).
II. S YSTEM M ODEL
N −1 M −1
This study adopts a hybrid CNN-LSTM-based technique for
X X
x(t) = X[n, m]gtx (t − nT )ej2πm∆f (t−nT ) . (3)
AMC in the OTFS system, enabling modulation recognition n=0 m=0

Authorized licensed use limited to: Indian Institute of Technology - Jodhpur. Downloaded on March 25,2025 at 17:29:56 UTC from IEEE Xplore. Restrictions apply.
KUMAR et al.: RESIDUAL STACK-AIDED HYBRID CNN-LSTM-BASED AMC FOR OTFS SYSTEM 3257

for stabilization. Further processing includes global average

pooling to condense the data, followed by a dense layer with
128 neurons. Finally, the output culminates with a softmax
classification layer with 6 neurons corresponding to the 6 clas-
sification classes, producing the probability distribution for
the predicted modulation classes. This comprehensive network
architecture effectively extracts hierarchical features from the
Fig. 2. Structure of the residual stack. IQ samples, enabling accurate classification of modulation
schemes.
The signal x(t) that is sent through the channel experiences
the effects of the channel’s complex baseband response in the
DD domain, represented by h(τ, ν). B. Dataset Generation

Z
We generate a set of six digital modulated baseband signals
X in OTFS, where the considered modulation schemes include
h(τ, ν) = hi δ(τ − τi )δ(ν − νi ), (4)
i=1
BPSK, QPSK, MSK, OOK, 4ASK, and 8ASK. We analyze
the OTFS signal consisting of M = 64 subcarriers in the
where Z represents the number of paths in the DD domain. frequency domain and N = 32 symbols in the time domain.
The variables hi , τi , and νi refer to the channel gain, delay, and We gather a dataset by capturing the received signals of the six
Doppler shift, respectively. The signal received at the receiver, modulated signals. Each signal in the dataset covers a signal-
r(t), is the time-domain representation of the transmitted to-noise ratio (SNR) range from −10 dB to +20 dB, with
signal after passing through the channel, given as, increments of 5 dB. For each modulation type at a specific
Z ∞Z ∞
SNR level, we generate a total of 4096 examples, each con-
r(t) = h(τ, ν)x(t − τ )ej2πν(t−τ ) dτ dν + v(t), sisting of 2048 samples. In total, we have 172,032 examples of
−∞ −∞
data. During the neural network training, we employ one-hot
(5)
encoding to label the collected data.
where v(t) denotes the additive white Gaussian noise. The received signal can be written in the form of a
In this letter, the input to the proposed network for modu- 2048 × 2 matrix, such as
lation classification is represented by the in-phase component  K
(I) and quadrature component (Q) of r(k) which is a discrete D = [I Q]2048×2 i=1 , (7)
sampled version of r(t), commonly referred to as IQ signals, where K is the number of training examples.
which can be expressed as
K K
I = {real[r[k]]}k=0 , and Q = {imag[r[k]]}k=0 , (6) C. Training Process
where K is the number of training examples. To ensure effective training and evaluation, we divide the
generated dataset into two sets: 70 percent of the data is
used for training the neural network, while the remaining
III. P ROPOSED M ETHODOLOGY
30 percent is reserved for validation. The weights are learned
In this section, we present a detailed explanation of the using the Adam optimizer, with a fixed learning rate of 0.01,
proposed CNN-LSTM with residual stack-based AMC for the and the objective is to minimize the categorical cross-entropy
OTFS system. loss function. Throughout the training process, which spans
100 epochs, we employ a batch size of 512 examples. This
A. Proposed DL-Based AMC Architecture setup allows us to assess the performance and generalization
capabilities of the trained networks on unseen data.
The neural network architecture begins with an input shape
of (2048, 2), representing IQ samples of received signals.
These samples are passed through two cascaded CNN layers IV. S IMULATION R ESULTS AND D ISCUSSION
with a kernel size of 1×7 and 64 kernels each. Each CNN layer In this section, we simulate the classification performance
employs the scaled exponential linear unit (SELU) activation of the proposed hybrid CNN-LSTM-based AMC for OTFS
function, followed by batch normalization and a dropout of systems over delay-Doppler channels. In particular, we inves-
0.3 for regularization. The output of the CNN layers is then tigate the performance of OTFS-modulated signals in terms of
fed into a max-pooling layer with a pool size of 2 × 2 and classification accuracy in percentage. In Table I, all relevant
strides of 2. simulation parameters are listed. We consider the extensive
Subsequently, the pooled output enters a triple-stack residual vehicle A model as the channel model for path delays (or
network, following a design specified in Fig. 2. After this, the delay taps) and generate a single Doppler shift for each delay
result undergoes batch normalization and the SELU activation tap using Jakes’ formula [14].
function. The processed data then passed through two cas- Fig. 3 (a) illustrates the training loss and validation loss
caded LSTM layers to track the long-term dependency of the of the proposed model. It can be observed from the figure
extracted pattern from the input data [16]. Each LSTM has that the model is effectively learning with rising epochs from
a memory size of 48. Batch normalization is applied again the training data and generalizing well to new unseen data as

Authorized licensed use limited to: Indian Institute of Technology - Jodhpur. Downloaded on March 25,2025 at 17:29:56 UTC from IEEE Xplore. Restrictions apply.
3258 IEEE COMMUNICATIONS LETTERS, VOL. 27, NO. 12, DECEMBER 2023

TABLE I
S IMULATION PARAMETERS

Fig. 4. (a) Individual classification accuracy for the proposed method,

(b) t-SNE plot of discriminant features learned through our proposed hybrid
CNN-LSTM method.

TABLE II
C LASSIFICATION ACCURACY C OMPARISON

Fig. 3. (a) Training and validation curve, (b) Classification accuracy vs. SNR
plot for different activation functions.

the training advances by studying the training and validation

curves. Testing is performed on the trained model where
each test example has 2048 samples. Fig. 3 (b) presents
the classification performance analysis for the SELU and
ReLU activation functions. As the SNR value increases, the enough for each class. The pairwise neighboring similarities
classification performance for SELU outperforms the ReLU are measured by t-SNE using a probabilistic method, first
activation function. Unlike ReLU, SELU not only effectively in high-dimensional space and then in the corresponding
processes positive-valued features but also has the ability low-dimensional space using the l2 -norm metric. The low-
to capture useful information from negative-valued features, dimensional (2-D) feature visualization for six classes using
which can significantly enhance classification accuracy as the t-SNE is displayed in Fig. 4 (b). The t-SNE graph demonstrates
received signal contains both positive and negative values the capacity of the CNN-LSTM network in learning the
exhibiting phase information in the time domain. discriminant features for different modulation classes [17].
The classification accuracy of six individual modulation The comparative analysis is presented in Fig. 6 and Table II.
formats is shown in Fig. 4 (a) using our proposed method The proposed model for the OTFS system demonstrates supe-
at different SNRs. All modulation classes have accuracy more rior classification accuracy when compared to DL-SMI [10]
than 90% at 10 dB SNR, except for QPSK, which is more and ResNet [11] designed for the OFDM system. To facilitate
complex and susceptible to phase ambiguity compared to other this comparison, we adapted the existing deep learning model
modulations. Fig. 5 presents the confusion matrix visualization designed for OFDM to evaluate with the OTFS system. The
of the test data across all SNR values. The results show that improved classification accuracy of our proposed network for
at 0 dB, BPSK is misclassified as QPSK, MSK, 4ASK, and the OTFS system can be attributed to its ability to effectively
8ASK at few instances. Moreover, the test outcomes also handle scenarios with high mobility. In contrast, existing
indicate misclassifications among QPSK, MSK, 4ASK, and models are ill-equipped to cope with such high mobility
8ASK. Notably, the confusion matrix almost perfectly forms scenarios. At lower SNRs, the proposed CNN-LSTM model
a diagonal at high SNR levels, indicating highly accurate with the OFDM system exhibits better performance compared
categorization. However, there is some ambiguity in the classi- to the CNN-LSTM model with the OTFS system. However,
fication at 0 dB SNR. However, the classification performance it’s important to note that the CNN-LSTM model with OFDM
considerably improves as the SNR rises. As shown in 5 (c), struggles in scenarios with high mobility, characterized by high
the best classification outcome is at 10 dB SNR. Doppler effects, which disrupt orthogonality and result ICI and
To learn more about the high-dimensional feature space and inter-symbol interference (ISI). In contrast, the classification
our system performance, we visualize it using t-distributed accuracy remains consistent when employing the CNN-LSTM
stochastic neighbor embedding (t-SNE) which is especially model with the OTFS system at high SNRs.
well-suited for the visualization of large datasets. From the The complexity analysis of the proposed neural network
incoming input, the convolution layers extract the features. reveals that the training process takes approximately 3 hours,
We fed the features from the previous layer (before the clas- while making predictions on each test dataset requires about
sification layer) into the t-SNE algorithm with the appropriate 126.9 milliseconds. Table III compares the proposed model
labels to examine how these feature spaces are discriminating with DL-SMI [10] and ResNet [11]. Notably, the proposed

Authorized licensed use limited to: Indian Institute of Technology - Jodhpur. Downloaded on March 25,2025 at 17:29:56 UTC from IEEE Xplore. Restrictions apply.
KUMAR et al.: RESIDUAL STACK-AIDED HYBRID CNN-LSTM-BASED AMC FOR OTFS SYSTEM 3259

Fig. 5. (a) Confusion matrix at 0 dB SNR, (b) Confusion matrix at 5 dB SNR, (c) Confusion matrix at 10 dB SNR.

[2] R. Hadani et al., “Orthogonal time frequency space modulation,”

in Proc. IEEE Wireless Commun. Netw. Conf. (WCNC), Mar. 2017,
pp. 1–6.
[3] W. Yuan et al., “New delay Doppler communication paradigm in 6G era:
A survey of orthogonal time frequency space (OTFS),” China Commun.,
vol. 20, no. 6, pp. 1–25, Jun. 2023.
[4] M. K. Ramachandran, G. D. Surabhi, and A. Chockalingam, “OTFS:
A new modulation scheme for high-mobility use cases,” J. Indian Inst.
Sci., vol. 100, no. 2, pp. 315–336, Apr. 2020.
[5] A. P. Hermawan, R. R. Ginanjar, D.-S. Kim, and J.-M. Lee, “CNN-based
automatic modulation classification for beyond 5G communications,”
IEEE Commun. Lett., vol. 24, no. 5, pp. 1038–1041, May 2020.
[6] L. Xiao, S. Li, Y. Qian, D. Chen, and T. Jiang, “An overview of OTFS for
Internet of Things: Concepts, benefits, and challenges,” IEEE Internet
Things J., vol. 9, no. 10, pp. 7596–7618, May 2022.
[7] U. Satija, M. Mohanty, and B. Ramkumar, “Automatic modulation
Fig. 6. Comparison analysis of the proposed and existing AMC methods. classification using S-transform based features,” in Proc. 2nd Int. Conf.
Signal Process. Integr. Netw. (SPIN), Feb. 2015, pp. 708–712.
[8] T. Yucek and H. Arslan, “A novel sub-optimum maximum-likelihood
TABLE III modulation classification algorithm for adaptive OFDM systems,”
C OMPARATIVE C OMPLEXITY A NALYSIS in Proc. IEEE Wireless Commun. Netw. Conf., vol. 2, Mar. 2004,
pp. 739–744.
[9] W. Wei and J. M. Mendel, “A new maximum-likelihood method for
modulation classification,” in Proc. Conf. Rec. 29th Asilomar Conf.
Signals, Syst., Comput., vol. 2, 1995, pp. 1132–1136.
[10] S. Hong, Y. Zhang, Y. Wang, H. Gu, G. Gui, and H. Sari, “Deep learning-
based signal modulation identification in OFDM systems,” IEEE Access,
vol. 7, pp. 114631–114638, 2019.
model consists of a smaller parameter count at 151,656, which [11] A. Kumar, K. K. Srinivas, and S. Majhi, “Automatic modulation classifi-
cation for adaptive OFDM systems using convolutional neural networks
is less than that of DL-SMI [10] and ResNet [11]. Additionally, with residual learning,” IEEE Access, vol. 11, pp. 61013–61024,
its prediction time is significantly lower than DL-SMI [10] 2023.
and ResNet [11]. Furthermore, the proposed model exhibits [12] S. Li, C. Ding, L. Xiao, X. Zhang, G. Liu, and T. Jiang, “Expec-
tation propagation aided model driven learning for OTFS signal
a reduced floating-point operations (flop) count, indicating detection,” IEEE Trans. Veh. Technol., vol. 72, no. 9, pp. 1–6,
lower computational complexity and reduced time complexity Sep. 2023.
compared to DL-SMI [10] and ResNet [11]. [13] C. Liu, S. Li, W. Yuan, X. Liu, and D. W. K. Ng, “Predictive precoder
design for OTFS-enabled URLLC: A deep learning approach,” IEEE J.
Sel. Areas Commun., vol. 41, no. 7, pp. 2245–2260, Jul. 2023.
V. C ONCLUSION
[14] P. Raviteja, K. T. Phan, Y. Hong, and E. Viterbo, “Interference
In this work, we propose CNN-LSTM with a residual stack cancellation and iterative detection for orthogonal time frequency
for categorizing six different modulation schemes in high space modulation,” IEEE Trans. Wireless Commun., vol. 17, no. 10,
pp. 6501–6515, Oct. 2018.
mobility delay-Doppler OTFS system. The proposed method [15] K. Deka, A. Thomas, and S. Sharma, “OTFS-SCMA: A code-domain
achieves a classification accuracy over 98.5% at an SNR of NOMA approach for orthogonal time frequency space modulation,”
20 dB and works well even at low SNR. IEEE Trans. Commun., vol. 69, no. 8, pp. 5043–5058, Aug. 2021.
[16] J. Cheng, L. Dong, and M. Lapata, “Long short-term memory-networks
for machine reading,” 2016, arXiv:1601.06733.
R EFERENCES
[17] Y. Zhang, S. Dong, C. Zhu, M. Balle, B. Zhang, and L. Ran,
[1] Z. Wei et al., “Orthogonal time-frequency space modulation: A promis- “Hand gesture recognition for smart devices by classifying deterministic
ing next-generation waveform,” IEEE Wireless Commun., vol. 28, no. 4, Doppler signals,” IEEE Trans. Microw. Theory Techn., vol. 69, no. 1,
pp. 136–144, Aug. 2021. pp. 365–377, Jan. 2021.

Authorized licensed use limited to: Indian Institute of Technology - Jodhpur. Downloaded on March 25,2025 at 17:29:56 UTC from IEEE Xplore. Restrictions apply.