1702 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL.

68, 2020

Model-Driven Deep Learning for MIMO Detection

Hengtao He , Student Member, IEEE, Chao-Kai Wen , Member, IEEE, Shi Jin , Senior Member, IEEE,
and Geoffrey Ye Li, Fellow, IEEE

Abstract—In this paper, we investigate the model-driven deep exponentially with the number of decision variables. Some
learning (DL) for MIMO detection. In particular, the MIMO de- suboptimal linear detectors, such as zero-forcing (ZF) and linear
tector is specially designed by unfolding an iterative algorithm and minimum mean-squared error (LMMSE) detectors, are with
adding some trainable parameters. Since the number of trainable
parameters is much fewer than the data-driven DL based signal reduced computational complexity, but have a huge performance
detector, the model-driven DL based MIMO detector can be rapidly degradation compared with the ML detection.
trained with a much smaller data set. The proposed MIMO detector With excellent performance and moderate complexity, itera-
can be extended to soft-input soft-output detection easily. Further- tive detectors, based on approximate message passing (AMP) [8]
more, we investigate joint MIMO channel estimation and signal and expectation propagation (EP) [9] algorithms, have been
detection (JCESD), where the detector takes channel estimation
error and channel statistics into consideration while channel esti- proposed for MIMO detection [6], [7]. The AMP-based detector
mation is refined by detected data and considers the detection error. approximates the posterior probability on a dense factor graph
Based on numerical results, the model-driven DL based MIMO by using the central limit theorem and the Taylor expansion,
detector significantly improves the performance of corresponding which can achieve Bayes-optimal performance in the large-
traditional iterative detector, outperforms other DL-based MIMO
scale systems when the elements of the channel matrix are
detectors and exhibits superior robustness to various mismatches.
with independent and identically sub-Gaussian distribution. The
Index Terms—Deep learning, Model-driven, MIMO detection, EP-based detector [7], [9] is derived by approximating the pos-
Iterative detector, Neural network, JCESD. terior distribution with factorized Gaussian distributions and can
I. INTRODUCTION achieve Bayes-optimal performance when the channel matrix is
unitarily-invariant1 and with a large scale. However, for practical
ULTIPLE-INPUT multiple-output (MIMO) technology small-size (e.g., 4 × 4 or 8 × 8) MIMO systems, the perfor-
M can dramatically improve the spectral efficiency and link
reliability and has been applied to many wireless communication
mance of these iterative detectors is still far from Bayes-optimal
solution and has serious deterioration with correlated MIMO
systems. To obtain the benefits of MIMO [2], efficient chan- channels and imperfect channel state information (CSI) [10].
nel estimation and signal detection algorithms, which balance Owing to strong learning ability from the data, deep learn-
performance and complexity, are essential in receiver design ing (DL) has been successfully introduced to computer vision,
and have arouse a series of research [3]–[7]. Among existing automatic speech recognition, and natural language process-
detectors, maximum likelihood (ML) detection can achieve ing. Recently, it has been applied in physical layer communi-
the optimal performance. However, its complexity increases cations [11]–[13], such as channel estimation [14]–[16], CSI
feedback [17], signal detection [18]–[26], channel coding [27],
Manuscript received July 22, 2019; revised December 13, 2019 and February [28], and end-to-end transceiver design [29], [30]. In particular,
10, 2020; accepted February 23, 2020. Date of publication February 28, 2020;
date of current version March 16, 2020. The associate editor coordinating the a five-layer fully connected deep neural network (DNN) is
review of this manuscript and approving it for publication was Prof. B. Shim. embedded into an orthogonal frequency-division multiplexing
This work was supported in part by the National Key Research and Development (OFDM) system for joint channel estimation and signal de-
Program 2018YFA0701602, in part by the National Science Foundation of China
(NSFC) for Distinguished Young Scholars with Grant 61625106, and in part by tection (JCESD) by treating the receiver as a black box and
the NSFC under Grant 61941104. The work of Hengtao He was supported in without exploiting domain knowledge [18]. However, training
part by the Scientific Research Foundation of Graduate School of Southeast such a black-box-based network requires a lot of training time
University under Grant YBPY1939 and in part by the Scholarship from the
China Scholarship Council under Grant 201806090077. The work of Chao- in addition to a huge data set. On the other hand, model-driven
Kai Wen was supported in part by the Ministry of Science and Technology DL constructs the network topology based on known domain
of Taiwan under Grants MOST 108-2628-E-110-001-MY3 and in part by the knowledge and has been successfully applied to image recon-
ITRI in Hsinchu, Taiwan. This paper was presented in part at the IEEE Global
Conference Signal and Information Processing (Globalsip), Anaheim, CA, Nov. struction [32], sparse signal recovery [33]–[36], and wireless
2018 [1]. (Corresponding author: Shi Jin.) communications recently [1], [12].
Hengtao He and Shi Jin are with the National Mobile Communications For MIMO detection, a specifically designed network, named
Research Laboratory, Southeast University, Nanjing 210096, China (e-mail:
[email protected]; [email protected]). DetNet, has been proposed in [19] by unfolding the iteration of
Chao-Kai Wen is with the Institute of Communications Engineering, Na- a projected gradient descent algorithm and adding considerable
tional Sun Yat-sen University, Kaohsiung 804, Taiwan (e-mail: chaokai.wen@ trainable variables. DetNet has comparable performance with
mail.nsysu.edu.tw).
Geoffrey Ye Li is with the School of Electrical and Computer Engineering,
Georgia Institute of Technology, Atlanta, GA 30332 USA (e-mail: liye@ece. 1 A matrix A = UΣV is unitarily-invariant if U, Σ and V are mutually
gatech.edu). independent, and U, V are Haar-distributed. The independent and identically
Digital Object Identifier 10.1109/TSP.2020.2976585 distributed (i.i.d.) Gaussian matrix is a typical unitarily-invariant matrix.

1053-587X © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
HE et al.: MODEL-DRIVEN DEEP LEARNING FOR MIMO DETECTION 1703

the AMP-based detector and is more robust to ill-conditioned architecture,2 we introduce the signal detection considering
channels [19]. To further reduce the number of learnable param- channel estimation error and data-aided channel estimation.3
eters and improve convergence, the approaches in [20] and [21] We assume that channel matrix H ∈ CNr ×Nt does not
use DL techniques for the belief propagation and message pass- change in a time slot. In each time slot, Np pilot vec-
ing detector, respectively. In [26], a DL-based sphere decoding tors xp [n] ∈ CNt ×1 for n = 1, . . . , Np , are first transmitted,
algorithm is proposed, where the radius of the decoding hyper- which are followed Nd data vectors, xd [n] ∈ CNt ×1 . The re-
sphere is learned by DNN. The performance achieved by this ceived signal vectors are yp [n] ∈ CNr ×1 for n = 1, . . . , Np and
algorithm is very close to the optimal ML detection. However, yd [n] ∈ CNr ×1 for n = 1, . . . , Nd corresponding to the pilot
most of the existing DL-based detector assume accurate CSI at and data vectors, respectively. We can also express them into
the receiver and ignore the channel estimation error. matrix forms as Xp = (xp [1], . . . , xp [Np ]) ∈ CNt ×Np , Yp =
Motivated by existing works, we develop a model-driven DL (yp [1], . . . , yp [Np ]) ∈ CNr ×Np , Xd = (xd [1], . . . , xd [Nd ]) ∈
network, named OAMP-Net2, for MIMO detection in this arti- CNt ×Nd , and Yd = (yd [1], . . . , yd [Nd ]) ∈ CNr ×Nd .
cle, where the iterative detector is improved with a few number
of trainable variables to adapt to various channel environments. A. JCESD Architecture
The structure of the detector is obtained by unfolding the OAMP
detector, which is similar to our early work in [1] and inspired As in Fig. 1, we consider a turbo-like JCESD architecture for
by the TISTA network [34] but adds more trainable parame- MIMO systems in this paper, which shares the same spirit as iter-
ters. Furthermore, an OAMP-Net2-based JCESD architecture is ative decoding. In JCESD, channel estimator and signal detector
proposed for imperfect CSI and data-aided scheme is utilized exchange information iteratively until convergence [40]. In the
to further improve channel estimation. The trainable parameters first iteration, pilot-only based channel estimation is performed.
are optimized by DL technique to adapt to various channel envi- In the subsequent iterations, data-aided channel estimation is
ronments and take channel estimation error into consideration. employed with the help of the detected data.
The main contributions of this paper are summarized as follows: The input of the JCESD architecture is the pilot signal matrix,
• Different from the existing DL-based MIMO detector [19]– Xp , received signal matrix corresponding to the pilot matrix,
[24], [26] with perfect CSI, we consider the MIMO detec- Yp , corresponding to the data matrix, Yd in each time slot. In
tion with estimated channel, which improves the perfor- the l-th turbo iteration, Ĥ(l) is the estimated channel matrix,
(l) (l) (l)
mance of MIMO receiver by considering the characteristics X̂d is the estimated data matrix, and V̂est and V̂det are used
of channel estimation error and channel statistics and using to compute the covariance matrix for equivalent noise in signal
the estimated payload data to refine the channel estimation. detector and channel estimator, respectively. The final output of
(L)
• Compared with the existing DL-based MIMO detec- the signal detector is finally detected data matrix X̂d , where
tor [19]–[21], [26], our proposed detector can provide L is the total number of turbo iterations.
soft-output information for decoder and absorb the soft Compared with the conventional receiver design where the
information. In addition, only a few trainable parameters channel estimator and signal detector are designed separately,
are required to be learned, which can reduce the demand this architecture can improve the performance of the receiver
for computing resources and training time significantly. by considering the characteristics of channel estimation error in
• Based on our numerical results, the OAMP-Net2 has con- addition to channel statistics when performing signal detection
siderable performance gain compared with the OAMP and using the estimated payload data for channel estimation, as
detector. Furthermore, OAMP-Net2 has strong robustness we will illustrate subsequently.
to signal-to-noise (SNR), channel correlation, modulation
symbol and MIMO configuration mismatches. B. Signal Detection With Channel Estimation Error
Notations: For any matrix A, AT , AH , and tr(A) denote the In the MIMO system, the received data signal vector yd [n]
transpose, conjugated transpose, and trace of A, respectively. corresponding to the n-th data vector can be expressed by
In addition, I is the identity matrix, 0 is the zero matrix. A
proper complex Gaussian with mean µ and covariance Ω can yd [n] = Hxd [n] + nd [n], (1)
be described by the probability density function,
where nd [n] ∼ NC (0, σ 2 INr ) is the additive white Gaussian
1 H −1 noise (AWGN) vector. Note that (1) can also be expressed into
NC (z; µ, Θ) = e−(z−µ) Θ (z−µ) . matrix from as
det(πΘ)
Yd = HXd + Nd , (2)
The rest of this paper is organized as follows. After introducing
Nr ×Nd
the JCESD architecture in Section II, we develop channel esti- where Nd = (nd [1], . . . , nd [Nd ]) ∈ C is the AWGN ma-
mator in Section III and propose the model-driven DL detector in trix in the data transmission stage. Denote the estimated channel
Section IV. Then, numerical results are presented in Section V.
Finally, Section VI concludes the paper. 2 One can introduce various JCESD architectures for MIMO systems, includ-
ing the schemes in [37]–[41] we present a turbo-like JCESD architecture similar
II. JOINT CHANNEL ESTIMATION AND SIGNAL DETECTION to [40] in this paper.
3 Although we mainly investigate the model-driven-DL-based MIMO detector
In this section, we consider a MIMO system with Nt trans- in this paper, we first introduce a JCESD architecture and then elaborate the
mit and Nr receive antennas. After presenting the JCESD channel estimator and signal detector modules, respectively.

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
1704 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 68, 2020

Fig. 1. The diagram of the turbo-like JCESD architecture. The channel estimator and signal detector exchange information iteratively until convergence.

Ĥ ∈ CNr ×Nt as estimator as additional pilot. Then, the received signal matrix
Yd corresponding to X̂d can be expressed as
Ĥ = H + ΔH, (3)
Yd = HXd + Nd
where ΔH is the channel estimation error. If the estimated
channel is used for signal detector, the signal detection problem = H(X̂d − Ed ) + Nd
can be formulated as
= HX̂d + (Nd − HEd )
yd [n] = Hxd [n] + nd [n]
= HX̂d + N̂p , (7)
= (Ĥ − ΔH)xd [n] + nd [n]

= Ĥxd [n] + nd [n] − ΔHxd [n] where N̂p = Nd − HEd is the equivalent noise for additional
pilot part X̂d . The statistical information of the n-th column of
= Ĥxd [n] + n̂d [n], (4) N̂p , n̂p [n] ∼ NC (0, V̂p [n]) for n = 1, . . . , Nd , where V̂p [n]
is calculated in Appendix A and will be utilized in data-aided
where n̂d [n] = nd [n] − ΔHxd [n] is the equivalent noise in
channel estimation stage. Then, we denote Y = (Yp , Yd ) as re-
signal detector which includes the contribution of channel esti-
ceived signal matrix corresponding to overall transmitted signal
mation error and original additive noise. n̂d [n] ∼ NC (0, V̂d [n])
matrix. Based on (5) and (7), we have
is assumed to be Gaussian distribution,4 and the covariance
matrix V̂d [n] can be obtained by considering the statistical
Y = (Yp Yd )
properties of the channel estimation error and detailed calculated
process is shown in Appendix A. = (HXp + Np , HX̂d + N̂p )

C. Data-Aided Channel Estimation = H(Xp , X̂d ) + (Np , N̂p )

In this section, we introduce the data-aided channel estimation = HX + N, (8)

approach. Conventional pilot-only based channel estimation is
performed and transmitted symbols are detected in the first where X = (Xp , X̂d ), N = (Np , N̂p ) can be interpreted as the
iteration. Then, the detected symbols are fed back to the chan- equivalent pilot signal and noise in data-aided channel estima-
nel estimator as additional pilot symbols to refine the channel tion stage, respectively.
estimation.
In the channel training stage, pilot matrix Xp is transmitted, D. Model-Driven DL for JCESD
similar to (1), the received signal matrix corresponding to the
pilot matrix Yp ∈ CNr ×Np can be expressed as In Section II-A, we have introduced the principle of the
JCESD architecture. Compared with other DL-based JCESD
Yp = HXp + Np , (5) architecture [16], [18], which uses a large number of data to train
where Np = (np [1], . . . , np [Np ]) ∈ C Nr ×Np
is the AWGN ma- the black-box-based network, we construct the network archi-
trix and each column np [n] ∼ NC (0, σ 2 INr ) for n = 1, . . . , Np . tecture by employing model-driven DL and domain knowledge.
In particular, we employ the LMMSE channel estimator and
The estimated data matrix X̂d can be expressed as
construct the model-driven DL detector. As in [1], the proposed
X̂d = Xd + Ed , (6) model-driven DL detector is obtained by unfolding the existing
iterative detector and adding several trainable parameters, which
where Ed is the signal detection error matrix. In data-aided fully exploiting the domain knowledge. This architecture is
channel estimation stage, estimated X̂d are fed back to channel promising as it inherits the superiority of traditional approach
and uses DL technique to improve the performance. We will
4 The assumption that n̂ [n] is Gaussian distribution is reasonable as we
d
introduce the channel estimator and signal detector in detail in
consider the LMMSE channel estimator in next section. Sections III and IV, respectively.

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
HE et al.: MODEL-DRIVEN DEEP LEARNING FOR MIMO DETECTION 1705

III. LMMSE CHANNEL ESTIMATOR

Algorithm 1: OAMP Algorithm for MIMO Detection.
To facilitate the representation of the channel estimation Input: Received signal yd , estimated channel matrix Ĥ,
problem, we apply matrix vectorization to (5) and rewrite it as equivalent noise covariance matrix Rn̂d n̂d .
y p = Ap h + n p , (9) Output: Recovered signal x̂d,T +1 .
Initialize: τ1 ← 1, x̂d,1 ← 0
where Ap = XTp ⊗ INr ∈ CNp Nr ×Nt Nr , yp = vec(Yp ) ∈  
CNp Nr ×1 , h = vec(H) ∈ CNr Nt ×1 and np = vec(Np ) ∈ rt = x̂d,t + Wt yd − Ĥx̂d,t , (14)
CNp Nr ×1 . We denote ⊗ as the matrix Kronecker product
and vec(·) as the vectorization operation. In pilot-only based x̂d,t+1 = E {x|rt , τt } , (15)
channel estimation stage, the LMMSE estimate of h is given by
y − Ĥx̂d,t 22 − tr (Rn̂d n̂d )
 −1 vt2 =   , (16)
ĥp = Rhh AH H
p Ap Rhh Ap + σ INp Nr
2
yp , (10) tr ĤH Ĥ
where Rhh is the channel covariance matrix. Based on the
1   2 1  
property of LMMSE estimate, ĥp is a Gaussian random vector. τt2 = tr Bt BH
t vt + tr Wt Rn̂d n̂d WtH .
Nt Nt
The channel estimation error vector Δhp = ĥp − h is also a (17)
Gaussian random vector with zero-mean and the covariance
matrix RΔhp can be computed as
 −1
RΔhp = Rhh −Rhh AH H
p Ap Rhh Ap +σ INp Nr
2
Ap Rhh .
(11) way, when estimated channel Ĥ and received signal yd are
In data-aided channel estimation stage, considering data feed- available.
back to channel estimator, the LMMSE channel estimation is The OAMP detector is presented in Algorithm 1 and mainly
given by contains two modules: linear estimator (14) and nonlinear es-
 −1 timator (15). The error variance estimators (16) and (17) are
ĥ = Rhh AH ARhh AH + Rnn y. (12) the average variance of the two error vectors pt and qt , where
pt = rt − xd and qt = x̂d,t − xd are used to measure the ac-
where A = XT ⊗ INr ∈ CNc Nr ×Nt Nr , y = vec(Y) ∈
Nc Nr ×1 curacy of the output in the linear and nonlinear estimators,
C , n = vec(N) ∈ CNc Nr ×1 and Nc = Np + Nd .
respectively. They are defined as
The covariance matrix RΔh of the channel estimation error
vector Δh = ĥ − h can be computed as E[qt 22 ] 2 E[pt 22 ]
 −1 vt2 = , τt = , (18)
RΔh = Rhh − Rhh AH ARhh AH + Rnn ARhh . Nt Nt
(13) and can be computed by employing (16) and (17).
The covariance matrix Rnn contains the equivalent noise power 1) Linear Estimator: The matrix Wt in linear estimator (14)
of the actual pilot Xp and additional pilot part X̂d , which is can be the transpose of Ĥ, the pseudo inverse of Ĥ or the
calculated in Appendix A. LMMSE matrix. From [42], the optimal one is

IV. MODEL-DRIVEN DL DETECTOR Nt

Wt =   Ŵt , (19)
In this section, we develop a model-driven DL detector, called tr Ŵt Ĥ
OAMP-Net2, which has been partly presented in our early
work [1]. After introducing the principle of the OAMP detector, where Ŵt is the LMMSE matrix given by
we present the structure of OAMP-Net2 in detail. Afterwards,  −1
computational complexity and extension to soft-input and soft- Ŵt = vt2 ĤH vt2 ĤĤH + Rn̂d n̂d , (20)
output detection are also investigated.
where vt2 is expressed in (16), and Rn̂d n̂d is the covariance ma-
A. OAMP Detector trix of the equivalent noise n̂d in signal detector, which includes
The OAMP algorithm has been proposed to solve sparse linear the contribution of the channel estimation error ΔH and original
inverse problems in compressed sensing [42] and can be utilized additive noise nd , and has been discussed in Section II-B.
for MIMO detection in Algorithm 1. The goal of the OAMP The matrix Wt is called de-correlated when tr(Bt ) = 0, where
algorithm is to recover the transmitted signal vector xd from the Bt = I − Wt Ĥ, and thus ensures the entries of pt are uncorre-
received signal vector yd = Hxd + nd .5 The principle of the lated with those of xd and mutually uncorrelated with zero-mean
algorithm is to decouple the posterior probability P(xd |yd , Ĥ) and identical variances.
into a series of P(xi |yd , Ĥ)(i = 1, 2, . . . , Nt ) in an iterative 2) Nonlinear Estimator: The nonlinear estimator in the
OAMP detector is constructed by MMSE estimate of xd , which
5 As each transmitted symbol vector x (n) for n = 1, . . . , N in each time
d d
is with respect to the equivalent AWGN channel
slot shares the same channel H, we omit the time index n and use xd to refer
xd [n] for simplicity. rt = xd,t + wt , (21)

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
1706 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 68, 2020

Fig. 2. Block diagram of OAMP-Net2 detector. The network consists of T cascade layers, and each layer has the same structure that contains the linear estimator
Wt , nonlinear estimator ηt (·), error variance τt2 and vt2 , and tied weights.

where wt ∼ NC (0, τt2 I). As the transmitted symbol xd is from follows

the discrete constellation set S = {s1 , s2 , . . . , sM }, correspond-  
rt = x̂d,t + γt Wt yd − Ĥx̂d,t , (24)
ing MMSE estimate for each element of the estimated symbol
vector is given by  
x̂d,t+1 = ηt rt , τt2 ; φt , ξt , (25)

si NC (si ; ri , τt2 )p(si )
x̂d,t+1 = E {xi |ri , τt } = si y − Ĥx̂d,t 22 − tr(Rn̂d n̂d )
(i)
, (22)
si NC (si ; ri , τt )p(si )
2 vt2 =   , (26)
tr ĤH Ĥ
where p(si ) is the prior distribution of the transmitted symbol
xi and is given by 1   2 θ2  
τt2 = tr Ct CH vt + t tr Wt Rn̂d n̂d WtH . (27)
 1 Nt t
Nt
p(xi ) = √ δ(xi − sj ). (23)
j∈M
M The critical difference between the OAMP and OAMP-Net2
detectors is there are learnable variables Ωt = {γt , φt , ξt , θt }
From (14), (15), (17) and (21), we observe that rt and τt2 in each layer. By optimizing these parameters in the training
are the prior mean and variance in MMSE estimator (15) that process, the detection performance can be improved. When
control the accuracy and convergence of estimated result x̂d,t+1 . γt = θt = 1, φt = 1 and ξt = 0, the OAMP-Net2 is reduced
The OAMP detector uses an iterative manner to obtain rt and to the OAMP detector. The matrix Ct = I − θt Wt Ĥ in the
τt2 , and the step-size for the update of rt and τt2 will influence OAMP-Net2 is a revised structure of Bt in the OAMP detector
the final performance. Instead of using complicated analytical with learnable parameter θt to regulate the error variance τt2 .
method to find an optimal step-size, we use a DL approach We will introduce the linear estimator, Wt , nonlinear estimator,
for providing an appropriate step-size to update rt and τt2 ηt (·), error estimators, τt2 and vt2 , and learnable variables, Ωt ,
and constructing learnable nonlinear estimator to improve the respectively.
detection performance. 2) Linear and Nonlinear Estimator Modules: The linear es-
timator module in (24) is a revised structure of (14) in the OAMP
B. OAMP-Net2 Detector algorithm by adding corresponding learnable parameter γt . The
In this section, we introduce the OAMP-Net2 detector, which learnable parameter γt can be interpreted as the step-size for
is developed by unfolding the OAMP detector. After presenting the update of rt , and is γt = 1 in each iteration in the OAMP
the network architecture, we elaborate the linear and nonlinear detector. The de-correlated matrix Wt is given in (19). The
estimators, error estimators, and learnable variables in OAMP- nonlinear estimator ηt (·) in OAMP-Net2 is constructed by the
Net2 detector. divergence-free estimator
1) Network Architecture: The structure of the OAMP-Net2  
ηt rt , τt2 ; φt , ξt = φt (E {xd |rt , τt } − ξt rt ) , (28)
is illustrated in Fig. 2, which is obtained by unfolding the OAMP
detector and adding several trainable parameters. The input where E{x|rt , τt } is the MMSE estimate of xd with the equiva-
of the OAMP-Net2 is the received signal vector, yd , and the lent AWGN channel (21), and has the same expression with (22).
estimated channel matrix, Ĥ, while the final output is x̂d,T +1 . The nonlinear estimator, ηt (rt , τt2 ; φt , ξt ), can be interpreted as
The network consists of T cascade layers and each has the a linear combination of the priori mean, rt , and the posteriori
same architecture but different trainable parameters. For the t- mean, E{xd |rt , τt }, which has been investigated in [42] and [45].
th layer of the OAMP-Net2, the input is the estimated signal Compared with the MMSE estimator (22) in the OAMP detec-
x̂d,t from the (t − 1)-th layer. Signal detection is performed as tor, the divergence-free estimator (28) considers the contribution

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
HE et al.: MODEL-DRIVEN DEEP LEARNING FOR MIMO DETECTION 1707

of the linear estimator and learnable parameters (φt , ξt ). The multiplication, but has performance deterioration in small-size
MMSE estimator (22) can be interpreted as a special case of MIMO systems. The LMMSE detector has a complexity of
(28) by setting φt = 1 and ξt = 0. O(Nt3 ) as one matrix inversion is needed, but it is non-iterative
3) Error Variance Estimators: The error variance estimators and no training is required. Although ML can achieve the opti-
vt2 in (26) and τt2 in (27) play important roles in providing appro- mal performance, it has a complexity of O(|S|Nt ). In Table I,
priate variance estimates required for the linear and nonlinear we compare the computational complexity of the OAMP-net2
estimators in the OAMP-Net2 detector. For error variance vt2 , detector with the state-of-art DL-based MIMO detectors in [19]–
we adopt the same estimator with the OAMP detector in (26). [23] including DetNet [19], DNN-dBP [20], DNN-MPD [21],
For error variance τt2 , we construct the estimator based on two TPG [22], LcgNet [23]. Although the state-of-art DL-based
assumptions on error vectors pt and qt in [42] and incorporate MIMO detectors have lower complexity than the OAMP-Net2
learnable variables (γt , θt ). The detailed derivation for the two detector, their performance is deteriorated in the small-MIMO
variance estimators vt2 and τt2 is provided in Appendix B. Fur- systems to some extent.
thermore, we substitute vt2 by max(vt2 , ) for a small positive Furthermore, we investigate the number of learnable vari-
constant ε = 5 × 10−13 to avoid stability problem. ables in different DL-based MIMO detectors. From Fig. 2, the
4) Learnable Variables: The learnable variables Ω = total number of trainable variables is equal to 4T since each
{Ωt }Tt=1 are optimized in the training process for OAMP-Net2. layer of the OAMP-Net2 contains only four trainable variables
The Bayes-optimal property of the OAMP algorithm has been Ωt = (γt , φt , ξt , θt ). By contrast, 2T trainable variables (γt , θt )
proven in [43], but it is derived in the large system with are required to train in OAMP-Net. However, the numbers of
unitarily-invariant matrix H. In fact, the performance of the learnable variables in DetNet, DNN-MPD and LcgNet heavily
OAMP detector is far from the Bayes-optimal performance in are dominated by the number of antennas in the transmitter or
practical finite-dimensional MIMO systems, especially when the receiver. By contrary, the number of trainable variables of
there are strong spatial correlation and channel estimation error. the OAMP-Net2 and OAMP-Net are independent of the number
These observations motivate us to improve the original iterative of antennas Nr and Nt , and only determined by the number
detector with several trainable parameters to adapt to various of layers T . This is an advantageous feature for large-scale
channel environments. problems, such as massive MIMO detection. With only few
Similar to the TPG detector [22] and OAMP-Net [1], the trainable variables, the stability and speed of convergence can
OAMP-Net2 uses two learnable parameters (γt , θt ) to adjust be improved in the training process.
the linear estimator, and to provide appropriate stepsize for the
update of the mean rt and variance τt2 in the MMSE estima- C. Soft-Input and Soft-Output
tor. On the other hand, the linear estimator (24) is related to
As many modern digital communication systems need to
gradient descent algorithm and its convergence behavior and
produce a probabilistic estimation of the transmitted data given
performance are determined by appropriate step-size of moving
the observations to probabilistic channel decoder. A significant
to the search point. The optimal step-size γt can be learned
issue is whether the MIMO detector can use the soft-information
from the data for the update of the prior mean rt in the MMSE
from the decoder and produce the soft-output. Different from
estimator. Furthermore, the parameter θt has the similar function
the DetNet in [19] that only can provide the soft-output, our
for the error variance τt2 , which can compensate for the channel
proposed detector are the soft-input and soft-output receiver and
estimation error and regulate the τt2 to provide appropriate value
therefore can achieve the turbo equalization. We only provide
for the update of the prior variance in the MMSE estimator.
the principle of the OAMP-Net2-based turbo receiver and the
The parameters (φt , ξt ) in the nonlinear estimator ηt (·) play
specific experimental results are outside the scope of this paper
important roles in constructing an appropriate divergence-free
and will be conducted in the future.
estimator, which has been discussed in [42]. In precise, the
From (28), the OAMP-Net2 can decouple the joint posterior
divergence-free estimator (28) can be applied in the OAMP
probability P(x|yd , Ĥ) into a series of marginal posterior prob-
detector, but the φt and ξt are related to the prior distribution of
ability P(xj |rt , τt ). The marginal posterior probability is used
the original signal and difficult to calculate. Therefore, MMSE
to produce the soft output log-likelihood ratios (LLR), which is
estimator (22) is considered for simplicity in the OAMP detector.
given by
In our early work [1], we set φt = 1 and ξt = 0 in OAMP-Net 
and use MMSE estimator (22). By contrary, we adaptively S + P(xj |rt , τt )
learn two parameters φt and ξt in OAMP-Net2 to construct the LA (bj,k ) = log  j , (29)
S − P(xj |rt , τt )
nonlinear estimator ηt (·) satisfying the divergence-free property. j

5) Complexity Analysis: The computational complexity re- where bj,k is the k-th bit in the transmitted symbol xj , and
quired for the OAMP-Net2 is O(T Nt3 ). Similar to the OAMP Sj+ and Sj− denote the subsets of the constellation symbols
detector and OAMP-Net, the computational complexity is dom- with the k-th bit being 1 and 0, respectively. After interleaved
inant by the matrix inverse in each layer in (20). When Nt and delivered to the channel encoder, extrinsic LLR can be
is relatively small (e.g., 4 or 8), the matrix inverse operation computed and given to the OAMP-Net2 detector as updated
is always acceptable. By contrary, the AMP detector [5] has prior information. The detector and decoder iteratively exchange
a complexity of O(T Nt2 ), which is dominated by the matrix information until convergence.

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
1708 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 68, 2020

TABLE I
COMPUTATIONAL COMPLEXITY OF DIFFERENT DETECTORS

D. Practical Implementation The training data consists of a number of randomly generated

pairs d(i)  (x(i) , y(i) ). For each pair d(i) , channel H is ran-
The developed detector can be divided into two stages. In the
offline training stage, we obtain the optimized parameters, Ω = domly generated from the i.i.d. or correlated Rayleigh MIMO
{Ωt }Tt=1 , for different SNRs and channel correlation coefficients channel model. The data x(i) is generated from the M -QAM
modulation symbol with M being the modulation order. We
based on TensorFlow platform and GPUs’ powerful computing
ability. The optimized parameters will be stored to detect the train the network with 1,000 epochs. At each epoch, the training
modulated symbols in the deployment stage. The OAMP-Net2 set contains 5,000 different samples d(i) , and 1,000 different
validation samples. The validation sets are used to choose the
detector can be interpreted as a new iterative detector after
training. The incorporated learnable parameters can adapt to best network for each epoch in the training stage. In the test
practical channel and compensate for channel estimation error stage, we generate the data to test the network until the number
of bit errors exceed 10,000. The network is trained using the
to improve the detection performance. Although aforementioned
implementation process is in an offline manner, the proposed de- stochastic gradient descent method and Adam optimizer. The
learning rate is set to be 0.001 and the batch size is set to 100.
tector can also be implemented by online training to adapt to the
To train the learnable variables well, the learning rate is set to
fluctuations in the channel conditions owing to the superiority
of its low demand for training data and computational resources. be 0.0001 and validation samples is 10,000 in high SNR regime
(SNR = 30 or above). Except for special instructions, we choose
the l2 loss6 as the cost function in all experiment settings, which
V. NUMERICAL RESULTS
is defined as
In this section, we provide numerical results to show the 1  (i)
performance of the proposed model-driven detector for MIMO l2 (Ω) = xd − x̂d,T +1 (y(i) )22 , (32)
D (i)
detection. First, we elaborate the implementation details and d ∈D
parameter settings. Then, the performance of the OAMP-Net2 is where D is the training set of a mini-batch and D = |D|.
presented under i.i.d. and correlated Rayleigh MIMO channels In general, the optimal variables Ω may be different for
with perfect CSI. Afterwards, we investigate the performance different SNRs. Therefore, we train the OAMP-Net2 for each
of the OAMP-Net2-based JCESD architecture. Finally, the ro- SNR separately, which means that the training and test data
bustness of the OAMP-Net2 to SNR and channel correlation is are generated with the same fixed discrete SNR value. For the
demonstrated. correlated Rayleigh MIMO channel, we train the network for
each correlation coefficient ρ, respectively. The network needs
A. Implementation Details to be retrained if different modulation symbols are utilized.
In our simulation, OAMP-Net2 is implemented in Tensorflow Furthermore, we also consider the robustness of the OAMP-Net2
by using a PC with GPU NVIDIA GeForce GTX 1080 Ti. The to various mismatches.
SNR of the system, defined as
B. Perfect CSI
EHxd 22
SNR = , (30) In this section, we study the performance of OAMP-Net2 for
End 22
MIMO detection with perfect CSI. After analyzing its conver-
is used to measure the noise level. We assume the same SNR gence, the detection performance of the OAMP-Net2 under i.i.d.
in pilot and data transmission stage. Each element of the and correlated Rayleigh MIMO channels is investigated.
i.i.d. Rayleigh MIMO channel H is hi,j ∼ NC (0, 1/Nr ) while 1) Convergence Analysis: First, we analyze the convergence
the correlated Rayleigh MIMO channel is described by the of the OAMP-Net2 and OAMP detectors. Fig. 3 illustrates the
Kronecker model, bit-error rate (BER) performance versus the number of layers
1/2
Hc = RR GRT ,
1/2
(31) (iterations) under SNR = 15 dB and 25 dB with QPSK and
16-QAM constellations. From Fig. 3, both the OAMP-Net2 and
where RR and RT are the spatial correlation matrix at the OAMP detectors converge within four layers (iterations) for all
receiver and the transmitter, respectively, and are generated the cases and OAMP-Net2 can obtain more performance gain
according to the exponential correlation model [44] with the when SNR = 25 dB, which means the OAMP-Net2 detector
same correlation coefficient ρ, and G is the i.i.d. Rayleigh fading improves OAMP detector significantly in high SNR regime.
channel matrix. Furthermore, to implement OAMP-Net2 in the
Tensorflow, we consider equivalent real-valued representation 6 We train the OAMP-Net2 with l loss and cross-entropy, respectively, and
2
for all aforementioned problem models. find the network trained with l2 loss outperforms that trained with cross-entropy.

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
HE et al.: MODEL-DRIVEN DEEP LEARNING FOR MIMO DETECTION 1709

Fig. 3. BERs performance of the OAMP-Net2 and OAMP detector versus the Fig. 4. BERs performance comparison of the OAMP-Net2 with other MIMO
number of layers under QPSK and 16-QAM modulation. detectors under i.i.d. Rayleigh MIMO channels with different orders of
modulation.

Furthermore, compared with 16-QAM, the performance gap

between OAMP and OAMP-Net2 is larger for QPSK. Based significant performance improvement in all setting. The reason
on the above observations, we set OAMP and OAMP-Net2 is the OAMP-Net2 can learn the optimal parameter (φt , ξt ) to
detectors four layers (T = 4) in following simulation when SNR construct the nonlinear estimator satisfying the divergence-free
 25 dB, and ten layers (T = 10) when SNR = 30 dB or above property. For high-dimensional MIMO systems in Fig. 5, similar
except for specific instructions. conclusions can be obtained.
2) Performance Comparison: Fig. 4 compares the BER per- 3) Correlated MIMO Channel Performance: Fig. 6 illus-
formance of the OAMP, OAMP-Net [1], TPG [22], DetNet [19], trates the BER of the OAMP-Net2 and the OAMP detector
AMP [5], SD, ML, MMSE-SIC [46], LMMSE and OAMP-Net2 under correlated Rayleigh MIMO channel with ρ = 0.5. We set
detectors. Both 4 × 4 and 8 × 8 MIMO systems are investigated. the OAMP and OAMP-Net2 detectors ten layers with different
Among these detectors, the TPG detector [22] is developed orders of modulation. From the figure, all algorithms have per-
with two trainable parameters in each layer and has a simi- formance degradation compared with the i.i.d. Rayleigh MIMO
lar architecture to OAMP-Net. Furthermore, the OAMP and channel, But the OAMP-Net2 detector still have the performance
MMSE-SIC detectors can achieve soft decision. From the figure, improvement over the OAMP detector in all setting and can
the OAMP-Net2 outperforms other MIMO detectors except for obtain more performance improvement in 8 × 8 MIMO system.
SD and ML, but the computational complexity of the ML and Specifically, if we target for BER = 10−2 with QPSK modula-
SD detectors is prohibitive. The DetNet and AMP detectors tion, BER performance improves about 1.8 dB for 4 × 4 MIMO.
have serious performance deterioration in small-size MIMO By contrast, it is approximately 2.2 dB for 8 × 8 MIMO. But the
systems. Furthermore, the OAMP-Net, TPG, and OAMP-Net2 performance gain of the OAMP-Net2 detector decreases with the
detectors outperform the OAMP detector in all setting, which increase of the modulation order. For example, if we target for
demonstrates DL can improve the OAMP detector by learning BER = 10−2 in 4 × 4 MIMO, SNR improves about 1.5 dB for
the optimal parameters. But the performance gain is limited for QPSK modulation. By contrast, it decreases to approximately
64-QAM. Compared with OAMP-Net in [1], OAMP-Net2 has a 1.1 dB for 16-QAM.

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
1710 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 68, 2020

Fig. 5. BERs performance comparisons of the OAMP and OAMP-Net2 under Fig. 6. BERs performance comparison of the OAMP and OAMP-Net2 un-
high-dimensional i.i.d. Rayleigh MIMO channels with different modulation der correlated Rayleigh MIMO channels with ρ = 0.5 for different orders of
symbols. modulation.

4) 3GPP MIMO Channels Performance: In the aforemen-

tioned sections, we have investigated the OAMP-Net2 detector
with Rayleigh fading channels. Here, we conduct numerical ex-
periments by utilizing a dataset of the realistic channel samples
from the 3GPP 3D MIMO channel [47], as implemented in the
QuaDRiGa channel simulator [48]. The channel parameters are
set to be same as [24]. As shown in Fig. 7, OAMP-Net2 indeed
has performance loss with the realistic channel. The reason is
that the OAMP-Net2 requires the channel matrix to be unitarily-
invariant while the practical channel is very ill-conditioned and
the channel correlation is serious. which incurs performance
degradation. Furthermore, only 4 trainable parameters in each
layer may restrict the ability to adapt to realistic 3GPP MIMO
channels. Nevertheless, the OAMP-Net2 still outperforms the Fig. 7. BERs performance comparison of the OAMP-Net2 with other MIMO
OAMP and LMMSE detectors with limited performance gain. detectors under 3GPP MIMO channels with different orders of modulation.

C. JCESD Performance
which is defined as
In the above, all detectors are investigated with accurate CSI.
L 
 T
In this section, we consider OAMP-Net2-based JCESD architec- 1  (i) (l)
l2 (Ω) = xd − x̂d,T (y(i) )22 , (33)
ture for a 4 × 4 MIMO system. Both OAMP and OAMP-Net2 D
t=1
l=1 (i) d ∈D
has four layers. In order to avoid gradient vanishing, we use the
summation of l2 -loss over all L × T layers to train the model, and the learning rate is set to be 0.0001.

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
HE et al.: MODEL-DRIVEN DEEP LEARNING FOR MIMO DETECTION 1711

Fig. 8. BERs performance comparisons of the OAMP and OAMP-Net2 in Fig. 9. BERs performance comparisons of the OAMP and OAMP-Net2 with
JCESD architecture with Np = 4 and Nc = 16. SNR and correlation mismatches for QPSK modulation.

We consider orthogonal pilot in channel training stage, where and tested with mismatched channel correlation coefficient ρ and
the pilot matrix Xp ∈ CNt ×Np is chosen by selecting Nt SNR, which are shown in the figure. From the figure, the trained
columns of the discrete Fourier transformation (DFT) matrix network with correlation mismatch still outperforms the OAMP
F ∈ CNp ×Np . Fig. 8 shows the BERs of the OAMP and OAMP- detector significantly in all setting and has 0.3 dB performance
Net2 detectors in the JCESD architecture, where L = 1 means loss, if we target for BER = 10−2 and ρ = 0.5, which demon-
no data feedback to channel estimator and L = 3 means the strates the OAMP-Net2 has strong robustness to mismatch.
detected data are feedback twice to channel estimator. From Interestingly, the trained network even outperforms the OAMP
the figure, the OAMP-Net2 detector outperforms the OAMP detector with ρ = 0.1 in high SNR regime (SNR = 20–30 dB).
detector in all settings significantly. Specifically, if we target Since the learnable variables successfully compensates for the
for BER = 10−2 with 16-QAM, BER performance improves disadvantages of channel correlation. Compared with perfect
about 2.1 dB for L = 1 and improves about 2.9 dB for L = 3. SNR,7 the OAMP-Net2 detector with SNR mismatch has little
Furthermore, we observe that OAMP-Net2 detector without data performance deterioration except for SNR = 25 dB or above,
feedback (L = 1) outperforms OAMP detector with data feed- when we target ρ = 0.5.
back twice (L = 3), which demonstrates learning some trainable 2) Robustness to MIMO Configuration and Modulation:
variables can compensate for the channel estimation error to Fig. 10 presents the BER performance of OAMP and OAMP-
improve the equivalent SNR in detection stage. For QPSK mod- Net2 with the MIMO configuration and modulation symbol
ulation, the performance gain are 5.1 dB and 3.1 dB for L = 1 mismatches. We train the OAMP-Net2 with 16-QAM symbol
and L = 3, respectively, when we target for BER = 10−2 . In in 8 × 8 MIMO system and test the robustness of the trained
addition, marginal performance is obtained by data feedback for parameters. Fig. 10(a) exhibits the numerical results of the
the OAMP-Net2 detector with QPSK because learnable param- OAMP-Net2 with MIMO configuration mismatch, where the
eters have strong ability to compensate for channel estimation network is tested in 4 × 4 systems with 16-QAM. Although
error. Therefore, no data feedback is needed in this case. the network is employed in different MIMO configurations, the
OAMP-Net2 still outperforms the LMMSE and OAMP detectors
D. Robustness and has little performance loss. The robustness demonstrates
the OAMP-Net2 is flexible to different MIMO configuration.
In this section, we analyze the robustness of the OAMP-Net2
Furthermore, Fig. 10(b) shows the performance of OAMP-Net2
against various mismatches, including the SNR, channel cor-
with modulation symbol mismatch, where the network is tested
relation, MIMO configuration and modulation mismatches. As
in 8 × 8 MIMO system with QPSK symbol. Only 0.4 dB perfor-
aforementioned numerical results are performed when training
mance loss is incurred owing to modulation symbol mismatch if
and test data are generated with same system parameters, an
we target for BER = 10−2 , which demonstrates the OAMP-Net2
interesting question is whether the trained parameters are robust
is robust to modulation symbol mismatch. The OAMP-Net2
to various mismatches. Because of limited data and computing
detector contains a linear and a nonlinear estimators. The linear
resources are available for online training, verifying the robust-
estimator is related to gradient descent algorithm and its conver-
ness of offline-trained network is particularly meaningful.
gence behavior and performance are determined by appropriate
1) Robustness to SNR and Channel Correlation: Fig. 9
step-size γt moving to the search point. As the linear estimator
presents the BER performance of OAMP and OAMP-Net2 with
SNR and correlation mismatches in 8 × 8 MIMO system. The
network is trained in the correlated Rayleigh MIMO channel (31) 7 Perfect SNR means that network in the training and test stage have the same
with channel correlation coefficient ρ = 0.5 and SNR = 20 dB, SNR.

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
1712 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 68, 2020

APPENDIX A
DERIVATION FOR COVARIANCE MATRICES
In order to derive the covariance matrix V̂d [n] of the equiv-
alent noise n̂d [n] in signal detection stage, we should evalu-
ate the statistical properties of the zd [n] = ΔHxd [n] for n =
1, . . . , Nd , which can be expressed as

Cov[zd [n]] = E zd [n](zd [n])H . (34)

As the covariance matrix Cov[zd [n]] for each symbol index n =

1, . . . , Nd is same and the data symbol is independent between
different antennas, we omit the index n and have

0, ∀i = k
H
E[zd,i zd,k ]=  Nt (35)
j=1 σΔhi,j , i=k
2

 t
where zd,i = N j=1 Δhi,j xd,j is the i-th element of the zd ,
Δhi,j is the (i, j)-th element of the channel estimation error
matrix ΔH, and xd,j is j-th element of the xd . The σΔh
2
i,j
is the
variance of (i, j)-th element in channel estimation error matrix
ΔH which can be obtained from RΔhp in (11) or RΔh in (13).
By considering the contribution of original additive noise, the
covariance matrix V̂est can be obtained as
⎛ ⎞
Nt
 Nt

V̂est = diag ⎝ σΔh
2
i,j
+ σ2 , . . . , σΔh
2
i,j
+ σ2 ⎠ .
j=1 j=1
(36)
Fig. 10. BERs performance comparisons of the OAMP and OAMP-Net2 with
The covariance matrix V̂est will be utilized in the OAMP-Net2
the MIMO configuration and modulation symbol mismatches. detector as the covariance matrix of the equivalent noise Rn̂d n̂d .
Next, we compute the covariance matrix V̂p [n] of the equiv-
alent noise n̂p [n] in data-aided channel estimation stage using
is independent of the modulation symbols, the learnable pa-  t For each time index n, We denote zn = Hen
similar approach.
rameter γt shows strong robustness to modulation symbol mis- and zi,n = N j=1 hi,j ej,n , where en is the n-th column of the
matches. Therefore, the trained network can be utilized directly signal detection error matrix Ed . In a similar way, we have
in different MIMO configurations with different modulation
symbols. 0, ∀i = k
H
E[zi,n zk,n ]= N t (37)
j=1 σej,n /Nr , i=k
2
VI. CONCLUSION
We have developed a novel model-driven DL network for because hi,j ∼ NC (0, 1/Nr ) and σe2j,n is the variance for j-th
MIMO detection, named OAMP-Net2, which is obtained by element of the en . Therefore, the covariance matrix V̂p [n] is
unfolding the OAMP detection algorithm. The OAMP-Net2 given by
detector inherits the superiority of the Bayes-optimal signal re-
covery algorithm and DL technique, and thus presents excellent ⎛ ⎞
Nt

performance. The network are easy and fast to train because only V̂p [n] = ⎝ σe2j,n + σ 2 ⎠ INt . (38)
few adjustable parameters are required to be optimized. To han- j=1
dle imperfect CSI, an OAMP-Net2-based JCESD is proposed,
where the detector takes channel estimation error and channel
The covariance matrix V̂p [n] will be utilized in channel estima-
statistics into consideration while channel estimation is refined
tor to evaluate the covariance matrix of the equivalent noise Rnn
by detected data and considers the detection error. Simulation
where n = vec(N̂). By considering the different covariance
results demonstrate that significant performance gain can be
matrix of the actual pilot and additional pilot, we have
obtained by learning corresponding optimal parameters from
the data to improve the detector and compensate for the channel  
estimation error. Furthermore, the OAMP-Net2 exhibits strong σ 2 IN p N r
Rnn = (39)
robustness to various mismatches. V̂det

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
HE et al.: MODEL-DRIVEN DEEP LEARNING FOR MIMO DETECTION 1713

where σ 2 INp Nr denotes the covariance matrix of the noise Then, for the error variance estimator τt2 , we have
vec(Np ) for actual pilot Xp while
⎡ ⎤ E[qt 22 ] 1
τt2 = = E[(I − γt Wt H)(x̂t − x) + γt Wt n22 ]
V̂p [1] Nt Nt
⎢ .. ⎥
V̂det = ⎣ . ⎦ (40) 1
= E[(x̂t − x)H (I − γt Wt H)(I − γt Wt H)H (x̂t − x)]
V̂p [Nd ] Nt
γt2
denotes the covariance matrix of the equivalent noise vec(N̂p ) + E[nH WtH Wt n]
Nt
for the additional pilot X̂d .
2γt
+ E[(x̂t − x)H (I − γt Wt H)H Wt n]
APPENDIX B Nt
DERIVATION FOR VARIANCE ESTIMATORS 1  
= tr (I − γt Wt H)(I − γt Wt H)H vt2
To derive the expressions for the error variance estimators Nt
vt2 and τt2 in OAMP-Net2, we use similar method to [34]. We γt2 2(γt − γt2 )
should indicate that vt2 and τt2 in OAMP-detector are based on + tr(Wt Wt H )σ 2 + E[(x̂t − x)H HH n].
Nt Nt
the following two assumptions about error vectors pt and qt (44)
[42].
Assumption 1: pt consists of i.i.d. zero-mean Gaussian en- The last term in the (44) vanishes as the E[(x̂t − x)H HH n] = 0.
tries independent of xd . Therefore, the error variance estimator τt2 is given by
Assumption 2: qt consists of i.i.d. zero-mean Gaussian en-
1  
tries independent of H and n̂d . τt2 = tr (I − γt Wt H)(I − γt Wt H)H vt2
From the similar viewpoint, we obtain the error variance Nt
estimators vt2 and τt2 (26) and (27) in OAMP-Net2 using As- γt2
+ tr(Wt Wt H )σ 2 . (45)
sumption 1 and 2, and considering the effect of the learnable Nt
variables γt and θt . For convenience, we will use H, x̂t , and n
to refer Ĥ, x̂d,t , and n̂d , respectively. Based on Assumption Then, we replace the parameter γt with θt in (45) as the simu-
2, qt consists of i.i.d. zero-mean Gaussian entries which is lation results show that two parameters (γt , θt ) can preferably
independent of H and n, which means that regulate the error variance estimator vt2 and τt2 . The superior
performance is attributed to more parameters, which can incor-
E[(x̂t − x)H HH n] = 0. (41) porate more side information from the data. Therefore, the final
expression for τt2 is obtained by
Therefore, The error variance estimator vt2 is given by,
E[qt 22 ] 1 tr(HH H)E[qt 22 ] 1   2 θt2 σ 2  
vt2 = = τt2 = tr Ct CH
t vt + tr Wt WtH , (46)
Nt Nt tr(HH H) Nt Nt

E[(x − x̂t )H HH H(x − x̂t )] where Ct = I − θt Wt H.

=
tr(HH H)
ACKNOWLEDGMENT
aE[(H(x − x̂t ))H H(x − x̂t ) + (H(x − x̂t ))H n]
= The authors would like to thank Dr. Junjie Ma for discussing
tr(HH H) the structure of the OAMP algorithm.
E[H(x − x̂t ) + n22 ] − E[nH n]
=
tr(HH H) REFERENCES
E[Hx + n − Hx̂t 22 ] − Nr σ 2 [1] H. He, C.-K. Wen, S. Jin, and G. Y. Li, “A model-driven deep learning
= network for MIMO detection,” in Proc. IEEE Glob. Conf. Signal Inf.
tr(HH H) Process., Nov. 2018, pp. 584–588.
[2] L. Zheng and D. N. C. Tse, “Diversity and multiplexing: A fundamental
E[y − Hx̂t 22 ] − Nr σ 2 tradeoff in multiple-antenna channels,” IEEE Trans. Inf. Theory, vol. 49,
= , (42)
tr(HH H) no. 5, pp. 1073–1096, May 2003.
[3] S. Yang and L. Hanzo, “Fifty years of MIMO detection: The road to large-
a scale MIMOs,” IEEE Commun. Surveys Tuts., vol. 17, no. 4, pp. 1941–
where = is obtained by using the equation (41). The final 1988, Oct.-Dec. 2015.
expression for vt2 is obtained by (42). [4] Z. Guo and P. Nilsson, “Algorithm and implementation of the K-best sphere
Next, we derive the expression for τt2 . The error vector rt − x decoding for MIMO detection,” IEEE J. Sel. Areas Commun., vol. 24, no. 3,
pp. 491–503, Mar. 2006.
can be rewritten as [5] C. Jeon, R. Ghods, A. Maleki, and C. Studer, “Optimality of large mimo
detection via approximate message passing,” in Proc. IEEE Int. Symp. Inf.
rt − x = x̂t + γt Wt (y − Hx̂t ) − x Theory, Jun. 2017, pp. 2333–2337.
[6] S. Wu, L. Kuang, Z. Ni, J. Lu, D. Huang, and Q. Guo, “Low-complexity
= x̂t + γt (Hx + n) − γt Wt Hx̂t − x iterative detection for large-scale multiuser MIMO-OFDM systems using
approximate message passing,” IEEE J. Sel. Topics Signal Process., vol. 8,
= (I − γt Wt )(x̂t − x) + γt Wt n. (43) no. 5, pp. 902–915, Oct. 2014.

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
1714 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 68, 2020

[7] J. Céspedes, P. M. Olmos, M. Sánchez-Fernandez, and F. Pérez-Cruz, “Ex- [32] Y. Yang, J. Sun, H. Li, and Z. Xu, “ADMM-Net: A deep learning approach
pectation propagation detection for high-order high-dimensional MIMO for compressive sensing MRI,” IEEE Trans. Pattern Anal. Mach. Intell.,
systems,” IEEE Trans. Commun., vol. 62, no. 8, pp. 2840–2849, Aug. 2014. vol. 42, no. 3, pp. 521–538, Mar. 2020.
[8] D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algorithms [33] M. Borgerding, P. Schniter, and S. Rangan, “AMP-inspired deep networks
for compressed sensing,” Proc. Nat. Acad. Sci., vol. 106, no. 45, pp. 18914– for sparse linear inverse problems,” IEEE Trans. Signal Process. vol 65,
18919, 2009. no. 16, pp. 4293–4308, Aug. 2017.
[9] T. P. Minka, “A family of algorithms for approximate Bayesian Inference,” [34] D. Ito, S. Takabe, and T. Wadayama, “Trainable ISTA for sparse signal
Ph.D. dissertation, Dept. Elect. Eng. Comput. Sci., MIT, Cambridge, MA, recovery,” IEEE Trans. Signal. Process., vol. 67, no. 12, pp. 3113–3125,
USA, 2001. Jun. 2019.
[10] K. Ghavami and M. Naraghi-Pour, “MIMO detection with imperfect [35] X. Chen, J. Liu, Z. Wang, and W. Yin, “Theoretical linear convergence of
channel state information using expectation propagation,” IEEE Trans. unfolded ISTA and its practical weights and thresholds,” in Adv. Neural
Veh. Technol., vol. 66, no. 9, pp. 8129–8138, Sep. 2017. Inf. Process. Syst., pp. 9061–9071, 2018.
[11] T. Wang, C.-K. Wen, H. Wang, F. Gao, T. Jiang, and S. Jin, “Deep [36] K. Gregor and Y. LeCun, “Learning fast approximations of sparse coding,”
learning for wireless physical layer: Opportunities and challenges,” China in Proc. Int’l Conf. Mach. Learn., 2010, pp. 399–406.
Commun., vol. 14, no. 11, pp. 92–111, Nov. 2017. [37] K. Takeuchi, R. R. Müller, M. Vehkaperä, and T. Tanaka, “On an achievable
[12] H. He, S. Jin, C.-K. Wen, F. Gao, G. Y. Li, and Z. Xu, “Model-driven deep rate of large Rayleigh block-fading MIMO channels with no CSI,” IEEE
learning for physical layer communications,” IEEE Wireless Commun., Trans. Inf. Theory., vol. 59, no. 10, pp. 6517–6541, Oct. 2013.
vol. 26, no. 5, pp. 77–83, Oct. 2019. [38] J. Ma and L. Ping, “Data-aided channel estimation in large antenna
[13] Z.-J. Qin, H. Ye, G. Y. Li, and B.-H. Juang, “Deep learning in physical systems,” IEEE Trans. Signal Process., vol. 62, no. 12, pp. 3111–3124,
layer communications,” IEEE Wireless Commun., vol. 26, no. 2, pp. 93–99, Jun. 2014.
Apr. 2019. [39] C.-K. Wen, C.-J. Wang, S. Jin, K.-K. Wong, and P. Ting, “Bayes-optimal
[14] H. He, C.-K. Wen, S. Jin, and G. Y. Li, “Deep learning-based channel esti- joint channel-and-data estimation for massive MIMO with low-precision
mation for beamspace mmWave massive MIMO systems,” IEEE Wireless ADCs,” IEEE Trans. Signal Process., vol. 64, no. 10, pp. 2541–2556,
Commun. Lett., vol. 7, no. 5, pp. 852–855, Oct. 2018. May 2016.
[15] Y. Yang, F. Gao, X. Ma, and S. Zhang, “Deep learning-based channel [40] F. Steiner, A. Mezghani, A. Lee Swindlehurst, J. A. Nossek, and
estimation for doubly selective fading channels,” IEEE Access, vol. 7, W. Utschick, “Turbo-like joint data-and-channel estimation in quantized
pp. 36579–36589, 2019. massive MIMO systems,” in Proc. 20th Int. ITG Workshop Smart Antennas,
[16] C.-J. Chun, J.-M. Kang, and I.-M. Kim, “Deep learning based channel Mar. 2016, pp. 1–5.
estimation for massive MIMO systems,” IEEE Wireless Commun. Lett., [41] S. Talwar, “Blind separation of synchronous co-channel digital signals
vol. 8, no. 4, pp. 1228–1231, Aug. 2019. using an antenna array–part II: performance analysis,” IEEE Trans. Signal
[17] C.-K. Wen, W. T. Shih, and S. Jin, “Deep learning for massive MIMO Process., vol. 45, no. 3, pp. 706–718, Mar. 1997.
CSI feedback,” IEEE Wireless Commun. Lett., vol. 7, no. 5, pp. 748–751, [42] J. Ma and L. Ping, “Orthogonal AMP,” IEEE Access, vol. 5, pp. 2020–
Oct. 2018. 2033, 2017.
[18] H. Ye, G. Y. Li, and B.-H. F. Juang, “Power of deep learning for chan- [43] K. Takeuchi, “Rigorous dynamics of expectation-propagation-based signal
nel estimation and signal detection in OFDM systems,” IEEE Wireless recovery from unitarily invariant measurements, IEEE Trans. Inf. Theory,
Commun. Lett., vol. 7, no. 1, pp. 114–117, Feb. 2018. vol. 66, no. 1, pp. 368–386, Jan. 2020.
[19] N. Samuel, T. Diskin, and A. Wiesel, “Deep MIMO detection,” in [44] S. L. Loyka, “Channel capacity of MIMO architecture using the exponen-
Proc. 18th IEEE Int. Workshop Signal Process. Adv. Wireless Commun. tial correlation matrix,” IEEE Commun. Lett., vol. 5, no. 9, pp. 369–371,
Hokkaido, Japan, Jul. 2017, pp. 1–5. Sep. 2001.
[20] X. Tan, W. Xu, Y. Be’ery, Z. Zhang, X. You, and C. Zhang, “Improving [45] Z. Xue, J. Ma, and X. Yuan, “Denoising-based turbo compressed sensing,”
massive MIMO belief propagation detector with deep neural network,” IEEE Access, vol. 5, pp. 7193–7204, 2017.
2018, arXiv:1804.01002. [46] P. Liu and I.-M. Kim, “Exact and closed-form error performance analysis
[21] X. Tan, Z. Zhong, Z. Zhang, X. You, and C. Zhang, “Low-Complexity for hard MMSE-SIC detection in MIMO systems,” IEEE Trans. Commun.,
message passing MIMO detection algorithm with deep neural network,” vol. 59, no. 9, pp. 2463–2477, Sep. 2011.
in Proc. IEEE Glob. Conf. Signal Inf. Process., Nov. 2018, pp. 559–563. [47] 3GPP, “Technical specification group radio access network; study on 3D
[22] S. Takabe, M. Imanishi, T. Wadayama, and K. Hayashi, “Trainable pro- channel model for LTE (Release 12),” in Proc. 3rd Generation Partnership
jected gradient detector for massive overload MIMO channels: Data-driven Project (3GPP), TR 36.873 V12.2.0, Jun. 2015. [Online]. Available:
tuning approach,” IEEE Access, vol. 7, pp. 93326–93338, 2019. [Online]. Available: http://www.3gpp.org/dynareport/36873.htm
[23] Y. Wei, M.-M. Zhao, M. Hong, M.-J. Zhao, and M. Lei, “Learned con- [48] S. Jaeckel, L. Raschkowski, K. Börner, and L. Thiele, “QuaDRiGa: A
jugate gradient descent network for massive MIMO detection,” 2019, 3-D multi-cell channel model with time evolution for enabling virtual
arXiv:1906.03814. field trials,” IEEE Trans. Antennas Propag., vol. 62, no. 6, 3242–3256,
[24] M. Khani, M. Alizadeh, J. Hoydis, and P. Fleming, “Adaptive neural signal Mar. 2014.
detection for massive MIMO,” 2019, arXiv:1906.04610.
[25] X. Gao, S. Jin, C.-K. Wen, and G. Y. Li, “ComNet: Combination of deep
learning and expert knowledge in OFDM receivers,” IEEE Commun. Lett.,
vol. 22, no. 12, pp. 2627–2630, Dec. 2018.
[26] M. Mohammadkarimi, M. Mehrabi, M. Ardakani, and Y. Jing, “Deep
learning based sphere decoding,” IEEE Trans. Wireless Commun., vol. 18,
no. 9, pp. 4368–4378, Sep. 2019.
[27] E. Nachmani, E. Marciano, L. Lugosch, W. J. Gross, D. Burshtein, and
Y. Beery, “Deep learning methods for improved decoding of linear codes,”
IEEE J. Sel. Topics Signal Process., vol. 12, no. 1, pp. 119–131, Feb. 2018. Hengtao He (Student Member, IEEE) received the
[28] Y. He, J. Zhang, C.-K. Wen, and S. Jin, “TurboNet: A model-driven DNN B.S. degree in communications engineering from
decoder based on max-log-MAP algorithm for turbo code,” in Proc. IEEE the Nanjing University of Science and Technology,
VTS Asia Pacific Wireless Commun. Symp., Singapore, 2019, pp. 1–5. Nanjing, China, in 2015. He is currently working
[29] T. O’Shea and J. Hoydis, “An introduction to deep learning for the physical toward the Ph.D. degree in information and com-
layer,” IEEE Trans. Cogn. Commun. Netw., vol. 3, no. 4, pp. 563–575, munications engineering with Southeast University,
Dec. 2017. Nanjing, China, under the supervision of Prof. S. Jin.
[30] H. Ye, L. Liang, G. Y. Li, and B.-H. F. Juang, “Deep learning From October 2018 to January 2020, he was a Visiting
based end-to-end wireless communication systems with GAN as un- Student with the Department of Electrical and Com-
known channel,” IEEE Trans. Wireless Commun., to be published, doi: puter Engineering, Georgia Institute of Technology,
10.1109/TWC.2020.2970707. Atlanta, GA, USA. His research interests include
[31] I. Santos and J. J. Murillo-Fuentes, “EP-based turbo detection for millimeter wave communications, massive MIMO, and machine learning for
MIMO receivers and large-scale systems,” vol. 8, no. 4, pp. 1095–1098, wireless communications. He was the recipient of the exemplary of IEEE
Aug. 2019. WIRELESS COMMUNICATIONS LETTERS in 2019.

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.
HE et al.: MODEL-DRIVEN DEEP LEARNING FOR MIMO DETECTION 1715

Chao-Kai Wen (Member, IEEE) received the Ph.D. Geoffrey Ye Li (Fellow, IEEE) is currently a Pro-
degree from the Institute of Communications Engi- fessor with the Georgia Institute of Technology, At-
neering, National Tsing Hua University, Taiwan, in lanta, GA, USA. Before moving to Georgia Tech,
2004. He was with the Industrial Technology Re- he was with AT&T Labs-Research, Red Bank, NJ,
search Institute, Hsinchu, Taiwan and MediaTek Inc., USA, as a Senior and then a Principal Technical
Hsinchu, Taiwan, from 2004 to 2009. Since 2009, Staff Member, from 1996 to 2000, and a Postdoctoral
he has been with the National Sun Yat-sen Univer- Research Associate with the University of Maryland
sity, Taiwan, where he is Professor with the Institute at College Park, MD, USA, from 1994 to 1996.
of Communications Engineering. His research inter- He has authored/coauthored more than 500 journal
ests include the optimization in wireless multimedia and conference papers in addition to more than 40
networks. granted patents. His publications have been cited
around 40,000 times and he has been recognized as the World’s Most Influential
Scientific Mind, also known as a Highly Cited Researcher, by Thomson Reuters
almost every year. His research interests include statistical signal processing and
machine learning for wireless communications. Dr. G. Y. Li was the recipient
of the IEEE Fellow for his contributions to signal processing for wireless
communications in 2005. He was also the recipient of several prestigious awards
from IEEE Signal Processing Society (Donald G. Fink Overview Paper Award in
Shi Jin (Senior Member, IEEE) received the B.S. 2017), IEEE Vehicular Technology Society (James Evans Avant Garde Award in
degree in communications engineering from Guilin 2013 and Jack Neubauer Memorial Award in 2014), and IEEE Communications
University of Electronic Technology, Guilin, China, Society (Stephen O. Rice Prize Paper Award in 2013, Award for Advances in
in 1996, the M.S. degree from Nanjing University Communication in 2017, and Edwin Howard Armstrong Achievement Award
of Posts and Telecommunications, Nanjing, China, in 2019). He also the recipient of 2015 Distinguished Faculty Achievement
in 2003, and the Ph.D. degree in information and Award from the School of Electrical and Computer Engineering, Georgia Tech.
communications engineering from the Southeast Uni- He has been involved in editorial activities for more than 20 technical journals,
versity, Nanjing, China, in 2007. From June 2007 to including the founding Editor-in-Chief of IEEE 5G Tech Focus and the founding
October 2009, he was a Research Fellow with the Editor-in-Chief of IEEE JSAC Special Series on ML in Communications and
Adastral Park Research Campus, University College Networking. He has organized and chaired many international conferences,
London, London, U.K. He is currently with the fac- including technical program Vice-Chair of IEEE ICC’03, technical program
ulty of the National Mobile Communications Research Laboratory, Southeast Co-Chair of IEEE SPAWC’11, General Chair of IEEE GlobalSIP’14, techni-
University. His research interests include space time wireless communications, cal program Co-Chair of IEEE VTC’16 (Spring), General Co-Chair of IEEE
random matrix theory, and information theory. He serves as an Associate Editor VTC’19 (Fall), and General Co-Chair of IEEE SPAWC’20.
for the IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, and IEEE COM-
MUNICATIONS LETTERS, and IET COMMUNICATIONS. Dr. Jin and his coauthors
was the recipient of the 2011 IEEE Communications Society Stephen O. Rice
Prize Paper Award in the field of communication theory and a 2010 Young
Author Best Paper Award by the IEEE Signal Processing Society.

Authorized licensed use limited to: University of Canberra. Downloaded on April 28,2020 at 07:37:09 UTC from IEEE Xplore. Restrictions apply.