1

Model-Driven Deep Learning forMIMO Detection

Hengtao He, Student Member, IEEE, Chao-Kai Wen, Member, IEEE,Shi Jin, Senior Member, IEEE, and Geoffrey Ye Li, Fellow, IEEE

Abstract—In this paper, we investigate the model-driven deeplearning (DL) for MIMO detection. In particular, the MIMOdetector is specially designed by unfolding an iterative algorithmand adding some trainable parameters. Since the number oftrainable parameters is much fewer than the data-driven DLbased signal detector, the model-driven DL based MIMO detectorcan be rapidly trained with a much smaller data set. Theproposed MIMO detector can be extended to soft-input soft-output detection easily. Furthermore, we investigate joint MIMOchannel estimation and signal detection (JCESD), where thedetector takes channel estimation error and channel statisticsinto consideration while channel estimation is refined by detecteddata and considers the detection error. Based on numericalresults, the model-driven DL based MIMO detector significantlyimproves the performance of corresponding traditional iterativedetector, outperforms other DL-based MIMO detectors andexhibits superior robustness to various mismatches.

Index Terms—Deep learning, Model-driven, MIMO detection,Iterative detector, Neural network, JCESD

I. INTRODUCTION

Multiple-input multiple-output (MIMO) technology can dra-matically improve the spectral efficiency and link reliabilityand has been applied to many wireless communication sys-tems. To obtain the benefits of MIMO [2], efficient channelestimation and signal detection algorithms, which balanceperformance and complexity, are essential in receiver designand have arouse a series of research [3]–[7]. Among existingdetectors, maximum likelihood (ML) detection can achieve

Manuscript received July 22, 2019; revised December 13, 2019 andFebruary 10, 2020; accepted Feb 23, 2020. Date of publication March XX,2020; date of current version May XX, 2020. The work was supported in partby the National Key Research and Development Program 2018YFA0701602,the National Science Foundation of China (NSFC) for Distinguished YoungScholars with Grant 61625106, and the NSFC under Grant 61941104. Thework of H. He was supported in part by the Scientific Research Foundationof Graduate School of Southeast University under Grant YBPY1939 and theScholarship from the China Scholarship Council under Grant 201806090077.The work of C.-K. Wen was supported in part by the Ministry of Scienceand Technology of Taiwan under grants MOST 108-2628-E-110-001-MY3and the ITRI in Hsinchu, Taiwan. The associate editor coordinating thereview of this paper and approving it for publication was Prof. ByonghyoShim. (Corresponding author: Shi Jin.)

This paper has been presented in part at IEEE Global Conference Signaland Information Processing (Globalsip), Anaheim, CA, Nov. 2018 [1].

H. He and S. Jin are with the National Mobile Communications ResearchLaboratory, Southeast University, Nanjing 210096, China (e-mail: heheng-tao@seu.edu.cn, and jinshi@seu.edu.cn).

C.-K. Wen is with the Institute of Communications Engineer-ing, National Sun Yat-sen University, Kaohsiung 804, Taiwan (e-mail:chaokai.wen@mail.nsysu.edu.tw).

G. Y. Li is with the School of Electrical and Computer Engineer-ing, Georgia Institute of Technology, Atlanta, GA 30332 USA (e-mail:liye@ece.gatech.edu).

the optimal performance. However, its complexity increasesexponentially with the number of decision variables. Somesuboptimal linear detectors, such as zero-forcing (ZF) andlinear minimum mean-squared error (LMMSE) detectors, arewith reduced computational complexity, but have a hugeperformance degradation compared with the ML detection.

With excellent performance and moderate complexity, iter-ative detectors, based on approximate message passing (AMP)[8] and expectation propagation (EP) [9] algorithms, have beenproposed for MIMO detection [6], [7]. The AMP-based de-tector approximates the posterior probability on a dense factorgraph by using the central limit theorem and the Taylor ex-pansion, which can achieve Bayes-optimal performance in thelarge-scale systems when the elements of the channel matrixare with independent and identically sub-Gaussian distribution.The EP-based detector [7], [9] is derived by approximatingthe posterior distribution with factorized Gaussian distributionsand can achieve Bayes-optimal performance when the channelmatrix is unitarily-invariant 1 and with a large scale. However,for practical small-size (e.g., 4×4 or 8×8) MIMO systems, theperformance of these iterative detectors is still far from Bayes-optimal solution and has serious deterioration with correlatedMIMO channels and imperfect channel state information (CSI)[10].

Owing to strong learning ability from the data, deeplearning (DL) has been successfully introduced to computervision, automatic speech recognition, and natural languageprocessing. Recently, it has been applied in physical layercommunications [11]–[13], such as channel estimation [14]–[16], CSI feedback [17], signal detection [18]–[26], channelcoding [27], [28], and end-to-end transceiver design [29], [30].In particular, a five-layer fully connected deep neural network(DNN) is embedded into an orthogonal frequency-divisionmultiplexing (OFDM) system for joint channel estimation andsignal detection (JCESD) by treating the receiver as a blackbox and without exploiting domain knowledge [18]. However,training such a black-box-based network requires a lot oftraining time in addition to a huge data set. On the other hand,model-driven DL constructs the network topology based onknown domain knowledge and has been successfully appliedto image reconstruction [32], sparse signal recovery [33]–[36],and wireless communications recently [1], [12].

For MIMO detection, a specifically designed network,named DetNet, has been proposed in [19] by unfolding the

1A matrix A = UΣV is unitarily-invariant if U, Σ and V are mutuallyindependent, and U, V are Haar-distributed. The independent and identicallydistributed (i.i.d.) Gaussian matrix is a typical unitarily-invariant matrix.

arX

iv:1

907.

0943

9v2

[cs

.IT

] 1

9 M

ar 2

021

2

iteration of a projected gradient descent algorithm and addingconsiderable trainable variables. DetNet has comparable per-formance with the AMP-based detector and is more robust toill-conditioned channels [19]. To further reduce the number oflearnable parameters and improve convergence, the approachesin [20] and [21] use DL techniques for the belief propaga-tion and message passing detector, respectively. In [26], aDL-based sphere decoding algorithm is proposed, where theradius of the decoding hypersphere is learned by DNN. Theperformance achieved by this algorithm is very close to theoptimal ML detection. However, most of the existing DL-based detector assume accurate CSI at the receiver and ignorethe channel estimation error.

Motivated by existing works, we develop a model-drivenDL network, named OAMP-Net2, for MIMO detection inthis article, where the iterative detector is improved with afew number of trainable variables to adapt to various channelenvironments. The structure of the detector is obtained byunfolding the OAMP detector, which is similar to our earlywork in [1] and inspired by the TISTA network [34] butadds more trainable parameters. Furthermore, an OAMP-Net2-based JCESD architecture is proposed for imperfectCSI and data-aided scheme is utilized to further improvechannel estimation. The trainable parameters are optimizedby DL technique to adapt to various channel environmentsand take channel estimation error into consideration. The maincontributions of this paper are summarized as follows:

• Different from the existing DL-based MIMO detector[19]–[24], [26] with perfect CSI, we consider the MIMOdetection with estimated channel, which improves theperformance of MIMO receiver by considering the char-acteristics of channel estimation error and channel statis-tics and using the estimated payload data to refine thechannel estimation.

• Compared with the existing DL-based MIMO detector[19]–[21], [26], our proposed detector can provide soft-output information for decoder and absorb the soft infor-mation. In addition, only a few trainable parameters arerequired to be learned, which can reduce the demand forcomputing resources and training time significantly.

• Based on our numerical results, the OAMP-Net2 hasconsiderable performance gain compared with the OAMPdetector. Furthermore, OAMP-Net2 has strong robustnessto signal-to-noise (SNR), channel correlation, modulationsymbol and MIMO configuration mismatches.

Notations—For any matrix A, AT , AH , and tr(A) denotethe transpose, conjugated transpose, and trace of A, respec-tively. In addition, I is the identity matrix, 0 is the zero matrix.A proper complex Gaussian with mean µ and covariance Ωcan be described by the probability density function,

NC(z;µ,Θ) =1

det(πΘ)e−(z−µ)HΘ−1(z−µ).

The rest of this paper is organized as follows. After in-troducing the JCESD architecture in Section II, we developchannel estimator in Section III and propose the model-driven DL detector in Section IV. Then, numerical results

are presented in Section V. Finally, Section VI concludes thepaper.

II. JOINT CHANNEL ESTIMATION AND SIGNALDETECTION

In this section, we consider a MIMO system with Nt

transmit and Nr receive antennas. After presenting the JCESDarchitecture2, we introduce the signal detection consideringchannel estimation error and data-aided channel estimation3.

We assume that channel matrix H ∈ CNr×Nt does notchange in a time slot. In each time slot, Np pilot vectorsxp[n] ∈ CNt×1 for n = 1, . . . , Np, are first transmitted, whichare followed Nd data vectors, xd[n] ∈ CNt×1.

The received signal vectors are yp[n] ∈ CNr×1 forn = 1, . . . , Np and yd[n] ∈ CNr×1 for n = 1, . . . , Nd

corresponding to the pilot and data vectors, respectively.We can also express them into matrix forms as Xp =(xp[1], . . . ,xp[Np]) ∈ CNt×Np , Yp = (yp[1], . . . ,yp[Np]) ∈CNr×Np , Xd = (xd[1], . . . ,xd[Nd]) ∈ CNt×Nd , and Yd =(yd[1], . . . ,yd[Nd]) ∈ CNr×Nd .

A. JCESD Architecture

As in Fig. 1, we consider a turbo-like JCESD architecturefor MIMO systems in this paper, which shares the same spiritas iterative decoding. In JCESD, channel estimator and signaldetector exchange information iteratively until convergence[40]. In the first iteration, pilot-only based channel estimationis performed. In the subsequent iterations, data-aided channelestimation is employed with the help of the detected data.

The input of the JCESD architecture is the pilot signalmatrix, Xp, received signal matrix corresponding to the pilotmatrix, Yp, corresponding to the data matrix, Yd in each timeslot. In the l-th turbo iteration, H(l) is the estimated channelmatrix, X

(l)d is the estimated data matrix, and V

(l)est and V

(l)det

are used to compute the covariance matrix for equivalent noisein signal detector and channel estimator, respectively. The finaloutput of the signal detector is finally detected data matrixX

(L)d , where L is the total number of turbo iterations.Compared with the conventional receiver design where the

channel estimator and signal detector are designed separately,this architecture can improve the performance of the receiverby considering the characteristics of channel estimation er-ror in addition to channel statistics when performing signaldetection and using the estimated payload data for channelestimation, as we will illustrate subsequently.

B. Signal Detection with Channel Estimation Error

In the MIMO system, the received data signal vector yd[n]corresponding to the n-th data vector can be expressed by

yd[n] = Hxd[n] + nd[n], (1)

2One can introduce various JCESD architectures for MIMO systems, in-cluding the schemes in [37]–[41], we present a turbo-like JCESD architecturesimilar to [40] in this paper.

3Although we mainly investigate the model-driven-DL-based MIMO detec-tor in this paper, we first introduce a JCESD architecture and then elaboratethe channel estimator and signal detector modules, respectively.

3

Signal

Detector

Channel

Estimator

JCESD( )p p p p[1], , [ ]N=X x x

( )p p p p[1], , [ ]N=Y y y

( )d d d d[1], , [ ]N=Y y y

p p,X Y ( ) ( )est

ˆ ˆ,l l

H V

( ) ( )

d detˆ ˆ,

l lX V

( ) ( ) ( )( )d d d dˆ ˆ ˆ[1], , [ ]

L L LN=X x x( )

dˆ LX

Fig. 1. The diagram of the turbo-like JCESD architecture. The channel estimator and signal detector exchange informationiteratively until convergence.

where nd[n] ∼ NC(0, σ2INr) is the additive white Gaussian

noise (AWGN) vector. Note that (1) can also be expressed intomatrix from as

Yd = HXd + Nd, (2)

where Nd = (nd[1], . . . ,nd[Nd]) ∈ CNr×Nd is the AWGNmatrix in the data transmission stage. Denote the estimatedchannel H ∈ CNr×Nt as

H = H + ∆H, (3)

where ∆H is the channel estimation error. If the estimatedchannel is used for signal detector, the signal detection prob-lem can be formulated as

yd[n] = Hxd[n] + nd[n]

= (H−∆H)xd[n] + nd[n]

= Hxd[n] + nd[n]−∆Hxd[n]

= Hxd[n] + nd[n], (4)

where nd[n] = nd[n] − ∆Hxd[n] is the equivalent noisein signal detector which includes the contribution of chan-nel estimation error and original additive noise. nd[n] ∼NC(0, Vd[n]) is assumed to be Gaussian distribution4, andthe covariance matrix Vd[n] can be obtained by consideringthe statistical properties of the channel estimation error anddetailed calculated process is shown in Appendix A.

C. Data-Aided Channel Estimation

In this section, we introduce the data-aided channel esti-mation approach. Conventional pilot-only based channel esti-mation is performed and transmitted symbols are detected inthe first iteration. Then, the detected symbols are fed back tothe channel estimator as additional pilot symbols to refine thechannel estimation.

In the channel training stage, pilot matrix Xp is transmitted,similar to (1), the received signal matrix corresponding to thepilot matrix Yp ∈ CNr×Np can be expressed as

Yp = HXp + Np, (5)

where Np = (np[1], . . . ,np[Np]) ∈ CNr×Np is the AWGNmatrix and each column np[n] ∼ NC(0, σ2INr

) for n =

4The assumption that nd[n] is Gaussian distribution is reasonable as weconsider the LMMSE channel estimator in next section.

1, . . . , Np. The estimated data matrix Xd can be expressedas

Xd = Xd + Ed, (6)

where Ed is the signal detection error matrix. In data-aidedchannel estimation stage, estimated Xd are fed back to channelestimator as additional pilot. Then, the received signal matrixYd corresponding to Xd can be expressed as

Yd = HXd + Nd

= H(Xd −Ed) + Nd

= HXd + (Nd −HEd)

= HXd + Np, (7)

where Np = Nd−HEd is the equivalent noise for additionalpilot part Xd. The statistical information of the n-th column ofNp, np[n] ∼ NC(0, Vp[n]) for n = 1, . . . , Nd, where Vp[n]is calculated in Appendix A and will be utilized in data-aidedchannel estimation stage. Then, we denote Y = (Yp,Yd)as received signal matrix corresponding to overall transmittedsignal matrix. Based on (5) and (7), we have

Y = (Yp Yd)

= (HXp + Np,HXd + Np)

= H(Xp, Xd) + (Np, Np)

= HX + N, (8)

where X = (Xp, Xd), N = (Np, Np) can be interpretedas the equivalent pilot signal and noise in data-aided channelestimation stage, respectively.

D. Model-Driven DL for JCESD

In Section II-A, we have introduced the principle of theJCESD architecture. Compared with other DL-based JCESDarchitecture [16], [18], which uses a large number of datato train the black-box-based network, we construct the net-work architecture by employing model-driven DL and domainknowledge. In particular, we employ the LMMSE channelestimator and construct the model-driven DL detector. Asin [1], the proposed model-driven DL detector is obtainedby unfolding the existing iterative detector and adding sev-eral trainable parameters, which fully exploiting the domainknowledge. This architecture is promising as it inherits the

4

superiority of traditional approach and uses DL techniqueto improve the performance. We will introduce the channelestimator and signal detector in detail in Sections III and IV,respectively.

III. LMMSE CHANNEL ESTIMATOR

To facilitate the representation of the channel estimationproblem, we apply matrix vectorization to (5) and rewrite itas

yp = Aph + np, (9)

where Ap = XTp ⊗ INr ∈ CNpNr×NtNr , yp = vec(Yp) ∈

CNpNr×1, h = vec(H) ∈ CNrNt×1 and np = vec(Np) ∈CNpNr×1. We denote ⊗ as the matrix Kronecker productand vec(·) as the vectorization operation. In pilot-only basedchannel estimation stage, the LMMSE estimate of h is givenby

hp = RhhAHp (ApRhhAH

p + σ2INpNr)−1yp, (10)

where Rhh is the channel covariance matrix. Based on theproperty of LMMSE estimate, hp is a Gaussian random vector.The channel estimation error vector ∆hp = hp − h is alsoa Gaussian random vector with zero-mean and the covariancematrix R∆hp can be computed as

R∆hp = Rhh−RhhAHp (ApRhhAH

p +σ2INpNr)−1ApRhh.

(11)In data-aided channel estimation stage, considering data

feedback to channel estimator, the LMMSE channel estimationis given by

h = RhhAH(ARhhAH + Rnn)−1y. (12)

where A = XT ⊗ INr∈ CNcNr×NtNr , y = vec(Y) ∈

CNcNr×1, n = vec(N) ∈ CNcNr×1 and Nc = Np + Nd.The covariance matrix R∆h of the channel estimation errorvector ∆h = h− h can be computed as

R∆h = Rhh −RhhAH(ARhhAH + Rnn)−1ARhh. (13)

The covariance matrix Rnn contains the equivalent noisepower of the actual pilot Xp and additional pilot part Xd,which is calculated in Appendix A.

IV. MODEL-DRIVEN DL DETECTOR

In this section, we develop a model-driven DL detector,called OAMP-Net2, which has been partly presented in ourearly work [1]. After introducing the principle of the OAMPdetector, we present the structure of OAMP-Net2 in detail.Afterwards, computational complexity and extension to soft-input and soft-output detection are also investigated.

A. OAMP Detector

The OAMP algorithm has been proposed to solve sparselinear inverse problems in compressed sensing [42] and canbe utilized for MIMO detection in Algorithm 1. The goal of theOAMP algorithm is to recover the transmitted signal vector xd

from the received signal vector yd = Hxd+nd5. The principle

of the algorithm is to decouple the posterior probabilityP(xd|yd, H) into a series of P(xi|yd, H)(i = 1, 2, . . . , Nt)in an iterative way, when estimated channel H and receivedsignal yd are available.

Algorithm 1: OAMP algorithm for MIMO detectionInput: Received signal yd, estimated channel matrixH, equivalent noise covariance matrix Rndnd

.Output: Recovered signal xd,T+1.Initialize: τ1 ← 1, xd,1 ← 0

rt = xd,t + Wt(yd − Hxd,t), (14)

xd,t+1 = E x|rt, τt , (15)

v2t =‖y − Hxd,t‖22 − tr(Rndnd

)

tr(HHH), (16)

τ2t =

1

Nttr(BtB

Ht )v2

t +1

Nttr(WtRndnd

WHt ). (17)

The OAMP detector is presented in Algorithm 1 and mainlycontains two modules: linear estimator (14) and nonlinearestimator (15). The error variance estimators (16) and (17)are the average variance of the two error vectors pt and qt,where pt = rt−xd and qt = xd,t−xd are used to measure theaccuracy of the output in the linear and nonlinear estimators,respectively. They are defined as

v2t =

E[‖qt‖22]

Nt, τ2

t =E[‖pt‖22]

Nt, (18)

and can be computed by employing (16) and (17).1) Linear Estimator: The matrix Wt in linear estimator

(14) can be the transpose of H, the pseudo inverse of H orthe LMMSE matrix. From [42], the optimal one is

Wt =Nt

tr(WtH)Wt, (19)

where Wt is the LMMSE matrix given by

Wt = v2t H

H(v2t HHH + Rndnd

)−1, (20)

where v2t is expressed in (16), and Rndnd

is the covariancematrix of the equivalent noise nd in signal detector, whichincludes the contribution of the channel estimation error ∆Hand original additive noise nd, and has been discussed inSection II-B. The matrix Wt is called de-correlated whentr(Bt) = 0, where Bt = I −WtH, and thus ensures theentries of pt are uncorrelated with those of xd and mutuallyuncorrelated with zero-mean and identical variances.

2) Nonlinear Estimator: The nonlinear estimator in theOAMP detector is constructed by MMSE estimate of xd,which is with respect to the equivalent AWGN channel

rt = xd,t + wt, (21)

5As each transmitted symbol vector xd(n) for n = 1, . . . , Nd in eachtime slot shares the same channel H, we omit the time index n and use xd

to refer xd[n] for simplicity.

5

-

... ...1-th layer

OAMP-Net2

t-th layer T-th layerd,1ˆ 0=x d,2x

d,ˆ

txd, 1

ˆt+x d,

ˆTx

d, 1ˆ

T +x

d,tx trd, 1

ˆt+x

γt tW2

t2

tv

H

d[n]yH

d[n]y

H

dˆ [n]x

Fig. 2. Block diagram of OAMP-Net2 detector. The network consists of T cascade layers, and each layer has the samestructure that contains the linear estimator Wt, nonlinear estimator ηt(·), error variance τ2

t and v2t , and tied weights.

where wt ∼ NC(0, τ2t I). As the transmitted symbol xd is

from the discrete constellation set S = s1, s2, . . . , sM, cor-responding MMSE estimate for each element of the estimatedsymbol vector is given by

x(i)d,t+1 = E xi|ri, τt =

∑sisiNC(si; ri, τ

2t )p(si)∑

siNC(si; ri, τ2

t )p(si), (22)

where p(si) is the prior distribution of the transmitted symbolxi and is given by

p(xi) =∑j∈M

1√Mδ(xi − sj). (23)

From (14), (15), (17) and (21), we observe that rt and τ2t

are the prior mean and variance in MMSE estimator (15)that control the accuracy and convergence of estimated resultxd,t+1. The OAMP detector uses an iterative manner to obtainrt and τ2

t , and the step-size for the update of rt and τ2t will

influence the final performance. Instead of using complicatedanalytical method to find an optimal step-size, we use a DLapproach for providing an appropriate step-size to updatert and τ2

t and constructing learnable nonlinear estimator toimprove the detection performance.

B. OAMP-Net2 Detector

In this section, we introduce the OAMP-Net2 detector,which is developed by unfolding the OAMP detector. Afterpresenting the network architecture, we elaborate the linearand nonlinear estimators, error estimators, and learnable vari-ables in OAMP-Net2 detector.

1) Network Architecture: The structure of the OAMP-Net2is illustrated in Fig. 2, which is obtained by unfolding theOAMP detector and adding several trainable parameters. Theinput of the OAMP-Net2 is the received signal vector, yd,and the estimated channel matrix, H, while the final output isxd,T+1.

The network consists of T cascade layers and each has thesame architecture but different trainable parameters. For the

t-th layer of the OAMP-Net2, the input is the estimated signalxd,t from the (t − 1)-th layer. Signal detection is performedas follows

rt = xd,t + γtWt(yd − Hxd,t), (24)

xd,t+1 = ηt(rt, τ2t ;φt, ξt), (25)

v2t =‖y − Hxd,t‖22 − tr(Rndnd

)

tr(HHH), (26)

τ2t =

1

Nttr(CtC

Ht )v2

t +θ2t

Nttr(WtRndnd

WHt ). (27)

The critical difference between the OAMP and OAMP-Net2detectors is there are learnable variables Ωt = γt, φt, ξt, θtin each layer. By optimizing these parameters in the trainingprocess, the detection performance can be improved. Whenγt = θt = 1, φt = 1 and ξt = 0, the OAMP-Net2 is reducedto the OAMP detector. The matrix Ct = I − θtWtH in theOAMP-Net2 is a revised structure of Bt in the OAMP detectorwith learnable parameter θt to regulate the error varianceτ2t . We will introduce the linear estimator, Wt, nonlinear

estimator, ηt(·), error estimators, τ2t and v2

t , and learnablevariables, Ωt, respectively.

2) Linear and Nonlinear Estimator Modules: The linearestimator module in (24) is a revised structure of (14) inthe OAMP algorithm by adding corresponding learnable pa-rameter γt. The learnable parameter γt can be interpreted asthe step-size for the update of rt, and is γt = 1 in eachiteration in the OAMP detector. The de-correlated matrix Wt

is given in (19). The nonlinear estimator ηt(·) in OAMP-Net2is constructed by the divergence-free estimator

ηt(rt, τ2t ;φt, ξt) = φt(E xd|rt, τt − ξtrt), (28)

where E x|rt, τt is the MMSE estimate of xd with theequivalent AWGN channel (21), and has the same expressionwith (22). The nonlinear estimator, ηt(rt, τ2

t ;φt, ξt), can beinterpreted as a linear combination of the priori mean, rt, andthe posteriori mean, E xd|rt, τt, which has been investigatedin [42] and [45].

6

Compared with the MMSE estimator (22) in the OAMPdetector, the divergence-free estimator (28) considers thecontribution of the linear estimator and learnable parameters(φt, ξt). The MMSE estimator (22) can be interpreted as aspecial case of (28) by setting φt = 1 and ξt = 0.

3) Error Variance Estimators: The error variance estima-tors v2

t in (26) and τ2t in (27) play important roles in providing

appropriate variance estimates required for the linear andnonlinear estimators in the OAMP-Net2 detector. For errorvariance v2

t , we adopt the same estimator with the OAMPdetector in (26). For error variance τ2

t , we construct theestimator based on two assumptions on error vectors pt andqt in [42] and incorporate learnable variables (γt, θt). Thedetailed derivation for the two variance estimators v2

t and τ2t

is provided in Appendix B. Furthermore, we substitute v2t by

max(v2t , ε) for a small positive constant ε = 5 × 10−13 to

avoid stability problem.4) Learnable Variables: The learnable variables Ω =

ΩtTt=1 are optimized in the training process for OAMP-Net2. The Bayes-optimal property of the OAMP algorithm hasbeen proven in [43], but it is derived in the large system withunitarily-invariant matrix H. In fact, the performance of theOAMP detector is far from the Bayes-optimal performance inpractical finite-dimensional MIMO systems, especially whenthere are strong spatial correlation and channel estimationerror. These observations motivate us to improve the originaliterative detector with several trainable parameters to adapt tovarious channel environments.

Similar to the TPG detector [22] and OAMP-Net [1], theOAMP-Net2 uses two learnable parameters (γt, θt) to adjustthe linear estimator, and to provide appropriate stepsize forthe update of the mean rt and variance τ2

t in the MMSEestimator. On the other hand, the linear estimator (24) is relatedto gradient descent algorithm and its convergence behaviorand performance are determined by appropriate step-size ofmoving to the search point. The optimal step-size γt canbe learned from the data for the update of the prior meanrt in the MMSE estimator. Furthermore, the parameter θthas the similar function for the error variance τ2

t , which cancompensate for the channel estimation error and regulate theτ2t to provide appropriate value for the update of the prior

variance in the MMSE estimator.The parameters (φt, ξt) in the nonlinear estimator ηt(·) play

important roles in constructing an appropriate divergence-freeestimator, which has been discussed in [42]. In precise, thedivergence-free estimator (28) can be applied in the OAMPdetector, but the φt and ξt are related to the prior distribution ofthe original signal and difficult to calculate. Therefore, MMSEestimator (22) is considered for simplicity in the OAMPdetector. In our early work [1], we set φt = 1 and ξt = 0in OAMP-Net and use MMSE estimator (22). By contrary,we adaptively learn two parameters φt and ξt in OAMP-Net2 to construct the nonlinear estimator ηt(·) satisfying thedivergence-free property.

5) Complexity Analysis: The computational complexity re-quired for the OAMP-Net2 is O(TN3

t ). Similar to the OAMPdetector and OAMP-Net, the computational complexity isdominant by the matrix inverse in each layer in (20). When Nt

is relatively small (e.g., 4 or 8), the matrix inverse operationis always acceptable. By contrary, the AMP detector [5] has acomplexity of O(TN2

t ), which is dominated by the matrixmultiplication, but has performance deterioration in small-size MIMO systems. The LMMSE detector has a complexityof O(N3

t ) as one matrix inversion is needed, but it is non-iterative and no training is required. Although ML can achievethe optimal performance, it has a complexity of O(|S|Nt).In Table I, we compare the computational complexity of theOAMP-net2 detector with the state-of-art DL-based MIMOdetectors in [19]–[23] including DetNet [19], DNN-dBP [20],DNN-MPD [21], TPG [22], LcgNet [23]. Although the state-of-art DL-based MIMO detectors have lower complexity thanthe OAMP-Net2 detector, their performance is deteriorated inthe small-MIMO systems to some extent.

Furthermore, we investigate the number of learnable vari-ables in different DL-based MIMO detectors. From Fig. 2,the total number of trainable variables is equal to 4T sinceeach layer of the OAMP-Net2 contains only four trainablevariables Ωt = (γt, φt, ξt, θt). By contrast, 2T trainablevariables (γt, θt) are required to train in OAMP-Net. However,the numbers of learnable variables in DetNet, DNN-MPD andLcgNet heavily are dominated by the number of antennas inthe transmitter or the receiver. By contrary, the number oftrainable variables of the OAMP-Net2 and OAMP-Net areindependent of the number of antennas Nr and Nt, and onlydetermined by the number of layers T . This is an advantageousfeature for large-scale problems, such as massive MIMOdetection. With only few trainable variables, the stability andspeed of convergence can be improved in the training process.

C. Soft-Input and Soft-Output

As many modern digital communication systems need toproduce a probabilistic estimation of the transmitted datagiven the observations to probabilistic channel decoder. Asignificant issue is whether the MIMO detector can use thesoft-information from the decoder and produce the soft-output.Different from the DetNet in [19] that only can provide thesoft-output, our proposed detector are the soft-input and soft-output receiver and therefore can achieve the turbo equaliza-tion. We only provide the principle of the OAMP-Net2-basedturbo receiver and the specific experimental results are outsidethe scope of this paper and will be conducted in the future.

From (28), the OAMP-Net2 can decouple the joint posteriorprobability P(x|yd, H) into a series of marginal posteriorprobability P(xj |rt, τt). The marginal posterior probability isused to produce the soft output log-likelihood ratios (LLR),which is given by

LA(bj,k) = log

∑S+jP(xj |rt, τt)∑

S−jP(xj |rt, τt)

, (29)

where bj,k is the k-th bit in the transmitted symbol xj , and S+j

and S−j denote the subsets of the constellation symbols withthe k-th bit being 1 and 0, respectively. After interleaved anddelivered to the channel encoder, extrinsic LLR can be com-puted and given to the OAMP-Net2 detector as updated prior

7

TABLE ICOMPUTATIONAL COMPLEXITY OF DIFFERENT DETECTORS

ComplexityDetectors OAMP-Net2 DetNet DNN-dBP DNN-MPD TPG LcgNet

Computational complexity O(TN3t ) O(TN2

t ) O(TNrNt) O(TNrNt) O(TNrNt) O(TNrNt)Learnable variables 4T (6NrNt + 2Nr +Nt)T T (Nr +Nt)T 2T 4NtT

information. The detector and decoder iteratively exchangeinformation until convergence.

D. Practical Implementation

The developed detector can be divided into two stages. Inthe offline training stage, we obtain the optimized parameters,Ω = ΩtTt=1, for different SNRs and channel correlationcoefficients based on TensorFlow platform and GPUs’ pow-erful computing ability. The optimized parameters will bestored to detect the modulated symbols in the deploymentstage. The OAMP-Net2 detector can be interpreted as a newiterative detector after training. The incorporated learnableparameters can adapt to practical channel and compensate forchannel estimation error to improve the detection performance.Although aforementioned implementation process is in anoffline manner, the proposed detector can also be implementedby online training to adapt to the fluctuations in the channelconditions owing to the superiority of its low demand fortraining data and computational resources.

V. NUMERICAL RESULTS

In this section, we provide numerical results to show theperformance of the proposed model-driven detector for MIMOdetection. First, we elaborate the implementation details andparameter settings. Then, the performance of the OAMP-Net2 is presented under i.i.d. and correlated Rayleigh MIMOchannels with perfect CSI. Afterwards, we investigate theperformance of the OAMP-Net2-based JCESD architecture.Finally, the robustness of the OAMP-Net2 to SNR and channelcorrelation is demonstrated.

A. Implementation Details

In our simulation, OAMP-Net2 is implemented in Tensor-flow by using a PC with GPU NVIDIA GeForce GTX 1080Ti. The SNR of the system, defined as

SNR =E‖Hxd‖22E‖nd‖22

, (30)

is used to measure the noise level. We assume the same SNRin pilot and data transmission stage. Each element of the i.i.d.Rayleigh MIMO channel H is hi,j ∼ NC(0, 1/Nr) whilethe correlated Rayleigh MIMO channel is described by theKronecker model,

Hc = R1/2R GR

1/2T , (31)

where RR and RT are the spatial correlation matrix at thereceiver and the transmitter, respectively, and are generatedaccording to the exponential correlation model [44] with the

same correlation coefficient ρ, and G is the i.i.d. Rayleighfading channel matrix. Furthermore, to implement OAMP-Net2 in the Tensorflow, we consider equivalent real-valuedrepresentation for all aforementioned problem models.

The training data consists of a number of randomly gener-ated pairs d(i) , (x(i),y(i)). For each pair d(i), channel His randomly generated from the i.i.d. or correlated RayleighMIMO channel model. The data x(i) is generated from the M -QAM modulation symbol with M being the modulation order.We train the network with 1, 000 epochs. At each epoch, thetraining set contains 5, 000 different samples d(i), and 1, 000different validation samples. The validation sets are used tochoose the best network for each epoch in the training stage.In the test stage, we generate the data to test the networkuntil the number of bit errors exceed 10, 000. The networkis trained using the stochastic gradient descent method andAdam optimizer. The learning rate is set to be 0.001 and thebatch size is set to 100. To train the learnable variables well,the learning rate is set to be 0.0001 and validation samplesis 10, 000 in high SNR regime (SNR = 30 or above). Exceptfor special instructions, we choose the l2 loss 6 as the costfunction in all experiment settings, which is defined as

l2(Ω) =1

D

∑d(i)∈D

‖x(i)d − xd,T+1(y(i))‖22, (32)

where D is the training set of a mini-batch and D = |D|.In general, the optimal variables Ω may be different for

different SNRs. Therefore, we train the OAMP-Net2 for eachSNR separately, which means that the training and test dataare generated with the same fixed discrete SNR value. For thecorrelated Rayleigh MIMO channel, we train the network foreach correlation coefficient ρ, respectively. The network needsto be retrained if different modulation symbols are utilized.Furthermore, we also consider the robustness of the OAMP-Net2 to various mismatches.

B. Perfect CSI

In this section, we study the performance of OAMP-Net2for MIMO detection with perfect CSI. After analyzing its con-vergence, the detection performance of the OAMP-Net2 underi.i.d. and correlated Rayleigh MIMO channels is investigated.

1) Convergence analysis: First, we analyze the convergenceof the OAMP-Net2 and OAMP detectors. Fig. 3 illustrates thebit-error rate (BER) performance versus the number of layers(iterations) under SNR = 15dB and 25dB with QPSK and 16-QAM constellations. From Fig. 3, both the OAMP-Net2 and

6We train the OAMP-Net2 with l2 loss and cross-entropy, respectively, andfind the network trained with l2 loss outperforms that trained with cross-entropy.

8

1 2 3 4 5 6 7 8 9 10

Number of layers

10-4

10-3

10-2

10-1

100

BE

R

OAMPOAMP-Net2

SNR = 15dB

SNR = 25dB

(a) QPSK

1 2 3 4 5 6 7 8 9 10

Number of layers

10-4

10-3

10-2

10-1

100

BE

R

OAMPOAMP-Net2

SNR = 15dB

SNR = 25dB

(b) 16-QAM

Fig. 3. BERs performance of the OAMP-Net2 and OAMPdetector versus the number of layers under QPSK and 16-QAM modulation.

OAMP detectors converge within four layers (iterations) for allthe cases and OAMP-Net2 can obtain more performance gainwhen SNR = 25dB, which means the OAMP-Net2 detectorimproves OAMP detector significantly in high SNR regime.Furthermore, compared with 16-QAM, the performance gapbetween OAMP and OAMP-Net2 is larger for QPSK. Basedon the above observations, we set OAMP and OAMP-Net2detectors four layers (T=4) in following simulation whenSNR6 25 dB, and ten layers (T=10) when SNR = 30dB orabove except for specific instructions.

2) Performance Comparison: Fig. 4 compares the BERperformance of the OAMP, OAMP-Net [1], TPG [22], DetNet[19], AMP [5], SD, ML, MMSE-SIC [46], LMMSE andOAMP-Net2 detectors. Both 4×4 and 8×8 MIMO systems areinvestigated. Among these detectors, the TPG detector [22] isdeveloped with two trainable parameters in each layer and hasa similar architecture to OAMP-Net. Furthermore, the OAMPand MMSE-SIC detectors can achieve soft decision. From thefigure, the OAMP-Net2 outperforms other MIMO detectorsexcept for SD and ML, but the computational complexity ofthe ML and SD detectors is prohibitive. The DetNet and AMPdetectors have serious performance deterioration in small-

0 5 10 15 20 25 30 35SNR (dB)

10-4

10-3

10-2

10-1

100

BE

R

OAMPOAMP-NetOAMP-Net2TPGLMMSESDMLMMSE-SICAMPDetNet

QPSK

64-QAM

(a) 4× 4 MIMO

0 5 10 15 20 25 30 35SNR (dB)

10-4

10-3

10-2

10-1

100

BE

R

OAMPOAMP-NetOAMP-Net2TPGLMMSESDMLMMSE-SICAMPDetNet

QPSK

64-QAM

(b) 8× 8 MIMO

Fig. 4. BERs performance comparison of the OAMP-Net2with other MIMO detectors under i.i.d. Rayleigh MIMOchannels with different orders of modulation.

size MIMO systems. Furthermore, the OAMP-Net, TPG, andOAMP-Net2 detectors outperform the OAMP detector in allsetting, which demonstrates DL can improve the OAMP detec-tor by learning the optimal parameters. But the performancegain is limited for 64-QAM. Compared with OAMP-Net in[1], OAMP-Net2 has a significant performance improvementin all setting. The reason is the OAMP-Net2 can learn theoptimal parameter (φt, ξt) to construct the nonlinear estimatorsatisfying the divergence-free property. For high-dimensionalMIMO systems in Fig. 5, similar conclusions can be obtained.

3) Correlated MIMO Channel Performance: Fig. 6 illus-trates the BER of the OAMP-Net2 and the OAMP detector un-der correlated Rayleigh MIMO channel with ρ = 0.5. We setthe OAMP and OAMP-Net2 detectors ten layers with differentorders of modulation. From the figure, all algorithms haveperformance degradation compared with the i.i.d. RayleighMIMO channel, But the OAMP-Net2 detector still have theperformance improvement over the OAMP detector in allsetting and can obtain more performance improvement in 8×8MIMO system. Specifically, if we target for BER=10−2 withQPSK modulation, BER performance improves about 1.8 dBfor 4× 4 MIMO. By contrast, it is approximately 2.2 dB for

9

0 5 10 15 20 25 30 35

SNR (dB)

10-4

10-3

10-2

10-1

100

BE

R

OAMPOAMP-Net2

QPSK

16-QAM

64-QAM

(a) 16× 16 MIMO

0 5 10 15 20 25 30 35

SNR (dB)

10-4

10-3

10-2

10-1

100

BE

R

OAMPOAMP-Net2

QPSK

16-QAM

64-QAM

(b) 32× 32 MIMO

Fig. 5. BERs performance comparisons of the OAMP andOAMP-Net2 under high-dimensional i.i.d. Rayleigh MIMOchannels with different modulation symbols.

8 × 8 MIMO. But the performance gain of the OAMP-Net2detector decreases with the increase of the modulation order.For example, if we target for BER=10−2 in 4×4 MIMO, SNRimproves about 1.5 dB for QPSK modulation. By contrast, itdecreases to approximately 1.1 dB for 16-QAM .

4) 3GPP MIMO Channels Performance: In the afore-mentioned sections, we have investigated the OAMP-Net2detector with Rayleigh fading channels. Here, we conductnumerical experiments by utilizing a dataset of the realisticchannel samples from the 3GPP 3D MIMO channel [47],as implemented in the QuaDRiGa channel simulator [48].The channel parameters are set to be same as [24]. Asshown in Fig. 7, OAMP-Net2 indeed has performance losswith the realistic channel. The reason is that the OAMP-Net2 requires the channel matrix to be unitarily-invariant whilethe practical channel is very ill-conditioned and the channelcorrelation is serious. which incurs performance degradation.Furthermore, only 4 trainable parameters in each layer mayrestrict the ability to adapt to realistic 3GPP MIMO channels.Nevertheless, the OAMP-Net2 still outperforms the OAMPand LMMSE detectors with limited performance gain.

0 5 10 15 20 25 30 35

SNR (dB)

10-4

10-3

10-2

10-1

100

BE

R

OAMPOAMP-Net2

QPSK

16-QAM

64-QAM

(a) 4× 4 MIMO

5 10 15 20 25 30 35

SNR (dB)

10-4

10-3

10-2

10-1

100

BE

R

OAMPOAMP-Net2

QPSK16-QAM

64-QAM

(b) 8× 8 MIMO

Fig. 6. BERs performance comparison of the OAMP andOAMP-Net2 under correlated Rayleigh MIMO channels withρ = 0.5 for different orders of modulation.

C. JCESD Performance

In the above, all detectors are investigated with accurateCSI. In this section, we consider OAMP-Net2-based JCESDarchitecture for a 4 × 4 MIMO system. Both OAMP andOAMP-Net2 has four layers. In order to avoid gradient van-ishing, we use the summation of l2-loss over all L×T layersto train the model, which is defined as

l2(Ω) =

L∑l=1

T∑t=1

1

D

∑d(i)∈D

‖x(i)d − x

(l)d,T (y(i))‖22, (33)

and the learning rate is set to be 0.0001.We consider orthogonal pilot in channel training stage,

where the pilot matrix Xp ∈ CNt×Np is chosen by selectingNt columns of the discrete Fourier transformation (DFT)matrix F ∈ CNp×Np . Fig. 8 shows the BERs of the OAMPand OAMP-Net2 detectors in the JCESD architecture, whereL = 1 means no data feedback to channel estimator and L = 3means the detected data are feedback twice to channel esti-mator. From the figure, the OAMP-Net2 detector outperformsthe OAMP detector in all settings significantly. Specifically,if we target for BER=10−2 with 16-QAM, BER performanceimproves about 2.1 dB for L = 1 and improves about 2.9 dB

10

0 5 10 15 20 25 30 35SNR (dB)

10-4

10-3

10-2

10-1

100

BE

R

OAMP-Net2OAMPLMMSEML

QPSK

16-QAM

Fig. 7. BERs performance comparison of the OAMP-Net2with other MIMO detectors under 3GPP MIMO channels withdifferent orders of modulation.

0 5 10 15 20 25 30 35

SNR (dB)

10-4

10-3

10-2

10-1

100

BE

R

OAMP (L=1)OAMP (L=3)OAMP-Net2 (L=1)OAMP-Net2 (L=3)

QPSK

16-QAM

Fig. 8. BERs performance comparisons of the OAMP andOAMP-Net2 in JCESD architecture with Np = 4 and Nc =16.

for L = 3. Furthermore, we observe that OAMP-Net2 detectorwithout data feedback (L = 1) outperforms OAMP detectorwith data feedback twice (L = 3), which demonstrates learn-ing some trainable variables can compensate for the channelestimation error to improve the equivalent SNR in detectionstage. For QPSK modulation, the performance gain are 5.1dB and 3.1 dB for L = 1 and L = 3, respectively, whenwe target for BER=10−2. In addition, marginal performanceis obtained by data feedback for the OAMP-Net2 detectorwith QPSK because learnable parameters have strong abilityto compensate for channel estimation error. Therefore, no datafeedback is needed in this case.

D. Robustness

In this section, we analyze the robustness of the OAMP-Net2 against various mismatches, including the SNR, channelcorrelation, MIMO configuration and modulation mismatches.As aforementioned numerical results are performed whentraining and test data are generated with same system parame-ters, an interesting question is whether the trained parameters

5 10 15 20 25 30 35

SNR (dB)

10-4

10-3

10-2

10-1

100

BE

R

OAMPOAMP-Net2 (Mismatch)OAMP-Net2 (No Mismatch)

=0.9

=0.5

=0.1

Fig. 9. BERs performance comparisons of the OAMP andOAMP-Net2 with SNR and correlation mismatches for QPSKmodulation.

are robust to various mismatches. Because of limited data andcomputing resources are available for online training, verify-ing the robustness of offline-trained network is particularlymeaningful.

1) Robustness to SNR and Channel Correlation: Fig. 9presents the BER performance of OAMP and OAMP-Net2with SNR and correlation mismatches in 8×8 MIMO system.The network is trained in the correlated Rayleigh MIMOchannel (31) with channel correlation coefficient ρ = 0.5 andSNR=20dB, and tested with mismatched channel correlationcoefficient ρ and SNR, which are shown in the figure. Fromthe figure, the trained network with correlation mismatch stilloutperforms the OAMP detector significantly in all setting andhas 0.3 dB performance loss, if we target for BER=10−2 andρ = 0.5, which demonstrates the OAMP-Net2 has strongrobustness to mismatch. Interestingly, the trained networkeven outperforms the OAMP detector with ρ = 0.1 in highSNR regime (SNR=20-30 dB). Since the learnable variablessuccessfully compensates for the disadvantages of channelcorrelation. Compared with perfect SNR7, the OAMP-Net2detector with SNR mismatch has little performance deteriora-tion except for SNR = 25dB or above, when we target ρ = 0.5.

2) Robustness to MIMO Configuration and Modulation:Fig. 10 presents the BER performance of OAMP and OAMP-Net2 with the MIMO configuration and modulation symbolmismatches. We train the OAMP-Net2 with 16-QAM symbolin 8× 8 MIMO system and test the robustness of the trainedparameters. Fig. 10(a) exhibits the numerical results of theOAMP-Net2 with MIMO configuration mismatch, where thenetwork is tested in 4 × 4 systems with 16-QAM. Althoughthe network is employed in different MIMO configurations,the OAMP-Net2 still outperforms the LMMSE and OAMPdetectors and has little performance loss. The robustnessdemonstrates the OAMP-Net2 is flexible to different MIMOconfiguration. Furthermore, Fig. 10(b) shows the performanceof OAMP-Net2 with modulation symbol mismatch, where the

7Perfect SNR means that network in the training and test stage have thesame SNR.

11

0 5 10 15 20 25 30 35SNR (dB)

10-4

10-3

10-2

10-1

100

BE

R

OAMPOAMP-Net2 (Configuration Mismatch)OAMP-Net2 (No Mismatch)LMMSEML

(a) MIMO configuration mismatch

0 5 10 15 20 25 30 35SNR (dB)

10-4

10-3

10-2

10-1

100

BE

R

OAMPOAMP-Net2 (Modulation Mismatch)OAMP-Net2 (No Mismatch)LMMSEML

(b) Modulation symbol mismatch

Fig. 10. BERs performance comparisons of the OAMP andOAMP-Net2 with the MIMO configuration and modulationsymbol mismatches.

network is tested in 8× 8 MIMO system with QPSK symbol.Only 0.4 dB performance loss is incurred owing to modu-lation symbol mismatch if we target for BER=10−2, whichdemonstrates the OAMP-Net2 is robust to modulation symbolmismatch. The OAMP-Net2 detector contains a linear and anonlinear estimators. The linear estimator is related to gradientdescent algorithm and its convergence behavior and perfor-mance are determined by appropriate step-size γt moving tothe search point. As the linear estimator is independent of themodulation symbols, the learnable parameter γt shows strongrobustness to modulation symbol mismatches. Therefore, thetrained network can be utilized directly in different MIMOconfigurations with different modulation symbols.

VI. CONCLUSION

We have developed a novel model-driven DL networkfor MIMO detection, named OAMP-Net2, which is obtainedby unfolding the OAMP detection algorithm. The OAMP-Net2 detector inherits the superiority of the Bayes-optimalsignal recovery algorithm and DL technique, and thus presentsexcellent performance. The network are easy and fast to trainbecause only few adjustable parameters are required to beoptimized. To handle imperfect CSI, an OAMP-Net2-based

JCESD is proposed, where the detector takes channel estima-tion error and channel statistics into consideration while chan-nel estimation is refined by detected data and considers thedetection error. Simulation results demonstrate that significantperformance gain can be obtained by learning correspondingoptimal parameters from the data to improve the detector andcompensate for the channel estimation error. Furthermore, theOAMP-Net2 exhibits strong robustness to various mismatches.

VII. ACKNOWLEDGEMENT

The authors would like to thank Dr. Junjie Ma for discussingthe structure of the OAMP algorithm.

APPENDIX A: DERIVATION FOR COVARIANCE MATRICES

In order to derive the covariance matrix Vd[n] of theequivalent noise nd[n] in signal detection stage, we shouldevaluate the statistical properties of the zd[n] = ∆Hxd[n] forn = 1, . . . , Nd, which can be expressed as

Cov[zd[n]] = E[zd[n](zd[n])H ]. (34)

As the covariance matrix Cov[zd[n]] for each symbol indexn = 1, . . . , Nd is same and the data symbol is independentbetween different antennas, we omit the index n and have

E[zd,izHd,k] =

0, ∀i 6= kNt∑j=1

σ2∆hi,j

, i = k(35)

where zd,i =∑Nt

j=1 ∆hi,jxd,j is the i-th element of the zd,∆hi,j is the (i, j)-th element of the channel estimation errormatrix ∆H, and xd,j is j-th element of the xd. The σ2

∆hi,j

is the variance of (i, j)-th element in channel estimation errormatrix ∆H which can be obtained from R∆hp in (11) or R∆h

in (13). By considering the contribution of original additivenoise, the covariance matrix Vest can be obtained as

Vest = diag

Nt∑j=1

σ2∆hi,j

+ σ2, . . . ,

Nt∑j=1

σ2∆hi,j

+ σ2

.

(36)The covariance matrix Vest will be utilized in the OAMP-Net2 detector as the covariance matrix of the equivalent noiseRndnd

.Next, we compute the covariance matrix Vp[n] of the

equivalent noise np[n] in data-aided channel estimation stageusing similar approach. For each time index n, We denotezn = Hen and zi,n =

∑Nt

j=1 hi,jej,n, where en is the n-thcolumn of the signal detection error matrix Ed. In a similarway, we have

E[zi,nzHk,n] =

0, ∀i 6= kNt∑j=1

σ2ej,n/Nr, i = k

(37)

because hi,j ∼ NC(0, 1/Nr) and σ2ej,n is the variance for j-th

element of the en. Therefore, the covariance matrix Vp[n] isgiven by

Vp[n] =

Nt∑j=1

σ2ej,n + σ2

INt . (38)

12

The covariance matrix Vp[n] will be utilized in channelestimator to evaluate the covariance matrix of the equivalentnoise Rnn where n = vec(N). By considering the differentcovariance matrix of the actual pilot and additional pilot, wehave

Rnn =

[σ2INpNr

Vdet

](39)

where σ2INpNrdenotes the covariance matrix of the noise

vec(Np) for actual pilot Xp while

Vdet =

Vp[1]. . .

Vp[Nd]

(40)

denotes the covariance matrix of the equivalent noise vec(Np)for the additional pilot Xd.

APPENDIX B: DERIVATION FOR VARIANCE ESTIMATORS

To derive the expressions for the error variance estimatorsv2t and τ2

t in OAMP-Net2, we use similar method to [34]. Weshould indicate that v2

t and τ2t in OAMP-detector are based

on the following two assumptions about error vectors pt andqt [42].

Assumption 1: pt consists of i.i.d. zero-mean Gaussianentries independent of xd.

Assumption 2: qt consists of i.i.d. zero-mean Gaussianentries independent of H and nd.

From the similar viewpoint, we obtain the error varianceestimators v2

t and τ2t (26) and (27 in OAMP-Net2 using

Assumption 1 and 2, and considering the effect of the learnablevariables γt and θt. For convenience, we will use H, xt, andn to refer H, xd,t, and nd, respectively. Based on assumption2, qt consists of i.i.d. zero-mean Gaussian entries which isindependent of H and n, which means that

E[(xt − x)HHHn] = 0. (41)

Therefore, The error variance estimator v2t is given by,

v2t =

E[‖qt‖22]

Nt=

1

Nt

tr(HHH)E[‖qt‖22]

tr(HHH)

=E[(x− xt)

HHHH(x− xt)]

tr(HHH)

a=

E[(H(x− xt))H

H(x− xt) + (H(x− xt))H

n]

tr(HHH)

=E[‖H(x− xt) + n‖22]− E[nHn]

tr(HHH)

=E[‖Hx + n−Hxt‖22]−Nrσ

2

tr(HHH)

=E[‖y −Hxt‖22]−Nrσ

2

tr(HHH), (42)

where a= is obtained by using the equation (41). The final

expression for v2t is obtained by (42).

Next, we derive the expression for τ2t . The error vector rt−x

can be rewritten as

rt − x = xt + γtWt(y −Hxt)− x

= xt + γt(Hx + n)− γtWtHxt − x

= (I− γtWt)(xt − x) + γtWtn. (43)

Then, for the error variance estimator τ2t , we have

τ2t =

E[‖qt‖22]

Nt=

1

NtE[‖(I− γtWtH)(xt − x) + γtWtn‖22]

=1

NtE[(xt − x)H(I− γtWtH)(I− γtWtH)H(xt − x)]

+γ2t

NtE[nHWH

t Wtn] +2γtNt

E[(xt − x)H(I− γtWtH)HWtn]

=1

Nttr((I− γtWtH)(I− γtWtH)H

)v2t

+γ2t

Nttr(WtWt

H)σ2 +2(γt − γ2

t )

NtE[(xt − x)HHHn].

(44)

The last term in the (44) vanishes as the E[(xt−x)HHHn] =0. Therefore, the error variance estimator τ2

t is given by

τ2t =

1

Nttr((I− γtWtH)(I− γtWtH)H

)v2t

+γ2t

Nttr(WtWt

H)σ2. (45)

Then, we replace the parameter γt with θt in (45) as thesimulation results show that two parameters (γt, θt) canpreferably regulate the error variance estimator v2

t and τ2t .

The superior performance is attributed to more parameters,which can incorporate more side information from the data.Therefore, the final expression for τ2

t is obtained by

τ2t =

1

Nttr(CtC

Ht )v2

t +θ2t σ

2

Nttr(WtW

Ht ), (46)

where Ct = I− θtWtH.

REFERENCES

[1] H. He, C.-K. Wen, S. Jin, and G. Y. Li, “A model-driven deep learningnetwork for MIMO detection,” in Proc. IEEE Glob. Conf. Signal Inf.Process., Anaheim, CA, Nov. 2018, pp. 584-588.

[2] L. Zheng and D. N. C. Tse, “Diversity and multiplexing: A fundamentaltradeoff in multiple-antenna channels,” IEEE Trans. Inform. Theory, vol.49, no. 5, pp. 1073-1096, May 2003.

[3] S. Yang and L. Hanzo, “Fifty years of MIMO detection: The road tolarge-scale MIMOs,” IEEE Commun. Surveys Tuts., vol. 17, no. 4, pp.1941-1988, 4th Quart., 2015.

[4] Z. Guo and P. Nilsson, “Algorithm and implementation of the K-bestsphere decoding for MIMO detection,” IEEE J. Sel. Areas Commun.,vol. 24, no. 3, pp. 491-503, Mar. 2006.

[5] C. Jeon, R. Ghods, A. Maleki, and C. Studer. “Optimality of large mimodetection via approximate message passing,” in Proc. IEEE Int. Symp.Inf. Theory, Jun. 2017, pp. 2333-2337.

[6] S. Wu, L. Kuang, Z. Ni, J. Lu, D. Huang, and Q. Guo, “Low-complexityiterative detection for large-scale multiuser MIMO-OFDM systems usingapproximate message passing,” IEEE J. Sel. Topics Signal Process., vol.8, no. 5, pp. 902-915, Oct. 2014.

[7] J. Cespedes, P. M. Olmos, M. Sanchez-Fern´andez, and F. Perez-Cruz,“Expectation propagation detection for high-order high-dimensionalMIMO systems,” IEEE Trans. Commun., vol. 62, no. 8, pp. 2840-2849,Aug. 2014.

13

[8] D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algo-rithms for compressed sensing,” Proc. Nat. Acad. Sci., vol. 106, no. 45,pp. 18914-18919, 2009.

[9] T. P. Minka, “A family of algorithms for approximate Bayesian In-ference,” Ph.D. dissertation, Dept. Elect. Eng. Comput. Sci., MIT,Cambridge, MA, USA, 2001.

[10] K. Ghavami and M. Naraghi-Pour, “MIMO detection with imperfectchannel state information using expectation propagation” IEEE Trans.Veh. Technol., vol. 66, no. 9, pp. 8129-8138, Sep. 2017.

[11] T. Wang, C.-K. Wen, H. Wang, F. Gao, T. Jiang, and S. Jin, “Deeplearning for wireless physical layer: Opportunities and challenges,”China Communications, vol. 14, no. 11, pp. 92-111, Nov. 2017.

[12] H. He, S. Jin, C.-K. Wen, F. Gao, G. Y. Li, and Z. Xu, “Model-driven deep learning for physical layer communications,” IEEE WirelessCommunications, vol. 26, no. 5, pp. 77-83, Oct. 2019.

[13] Z.-J. Qin, H. Ye, G. Y. Li, and B.-H. Juang, “Deep learning in physicallayer communications,” IEEE Wireless Communications, vol. 26, no. 2,pp. 93-99, Apr. 2019.

[14] H. He, C.-K. Wen, S. Jin, and G. Y. Li, “Deep learning-based channelestimation for beamspace mmWave massive MIMO systems,” IEEEWireless Commun. Lett., vol. 7, no. 5, pp. 852–855, Oct. 2018.

[15] Y. Yang, F. Gao, X. Ma, and S. Zhang, “Deep learning-based channelestimation for doubly selective fading channels,” IEEE Access, vol. 7,no. 1, pp. 36579–36589, Dec. 2019.

[16] C.-J. Chun, J.-M. Kang, and I.-M. Kim, “Deep learning based channelestimation for massive MIMO systems,” IEEE Wireless Commun. Lett.,vol. 8, no. 4, pp. 1228–1231, Aug. 2019.

[17] C.-K. Wen, W. T. Shih, and S. Jin, “Deep learning for massive MIMOCSI feedback,” IEEE Wireless Commun. Lett., vol. 7, no. 5, pp. 748–751,Oct. 2018.

[18] H. Ye, G. Y. Li, and B.-H. F. Juang, “Power of deep learning for channelestimation and signal detection in OFDM systems,” IEEE WirelessCommun. Lett., vol. 7, no. 1, pp. 114-117, Feb. 2018.

[19] N. Samuel, T. Diskin, and A. Wiesel, “Deep MIMO detection,” in Proc.18th IEEE Int. Workshop Signal Process. Advances Wireless Commun(SPAWC), Hokkaido, Japan, Jul. 2017.

[20] X. Tan, W. Xu, Y. Be’ery, Z. Zhang, X. You and C. Zhang, “Improvingmassive MIMO belief propagation detector with deep neural network,”arxiv preprint arXiv:1804.01002, 2018.

[21] X. Tan, Z. Zhong, Z. Zhang, X. You, and C. Zhang, “Low-Complexitymessage passing MIMO detection algorithm with deep neural network,”in Proc. IEEE Glob. Conf. Signal Inf. Process., Anaheim, CA, Nov.2018, pp. 559-563.

[22] S. Takabe, M. Imanishi, T. Wadayama, and K. Hayashi, “Trainableprojected gradient detector for massive overload MIMO channels: Data-driven tuning approach,” arxiv preprint arXiv:1812.10044, 2018.

[23] Y. Wei, M.-M. Zhao, M. Hong, M.-J. Zhao and M. Lei, “Learnedconjugate gradient descent network for massive MIMO detection,” arxivpreprint arXiv:1906.03814, 2019.

[24] M. Khani, M. Alizadeh, J. Hoydis and P. Fleming, “Adaptive neuralsignal detection for massive MIMO,” arxiv preprint arXiv:1906.04610,2019.

[25] X. Gao, S. Jin, C.-K. Wen, and G. Y. Li, “ComNet: Combination of deeplearning and expert knowledge in OFDM receivers,” IEEE Commun.Lett., vol. 22, no. 12, pp. 2627-2630, Dec. 2018.

[26] M. Mohammadkarimi, M. Mehrabi, M. Ardakani, and Y. Jing, “Deeplearning based sphere decoding,” arXiv preprint arXiv:1807.03162,2018.

[27] E. Nachmani, E. Marciano, L. Lugosch, W. J. Gross, D. Burshtein, andY. Be’ery, “Deep learning methods for improved decoding of linearcodes,” IEEE J. Sel. Topics Signal Process., vol. 12, no. 1, pp. 119-131,Feb. 2018.

[28] Y. He, J. Zhang, C.-K. Wen, and S. Jin, “TurboNet: A model-drivenDNN decoder based on max-log-MAP algorithm for turbo code,” arXivpreprint arXiv:1905.10502, 2019.

[29] T. O’ Shea and J. Hoydis, “An introduction to deep learning for thephysical layer,” IEEE Trans. on Cogn. Commun. Netw., vol. 3, no. 4,pp. 563-575, Dec. 2017.

[30] H. Ye, L. Liang, G. Y. Li, and B.-H. F. Juang, “Deep learning basedend-to-end wireless communication systems with GAN as unknownchannel,” to appear in IEEE Trans. Wireless Commun.

[31] I. Santos and J. J. Murillo-Fuentes, “EP-based turbo detection for MIMOreceivers and large-scale systems,” arXiv preprint arXiv:1805.05065,2018.

[32] Y. Yang, J. Sun, H. Li, and Z. Xu, “ADMM-Net: A deep learning ap-proach for compressive sensing MRI,” arXiv preprint arXiv:1705.06869,2017.

[33] M. Borgerding, P. Schniter, and S. Rangan, “AMP-inspired deep net-works for sparse linear inverse problems,” IEEE Trans. Signal Process.vol 65, no. 16, pp. 4293-4308, Aug. 2017.

[34] D. Ito, S. Takabe, and T. Wadayama, “Trainable ISTA for sparse signalrecovery,” IEEE Trans. Signal. Process., vol. 67, no. 12, pp. 3113-3125,Jun. 2019.

[35] X. Chen, J. Liu, Z. Wang, and W. Yin. “Theoretical linear convergence ofunfolded ISTA and its practical weights and thresholds,” arXiv preprintarXiv:1808.10038, 2018.

[36] K. Gregor and Y. LeCun, “Learning fast approximations of sparsecoding,” in Proc. Int’l Conf. Machine Learning, 2010, pp. 399-406.

[37] K. Takeuchi, R. R. Muller, M. Vehkapera, and T. Tanaka, “On anachievable rate of large Rayleigh block-fading MIMO channels withno CSI,” IEEE Trans. Inf. Theory., vol. 59, no. 10, pp. 6517-6541, Oct.2013.

[38] J. Ma and L. Ping, “Data-aided channel estimation in large antennasystems,” IEEE Trans. Signal Process., vol. 62, no. 12, pp. 3111-3124,Jun. 2014.

[39] C.-K. Wen, C.-J. Wang, S. Jin, K.-K. Wong, and P. Ting, “Bayes-optimaljoint channel-and-data estimation for massive MIMO with low-precisionADCs,” IEEE Trans. Signal Process., vol. 64, no. 10, pp. 2541-2556,May. 2016.

[40] F. Steiner, A. Mezghani, A. Lee Swindlehurst, J. A. Nossek, W.Utschick,“Turbo-like joint data-and-channel estimation in quantized massiveMIMO systems,” in Proc. 20st Int. ITG Workshop Smart Antennas, Mar.2016, pp. 1-5.

[41] S. Talwar, “Blind Separation of Synchronous Co-Channel Digital SignalsUsing an Antenna Array—Part II: Performance Analysis,” IEEE Trans.Signal Process., vol. 45, no. 3, pp. 706-718, Mar. 1997.

[42] J. Ma and L. Ping, “Orthogonal AMP,” IEEE Access, vol. 5, no. 14, pp.2020-2033, Jan. 2017.

[43] K. Takeuchi, “Rigorous dynamics of expectation-propagation-based sig-nal recovery from unitarily invariant measurements,” arXiv preprintarXiv:1701.05284, 2017.

[44] S. L. Loyka, “Channel capacity of MIMO architecture using the ex-ponential correlation matrix,” IEEE Commun. Lett., vol. 5, no. 9, pp.369-371, Sep. 2001.

[45] Z. Xue, J. Ma, and X. Yuan, “Denoising-based turbo compressedsensing,” IEEE Access, vol. 5, pp. 7193-7204, 2017.

[46] P. Liu and I.-M. Kim, “Exact and closed-form error performanceanalysis for hard MMSE-SIC detection in MIMO systems,” IEEE Trans.Commun., vol. 59, no. 9, pp. 2463-2477, Sep. 2011.

[47] 3GPP TR 36.873. 2015. “Study on 3D channel model for LTE”.Technical report.

[48] S. Jaeckel, L. Raschkowski, K. Borner, and L. Thiele. “QuaDRiGa: A3-D multi-cell channel model with time evolution for enabling virtualfield trials,” IEEE Trans. Antennas Propag., vol. 62, no.6, 3242-3256,Mar. 2014.