low-complexity detection of high-order qam faster-than

Received May 6, 2017, accepted June 7, 2017, date of publication July 25, 2017, date of current version August 14, 2017.

Digital Object Identifier 10.1109/ACCESS.2017.2719628

Low-Complexity Detection of High-Order QAMFaster-Than-Nyquist SignalingEBRAHIM BEDEER1, (Member, IEEE), MOHAMED HOSSAM AHMED2, (Senior Member, IEEE),AND HALIM YANIKOMEROGLU1, (Fellow, IEEE)1System and Computer Engineering Department, Carleton University, Ottawa, ON K1S 5B6, Canada2Faculty of Engineering and Applied Science, Memorial University of Newfoundland, St. John’s, NL A1B 3X5, Canada

Corresponding author: Ebrahim Bedeer ([email protected])

This work was supported in part by DragonWave Inc., in part by the Mathematics of Information Technology and Complex SystemsCanada, and in part by the Natural Science and Engineering Research Council of Canada through Discovery Program.

ABSTRACT Faster-than-Nyquist (FTN) signaling is a promising non-orthogonal transmission technique toconsiderably improve the spectral efficiency. This paper presents the first attempt in the literature to estimatethe transmit data symbols of any high-order quadrature amplitude modulation (QAM) FTN signaling inpolynomial time complexity. In particular, we propose a generalized approach to model the finite alphabet ofany high-order QAMconstellation as a high degree polynomial constraint. Then, we formulate the high-orderQAM FTN signaling sequence estimation problem as a non-convex optimization problem. As an exampleof a high-order QAM, we consider 16-QAM FTN signaling and then transform the high degree polynomialconstraint, with the help of auxiliary variables, to multiple quadratic constraints. Such transformation allowsus to propose a generalized approach semidefinite relaxation (SDR)-based sequence estimation (GASDRSE)technique to efficiently provide a sub-optimal solution to the NP-hard non-convex FTN detection problem,with polynomial time complexity. For the particular case of 16-QAM FTN signaling, we additionallypropose a sequence estimation technique based on concepts from the set theory. We show that the set theorySDR-based sequence estimation (STSDRSE) technique is of lower complexity when compared with theproposed GASDRSE. Simulation results show the effectiveness of the proposed GASDRSE and STSDRSEtechniques in increasing the data rate and spectral efficiency of 16-QAM FTN signaling, without increasingthe bit-error-rate, the bandwidth, or the data symbols energy, when compared with the Nyquist signaling.

INDEX TERMS Faster-than-Nyquist (FTN), high-order QAM, intersymbol interference (ISI), Mazo limit,semidefinite relaxation (SDR), sequence estimation.

I. INTRODUCTIONHigher spectrally efficient transmission techniques, e.g.,faster-than-Nyquist (FTN) signaling, have attracted the atten-tion of industrial and academic communities to meet the everincreasing demands on data rates [1]. In FTN signaling, theT -orthogonal pulses are sent at rate 1

τT which is higher thanthe Nyquist signaling rate 1

T , where 0 < τ ≤ 1 is the timepacking parameter. Since FTN signaling violates the Nyquistlimit, inter-symbol interference (ISI) between the receivedpulses is unavoidable.

The pioneering work of Mazo in [2] showed thatFTN signaling does not affect the minimum distance ofuncoded binary sinc pulse transmission, and hence theasymptotic error probability, as long as the time packingparameter τ is above a certain limit, i.e., τ ≥ 0.802,later known as Mazo limit. Mazo limit has been extended

to root-raised cosine (rRC) pulses in [3], to frequency andtime domains simultaneously in [4], and to multiple-input-multiple-output systems in [5].

Most of the research literature focused on the detection ofbinary or low-order quadrature amplitudemodulation (QAM)FTN signaling [6]–[10]; this is mainly due to the exces-sive computational complexity involved in removing the ISIof high-order constellations FTN signaling. For instance,Bedeer et al. [6] proposed a reduced complexity spheredecoding algorithm to optimally detect binary FTN signaling.The reduction in complexity was achieved on average andnot in the worst case sense. However, extending such analgorithm to high-order QAM results in prohibitive com-putational complexity. Similarly, extending the truncatedstate Viterbi algorithm (VA) [7] and the reduced stateBahl-Cocke-Jelinek-Raviv (BCJR) algorithm [8] to

VOLUME 5, 20172169-3536 2017 IEEE. Translations and content mining are permitted for academic research only.

Personal use is also permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

14579

E. Bedeer et al.: Low-Complexity Detection of High-Order QAM FTN Signaling

high-order QAM are not straightforward and rather leadto significant computational complexity. The worst casecomputational complexities of the works in [6]–[9] are ingeneral exponential either in received block length [6] orthe ISI length [7]–[9]. Bedeer et al. [10] proposed reducedcomplexity algorithms to detect binary and 4-QAM FTNsignaling on symbol-by-symbol basis. Extending these algo-rithms to detect high-order QAM is not trivial and most likelywill lead to unsatisfactory performance. In [11], a modifiedBCJR algorithm was proposed to detect 4-pulse amplitudemodulation FTN signaling. In [12], a 9-QAM super-Nyquistsignaling scheme is proposed for optical fiber communica-tions to improve the spectral efficiency.

This paper presents the first attempt in the literature todetect FTN signaling of any high-order QAM constellation(and in particular 16-QAM constellation) in polynomial timecomplexity. More specifically, we propose a generalizedapproach to model the finite alphabet of any high-orderQAM constellation as a high degree polynomial constraint,and we formulate the FTN signaling detection problem asa non-convex optimization problem that turns out to benon-deterministic polynomial-time (NP)-hard. This meansthe FTN detection problem is at least as hard as the hardestproblems inNP [13]. As an example of a high-order QAM,weconsider 16-QAM FTN signaling and use auxiliary variablesto transform the high degree polynomial constraint to multi-ple quadratic constraint. That said, we propose a generalizedapproach semidefinite relaxation (SDR)-based sequence esti-mation (GASDRSE) technique to efficiently provide a sub-optimal solution to the NP-hard non-convex FTN signalingdetection problem, with polynomial time complexity. For theparticular case of 16-QAM FTN signaling, we additionallypropose a sequence estimation technique based on conceptsfrom the set theory. We show that the set theory SDR-based sequence estimation (STSDRSE) technique is of lowercomplexity when compared to the proposed GASDRSE.Simulation results show the effectiveness of the proposedGASDRSE and STSDRSE techniques, for moderate valuesof τ , in significantly increasing the data rate and spectrumefficiency of 16-QAM FTN signaling without increasingthe BER, the bandwidth, or the data symbols energy, whencompared to Nyquist signaling.

The remainder of this paper is organized as follows.Section II presents the system model of the high-orderQAMFTN signaling and formulates the maximum likelihoodsequence estimation (MLSE) detection problems. The pro-posed GASDRSE technique for the case of 16-QAM FTNsignaling is discussed in Section III; while in Section IV weintroduce the proposed STSDRSE technique. The computa-tional complexities of the proposed algorithms are discussedin Section V. Section VI provides the simulation results, andfinally the paper is concluded in Section VII.

Throughout the paper we use bold-faced upper case letters,e.g., X , to denote matrices, bold-faced lower case letters,e.g., x, to denote column vectors, and light-faced italicsletters, e.g., x, to denote scalars. The complex conjugate of

FIGURE 1. Block diagram of FTN signaling.

a complex number x is denoted as x∗, xi denotes the ithelement of vector x and tr(X), rank(X), and diag(X) denotethe trace, rank, and a column vector consisting of the diagonalelements of matrix X , respectively. The operator E(.) denotesthe expectation, [.]T denotes the transpose operator, I is theidentity matrix, ‖.‖p is the p-norm, and N (., .) representsthe Gaussian distribution. 1 denotes the vector of ones,<{.} and ={.} represent the real and imaginary parts ofcomplex numbers, respectively.

II. SYSTEM MODEL AND FTN SIGNALING DETECTIONPROBLEM FORMULATIONFig. 1 shows a block diagram of a communication systememploying FTN signaling. Data bits to be transmitted aregray mapped1 to data symbols through the bits-to-symbolsmapping block. Data symbols are transmitted, through thetransmit filter block, faster than the traditional Nyquist trans-mission, i.e., every τT , where 0 < τ ≤ 1 is the timepacking/acceleration parameter and T is the symbol duration.A possible receiver structure is shown in Fig. 1, where thereceived signal is passed through a filter matched to thetransmit filter followed by a sampler and an optional discrete-time filter. Since the transmission rate of the transmit pulsescarrying the data symbols intentionally violate the Nyquistcriterion, ISI occurs between the received samples. Accord-ingly, sequence estimation techniques are needed to removethe ISI and to estimate the transmitted data symbols. Theestimated data symbols are finally gray demapped to theestimated received bits.

The received FTN signal in case of additive white Gaussiannoise (AWGN) channel is written as

y(t) =√τEs

∑N

n=1ang(t − nτT )+ q(t), (1)

whereN is the total number of transmit data symbols, an, n =1, . . . ,N , is the independent and identically distributed datasymbols drawn from any high-order QAM constellation, Es isthe data symbol energy, g(t) =

∫p(x)p(x−t)dx, where p(t) is

a real-valued and unit-energy pulse, i.e.,∫∞

−∞|p(t)|2dt = 1,

q(t) =∫n(x)p(x − t)dx with n(t) is the additive white

Gaussian noise (AWGN) with zero mean and variance σ 2,and 1/(τT ) is the signaling rate.

1 In this paper, we focus on 16-QAM, and hence, graymapping is possible.Optimization of the bits-to-symbol mapping is an interesting topic itselfwhich is beyond the scope of this paper.

14580 VOLUME 5, 2017


Assuming perfect time synchronization between the trans-mitter and the receiver, the received high-order QAM FTNsignal y(t) is sampled every τT and the kth received sampleis expressed as

yk = y(kτT ),

=

√τEs

∑N

n=1ang(kτT − nτT )+ q(kτT ),

=

√τEsakg(0)︸︷︷︸

desired symbol

+

√τEs

∑N

n=1,n 6=kang((k − n)τT )︸︷︷︸

ISI+ q(kτT ). (2)

As can be seen, the kth received data symbol depends onthe kth transmit data symbol, as well as, ISI from adjacentsymbols.

A. FORMULATION OF THE HIGH-ORDER QAM FTNSIGNALING DETECTION PROBLEMFOR GENERAL PULSE SHAPEThe received FTN signal after the matched filter and samplercan be written in a vector form as

yc = Ga+ qc, (3)

where yc is the N × 1 received samples vector that containscolored noise samples, G is the N × N intersymbol interfer-ence (ISI) matrix,2 a is the N × 1 transmitted data symbolsvector, and qc ∼ N (0, σ 2G) is the N × 1 Gaussian noisesamples with zero-mean and covariance matrix σ 2G [6].

To avoid handling complex-valued variables, we use thefollowing equivalent real-valued model of (3) as follows3

yc,2N×1 = G2N×2Na2N×1 + qc,2N×1,[<{yc}={yc}

]=

[<{G} −={G}={G} <{G}

] [<{a}={a}

]+

[<{qc}={qc}

]. (4)

Assuming that G is invertible, one can rewrite (4) as

G−1yc = a+ G−1qc,

z = a+ η, (5)

where z = G−1yc and η = G−1qc. The high-order QAMFTNsignaling sequence estimation problem can be interpreted asfollows. Given the received samples z (or yc), we want to findan estimated data symbol vector a, to the transmit data symbolvector a, such that the probability of error is minimized.In other words, the FTN detection problem can be seen asa maximization of the probability that the data symbol vectora is sent given the received samples z (or yc). With the help ofBayes rule, such maximization problem can be re-expressedequivalently as

a = argmaxa∈D

P (z|a) . (6)

2The ISI matrix matrixG is a Toeplitz matrix, and hence, some errors mayoccur at the boundaries between data blocks. Ideas of overlap and reject canbe employed to mitigate such errors.

3One can use the structure in (4) (i.e., the fact that the real and imaginaryparts of the received signal and noise are independent and the fact that theimaginary part of the ISI matrix is zero for the case of AWGN channel) toimplement the proposed algorithms in parallel, i.e., by separately processingthe real and imaginary parts of the received signal.

The probability P (z|a) is usually known as the likelihoodprobability, and hence, the problem to estimate the receivedsymbol vector a is known as the maximum likelihoodsequence estimation (MLSE) problem.

Following (5), for a given set of data symbols a, thereceived samples z can be seen as Gaussian random variableswith a mean a and covariance matrix 1

2σ2G−1, i.e., η ∈

N (0, 12σ2G−1) (the proof is provided in Appendix). That

said, the likelihood probability in (6) to detect the high-orderQAM FTN signaling is expressed as [14]

P(z|a) =(

12πσ 2

)N/2 1√det[G−1]

× exp(−

1σ 2 (z− a)

TG(z− a)). (7)

One can see from (7) that maximizing the likelihood functionis equivalent to maximizing the exponent of the exponentialfunction (or minimize its negative value). Accordingly, theMLSE problem to detect high-order QAM FTN signaling iswritten as

OPMLSEc : a = argmina∈D

(z− a)TG(z− a). (8)

B. FORMULATION OF THE HIGH-ORDER QAM FTNSIGNALING DETECTION PROBLEMFOR rRC PULSE SHAPEEquation (5) assumes the ISI matrixG is invertible. However,this does not necessarily hold for common pulse shapes suchas rRC pulses operating at very small values of τ [7]. In suchcases, designing a causal and stable whitening matched filteris necessary, and we follow a designing approach similar tothe one in [7]. That said, (2) can be re-written as

yc =√τEsa ? g+ qc, (9)

where ? denotes the convolution operator. The whiteningfilter decorrelates the noise qc and can be constructed fromg by spectral factorization of its all-zero z-transform G(z)into V (z)V (1/z∗) [15]. After whitening the received vectoryc by 1/V (1/z∗), the whitened received vector can be repre-sented as

yw =√τEsa ? v+ qw, (10)

where qw is white Gaussian noise with zero mean andvariance σ 2 and v represents the causal ISI such that v[n] ?v[−n] = g. It is not possible to design an exact whiteningfilter, i.e. to have V (z) with all zeros strictly outside the unitcircle, especially for very small values of τ . That said, wedesign approximate whitening filter as follows [7]. We findG(z) with quartets of zeros on the unit circle (zeros hasto occur in quartets as V (z) and V (1/z∗) each requires aconjugate pair). Then, we split the quartet of zeros such thatone conjugate pair is slightly inside the unit circle and oneis outside. Our aim is to have the spectrum of approximationwhitening filter 1/V (1/z∗) and channel model V (z) as close

VOLUME 5, 2017 14581


as possible to the the spectrum of g. Equation (10) can bewritten in a vector form as

yw = Va+ qw, (11)

where V is the ISI matrix constructed from the vector v.Similar to the previous discussion, we use the followingequivalent real-valued model of (11) as follows

yw,2N×1 = V2N×2N a2N×1 + qw,2N×1,[<{yw}={yw}

]=

[<{V} −={V}={V} <{V}

] [<{a}={a}

]+

[<{qw}={qw}

]. (12)

Similar to the previous discussion, we can formulate the high-order QAM FTN signaling detection problem as

OPMLSEw : a = argmina∈D‖yw − Va‖

22. (13)

C. COMMENTS ON COMPLEXITY OF OPTIMALDETECTION OF HIGH-ORDER QAM FTNSIGNALING DETECTION PROBLEMSThe OPMLSEc and OPMLSEw problems can be solved by abrute force search with a complexity in the order of M2N ,where M is the modulation order of the transmit symbols ofthe real-valued model in (4) or (12). However, such exten-sive computational complexity hinders the MLSE practicalimplementations. Other techniques such as BCJR [8], [9],Viterbi algorithm [7], and sphere decoding [6] (that approx-imate the optimal detection of the OPMLSEc and OPMLSEwproblems) require an exponential (in ISI length at best) worstcase computational complexity. In the following sections,we propose reduced-complexity SDR-based sequence esti-mation techniques, namely, GASDRSE and STSDRSE tech-niques, to provide a suboptimal solution to theOPMLSEc andOPMLSE,w problems, with polynomial time complexity.

III. PROPOSED GASDRSE TECHNIQUEIn this section, we propose a generalized approach reduced-complexity SDR-based sequence estimation technique thatprovides a sub-optimal solution for the detection of any high-QAM FTN signaling.

A. FORMULATION OF QUADRATIC CONSTRAINTSIn order to transform the OPMLSEc and OPMLSE,w in(8) and (13), respectively, to an SDR problem, wemake use ofthe following observation. Any finite alphabet constellationcan be replaced by a high degree polynomial constraint [16],e.g., if x ∈ {x1, x2, . . . , xM }, then (x − x1)(x − x2) . . .(x−xM ) = 0. Next, by introducing appropriate auxiliary vari-ables, the high degree polynomial constraint can be replacedby multiple quadratic constraints.

In our case of interest, i.e., 16-QAM, the setD inOPMLSEcand OPMLSEw includes the constellation points whose realand imaginary parts belong to the set {±1,±3}, and hence,the high degree polynomial constraint can be written as

(an − 1)(an + 1)(an − 3)(an + 3) = 0, n = 1, . . . , 2N .

(14)

By introducing the auxiliary variables bn = a2n, n =1, . . . , 2N , the constraints in (14) can be further written as

a2n − bn = 0, n = 1, . . . , 2N , (15)

b2n − 10bn + 9 = 0, n = 1, . . . , 2N . (16)

To facilitate formulating the SDR sequence estimation prob-lem, we define the following 4N × 1 column vector

ωT= [aTbT1], (17)

and we find the rank-one positive semidefinite matrix � =ωωT

(of dimension (4N + 1)× (4N + 1)

)as

� =

aaT abT abaT bbT baT bT 1

. (18)

The constraints in (15) and (16) can be re-expressed in termsof �, as

diag{�1,1} −�2,3 = 0, (19)

diag{�2,2} − 10�2,3 + 9.1 = 0, (20)

�3,3 = 1, (21)

where �i,j, i, j = 1, 2, and 3, are the (i, j)th sub-blocks of� of appropriate sizes, e.g., �1,1 = aaT (of size 2N × 2N ),�2,3 = b (of size 2N × 1), �2,2 = bbT (of size 2N × 2N ),�1,3 = a, and �3,3 = 1.

B. FORMULATION OF OBJECTIVE FUNCTION1) GENERAL PULSE SHAPESince the matrix G is symmetric, the objective function ofOPMLSEc in (8) can be rewritten in terms of � as [16], [17]

ωT8cω = tr{8cωωT} = tr{8c�}, (22)

where

8c,(4N+1)×(4N+1) =

G 0 −yc0 0 0−yTc 0 zTGz

. (23)

2) rRC PULSE SHAPEFor the case of rRC and given that the matrix V is symmetric,the objective function of OPMLSEw in (13) can be rewrittenin terms of � as [16], [17]

tr{8w�}, (24)

where 8w is given by

8w,(4N+1)×(4N+1) =

VTV 0 −VTyw0 0 0−yTwV 0 yTwyw

. (25)

14582 VOLUME 5, 2017


C. SDR OPTIMIZATION PROBLEM AND PROPOSEDGASDRSE ALGORITHMFinally, the optimization problems OPMLSEc and OPMLSEwin (8) and (13), respectively, to detect 16-QAMFTN signalingis re-expressed as

min�

tr{8�}

subject to (19), (20), and (21),

� � 0, (26)

rank{�} = 1, (27)

where 8 ∈ {8c,8w} and the constraints in (26) and (27) areused to reflect that� is a positive semidefinite and a rank-onematrix, respectively. Such an optimization problem is non-convex due to the rank-one constraint in (27). By relaxing thisconstraint, we obtain an upper bound on its optimal solutionas

OPGASDRSE : min�

tr{8�}

subject to (19), (20), (21), and (26).

Please note that the objective function of OPGASDRSE islinear in � subject to affine equalities and a linear matrixinequality.

The optimization problem OPGASDRSE can be efficientlysolved to any arbitrary accuracy using well-developed reli-able numerical solvers [18]. By solving OPGASDRSE, weobtain the optimal solution �op which is the global optimalsolution to the convex problem OPGASDRSE. In order toobtain a feasible solution to the original non-convex FTNdetection problem, we use Gaussian randomization [17].The intuition behind Gaussian randomization is to solve astochastic version of OPGASDRSE, where the sub-optimalsolution w is considered as the random vector—generatedfrom a multivariate Gaussian random vector with zero meanand covariance matrix �op—that maximizes the objectivein OPGASDRSE. The Gaussian randomization procedure canbe explained as follows. We generate ξ `, ` = 1, . . . ,L,where L is the total number of randomization iterations,as ξ ` ∼ (0,�op), then we find ω` = quantize{ξ `},where quantize(x) rounds x to the nearest element in the set{±1,±3}. Then, we select the optimal value of `, i.e., òp,that minimizes the objective function in OPGASDRSE as

òp = arg min`=1,...,L

tr{8ω`ωT` }, (28)

where one can easily find the sub-optimal vector ω as

ω = ωòp . (29)

Finally, the estimated data symbols vector a can be foundfrom the first 2N elements in ω.

The proposed GASDRSE technique is formally expressedin Algorithm 1 as follows

Algorithm 1: Proposed GASDRSE Technique1) Input: Pulse shape p(t) (ISI matrix G), received

samples yc or yw, and number of randomizationiterations L.

2) Solve OPGASDRSE to find �op.3) Generate random variable ξ ` as ξ ` ∼ (0,�op), ` =

1, . . . ,L.4) Find ω` = quantize{ξ `}.5) Find òp = arg min

`=1,...,Ltr{8ω`ω

T` }.

6) Set ω = ωòp .7) Output: The estimated data symbols a which are the

first 2N elements of ω.

IV. PROPOSED STSDRSE TECHNIQUEIn this section, we propose a STSDRSE technique to detect16-QAM FTN signaling with reduced complexity. The pro-posed sequence estimation technique is more computation-ally efficient when compared to the proposed GASDRSE.As can be seen inOPGASDRSE, the dimension of the decisionmatrix � is (4N + 1) × (4N + 1), where N is the length ofthe complex data symbol vector. In this section, we reducethe dimension of the decision matrix, and hence, decrease thecomputational complexity.

A. FORMULATION OF QUADRATIC CONSTRAINTSWe replace the generic high degree polynomial constraint thatmodels any high-order QAM, using concepts from the set the-ory, with a set of quadratic constraints suited to our particularcase of interest in this paper, i.e., 16-QAM. Towards this goal,let us define the following sets

D1 := (−∞,−3),

D2 := (−3,−1),

D3 := (−1, 1),

D4 := (1, 3),

D5 := (3,∞). (30)

Then, the constraints in OPMLSEc and OPMLSEw can beexpressed as [16]

{±1,±3} = D1 ∪D2 ∪D3 ∪D4 ∪D5. (31)

With the help of De Morgan’s associative laws [19], theconstraints can be further rewritten as

{±1,±3} = D1 ∩D2 ∩D3 ∩D4 ∩D5,

= ((D1 ∩D5) ∩D3) ∩D2 ∩D4

= ([−3,−1] ∪ [1, 3]) ∩D2 ∩D4. (32)

Accordingly, [−3,−1] ∪ [1, 3], D2, and D2 can be respec-tively written as

1 ≤ a2n ≤ 9, (33)

(an + 1)(an + 3) ≥ 0, (34)

(an − 1)(an − 3) ≥ 0. (35)

VOLUME 5, 2017 14583


To facilitate formulating the new SDR problem, let us refor-mulate OPMLSEc and OPMLSEw in a higher dimension bydefining

ψT= [aT1],

9(2N+1)×(2N+1) = ψψT=

[aaT aaT 1

]. (36)

The constraints in (33) - (35) can be re-written in terms of 9as

1 � diag{91,1} � 9.1, (37)

diag{91,1} + 491,2 + 3.1 � 0, (38)

diag{91,1} − 491,2 + 3.1 � 0, (39)

92,2 = 1, (40)

where 9 i,j, i, j = 1, 2 are the (i, j)th sub-blocks of 9 ofappropriate sizes, e.g., 91,1 = aaT (of size 2N × 2N ),91,2 = a, 92,1 = aT, and 92,2 = 1. The constraints in(37), (38), and (39) are equivalent to (33), (34), and (35),respectively.

B. FORMULATION OF OBJECTIVE FUNCTION1) GENERAL PULSE SHAPESince the matrix G is symmetric, the objective function ofOPMLSEc can be expressed in terms of 9 as [16], [17]

ψT2cψ = tr{2cψψT} = tr{2c9}, (41)

where

2c,(2N+1)×(2N+1) =

[G −yc−yTc zTGz

]. (42)

2) rRC PULSE SHAPEFor the case of rRC and for symmetric ISI matrix V , theobjective function of OPMLSEw in (13) can be rewritten interms of 9 as [16], [17]

tr{2w9}, (43)

where2w is given by

2w =

[VTV −VTyw−yTwV yTwyw

]. (44)

C. SDR OPTIMIZATION PROBLEM AND PROPOSEDSTSDRSE ALGORITHMFinally,OPMLSEc andOPMLSEw can be casted as the follow-ing optimization problem

min9

tr{29}

subject to (37), (38), (39), and (40)

9 � 0, (45)

rank{9} = 1, (46)

where 2 ∈ {2c,2w} and the constraints (45) and (46)reflect that9 is a positive semidefinite and a rank-onematrix,

respectively. It is clear that the above optimization problem isnot convex due to the rank-one constraint in (46). By relaxingthis constraint, we obtain an upper bound on its optimalsolution as

OPSTSDRSE : min9

tr{29}

subject to (37), (38), (39), (40), and (45),

which can be solved efficiently [18] with polynomial timecomplexity.

Similar to the previous discussion of the proposedGASDRSE, the optimal solution9op ofOPSTSDRSE will notnecessarily be a feasible solution of the original non-convexoptimization problem; hence, we use Gaussian randomiza-tion to obtain an efficient feasible solution. We generateζ `, ` = 1, . . . ,L, as ζ ` ∼ (0,9op), then we quantized eachrandom variable as ψ` = quantize{ζ `}. Finally, we find thesolution as

òp = arg min`=1,...,L

tr{2ψ`ψT` }, (47)

ψ = ψòp . (48)

Finally, the estimated 16-QAMconstellation vector a is foundfrom the first 2N elements in the vector ψ .The proposed STSDRSE technique is formally expressed

in Algorithm 2 as follows

Algorithm 2: Proposed STSDRSE Technique1) Input: Pulse shape g(t) (ISI matrix G), received

samples yc and yw, and number of randomizationiterations L.

2) Solve OPSTSDRSE to find 9op.3) Generate random variable ζ ` as ζ ` ∼ (0,9op),` = 1, . . . ,L.

4) Find ψ` = quantize{ζ `}.5) Find òp = arg min

`=1,...,Ltr{2ψ`ψ

T` }.

6) Set ψ = ψòp .7) Output: Estimated data symbols a are the first

2N elements of ψ .

V. COMMENTS ON THE PROPOSED GASDRSE ANDSTSDRSE TECHNIQUESA. COMPLEXITY DISCUSSIONThe worst case computational complexity of solving therelaxedOPGASDRSE isO((4N + 1)3.5 [17]. Additionally, thecomplexity of generating and evaluating the objective func-tion corresponding to the L random samples of the Gaussianrandomization is O((4N + 1)2L [17]. Hence, the total com-putational complexity of the proposed GASDRSE techniqueis O((4N + 1)3.5 + (4N + 1)2L), which is polynomial in thereceived block length.

As mentioned earlier, the proposed GASDRSE techniquecan be extended to any high-order QAM FTN signaling, withpolynomial time computational complexity. For instance, for

14584 VOLUME 5, 2017


the case of 64-QAM FTN signaling, we need to use anadditional auxiliary variable cn = b2n, n = 1, . . . , 2N ,in order to formulate the detection problem. In this case,the complexity of solving the relaxed OPGASDRSE isO((6N + 1)3.5, and the complexity of generating and evalu-ating the objective function corresponding to the L randomsamples of the Gaussian randomization is O((6N + 1)2L.Hence, the overall complexity of the proposed GASDRSEto detect 64-QAM FTN signaling will be in the order ofO((6N + 1)3.5 + (6N + 1)2L), which is still polynomial.The computational complexity reduction of the pro-

posed STSDRSE technique, when compared to the proposedGASDRSE, comes from the fact that the dimension of theunknown matrix 9 is (2N + 1)× (2N + 1) which is smallerthan its counterpart � of (4N + 1) × (4N + 1) of the pro-posed GASDRSE. Accordingly, the computational complex-ity of the proposed STSDRSE technique is O((2N + 1)3.5 +(2N + 1)2L).

B. SOFT-OUTPUTSThe proposed GASDRSE and STSDRSE schemes producehard-output, and in their current forms they may not be suit-able to be used with channel coding. Fortunately, Gaussianrandomization can be effectively used to reduce the complex-ity of evaluating the log-likelihood ratio (LLR) as follows.Inspired by the idea of the list sphere decoding in [20], theGaussian randomization step can store a list of vectors thatachieve the lowest values of the objective function (instead ofstoring only one vector that achieves the minimum objectivevalue). This candidate list can then be used to approximatethe LLR calculations. In other words, instead of searching thewhole search space, Gaussian randomization can efficientlyproduce candidate vectors that contribute more towards thecalculation of the LLR equation. However, there is a possi-bility that some bit positions in the candidate list are empty.In this case, we can extend the candidate list to include all bitvectors with hamming distance of 1 of the bit vectors in theoriginal candidate list as in [21].

VI. SIMULATION RESULTSIn this section, we evaluate the performance of the proposedGASDRSE and STSDRSE techniques in detecting 16-QAMFTN signaling. We employ an rRC filter4 with roll-off factorsβ = 0.3 and 0.5. The length of the ISI is set to 23 symbolsand the number of randomization iterations L is set to 1000.

A. PERFORMANCE OF THE PROPOSED GASDRSE ANDSTSDRSE TECHNIQUESFig. 2 depicts the BER of 16-QAM FTN signaling as afunction of Eb/N0 for both the proposed GASDRSE andSTSDRSE techniques for β = 0.3 and τ = 0.7 and 0.8.As can be seen, in general both the proposed GASDRSE

4 It is worthy to mention that the proposed GASDRSE and STSDRSEtechniques are independent of the pulse shape, and they can be used withother pulse shapes, e.g., Gaussian pulses.

FIGURE 2. BER performance of 16-QAM FTN signaling as a function ofEb/N0 for the proposed GASDRSE and STSDRSE techniques for β = 0.3and τ = 0.7 and 0.8.

FIGURE 3. BER performance of 16-QAM FTN signaling as a function ofEb/N0 for the proposed GASDRSE and STSDRSE techniques for β = 0.5and τ = 0.7 and 0.8.

and STSDRSE techniques almost achieve the BER perfor-mance of Nyquist signaling at τ = 0.8. This means that theproposed GASDRSE and STSDRSE techniques can achieve10.8 − 1 = 25% increase in the data rate, when compared toNyquist signaling at β = 0.3, without increasing the BER,the bandwidth, or the data symbols energy. Additionally, onecan notice that as the value of τ increases from 0.8 and 0.7(i.e., the ISI between adjacent symbols increases), the BERperformance deteriorates as expected. This clearly shows thatour proposed algorithms provide sub-optimal solutions to the16-QAM FTN signaling detection problem, as they cannotachieve the orthogonal error performance with τ at the Mazolimit, which for β = 0.3 lies at τ = 0.7 [3].Fig. 3 plots the BER of 16-QAM FTN signaling as a

function of Eb/N0 for both the proposed GASDRSE andSTSDRSE techniques for β = 0.5 and τ = 0.7 and 0.8.

VOLUME 5, 2017 14585


FIGURE 4. Spectral efficiency of 16-QAM FTN signaling as a function of βusing the proposed STSDRSE at BER = 10−4.

One can see from Fig. 3 that up to 24.72% increase in the datarate of 16-QAM FTN signaling can be achieved for τ = 0.8and β = 0.5, without increasing the BER, the bandwidth, orthe data symbols energy, when compared to 16-QAMNyquistsignaling. Additionally, up to 42.86% increase in the data ratecan be achieved for τ = 0.7 and β = 0.5 at BER = 10−4 atthe expense of 0.7 dB increase in the SNR.Fig. 4 plots the spectral efficiency of 16-QAM Nyquist

(i.e., no ISI and τ = 1) and the proposed STSDRSE5 of16-QAM FTN signaling as a function of the roll-off factor βat the same SNR and BER = 10−4. In order to have a faircomparison, the value of τ of the 16-QAM FTN signalingis selected to be the smallest value such that the proposedSTSDRSE achieves the same BER = 10−4 of 16-QAMNyquist signaling at the same SNR. As can be seen, thespectral efficiency of 16-QAM FTN signaling is higher thanits counterpart of Nyquist signaling for all values of β. Forinstance, at β = 0 the proposed STSDRSE improves thespectral efficiency by 7.53% for the same BER and SNRvalues, when compared to Nyquist signaling. Further, theimprovement of the spectral efficiency increaseswith increas-ing the value of the roll-off factor β. Additionally, resultsrevealed that the proposed STSDRSE can achieve spectralefficiency higher than the maximum spectral efficiency of16-QAM Nyquist signaling (4 bits/s/Hz achieved at β = 0)for the range of β ∈ [0, 0.15].

B. PERFORMANCE COMPARISON WITHQUASI-OPTIMAL DETECTORSIn Fig. 5, we compare the performance of 16-QAM FTNsignaling detection using the proposed STSDRSE techniqueand QPSK FTN signaling detection using the scheme in [9],for β = 0.3 and same spectral efficiencies of 4.3956 and3.8462 bits/s/Hz. As can be seen, the proposed STSDRSEprovides a trade-off between complexity and performance.

5Spectral efficiency results of the proposed GASDRSE are not includedas it is similar to that of the STSDRSE as shown in Figs. 2 and 3.

FIGURE 5. Performance comparison between 16-QAM FTN detectionusing the proposed STSDRSE technique and QPSK FTN signaling detectionusing the scheme in [9], for β = 0.3 and same spectral efficiencies of4.3956 and 3.8462 bits/s/Hz.

FIGURE 6. Performance comparison between 16-QAM FTN detectionusing the proposed GASDRSE and STSDRSE techniques and QPSK FTNsignaling optimal detection using the sphere decoding in [6],for β = 0.5 and same spectral efficiency of 3.33 bits/s/Hz.

For instance, the scheme in [9] (that operates at τ = 0.35 andspectral efficiency of 4.3956 bits/s/Hz) achieves better perfor-mance than the proposed STSDESE technique (that operatesat τ = 0.7 and spectral efficiency of 4.3956 bits/s/Hz), atthe expense of higher computational complexity. In addition,the proposed STSDRSE technique (that operates at τ = 0.8and spectral efficiency of 3.8462 bits/s/Hz) achieves the sameperformance as the scheme in [9] (that operates at τ = 0.4and spectral efficiency of 3.8462 bits/s/Hz), with reducedcomputational complexity.

Fig. 6 compares the performance of 16-QAM FTN signal-ing detection using the proposed GASDRSE and STSDRSEtechniques and QPSK FTN signaling optimal detection usingthe SDSE technique in [6], for β = 0.5 and same spectralefficiency of 3.33 bits/s/Hz. Although both 16-QAM andQPSK FTN signaling have the same BER performance for

14586 VOLUME 5, 2017


β = 0.5 and SE of 3.33 bits/s/Hz, but as mentioned earlier thecomputational complexity of the proposed GASDRSE andSTSDRSE is much lower than its counterpart of the SDSEtechnique required to detect QPSK FTN signaling.

VII. CONCLUSIONFTN signaling is a promising transmission technique to sig-nificantly improve the spectral efficiency, when compared toNyquist signaling. In this paper, we presented the first attemptin the literature to detect any high-order QAMFTN signaling,in polynomial time complexity. More specifically, we intro-duced a general approach to model the finite alphabet of anyhigh-order QAM constellation as a high degree polynomialconstraint, and we formulated the high-order QAM FTNsignaling detection problem as a non-convex optimizationproblem that turns out to be NP-hard to solve. We showedthat for 16-QAM, the high degree polynomial can be replacedwith multiple quadratic constraints, with the help of auxiliaryvariables. This enabled us to propose a GASDRSE techniquethat efficiently provides a sub-optimal solution to theNP-hardnon-convex FTN detection problem with polynomial timecomplexity. For the 16-QAM FTN signaling, we additionallyproposed a STSDRSE technique that is of lower complex-ity when compared to the GASDRSE technique. Simulationresults showed that up to 25% increase in the data rate canbe achieved at β = 0.3 without increasing the BER, thebandwidth, or the data symbols energy, when compared toNyquist signaling; or up to 42.86% increase in the data ratecan be achieved at β = 0.5 with 0.7 dB penalty in the SNR.Additionally, results revealed that the proposed STSDRSEcan achieve spectral efficiency higher than the maximumspectral efficiency of 16-QAM Nyquist signaling (4 bit/s/Hzachieved at β = 0) for the range of β ∈ [0, 0.15] for the sameBER and SNR.

APPENDIXCOVARIANCE OF NOISE VECTOR ηThe covariance matrix of the noise vector η can be calculatedas

E{ηηT} = E{G−1qcqTc (G−1)T}

= G−1E{qcqTc }(G−1)T. (49)

Given that the real noise vector is qc = [<{q}T,={q}T]T, thevalue of E{qcqTc } can be calculated as

E{qcqTc } =[E{<{qc}<{qc}T} E{<{qc}={qc}T}E{={qc}<{qc}T} E{={qc}={qc}T}

]. (50)

The values of E{<{qc}<{qc}T} and E{<{qc}={qc}T} can becalculated with the help of E{qcqHc } = σ 2G as follows.

E{qcqHc } = E{(<{qc} + j={qc})(<{qc}T − j={qc}T)}= E{<{qc}<{qc}T} + E{={qc}={qc}T}+ jE{={qc}<{qc}T} − jE{<{qc}={qc}T}.

= σ 2G (51)

We assume that E{={qc}<{qc}T} = E{<{qc}={qc}T} = 0(this assumption can be further verified from the fact thatp(t) is real-valued, and hence, ={G} = 0 for the consideredAWGN case; accordingly, σ 2G does not have imaginaryparts). Accordingly, (51) is re-written as

E{<{qc}<{qc}T} + E{={qc}={qc}T} = σ 2G. (52)

We also assume that the noise is a proper random variable,i.e. its real and imaginary parts have equal variance [22]; thatsaid we can conclude that

E{<{qc}<{qc}T} = E{={qc}={qc}T} =12σ 2G. (53)

By substituting (53) into (50), we get

E{qcqTc } =

12σ 2G 0

012σ 2G

,=

12σ 2G, (54)

which is reached by knowing that <{G} = G and ={G} = 0for the considered AWGN case. Finally, from (49) and (54),one can easily find that E{ηηT} = 1

2σ2G−1.

REFERENCES[1] J. B. Anderson, F. Rusek, and V. Öwall, ‘‘Faster-than-Nyquist signaling,’’

Proc. IEEE, vol. 101, no. 8, pp. 1817–1830, Aug. 2013.[2] J. E. Mazo, ‘‘Faster-than-Nyquist signaling,’’ Bell Syst. Tech. J., vol. 54,

no. 8, pp. 1451–1462, 1975.[3] A. D. Liveris and C. N. Georghiades, ‘‘Exploiting faster-than-Nyquist

signaling,’’ IEEE Trans. Commun., vol. 51, no. 9, pp. 1502–1511,Sep. 2003.

[4] F. Rusek and J. B. Anderson, ‘‘The two dimensional Mazo limit,’’ in Proc.Int. Symp. Inf. Theory (ISIT), Sep. 2005, pp. 970–974.

[5] F. Rusek, ‘‘On the existence of the Mazo-limit on MIMO chan-nels,’’ IEEE Trans. Wireless Commun., vol. 8, no. 3, pp. 1118–1121,Mar. 2009.

[6] E. Bedeer, H. Yanikomeroglu, and M. H. Ahmed, ‘‘Reduced complexityoptimal detection of binary faster-than-Nyquist signaling,’’ in Proc. IEEEInt. Conf. Commun. (ICC), Paris, France, 2017, pp. 1–6.

[7] A. Prlja, J. B. Anderson, and F. Rusek, ‘‘Receivers for faster-than-Nyquistsignaling with and without turbo equalization,’’ in Proc. IEEE Int. Symp.Inf. Theory, Jul. 2008, pp. 464–468.

[8] J. B. Anderson, A. Prlja, and F. Rusek, ‘‘New reduced state space BCJRalgorithms for the ISI channel,’’ inProc. IEEE Int. Symp. Inf. Theory (ISIT),Jun. 2009, pp. 889–893.

[9] J. B. Anderson and A. Prlja, ‘‘Turbo equalization and an M-BCJR algo-rithm for strongly narrowband intersymbol interference,’’ in Proc. IEEEInt. Symp. Inf. Theory Appl. (ISIT), Jun. 2010, pp. 261–266.

[10] E. Bedeer, M. H. Ahmed, and H. Yanikomeroglu, ‘‘A very lowcomplexity successive symbol-by-symbol sequence estimator forfaster-than-Nyquist signaling,’’ IEEE Access, vol. 5, pp. 7414–7422,Mar. 2017.

[11] J. B. Anderson, ‘‘A review of faster than Nyquist signaling with anextension to four-level modulation,’’ in Proc. 9th Int. Symp. Turbo CodesIterative Inf. Process. (ISTC), Sep. 2016, pp. 6–10.

[12] J. Zhang, J. Yu, and N. Chi, ‘‘Generation and transmission of 512-Gb/squad-carrier digital super-Nyquist spectral shaped signal,’’ Opt. Exp.,vol. 21, no. 25, pp. 31212–31217, Dec. 2013.

[13] S. Arora and B. Barak, Computational Complexity: A Modern Approach.Cambridge, U.K.: Cambridge Univ. Press, 2009.

[14] J. Proakis, Digital Communications. New York, NY, USA: McGraw-Hill,2001.

[15] A. V. Oppenheim and R. W. Schafer, Discrete-Time Signal Processing.Upper Saddle River, NJ, USA: Pearson Ed., 2010.

VOLUME 5, 2017 14587


[16] W.-K. Ma, Semidefinite Relaxation and its Applications in Signal Process-ing and Communications. Xi’an, China: MIIS Tutorial, Jul. 2012.

[17] Z.-Q. Luo, W.-K. Ma, A. M.-C. So, Y. Ye, and S. Zhang, ‘‘Semidefiniterelaxation of quadratic optimization problems,’’ IEEE Signal Process.Mag., vol. 27, no. 3, pp. 20–34, May 2010.

[18] M. Grant and S. Boyd. (2010). CVX: MATLAB Software for DisciplinedConvex Programming. [Online]. Available: https://cvxr.com/cvx

[19] K. H. Rosen and K. Krithivasan, Discrete Mathematics and Its Applica-tions. New York, NY, USA: McGraw-Hill, 1995.

[20] B. M. Hochwald and S. ten Brink, ‘‘Achieving near-capacity ona multiple-antenna channel,’’ IEEE Trans. Commun., vol. 51, no. 3,pp. 389–399, Mar. 2003.

[21] D. J. Love, S. Hosur, A. Batra, and R. W. Heath, ‘‘Space-time chasedecoding,’’ IEEE Trans. Wireless Commun., vol. 4, no. 5, pp. 2035–2039,Sep. 2005.

[22] F. D. Neeser and J. L. Massey, ‘‘Proper complex random processes withapplications to information theory,’’ IEEE Trans. Inf. Theory, vol. 39, no. 4,pp. 1293–1302, Jul. 1993.

EBRAHIM BEDEER (S’10–M’14) received theB.Sc. (Hons.) and M.Sc. degrees from TantaUniversity, Tanta, Egypt, and the Ph.D. degreefrom the Memorial University of Newfoundland,St. Johns, NL, Canada, all in Electrical Engineer-ing. He was a Post-Doctoral Fellow with The Uni-versity of British Columbia, BC, Canada. He iscurrently a Post-Doctoral Fellow with CarletonUniversity, Ottawa, ON, Canada. His currentresearch interests include wireless communica-

tions and signal processing, with a current focus of resource managementin wireless networks, and applications of optimization is signal processing.

Dr. Bedeer is an Editor of the IEEE COMMUNICATIONS LETTERS. He hasserved on the Technical Program Committees of numerous major inter-national communication conferences, such as the IEEE GLOBECOM, theIEEE ICC, the IEEE VTC, and CROWNCOM. He received numerousawards, including the Exemplary Reviewer of the IEEE COMMUNICATIONS

LETTERS and the IEEE WIRELESS COMMUNICATIONS LETTERS.

MOHAMED HOSSAM AHMED (SM’07)received the Ph.D. degree in electrical engineeringfrom Carleton University, Ottawa, in 2001. From2001 to 2003, he was with Carleton University asa Senior Research Associate. In 2003, he joinedthe Faculty of Engineering and Applied Science,Memorial University of Newfoundland, where heis currently a Full Professor. He has authoredover 135 papers in international journals andconferences. His research interests include radio

resource management in wireless networks, multi-hop relaying, cooperativecommunication, vehicular ad-hoc networks, cognitive radio networks, andwireless sensor networks. His research is sponsored by NSERC, CFI,QNRF, Bell/Aliant, and other governmental and industrial agencies. Hereceived the Ontario Graduate Scholarship for Science and Technology in1997, the Ontario Graduate Scholarship in 1998, 1999, and 2000, and theCommunication and Information Technology Ontario Graduate Award in2000. He served as a Co-Chair of the Signal Processing Track in ISSPIT’14,the Transmission Technologies Track in VTC’10-Fall, and the MultimediaAnd Signal Processing Symposium in CCECE’09. He serves as an Editor ofthe IEEE COMMUNICATION SURVEYS AND TUTORIALS and as an Associate Editorof theWiley International Journal of Communication Systems and theWileyCommunication and Mobile Computing. He served as a Guest Editor ofa special issue on Fairness of Radio Resource Allocation, the PEURASIPJWCN in 2009, and a special issue on Radio Resource Management inWireless Internet, theWileyWireless andMobile Computing Journal in 2003.He is a registered Professional Engineer in the province of Newfoundland,Canada.

HALIM YANIKOMEROGLU (F’17) was bornin Giresun, Turkey, in 1968. He received theB.Sc. degree in electrical and electronics engineer-ing from the Middle East Technical University,Ankara, Turkey, in 1990, and the M.A.Sc. degreein electrical engineering and the Ph.D. degreein electrical and computer engineering from theUniversity of Toronto, Canada, in 1992 and 1998,respectively.

During 1993–1994, he was with the Researchand Development Group, Marconi Kominikasyon A.S., Ankara. Since 1998,he has been with the Department of Systems and Computer Engineering,Carleton University, Ottawa, Canada, where he is currently a Full Professor.His research interests include wireless technologies with a special emphasison cellular networks. In recent years, his research has been funded byHuawei, Telus, Allen Vanguard, Blackberry, Samsung, CommunicationsResearch Center of Canada, and DragonWave. This collaborative researchresulted in about 25 patents.

Dr. Yanikomeroglu is a Distinguished Lecturer of the IEEE Commu-nications Society and a Distinguished Speaker for the IEEE VehicularTechnology Society in 5G wireless technologies. He has been involved inthe organization of the IEEE Wireless Communications and NetworkingConference (WCNC) from its inception in 1998 in various capacities,including serving as a Steering Committee Member, Executive CommitteeMember, and the Technical Program Chair or Co-Chair of WCNC, Atlanta,in 2004, WCNC, Las Vegas, in 2008, and WCNC, Istanbul, in 2014. He wasthe General Co-Chair of the IEEE 72nd Vehicular Technology Conference(VTC2010-Fall) held in Ottawa. He is currently the General Co-Chairof the IEEE 86th Vehicular Technology Conference (VTC2017-Fall)to be held in Toronto. He was the Chair of the Wireless Technical Com-mittee. He has served in the Editorial Board of the IEEE TRANSACTIONS ON

COMMUNICATIONS, the IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, andthe IEEE COMMUNICATIONS SURVEYS AND TUTORIALS.

He spent the 2011–2012 academic year with the TOBB Universityof Economics and Technology, Ankara, as a Visiting Professor. He is aregistered Professional Engineer in the province of Ontario, Canada. Hewas a recipient of the IEEE Ottawa Section Outstanding Educator Awardin 2014, the Carleton University Faculty Graduate Mentoring Award in2010, the Carleton University Graduate Students Association ExcellenceAward in Graduate Teaching in 2010, and the Carleton University ResearchAchievement Award in 2009.

14588 VOLUME 5, 2017

low-complexity detection of high-order qam faster-than

Documents