MAXIMUM LIKELIHOOD BINARY DETECTION IN

IMPROPER COMPLEX GAUSSIAN NOISE

Amirhossein S.Aghaei, Konstantinos N.Plataniotis, Subbarayan Pasupathy

E-mail: {aghaei, kostas, pas} @ comm.utoronto.ca

The Edward S. Rogers Sr. Dept. of Electrical and Computer Engineering
University of Toronto

ABSTRACT The pseudo variance γn2 is in fact the covariance between n and n∗ .
In a wide range of communication systems, including DS-CDMA When n is uncorrelated from n∗ (i.e. γn2 = 0), it will be called
and OFDM systems, the signal-of-interest might be corrupted by an a proper or circularly symmetric noise. This paper focuses on the
improper [1] (also called non circularly symmetric [2]) interfering problem of detecting the transmitted signal, which is deterministic
signal. This paper studies the maximum likelihood (ML) detection but unknown at the receiver, from the noisy observation r when the
of binary signals in the presence of additive improper complex Gaus- noise n is improper. Although [9] has studied the mean square es-
sian noise. Proposing a new measure for noncircularity of complex timation and detection of improper signals-of-interest observed in
random variables, we will derive the ML decision rule and its perfor- proper noise, the problem of ML detection in the presence of im-
mance based on this measure. It will be shown that the ML detector proper noise has not been studied in general yet; except few works
performs pseudo correlation [1] as well as conventional correlation which have studied some special systems with BPSK signaling (e.g.
of the observation to the signals-of-interest. As an alternative so- [4, 7, 8]). However, the results of this paper, specially the circular-
lution, we will propose a filter for converting improper signals to ization filter introduced in Section 4, are applicable to any M-ary
proper ones, called circularization filter, and will utilize it together signaling with fixed constellation in the complex plain (e.g. antipo-
with a conventional matched-filter (MF) to construct an ML detector. dal, orthogonal, M-QAM, or M-PSK signaling).
In Section 3, after derivation of ML decision rule, we will prove
Index Terms— Circularization, Gaussian noise, improper com- that the ML detector makes use of the correlation existing between
plex, matched filters, maximum likelihood detection. real and imaginary parts of the interfering noise to get the optimal
detection performance. Based on the results of Section 3, we will
1. INTRODUCTION propose a circularization filter for converting the improper additive
noise into a proper one in Section 4. This filter can be used as a pre-
Since the first studies of improper complex signals (also called non processing step for conventional tools (MFs in this paper) to enable
circularly symmetric signals) by [1, 2] there has been a growing in- them to deal with improper noises.
terest on characterization of such signals in various applications, es-
pecially communication systems. The issue of improper interference
was firstly studied by [3] in the context of multi-access interference 2. IMPROPER GAUSSIAN RANDOM VARIABLES
(MAI) in synchronous DS-CDMA systems using BPSK modulation.
However, improper interferences are not restricted to this case and Let z be a scalar complex Gaussian random variable (RV) with Carte-
can arise in many other applications (e.g. [4]-[8]). sian representation of the form z = x + jy, where x and y are
This paper considers the simple scalar case of binary transmis- two real valued RVs. It has been shown in [10] that z can be com-
sion over a typical additive Gaussian noise channel, where the noise pletely characterized by either wz  [x, y]T or vz  √12 [z, z ∗ ]T ,
is independent of the transmitted signal. Let’s assume that r is a as these two random vectors relate to each other through a unitary
scalar1 noisy observation of the unknown scalar signal-of-interest s linear transformation:
which might take either of the deterministic constant values s0 or s1 » –
1 1 1
with equal probabilities according to the following hypotheses: wz = Tvz , where T = √ (3)
j 2 −j j
H0 : r = s0 + n
(1)
H1 : r = s1 + n Moreover, complete second-order characterization of z requires knowl-
edge of the following statistics [10]:
where r, s0 , s1 , n ∈ C and the noise n has the following variance
and pseudo-variance [1]: σz2 = σx2 + σy2 , γz2 = (σx2 − σy2 ) + j(2 σxy ), (4)
σn2 2
 E{|r − r̄| } , γn2 2
 E{(r − r̄) }. (2)
or knowledge of the following covariance matrices:
1 Inthis paper we use the following notation: lowercase letters for scalar » –
variables (e.g. z), boldface lowercase letters for vectors (e.g. z), and bold- H 1 σz2 γz2
face uppercase letters for matrices (e.g. Z). R and C represent the real and Cvz vzH  E{(vz − vz̄ )(vz − vz̄ ) } = (5)
2 (γz2 )∗ σz2
complex domains. {z} and {z} represent the real and imaginary parts of » –
a complex variable z. σz2 , γz2 , and σxy represent the variance of z, pseudo σx2 σxy
Cwz wzT  E{(wz − wz̄ )(wz − wz̄ )T } = . (6)
variance of z, and the covariance between x and y, respectively. σxy σy2

1-4244-1484-9/08/$25.00 ©2008 IEEE 3209 ICASSP 2008

Note that by knowing one of the covariance matrices in (5) or (6), in (9). By taking the natural logarithm of both sides, the following
the other one can be determined since Cwz wzT = TCvz vzH TH . decision rule will be resulted:
Finally, it should be noted that the probability density function (pdf)
1 H −1
of a complex Gaussian RV z is defined in [10] as follows: vsH1 C−1 v −
vn vH r
vs C H vs
n 2 1 vn vn 1
˛ ˛− 1 n 1‚ ‚ o
˛ ˛ 2 ŝ=s1 1 H −1
pz (z) = ˛2πCwz wzT ˛ exp − ‚wz − wz̄ ‚C−1 ≷ vsH0 C−1 H vr − vs C H vs , (10)
2 T
wz wz ŝ=s0
vn vn 2 0 vn vn 0
˛ ˛− 1 n 1‚ ‚ o
˛ ˛
exp − ‚vz − vz̄ ‚C−1
2
= ˛2πCvz vzH ˛ , (7) where vsj = √1 [sj , s∗ T √1 [n, n∗ ]T , 1
[r, r ∗ ]T ,
2 H
vz vz 2 j] , vn = 2
vr = √
2
‚ ‚ and » –
where ‚x‚C  xH Cx. 2 1 −αn
C−1 = ∗ .
The latter equation in (4) reveals that z is proper (γz2 = 0) if vn vHn σn2 (1 − |αn |2 ) −αn 1
the power of z is equally distributed between its real and imaginary By substituting these values in (10), we will get
parts (σx2 = σy2 ) and there exists no correlation between these two
j ff
parts (σxy = 0). In this case, we will get γz2 = 0 and σz2 = 2σx2 , and ` 1 ´ ` 1 ´ ∗
the covariance matrices of (5) and (6) will be reduced to: Cwz wzT =  rs∗1 − |s1 |2 − rs1 − s21 αn
2 2
Cvz vzH = (σz2 /2)I, where I is the identity matrix. In the follow- j ff
ŝ=s1 ` ∗ 1 ´ ` 1 ´ ∗
ing definition, we will propose a new complex valued measure for ≷  rs0 − |s0 |2 − rs0 − s20 αn . (11)
ŝ=s0 2 2
noncircularity (or improperness) of RVs, which will be utilized in
analyzing the ML detector in next sections.
Fig. 1(a) shows the the block diagram of this detector. The term
Definition 1 Given a complex RV z, the noncircularity coefficient of rs∗j correlates the observation r with sj , while the the term rsj
z, denoted by αz , is defined as the ratio of its pseudo-variance to its pseudo correlates [1] r with sj . After the outputs of the correlator
variance: and pseudo correlator are adjusted by 12 |sj |2 and 12 s2j , respectively,
„ 2 « „ « they will be superposed according to the noncircularity coefficient
γz2 σx − σy2 2σxy αn , and the ML decision will be based on the real part of this re-
αz  2 = +j . (8)
σz σx2 + σy2 σx2 + σy2 sult. For proper noises, the outputs of pseudo correlators for both s1
We call αz the noncircularity coefficient of z, in that its real and and s0 are ignored (multiplied by αn = 0), and the decision only
imaginary parts are normalized measures of the power difference depends on the outputs of correlators according to
between x and y and the correlation between x and y, respectively. j ff ŝ=s j ff
` 1 ´ 1 ` 1 ´
Therefore, αz is independent from the changes in the power of z  rs∗1 − |s1 |2 ≷  rs∗0 − |s0 |2 (12)
and only conveys information about its properness. In fact, decom- 2 ŝ=s0 2
position of the pseudo-variance as γz2 = σz2 αz provides us with the
ability to distinguish between changes in γz2 caused by changing the which is the decision rule of well-known conventional ML detectors
power of z as opposed to changes caused by changing the properness for proper noises.
of z. By using (3), decision rule of (10) can also be rewritten as:

Theorem 1 The magnitude of the pseudo-variance of a complex 1 T −1

wsT1 C−1
wn wT
wr − ws C T ws
RV is upper bounded by the value of its conventional variance. n 2 1 wn wn 1
ŝ=s1 1 T −1
(Proof: Appendix-A) Theorem-1 implies that the newly defined co- ≷ wsT0 C−1 T wr − ws C T ws , (13)
ŝ=s0
wn wn 2 0 wn wn 0
efficient αz lies within the unit circle (i.e. 0 ≤ |αz | ≤ 1), and the
extreme case for improperness of z occurs when |γz2 | = σz2 .
where wsj = [{sj }, {sj }]T , wn = [{n}, {n}]T , wr =
[{r}, {r}]T , and
3. MAXIMUM LIKELIHOOD BINARY DETECTION IN
THE PRESENCE OF IMPROPER NOISE » –
−1 2 1 − {αn } −{αn }
Cwn wT = 2 .
n σn (1 − |αn |2 ) −{αn } 1 + {αn }
In this paper we consider the binary hypothesis model of (1) and as-
sume that the detector has complete knowledge of the noise statistics
The ML decision rule of (13) can be interpreted as follows. {r}
(i.e. n̄ as well as either Cvn vnH or Cwn wnT ); hence, the noise pdf
and {r} can be considered as two observations of signals {sj }
pN (n) is available. Without loss of generality, we will assume that and {sj } which are corrupted with noises {n} and {n}, re-
n has zero mean (For nonzero mean noises, we can use r = r − n̄). spectively. When n is improper (αn = 0), the off-diagonal elements
Under the maximum likelihood paradigm, the following decision of Cwn wnT are not zero and {n} and {n} will become correlated
rule could be defined: noises.
pR|S1 (r|s1 ) ŝ=s1 Therefore, the term wsT1 C−1 H −1
T wr (= vsj Cv vH vr ) can be
L(r) = ≷ 1, (9) wn wn n n
pR|S0 (r|s0 ) ŝ=s0 viewed as a whitened matched-filter for optimal joint detection of
{sj } and {sj } from the noisy observation vector of wr , when
By conditioning on sj , the detector assumes that sj is transmitted;
the elements of the jointly Gaussian noise vector wn are correlated
therefore, pR|Sj (r|sj ) = pN (r − sj ). Since sj is a deterministic
to each other. This relationship between ML detector for improper
constant value, we can substitute
scalar noise n and the well-known whitened MF for correlated wn
˛ ˛− 1 n 1‚ ‚ o
˛ ˛ 2 motivates us to study another alternative implementation of optimal
pR|Sj (r|sj ) = ˛2πCvn vnH ˛ exp − ‚vr − vsj ‚C−1
2 H
vn vn
ML detector in the next section.

3210
 Applying a circularization filter to the observation r, we get
1 1 1
wre = Λ− 2 UT wr = Λ− 2 UT ws + Λ− 2 UT wn = wse + wne ,
(15)
where se and n e denote the transformed signal and the circularized
noise (Cwne wT = I), and U and Λ are derived from eigen decom-
ne
position of Cwn wnT . As it is shown in Fig. 1(b), the ML detection
of se can now be accomplished using conventional matching of re to
se∗ , i.e.,
(a) Combination of Correlators and Pseudo Correlators j ff ŝ=s j ff
1 1 1
 re se∗1 − |e
s1 |2 ≷  re se∗0 − |es0 |2 (16)
2 ŝ=s0 2
or equivalently
1 T ŝ=s1 1 T
wsTe1 wre − wse wse1 ≷ wsTe0 wre − wse wse0 . (17)
2 1 ŝ=s0 2 0

Note that both decision rules of (16) and (17) are equivalent to the
ML decision rule of (10), in that
˘ ¯
 re se∗j = wsTej wre = wsTj C−1
wn wT
wr = vsHj C−1 v
vn vH r
n n

Accordingly, we will call the term C−1

vsHj H vr
vn vn
a circularized MF
(b) Circularization Filter followed by Matched-Filter H
to distinguish it from conventional MF (vsj vr ).
Fig. 1. Equivalent implementations of binary ML detector. In fact, circularization is a reversible preprocessing step and pre-
serves all the data included in vr . Thus, applying the ML criterion
to the preprocessed data will yield the same result as applying the
Finally, it should be mentioned that for proper noises, decision ML criterion to the original data (see [11] Page 289).
rules of (10) and (13) reduce to Finally, it should be noted that for proper noises (αn = 0), we
have Cwne wT = (σn2 /2)I. Consequently, the circularization filter
1 H ŝ=s1 1 H n
e p
vsH1 vr − vs vs ≷ vsH0 vr − vs vs simply scales r with factor 2/σn2 . Therefore, for proper noises, it
2 1 1 ŝ=s0 2 0 0
can be omitted and the optimal detector will be the conventional MF.
and
1 T ŝ=s1 1 T
wsT1 wr − ws ws1 ≷ wsT0 wr − ws ws0 , 5. PERFORMANCE
2 1 ŝ=s0 2 0
Let r = x + jy  be the representation of the observation r =
both of which make use of conventional MFs (vsHj vr or wsTj wr ). x + jy in a new coordinate system which has the following axes:
the line passing through the points s0 and s1 , and the perpendicular
4. CIRCULARIZATION FILTER FOLLOWED BY bisector of the line segment s0 s1 . Assuming that the angle between
MATCHED-FILTER the former line and x-axis is θ, we have [x , y  ]T = Rw(−θ) [x −
{s̄}, y − {s̄}]T , where Rw(−θ) is defined in Appendix-B and
In Section 3, it was shown that an improper scalar noise n can be s̄ = 12 (s0 + s1 ). It can be shown that when x -axes is parallel to one
represented by the noise vector wn with non i.i.d. elements. Con- of the eigenvectors of Cwn wnT , the ML detectors of Sections 3,4 will
sequently, by converting the elements of wn into i.i.d elements, the become equivalent to conventional MF. For all other cases, however,
corresponding noise will become proper. Thus, we can exploit the the ML detector is not a conventional MF and outperforms it. It can
concept of whitening filters in order to convert an improper noise be proved that the probability of detection error is
into a proper one as follows. s !
2|s1 − s0 |2 1 − |αn | cos(ϕ − 2θ)
Definition 2 Given an improper complex RV z, the circularization Pe = Q ×
(or proprization) filter is defined as the filter which converts z into a σn2 1 − |αn |2
s !
proper RV ze, using the following input/output relationship:
|s1 − s0 |2
= Q , (18)
wze =
1
Λ− 2 UT wz , (14) 4σn2  (1 − ρ2 )
I
.q
where U and Λ are the matrices including eigenvectors and eigen-
where nI , nQ , and ρ = σnI nQ σn2  σn2  are, respectively, the
values of Cwz wzT , respectively (i.e. Cwz wzT = UΛUT ). I Q
inphase part, quadrature part, and correlation factor of the noise in
According to Appendix-B, this filter firstly rotates the coordinate x -y coordinates, and ϕ is the phase of the noncircularity coefficient
system and uses the eigenvectors of Cwz wzT as the new coordinate of noise in x-y coordinates. Equation (18) together with Appendix-
system to remove the correlation between {z} and {z}. After- B reveals that for a fixed power constraint, the optimal signaling is
wards, the new coordinates should be scaled separately by the in- antipodal signaling (s1 = −s0 ) along the eigenvector corresponding
verse of the square root of corresponding eigenvalues to get a proper to the smallest eigenvalue of Cwn wnT . As mentioned before, the ML
ze with equal power on {e z } and {e
z }. detector for this signaling will become a conventional MF.

3211
 In order to illustrate how ML detectors of Section 3 outperform 0
10
ρ’ = 0.00
conventional MFs, we will consider an antipodal signaling such that −1 ρ’ = 0.25
10
θ = (ϕ − π)/2 and θ = ϕ/2 (in these two cases the ML detector ρ’ = 0.50
ρ’ = 0.75
is MF). For simplicity, let’s assume that θ = 0 and the power of −2
10

noise is equally distributed in x and y directions. Fig. 2 presents −3

10

Error Probability
the performance of the ML detector in this case for different values
of ρ . It can be seen that for improper noises with higher ρ , this
−4
10

performance improves since the optimal detector is provided with −5

10

more side information in y  . −6

10
Fig. 2 also illustrates the improvement which can be achieved
by using ML detectors instead of conventional MFs. In x -y co-
−7
10

ordinate system, the imaginary part of s is zero for both s1 and −8

10
0 5 10 15
s0 . Therefore, a conventional MF, which is based on a proper noise SNR (dB)

model, assumes that there is no relevant data in the imaginary part

of the observation (y  ); hence,qignores it. The performance of such Fig. 2. Error Probability for different values of ρ when σn2  = σn2 
I Q
detector is well-known as Q( |s1 − s0 |2 /4σn2  ), which does not and s1 = −s0 ∈ R .
I
change for different values of ρ and is always equivalent to the case
of ρ = 0 shown in Figure 2. However, using the correct improper ` ´
where ϕ denotes the phase of noncircularity coefficient αz , and Rw ϕ2
model for the noise, the ML detector takes advantage of the corre- `ϕ´
lation existing between the real and imaginary parts of observation and Rv 2 are unitary transformations on wz and vz , respectively,
r to give the optimal performance. Thus, its performance improves both of which result in the rotation of z by phase ϕ2 .
for improper noises when ρ increases.
C. REFERENCES
6. CONCLUSION AND REMARKS
[1] F. D. Neeser and J. L. Massey, “Proper complex random pro-
In this paper, the ML detection of binary signals in additive improper cesses with applications to information theory,” IEEE Trans.
Gaussian noise was studied. Using the noncircularity coefficient of Inf. Theory, vol. 39, pp. 1293–1302, July 1993.
the noise, the ML detector combines correlation and pseudo corre- [2] B. Picinbono, “On circularity,” IEEE Trans. Signal Process.,
lation of r with s during the detection process. As a more general vol. 42, pp. 3473–3482, Dec. 1994.
solution, we proposed that instead of designing a new ML detector [3] Y. C. Yoon and H. Leib, “Maximizing SNR in improper com-
for improper noises, we can equip conventional detectors, which are plex noise and applications to CDMA,” IEEE Commun. Lett.,
designed for proper noises, with a circularization filter. Finally, it vol. 1, pp. 5–8, Jan. 1997.
was shown that the resulting ML detector in both approaches be- [4] G. E. Bottomley, “CDMA downlink interference suppression
come a conventional MF when the noise is proper or the signaling is using I/Q projection,” IEEE Trans. Wireless Commun., vol. 2,
along eigenvectors of Cwn wnT . pp. 890–900, Sept. 2003.
Due to space limitations and for simplicity, this paper was re-
stricted to scalar RVs and binary hypotheses; however, the more [5] G. Taubock, “Complex noise analysis of DMT,” IEEE Trans.
general results for random vectors and M-ary hypotheses as well as Signal Process., vol. 55, pp. 5739–5754, Dec. 2007.
derivation of error probability will be presented in future works. [6] R. Nilsson, F. Sjoberg, and J. P. LeBlanc, “A rank-reduced
LMMSE canceller for narrowband interference suppression
in OFDM-based systems,” IEEE Trans. Commun., vol. 51,
A. PROOF OF THEOREM-1
pp. 2126–2140, Dec. 2003.
According to (4) we have σz4 − |γz4 | = 4[σx2 σy2 − σxy 2
]. From [7] Y. Yoon and H.-M. Kim, “Maximum likelihood multiuser de-
Schwartz inequality (σx σy − σxy ≥ 0), it follows that 0 ≤ |γz4 | ≤
2 2 2 tection of DS/CDMA signals in improper noise,” in 2005
σz4 ; hence, 0 ≤ |γz2 | ≤ σz2 . Fifth Int. Conference on Inf., Commun. and Signal Process.,
pp. 224–228, Dec. 2005.
B. EIGEN DECOMPOSITION OF CvZ vH AND CwZ wT [8] P. Chevalier and F. Pipon, “New insights into optimal widely
Z Z
linear array receivers for the demodulation of BPSK, MSK,
and GMSK signals corrupted by noncircular interferences-
Let Cvz vzH = QΛ1 QH and Cwz wzT = UΛ2 UT be the eigen de-
application to SAIC,” IEEE Trans. Signal Process., vol. 54,
composition of covariance matrices defined in (6), (5), where Λ1 and
pp. 870–883, Mar. 2006.
Λ2 are diagonal matrices including the eigen values of Cvz vzH and
Cwz wzT in non-increasing order, and Q and U are unitary matrices [9] P. J. Schreier, L. L. Scharf, and C. T. Mullis, “Detection and
including their corresponding eigenvectors. Then estimation of improper complex random signals,” IEEE Trans.
» – Inf. Theory, vol. 51, pp. 306–312, Jan. 2005.
σ2 1 + |αz | 0
1. Λ1 = Λ2 = Λ = 2z [10] B. Picinbono, “Second-order complex random vectors and
0 1 − |αz |
» – normal distributions,” IEEE Trans. Signal Process., vol. 44,
ϕ
cos 2 − sin 2 ϕ ` ´ pp. 2637–2640, Oct. 1996.
2. U = = Rw ϕ2
sin ϕ2 cos ϕ2 [11] H. L. Van Trees, Detection, Estimation, and Modulation The-
" ϕ
#
ej 2 0 ` ´ ory, vol. I: Detection, Estimation, and Linear Modulation The-
H
3. Q = T U = −j ϕ TH = Rv ϕ2 TH ory. John Wiley and Sons, 1968.
0 e 2

3212