Optimal Least-Squares FIR Digital Filters for

Compensation of Chromatic Dispersion in

Digital Coherent Optical Receivers

Amir Eghbali, Håkan Johansson, Oscar Gustafsson and Seb J. Savory

Linköping University Post Print

N.B.: When citing this work, cite the original article.

Original Publication:

Amir Eghbali, Håkan Johansson, Oscar Gustafsson and Seb J. Savory, Optimal Least-Squares
FIR Digital Filters for Compensation of Chromatic Dispersion in Digital Coherent Optical
Receivers, 2014, Journal of Lightwave Technology, (32), 8, 1449-1456.
http://dx.doi.org/10.1109/JLT.2014.2307916
Copyright: Optical Society of America
http://www.osa.org/

Postprint available at: Linköping University Electronic Press

http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-106013
JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 32, NO. 8, APRIL 15, 2014 1449

Optimal Least-Squares FIR Digital Filters

for Compensation of Chromatic Dispersion
in Digital Coherent Optical Receivers
Amir Eghbali, Member, IEEE, Håkan Johansson, Senior Member, IEEE, Oscar Gustafsson, Senior Member, IEEE,
and Seb J. Savory, Senior Member, IEEE

Abstract—This paper proposes optimal finite-length impulse re- sampling frequency of F = T1 . The CD can be compensated by
sponse (FIR) digital filters, in the least-squares (LS) sense, for com- designing a filter H(ej ω T ) to approximate a desired frequency
pensation of chromatic dispersion (CD) in digital coherent optical
receivers. The proposed filters are based on the convex minimiza-
response as
1 2
tion of the energy of the complex error between the frequency HD es (ej ω T ) = = ej K (ω T ) . (2)
responses of the actual CD compensation filter and the ideal CD C(ej ω T )
compensation filter. The paper utilizes the fact that pulse shaping This is also referred to as static channel equalization [2] for
filters limit the effective bandwidth of the signal. Then, the filter which FIR or infinite-length impulse response (IIR) filters1 can
design for CD compensation needs to be performed over a smaller be used [1], [2], [4]. This paper uses FIR filters because they
frequency range, as compared to the whole frequency band in the
existing CD compensation methods. By means of design examples, (i) are unconditionally stable [8], (ii) can efficiently be im-
we show that our proposed optimal LS FIR CD compensation plemented in the frequency domain [2], and (iii) do not have
filters outperform the existing filters in terms of performance, im- limitations on the maximal sampling frequency, as opposed to
plementation complexity, and delay. their IIR counterparts [9].
Index Terms—Chromatic dispersion (CD), digital filter, fiber
optics, optimal least-squares (LS) FIR filter. A. Contribution of the Paper and Relation to Previous Work
In [1] and [2], an FIR CD compensation filter was derived
I. INTRODUCTION with a complex impulse response given by
    
N optical fibers, the group velocity of the propagating signal
I is frequency dependent and optical pulses hence spread in
time. This results in CD [1]–[5] thus limiting the transmission
h(n) =
j −j n 2
4Kπ
e 4 K , −
N
2
≤n≤
N
2
(3)

distance and/or data rate [6]. The CD is traditionally compen- where the length of the filter h(n) is odd and given2 as [1]
sated using optical devices with opposite dispersion [5] but such N = 2 2Kπ + 1. (4)
approaches cannot be easily tuned/improved to accommodate
different fiber spans/properties and quality measures [6]. To decrease the hardware cost, it is generally of interest to reduce
With coherent detection schemes, richer constellations, and N which in turn reduces the number of arithmetic operations re-
fast analog to digital converters (ADCs), digital signal process- quired for implementation [8]. Besides hardware cost, a smaller
ing is playing a growing role in CD compensation [4], [6], [7]. In N decreases the overall delay of the filter which is an important
digital coherent optical receivers, CD is modeled as a frequency issue, especially, in high-speed communications.
response given by [1]–[5] A drawback of (3) is that it does not utilize the effects
of pulse shaping filters in limiting the bandwidth of signals.
Dλ2 z
C(ej ω T ) = e−j K (ω T ) , With band-limited signals, at the output of pulse shaping filters,
2
K= . (1)
4πcT 2 we do not need to compensate CD over the whole frequency
Here, D, λ, z, and c are the fiber dispersion parameter, wave- band of ωT ∈ [−π, π]. In other words, and as we will see in
length, propagation distance, and the speed of light, respec- Section III-B2, the band-limiting properties of pulse shaping
tively. This paper uses ωT = 2πf T to represent the “digital filters allow us to compensate CD in a smaller frequency band
frequency” with a sampling period of T thus corresponding to a thereby reducing the filter length and the implementation cost.
As discussed later in Section III-A2, (3) has another drawback
Manuscript received October 9, 2013; revised January 19, 2014 and February because it does not necessarily improve the CD compensation
18, 2014; accepted February 19, 2014. Date of publication February 23, 2014; quality if we increase the filter length. As we will also describe
date of current version March 9, 2014.
A. Eghbali, H. Johansson, and O. Gustafsson are with the Division of
in Sections III-A1 and III-B2, a third drawback of (3) is that it
Electronics Systems, Department of Electrical Engineering, Linköping Univer- has suboptimal performance, especially for modulation formats
sity, 581 83 Linköping, Sweden (e-mail: amire@isy.liu.se; hakanj@isy.liu.se; with higher spectral-efficiencies.
oscarg@isy.liu.se).
S. J. Savory is with the Optical Networks Group, Department of Electronic
1 The FIR filter is also called feed-forward equalizer or transversal tap filter [2].
and Electrical Engineering, University College London, London, WC1E 7JE,
U.K. (e-mail: s.savory@ee.ucl.ac.uk). 2 Note that (4) does not guarantee a desired performance, e.g., bit error rate
Digital Object Identifier 10.1109/JLT.2014.2307916 (BER), for a given K .

0733-8724 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications standards/publications/rights/index.html for more information.
1450 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 32, NO. 8, APRIL 15, 2014

With the ever increasing demands on optical communica- complex error as, for ωT ∈ [Ω1 , Ω2 ],
tion systems, to move towards high-speed traffic using high-  Ω2
1  
spectral-efficiency modulation formats, previous CD compen- E= H(ej ω T ) − HD es (ej ω T )2 d(ωT ). (6)
sation approaches need to be revisited so as to (i) improve the CD 2π Ω 1
compensation quality, (ii) relax the demands on the subsequent To obtain h(n), we formulate an LS problem to find an FIR filter
adaptive equalizers, and (iii) overcome the above-mentioned ĥ which minimizes E as [10]
drawbacks.
This paper proposes optimal FIR filters, in the LS sense, ĥ = argmin E (7)
which outperform (3). Our proposed filters can be designed to
where ĥ is defined by (8) as shown at the bottom of the page.
compensate CD in either a small frequency band or the whole
With signals having a flat frequency spectrum, (7) amounts to
frequency band. By increasing the filter length, our proposed
minimizing the error vector magnitude. The solution of (7) is
filters can obtain arbitrarily good CD compensation, even with
unique and globally optimal in the LS sense.3 It is given by
modulation formats having high spectral-efficiencies. As will
be seen in Section III-B2, considering the effects of pulse shap- ĥ = Q−1 D (9)
ing filters, in our proposed method, we can further reduce the
implementation cost. Even without considering the effects of where Q is an Nc × Nc matrix with elements as
pulse shaping filters, i.e., for the whole frequency band and the  Ω2
1
same filter length, our proposed filters still outperform (3), as Q(n, m) = ej (n −m )ω T d(ωT )
2π Ω 1
we will see in Sections III-A1 and III-A2.
In Section IV-A, we will also compare our proposed filters e−j (−n +m )Ω 1 − e−j (−n +m )Ω 2
to those obtained by the frequency sampling method (FSM). =
2jπ(−n + m)
We will show that our proposed filters require fewer number ⎧Ω −Ω
of taps, than those of FSM, in order to obtain the same BER ⎪
⎪
2 1
n=m
⎨ 2π
performance. Further, we will show that the superiority, of our = (10)
proposed filters, becomes more pronounced at modulation for- ⎪
⎪
−j (−n +m )Ω 1
− e−j (−n +m )Ω 2
⎩e n = m
mats with a high spectral-efficiency. 2jπ(−n + m)
for 0 ≤ n ≤ Nc − 1 and 0 ≤ m ≤ Nc − 1. Note that Q is a
B. Paper Outline Hermitian Toeplitz matrix and it thus suffices to only compute
Section II introduces the proposed optimal LS FIR filters for its first row with n = 0 in (10). This property of Q reduces
CD compensation whereas the numerical results are discussed the design complexity by reducing the number of computations
in Section III. Some design and implementation issues are dis- required in (10). Further, D is defined as in (11) as shown at the
cussed in Section IV. Finally, the concluding remarks are given bottom of the page, with
in Section V.  Ω2
1
D(n) = HD es (ej ω T )ej n ω T d(ωT )
2π Ω 1
II. PROPOSED CD COMPENSATION FILTERS
 Ω2
Consider a complex FIR filter of length Nc , which like N is 1
= ej ω T (K ω T +n ) d(ωT ). (12)
assumed to be odd here, with a frequency response as [8] 2π Ω 1
N c −1 After some manipulations, (12) gives the closed form solution
 2
of (13) as shown at the bottom of the page, in which erf(α) is
jωT −j n ω T
H(e )= h(n)e . (5)
n =− N c2−1
3 By optimality, we mean that for a given filter length N , no other filter
c
To measure the accuracy of the designed filter H(ej ω T ), which (having the same length) will result in an E which is smaller than that obtained
approximates HD es (ej ω T ) in (2), we define the energy of the by ĥ in (9).

     T
Nc − 1 Nc − 1 Nc − 1 Nc − 1
ĥ = ĥ − ĥ − +1 ... ĥ −1 ĥ (8)
2 2 2 2

     T
Nc − 1 Nc − 1 Nc − 1 Nc − 1
D= D − D − +1 ... D −1 D (11)
2 2 2 2
     
−j n2
+ 34π 3π 3π
e 4K
ej 4 (2Kπ − n) ej 4 (2Kπ + n) Nc − 1 Nc − 1
D(n) = √ erf √ + erf √ ,− ≤n≤ (13)
4 πK 2 K 2 K 2 2
EGHBALI et al.: OPTIMAL LEAST-SQUARES FIR DIGITAL FILTERS FOR COMPENSATION OF CD IN DIGITAL COHERENT OPTICAL RECEIVERS 1451

TABLE I
SIMULATION PARAMETERS USED IN EXAMPLES 1 AND 2
(a)

Here, c = 3×108 m/s, D = 17 ps/nm/km, λ = 1553 nm, and L = 2.

(b)

Fig. 1. Interpolation and decimation by L. (a) Interpolation. (b) Decimation.

defined as Fig. 2. Simulation chain composed of pulse shaping, CD model, and CD

 α compensation filter.
2
e−t dt.
2
erf(α) = √ (14)
π 0
where IN c is an Nc × Nc identity matrix which corresponds
Note that if α, in (14), is an imaginary term, we obtain the
to the (approximately) unit energy of the CD compensation
imaginary error function erfi(α) which is related to erf(α) as
filter. By using , we add a penalty term which is proportional
 jα
erf(jα) 1 2 to the unit energy of the CD compensation filter. Without this
e−t dt.
2
erfi(α) = = √ (15) penalty term, the frequency response of the CD compensation
j j π 0
filter can have undesired behaviors in the frequency range of
There exist efficient numerical methods to evaluate (15), e.g., ωT ∈ [ωs T, π]. For each , (17) gives an optimal filter but the
[11]. As discussed earlier, (3) targets the whole frequency band best solution is dependent on  as we will see in Section III-B1.
where Ω2 = −Ω1 = π. Utilizing the properties of pulse shaping
filters, the values of Ω2 and Ω1 (and hence Nc ) can be reduced III. NUMERICAL RESULTS
as will be discussed in Section II-A, below.
We here assume the same parameters5 given in Table I. Note
A. Effects of Pulse Shaping that Example 1 uses the same parameters as in [1]. However,
for Example 2, we have increased the symbol rate F , while us-
Pulse shaping comprises interpolation (decimation) at the ing the same oversampling ratio L, to show that our proposed
transmitter (receiver) side and it reduces the inter-carrier in- filters can actually be used for future high-speed communica-
terference and inter-symbol interference. In Fig. 1(a) [1(b)], in- tions requiring longer CD compensation filters. In Example 1,
terpolation [decimation] by the integer factor L > 1 requires an (4) gives N = 251 whereas for Example 2, we get N = 875.
upsampler [a downsampler] and an anti-imaging [anti-aliasing] Based on these parameters, we will compare the existing CD
filter GT X (ej ω T ) [GR X (ej ω T )] [12]. These filters usually have compensation filter, defined by (3), with our proposed filters in
lowpass characteristics4 with a roll-off of 0 < ρ < 1 leading (9) or (17). This comparison is carried out using a simulation
to a stopband edge at ωs T = π 1+ρ L . They are designed so that chain, shown in Fig. 2, where e(n) represents the additive white
L −1 j (ω T − 2Lπ l ) j (ω T − 2Lπ l )
l=0 GT X (e )GR X (e ) ≈ 1 [13]. A typical Gaussian noise (AWGN) channel. In Section IV-A, we will also
solution is the square-root raised cosine filter as [14] compare our proposed filters with those obtained by FSM.
    We can add other noise sources to this simulation setup, e.g.,
sin πTt (1 − ρ) + 4ρt cos πTt (1 + ρ)
gT X (t) = gR X (t) =   4ρt 2 
T
. laser phase noise [16] or impairments of ADCs and digital to
πt
T 1 − T analog converters (DACs). However, the CD compensation fil-
(16) ters are independent of such effects [1] and we have therefore
This paper deals with digital filters and we will hence use t = nLT not considered them6 . In (16), we use ρ = 0.25 and we have
in (16). As the interpolated signals have limited bandwidths, one chosen high orders for GT X (ej ω T ) and GR X (ej ω T ) so that
does not actually need to compensate CD in the whole frequency the errors, arising from pulse shaping, are negligible. Then, the
band of ωT ∈ [−π, π]. It suffices to design ĥ for the frequency only error sources are those due to (i) the CD model, (ii) the CD
band of ωT ∈ [−ωs T , ωs T ] by using Ω2 = −Ω1 = ωs T in (6)– compensation filter, and (iii) the AWGN channel.
(12). Our Monte Carlo simulations use quadrature amplitude
With Ω2 = −Ω1 = ωs T , one should however add a proper modulation (QAM) symbols. In the BER plots, the term
nonzero term to Q so as to avoid ill-conditioned matrices. One “BTB” stands for the back-to-back propagation obtained with
way to do so is to rewrite (9) as C(ej ω T ) = 1 and H(ej ω T ) = 1. In other words, BTB stands

ĥ = (Q + IN c )−1 D (17) 5 We can further reduce the implementation complexity by choosing 1 < L <
2 but this would require interpolation/decimation by rational factors [15] and
is beyond the scope of this paper. The main focus of this paper is to compare
4 Pulse shaping filters generally belong to a class of filters called Lth-band
the performance of different filter design methods and as long as the simulation
filters for which the transition band includes Lπ and the passband/stopband edges chain is the same, the choice of L is not crucial.
are equally distanced from Lπ . Therefore, having passband/stopband edges as 6 This paper compares different digital filters which compensate the same
π 1 L∓ρ is a customary way of defining these edges so that they, with 0 < ρ < 1 amount of CD and we assume other parts of the system to have a negligible
and integer L > 1, are guaranteed to be smaller than π which is necessary for effect. If we add other impairments, like laser phase noise or ADC/DAC errors,
digital filters [8], [12]. the system performance may ultimately be limited by other factors.
1452 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 32, NO. 8, APRIL 15, 2014

Fig. 3. Simulated uncoded BER for QAM data with K = 19.9227, L = Fig. 4. Simulated uncoded BER for QAM data with K = 69.605, L =
2, Ω 2 = −Ω 1 = π, and the filters in (9) and (3). 2, Ω 2 = −Ω 1 = π, and the filters in (9) and (3).

for the matched filter performance without the CD effects. In

our Monte Carlo simulations, we model C(ej ω T ) by solving a
problem like (7) with HD es (ej ω T ) = C(ej ω T ) but (as for the
pulse shaping filters) we have chosen a high order so that the
corresponding error becomes negligible.

A. Fullband Case
In this section, we will compare the filters in (9) and (3) using
Ω2 = −Ω1 = π and L = 2.
1) BER Performance With the Same Filter Lengths: In this
case, (10) becomes
⎧
⎨1 n=m
Q(n, m) = sin ((n − m)π)
⎩ n = m
(n − m)π

1 n=m Fig. 5. Simulated uncoded BER values of 16-QAM data with different values
= (18) of N = N c in (3) and (9) for Ω 2 = −Ω 1 = π, L = 2, K = 19.9227, and
0 n=m. E
some values of N b0 dB.
The matrix Q is thus an identity matrix and (9) hence amounts
to only finding D in (11) and (12). This simplifies the design for (3), tend to obtain a floor8 . These plots also show that for
complexity, associated with (9), as it gives simpler modulation schemes and with large BER values, the
filters obtained by (9) and (3) result in roughly similar BER
ĥ(n) = D(n) (19) curves. For much simpler modulation schemes, e.g., 4-QAM,
the BER curves of (9) and (3), are very similar and we have thus
which is given by (13). Figs. 3 and 4 respectively compare the not plotted them. However, with modulation formats having
simulated uncoded BER of Examples 1 and 2 over an AWGN higher spectral-efficiencies and if a small BER is desired, the
channel. As noted earlier, we use the parameters in Table I filter obtained by (9) clearly has a better performance.
along with (4) to estimate N . With the same number7 of taps 2) Comparison With Different Filter Lengths: As noted ear-
per 1000 ps/nm of CD, our proposed filter, in (9), gives a smaller lier, a drawback of (3) is that CD compensation does not nec-
BER as compared to that of (3). essarily improve if we increase the filter length. Fig. 5 shows
Eb
Figs. 3 and 4 show that beyond certain values of N 0
and for the uncoded BER values of 16-QAM data where we have con-
modulations with a high spectral-efficiency, the BER curves, sidered different values of N = Nc in (3) and (9) as well as

8 The Eb
7 In N BER curves of (9) will also become flat if further increases. How-
Example 1 and with N = N c = 251, we need z = 3.7 taps per N0
Eb
1000 ps/nm of CD [1]. ever, with (3), the BER curves become flat at much lower values of N0 .
EGHBALI et al.: OPTIMAL LEAST-SQUARES FIR DIGITAL FILTERS FOR COMPENSATION OF CD IN DIGITAL COHERENT OPTICAL RECEIVERS 1453

Fig. 7. Magnitude response and group delay of the filters obtained by (9)
Fig. 6. The real parts, the imaginary parts, and the absolute values of the im- and (3) with N = N c = 31 and z = 500 km along with the parameters of
pulse responses h(n) and ĥ(n) with N = N c = 31 and z = 500 km along Example 1. The solid line represents the filter obtained by (3).
with the parameters of Example 1. The plot with circles represents h(n).
(a) Real part. (b) Imaginary part. (c) Absolute value.
Eb
different values of N 0
. As can be seen, if we increase Nc in
(9), the value of BER decreases. This means that we can ob-
tain arbitrarily good CD compensation filters by increasing Nc .
However, this does not apply to (3) and the value of BER may
even increase if we increase N .
In conclusion, the filters given by (9) not only can be designed
to obtain arbitrarily good CD compensation but they also re-
quire fewer taps for the same BER performance. The reason is
two-fold with the first being that (9) gives a smaller E, than (3),
for the same filter length. This is a direct result (see Footnote
3) of optimality of (9). The second reason is that (this is easy
to verify using, e.g., MATLAB) the value of E decreases if the
length of the filter, given by (9), is increased. However, this does
not necessarily apply to the filter in (3). Note that if E becomes
very small, i.e., smaller than the noise of the AWGN channel,
the BER curve does not improve even if we increase Nc . The
BER curves, of (9), hence asymptotically reach the BER curve
of BTB propagation. According to Fig. 5, such a phenomenon Fig. 8. The values of E, in (6), obtained from (17), L = 2, Ω 2 = −Ω 1 =
K = 19.9227, and  = 10−k .
π (1 + ρ )
does not happen for the BER curves of (3) and beyond certain L ,
values of N , the BER even degrades.
3) Relationships Between Closed Form Impulse Responses:
A comparison of (3) and (9) [with its closed form solution in sponse and the group delay. Therefore, the group delay of ĥ(n)
(19)] reveals that the impulse response values are partly similar. tends to be more constant as opposed to that of h(n). This is
n2 another explanation for the superiority of (9) over (3).
For example, the term e−j 4 K appears in both (3) and (19). To
compare the values of h(n) and ĥ(n), Fig. 6 shows the real parts,
the imaginary parts, and the absolute values of these impulse re- B. Band-Limited Case
sponses. For illustration purposes, we have considered a smaller In this section, we will compare the filters in (17) and (3)
value of N = Nc = 31, obtained from z = 500 km and the pa- using Ω2 = −Ω1 = π (1+ρ)L and L = 2.
rameters of Example 1. Note that the absolute value of h(n), in 1) Choice of Penalty Factor: As discussed earlier, the
(3), has a constant value as |h(n)| = 2 √1K π . However, this does penalty factor , in (17), should have a small nonzero value
not apply to ĥ(n). Also, Fig. 7 shows the magnitude responses but very small values of  may result in numerical problems and
and the group delays of the impulse responses in Fig. 6. As can must be avoided. Based on our experiments, in MATLAB, Fig. 8
be seen, ĥ(n) results in fewer overshoots in the magnitude re- depicts the values of E, in (6), where we have used (17) along
1454 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 32, NO. 8, APRIL 15, 2014

data, this shows a reduction of about 1 − Nc

N = 52 percent in
the number of taps per 1000 ps/nm of CD.

IV. DESIGN AND IMPLEMENTATION COMPLEXITY

In digital filters, one generally has to consider two issues:
design complexity and implementation complexity. The design
complexity refers to the required (numerical) effort to obtain the
filter coefficients. Here, we have two cases for comparison of
the design complexity. If Ω2 = −Ω1 = π, (9) leads to a closed
form solution as in (19). Therefore, for the same filter length
and with Ω2 = −Ω1 = π, the design complexities of (3) and (9)
are comparable.
For the case with Ω2 = −Ω1 = π (1+ρ)L , (17) requires to com-
−1
pute D, Q, and (Q + IN c ) . For the same filter length along
with Ω2 = −Ω1 = π (1+ρ) L , our proposed method thus has a
Fig. 9. Simulated uncoded BER for QAM data in Example 1 and 2 along with higher design complexity than (3). However, as can clearly be
Ω 2 = −Ω 1 =
π (1 + ρ )
L , L = 2, and  = 10 −1 4 . Here, K 1 and K 2 stand for seen from Fig. 9, the choice of Ω2 = −Ω1 = π (1+ρ) L allows to
the values of K in Example 1 and 2, respectively. reduce Nc which in turn reduces the overall design complex-
ity. Our numerical results (not reported here) also show that if
with L = 2, K = 19.9227, and  = 10−k , k = 0, 1, . . . , 25 for L increases, the value of Nc will be much smaller because, in
some values of Nc . Our numerical results show that if  < 10−15 , such a case, the value of Ω2 = −Ω1 = π (1+ρ)L becomes smaller.
the value of E may be degraded as numerical problems occur Then, the design complexity of (17) will be even smaller. It is
due to the software precision limits. For any set {Nc , K, L}, generally desired to derive accurate formulas for estimation of
one must however find the best value of {, E} so as to improve Nc so as to guarantee a desired performance, e.g., BER. This
the CD compensation quality. For a given set {Nc , K, L}, the requires many Monte Carlo simulations in a design space com-
Eb
best value of {, E} can be determined by a simple exhaustive prised of , N 0
, K, L, and the desired BER value and is beyond
search which amounts to designing ĥ(n) for different values of the scope of this paper. However, as can be seen from Fig. 9,
 and selecting the one which gives the lowest E. Note that we, Nc can roughly be estimated as NL . This corresponds to the
here, only illustrate the trend of E for some choices of  and band ωT ∈ [0, ωs T ], over which the CD needs to actually be
Nc . To obtain a desired performance, say a given BER, and if compensated.
K is large, we would require a longer filter. Our simulations Generally, CD compensation filters need not be designed very
in Fig. 8, however, aim to show how to choose  and how the often. Thus, the major complexity-burden comes from their im-
software (MATLAB in this case) precision affects this choice. plementation rather than their design. By means of numerical
For very small values of E, the performance of the system, examples, we have shown that our proposed filters, in (9) and
e.g., BER, is anyhow mainly determined by other factors like the (17), clearly require fewer number of taps [to obtain a given
AWGN channel. Therefore, minor changes in  and Nc will not BER performance], than the filter in (3). Especially for mod-
be crucial although small values of Nc and E are always desired. ulation formats with severe requirements, the implementation
Our experiments show that below certain9 small values of , say complexity of our proposed filters will thus be lower than that
when moving from  = 10−12 to  = 10−14 , the lowest possible of (3).
Nc may vary by a small value, e.g., two. In our simulations, we This paper deals with optimization-based filter design and we
have therefore used  = 10−14 and we have ignored such minor differentiate between filter design and filter implementation. In
changes in Nc . other words, we optimize the impulse response of the filter with
2) BER Performance With Different Filter Lengths: In this constraints in the frequency domain. This is a common practice
case, Q is a general Hermitian Toeplitz matrix. Fig. 9 compares in the design of digital filters [8], [12]. For a given impulse
the simulated uncoded BER of Examples 1 and 2 over an AWGN response, one can generally implement the filter (or equiva-
channel where we have assumed  = 10−14 . In these figures, we lently, the linear convolution) either in the time domain (e.g.,
report the lowest possible length, i.e., Nc in (17), whose BER see [8, Sec. VIII.3]) or in the frequency domain (e.g., see [8, Sec.
is smaller than or equal to the BER of N in (3). For example, V.10]). In the latter method, one can use, e.g., the overlap-add or
in Fig. 9 and with a 16-QAM data, we can select Nc = 119 overlap-save methods, to reduce the implementation complex-
in (17) and still obtain a smaller BER as compared to the case ity [8], [17], [18]. The implementation complexity can also be
with N = 251 in (3). In case of Example 1 and with 16-QAM reduced in the time domain using, e.g., polyphase realizations
or multiple constant multiplication methods [8], [12]. Here, we
do not discuss such implementation issues as they (i) can be
9 In general, for high-spectral-efficiency modulation formats with small BER applicable to any given impulse response and (ii) do not affect
values, the values of  and E must be smaller. the filtering performance. Instead, this paper has focused on the
EGHBALI et al.: OPTIMAL LEAST-SQUARES FIR DIGITAL FILTERS FOR COMPENSATION OF CD IN DIGITAL COHERENT OPTICAL RECEIVERS 1455

Fig. 10. Simulated uncoded BER for 4-QAM data in the example of Fig. 11. Simulated uncoded BER for QAM data in the example of
, L = 2, and  = 10 −1 4 . , L = 2, and  = 10 −1 4 .
π (1 + ρ ) π (1 + ρ )
Section III-A3 with Ω 2 = −Ω 1 = L Section III-A3 with Ω 2 = −Ω 1 = L

number of filter taps per 1000 ps/nm of CD. Regardless of how format, like 4-QAM, both filters have a very good performance
the filter is implemented, it is always beneficial to reduce the although our proposed method needs fewer number of taps to
filter length from the design/implementation as well as the delay obtain the same BER as that of FSM. As discussed earlier, future
points of view. optical communication systems will move towards high-speed
traffic using high-spectral-efficiency modulation formats thus
A. Comparison With Filters Designed Using FSM necessitating efficient CD compensation filters. Fig. 11 com-
pares the simulated uncoded BER for filters obtained by our
A straightforward alternative, to our proposed optimization-
proposed method and FSM where we have considered some
based filter design method, is to use the inverse CD func-
very high-spectral-efficiency modulation formats. As can be
tion at N predefined uniformly-spaced frequencies ωk T
seen, our proposed method (with fewer number of taps) has
corresponding to those of a length-N discrete Fourier
clearly a better performance than FSM.
transform (DFT), i.e., ωk T = 2kNπ , k = 0, 1, . . . , N − 1. Then,
CD equalization is typically performed in the frequency
V. CONCLUSION
domain with a filter frequency response represented as
2
HFSM (k) = ej K (ω k T ) , k = 0, 1, . . . , N − 1. Still, there exists Optimal FIR digital filters, in the LS sense, for compensation
an underlying impulse response hFSM (n) where HFSM (k) = of CD were derived. For the same amount of CD, our proposed
N −1 −j n ω k T
n =0 hFSM (n)e , k = 0, 1, . . . , N − 1. This method filters outperform (especially for modulation formats having
corresponds to the so-called FSM which has a low design com- high spectral-efficiencies) the exiting ones as they require fewer
plexity. However, a drawback of FSM is that the CD is only (per- taps giving a lower implementation cost and delay. Design ex-
fectly) equalized at N predefined uniformly-spaced frequencies amples were provided for illustration and comparison. It was
ωk T . In such methods, there is an exact control of the system be- shown that considering the effects of pulse shaping filters, in
havior over these N predefined uniformly-spaced frequencies our proposed method, we can further reduce the implementa-
ωk T . Between these frequencies, the behavior of the system tion cost.
cannot be controlled. This is in contrast to our proposed method
which covers all frequencies through the integral in (6). REFERENCES
To illustrate this, let us revisit the example in [1] S. J. Savory, “Digital filters for coherent optical receivers,” Opt. Exp.,
Section III-A3. In this case, we can optimize our proposed fil- vol. 16, no. 2, pp. 804–817, 2008.
ters, as in (17). For FSM, we can obtain the underlying impulse [2] S. J. Savory, “Digital coherent optical receivers: Algorithms and subsys-
tems,” IEEE J. Sel. Topics Quantum Electron., vol. 16, no. 5, pp. 1164–
response hFSM (n) through the inverse DFT of HFSM (k). Then, 1179, Sep./Oct. 2010.
we can compute E, in (6), in the frequency band of interest. [3] D. Lavery, R. Maher, D. S. Millar, B. C. Thomsen, P. Bayvel, and
With N = Nc = 31, our proposed method gives E = −220 dB S. J. Savory, “Digital coherent receivers for long-reach optical access
networks,” J. Lightw. Technol., vol. 31, no. 4, pp. 609–620, Feb. 2013.
whereas the filter obtained by FSM gives E = −63 dB. As can [4] G. Goldfarb and G. Li, “Chromatic dispersion compensation using digital
be seen, our proposed method gives a smaller E which then al- IIR filtering with coherent detection,” IEEE Photon. Technol. Lett., vol. 19,
lows Nc to be reduced to obtain the same BER performance as no. 13, pp. 969–971, Jul. 2007.
[5] G. P. Agrawal, Fiber-Optic Communication Systems. New York, NY,
that of FSM. Fig. 10 compares the simulated uncoded BER for USA: Wiley, 2010.
filters obtained by our proposed method and FSM in the exam- [6] C. C. Davis and T. E. Murphy, “Fiber-optic communications,” IEEE Sig-
ple of Section III-A3. As can be seen, with a simple modulation nal Process. Mag., vol. 28, no. 4, pp. 152–150, Jul. 2011.
1456 JOURNAL OF LIGHTWAVE TECHNOLOGY, VOL. 32, NO. 8, APRIL 15, 2014

[7] P. Bower and I. Dedic, “High speed converters and DSP for 100G and Håkan Johansson (SM’06) received the Master of Science degree in computer
beyond,” Opt. Fiber Technol., vol. 17, no. 5, pp. 464–471, May 2011. science, and the Licentiate, Doctoral, and Docent degrees in electronics systems
[8] S. K. Mitra, Digital Signal Processing: A Computer Based Approach. from Linköping University, Linköping, Sweden, in 1995, 1997, 1998, and 2001,
New York, NY, USA: McGraw-Hill, Feb. 2006. respectively.
[9] M. Renfors and Y. Neuvo, “The maximum sampling rate of digital filters During 1998 and 1999, he held a Postdoctoral position at Signal Process-
under hardware speed constraints,” IEEE Trans. Circuits Syst., vol. 28, ing Laboratory, Tampere University of Technology, Tampere, Finland. He is
no. 3, pp. 196–202, Mar. 1980. currently a Professor of electronics systems in the Department of Electrical En-
[10] S. S. Kidambi and R. P. Ramachandran, “Complex coefficient nonrecur- gineering, Linköping University. His research encompasses theory, design, and
sive digital filter design using a least-squares method,” IEEE Trans. Signal implementation of efficient and flexible signal processing systems for various
Process., vol. 44, no. 3, pp. 710–713, Mar. 1996. purposes. He is one of the founders of the company Signal Processing Devices
[11] M. Abramowitz and I. A. Stegun, “Error function and Fresnel integrals,” Sweden AB that sells advanced signal processing solutions. He is the author
in Handbook of Mathematical Functions with Formulas, Graphs, and or coauthor of four books and some 170 international journal and conference
Mathematical Tables. New York, NY, USA: Dover, 1972, pp. 297–309. papers.
[12] P. P. Vaidyanathan, Multirate Systems and Filter Banks. Englewood Dr. Johansson is the coauthor of three papers that have received best paper
Cliffs, NJ, USA: Prentice-Hall, 1993. awards and he has authored one invited paper in IEEE Transactions and four
[13] A. Eghbali, T. Saramäki, and H. Johansson, “On two-stage Nyquist pulse invited chapters. He has served as an Associate Editor for IEEE TRANSACTION
shaping filters,” IEEE Trans. Signal Process., vol. 60, no. 1, pp. 483–488, ON CIRCUITS AND SYSTEMS I and II, IEEE TRANSACTION ON SIGNAL PROCESS-
Jan. 2012. ING, and IEEE Signal Processing Letters. He is currently an Associate Editor
[14] “Universal mobile telecommunications system (UMTS); base station (BS) of IEEE TRANSACTION ON CIRCUITS AND SYSTEMS I and the Area Editor of the
radio transmission and reception (FDD),” 3GPP TS 25.104 version 6.8.0 Elsevier Digital Signal Processing journal, and a Member of the IEEE Interna-
Release 6, Dec. 2004 tional Symposium on Circuits and Systems DSP track committee.
[15] A. Eghbali, H. Johansson, and P. Löwenborg, “A class of multimode
transmultiplexers based on the Farrow structure,” Circuits Syst. Signal
Process., vol. 31, no. 3, pp. 961–985, Jun. 2012.
[16] W. Shieh and K.-P. Ho, “Equalization-enhanced phase noise for coherent-
detection systems using electronic digital signal processing,” Opt. Exp.,
vol. 16, no. 20, pp. 15 718–15 727, 2008.
[17] J. G. Proakis and D. G. Manolakis, Digital Signal Processing: Principles,
Algorithms, and Applications, 4th ed. Upper Saddle River, NJ, USA: Oscar Gustafsson (S’98–M’03–SM’10) received the M.Sc., Ph.D., and Docent
Prentice-Hall, 2007. degrees from Linköping University, Linköping, Sweden, in 1998, 2003, and
[18] V. K. Madisetti and D. B. Walliams, Eds., The Digital Signal Processing 2008, respectively.
Handbook. Boca Raton, FL, USA: CRC Press, 1998. He is currently an Associate Professor and the Head of the Electronics Sys-
tems Division, Department of Electrical Engineering, Linköping University.
His research interests include design and implementation of DSP algorithms
and arithmetic circuits. He has authored and coauthored more than 140 papers
in international journals and conferences on these topics.
Dr. Gustafsson is a Member of the VLSI Systems and Applications and
the Digital Signal Processing technical committees of the IEEE Circuits and
Systems Society. Currently, he serves as an Associate Editor for the IEEE
TRANSACTIONS ON CIRCUITS AND SYSTEMS-PART II: EXPRESS BRIEFS AND
INTEGRATION, and the VLSI Journal. He has served and serves in various po-
sitions for conferences such as the International Symposium on Circuits and
Systems, International Workshop on Power and Timing Modeling, Optimiza-
tion and Simulation, PrimeAsia, Asilomar, Norchip, the European Conference
on Circuit Theory and Design, and the International Conference on Electronics
and Communication Systems.

Amir Eghbali (S’08–M’11) was born in Urmia, Iran, in 1980. He received the
Bachelor of Science degree from the Iran University of Science and Technology, Seb J. Savory (M’07–SM’11) received the M.Eng., M.A., and Ph.D. degrees
Tehran, Iran, in 2003, and the Master of Science, Licentiate, and Ph.D. degrees in engineering from the University of Cambridge, Cambridge, U.K., in 1996,
from Linköping University, Linköping, Sweden, in 2006, 2009, and 2010, all in 1999, and 2001, respectively, and the M.Sc. (Maths) degree in mathematics
electrical engineering, respectively. from the Open University, Milton Keynes, U.K., in 2007.
Since July 2011, he has been an Assistant Professor with the Department His interest in optical communications began in 1991, when he joined Stan-
of Electrical Engineering, Linköping University. Between November 2009 and dard Telecommunications Laboratories, Harlow, U.K., prior to being sponsored
January 2010, he was with the Signal Processing Algorithm Group, Department through his undergraduate and postgraduate studies, after which he rejoined
of Signal Processing, Tampere University of Technology, Tampere, Finland. Nortel’s Harlow Laboratories. In 2005, he joined the Optical Networks Group
He is the coauthor of the chapter titled “Reconfigurable Multirate Systems in at University College London, holding a Leverhulme Trust Early Career Fel-
Cognitive Radios” in the book Foundation of Cognitive Radio Systems. His lowship from 2005 to 2007, and was appointed a University Lecturer in 2007
main research interests include digital and multirate signal processing with and subsequently Reader in Optical Fibre Communication (OFC) in 2012. From
applications to communication systems, application specific integrated circuits, June 2009 to June 2010, he was also a Visiting Professor at the Politecnico di
and VLSI design. Torino, Torino, Italy. His current research interests include digital signal pro-
Dr. Eghbali received the Best Regular Paper Award in 2008 East-West cessing, optical transmission subsystems, systems and networks.
Design and Test Symposium. He was an invited author to 2011 IEEE Midwest Dr. Savory is the Editor-in-Chief of IEEE Photonics Technology Letters,
Symposium on Circuits and Systems; 2007 International Symposium on Image an IEEE Photonics Society representative on the Steering Committee of OFC
and Signal Processing and Analysis; 2011 European Conference on Circuit and will serve as a General Chair for OFC in 2015. He previously served as a
Theory and Design. He was also a Session Chair in 2013 IEEE Symposium Program Chair for OFC 2013, Signal Processing in Photonic Communications
on Circuits and Systems; 2011 European Conference on Circuit Theory and (SPPCom) 2012 and SPPCom 2013. In addition, he serves on the Editorial
Design. Currently, he is serving as an Editorial Board Member for Elsevier Board for IET Optoelectronics and the technical program committee for the
Digital Signal Processing journal. European Conference on Optical Communication.