Unit II

DISCRETE TIME RANDOM PROCESSES

Stationary random processes, Autocorrelation, Rational Power Spectra, Filters for generating
random Processes from white noise and inverse filter – AR, MA and ARMA processes –
relationship between autocorrelation and the filter parameters
Introduction

o Signals may be broadly classified into one of two types—deterministic and random.

o A deterministic signal is one that may be reproduced exactly with repeated measurements.

o The unit sample response of a linear shift-invariant filter is an example of a deterministic signal.

o A random signal, or random process, on the other hand, is a signal that is not repeatable in a predictable manner.

o Quantization noise produced by an A/D converter or a fixed-point digital filter is an example of a random process.

o Other examples include tape hiss or turntable rumble in audio signals, background clutter in radar images, speckle noise in synthetic
aperture radar (SAR) images, and engine noise in speech transmissions from the cockpit of an airplane.

o Some signals may be considered to be either deterministic or random, depending upon the application.

o Speech, for example, may be considered to be a deterministic signal if a specific speech waveform is to be processed or analyzed.

o However, speech may also be viewed as a random process if one is considering the ensemble of all possible speech waveforms in order
to design a system that will optimally process speech signals, in general.

o Many of the signals that we will be characterizing and analyzing will be random processes.

o Since a random process may only be described probabilistically or in terms of its average behavior, in this chapter we develop the
background that is necessary to understand how a random process may be described and how its statistical properties are affected by a
linear shift-invariant system.
Random Variable

o The concept of a random variable may best be illustrated by means of the following simple example.

o Given a fair coin that is equally likely to result in Heads or Tails when flipped, one would expect that if the coin were to be flipped repeatedly, then
the outcome would be Heads approximately half of the time and Tails the other half.

o For example, if flipping the coin 𝑁𝑇 times results in 𝑛𝐻 Heads and 𝑛𝑇 Tails, then, for 𝑁𝑇 large, we should find that

𝒏𝑯 𝒏𝑻
≈ 𝟎. 𝟓 ; ≈ 𝟎. 𝟓
𝑵𝑻 𝑵𝑻
o Therefore, with a fair coin there is an equal probability of getting either Heads or Tails and the following probability assignment is made for the two
possible experimental outcomes 𝑯 = {𝑯𝒆𝒂𝒅𝒔} and 𝑻 = {𝑻𝒂𝒊𝒍𝒔},
𝑷𝒓 𝑯 = 𝟎. 𝟓 ; 𝑷𝒓 𝑻 = 𝟎. 𝟓 → ①

o The set of all possible experimental outcomes is called the Sample Space or the Certain Event and the sample space, denoted by 𝜴, is always
assigned a probability of one 𝑷𝒓 𝜴 = 𝟏

o For the coin flipping experiment 𝜴 = {𝑯, 𝑻} and 𝑷𝒓 𝑯, 𝑻 = 𝟏

o Subsets of the sample space are called Events, and events consisting of a single element are called Elementary Events.

o For the coin tossing experiment there are only two elementary events, 𝝎𝟏 = 𝑯 ; 𝝎𝟐 = {𝑻}

o Given the coin flipping experiment, suppose that a real-valued variable, 𝒙, is defined in such a way that a value of 𝟏 is assigned to 𝑥 if the outcome of
a coin flip is Heads and a value of −𝟏 is assigned if the outcome is Tails.

o With this definition, a mapping has be defined between the set of experimental outcomes, 𝝎𝒊 ∈ 𝜴, and the set of real number, 𝑹, 𝑖. 𝑒. , 𝒇: 𝜴 → 𝑹.
o This mapping is given by 𝝎𝟏 = 𝑯 ⟹ 𝒙 = 𝟏 𝒂𝒏𝒅 𝝎𝟐 = 𝑻 ⟹ 𝒙 = −𝟏
o Based on the probability assignments defined for the elementary events Heads and Tails in ①, it follows that there is an equal probability that the
variable 𝑥 will be assigned a value of 1 or −1
𝑷𝒓 {𝒙 = 𝟏} = 𝟎. 𝟓 & 𝑷𝒓 𝒙 = −𝟏 = 𝟎. 𝟓 → ②
o Since the only two possible outcomes of the coin flip are Heads and Tails, then the only possible values that may be assigned to the variable 𝑥 and
𝑥 = 1 and 𝑥 = −1.
o Therefore, for any number 𝛼 that is different from 1 or −1, the probability that 𝑥 = 𝛼 is equal to zero
𝑷𝒓 𝒙 = 𝜶 = 𝟎 𝒊𝒇 𝜶 ≠ ±𝟏
o And the probability that 𝑥 assumes one of the two values, 𝑥 = 1 𝑜𝑟 𝑥 = −1, is equal to one
𝑷𝒓 𝜴 = 𝑷𝒓 𝒙 = ±𝟏 = 𝟏
o With this probability assignment on each of the elementary events in the sample space 𝜴, and with the mapping of each elementary event 𝜔𝑖 ∈ Ω
to a value of 𝒙, a Random Variable has been defined that is specified in terms of the likelihood (probability) that it assumes a particular value.
o This random variable has an equal probability of having a value of 𝟏 or −𝟏.
o More generally, a random variable 𝒙 that may assume only one of two values, 𝒙 = 𝟏 𝒂𝒏𝒅 𝒙 = −𝟏, with 𝑷𝒓 {𝒙 = 𝟏} = 𝒑 & 𝑷𝒓 𝒙 = −𝟏 = 𝟏 − 𝒑 is
referred to as Bernoulli Random Variable.
o A slightly more complicated random variable may be defined by replacing the coin tossing experiment with the roll of a fair dice.
o Specifically, if the number that is obtained with the roll of a fair die is assigned to the variable 𝒙, then 𝒙 will be a random variable that is equally likely
to take on any integer value from 𝟏 𝒕𝒐 𝟔.
o In a similar fashion, a complex random variable may be defined by assigning complex numbers to elementary events in the sample space.
Ensemble Averages
o A complete statistical characterization of a random variable requires that it be possible to determine the probability of any event that may be defined
on the sample space.
o In many applications, however, a complete statistical characterization of a random variable may not be necessary if the average behavior of the
random variable is known.
o For example, in making the decision whether or not to play blackjack, it is the expected rate of return on a hand that is of interest rather than a
complete statistical characterization of the game.
o Let 𝒙 be the random variable that was defined on the die experiment. Suppose that the die is rolled 𝑁𝑇 times and the number 𝑘 appears 𝑛𝑘 times.
Computing the average value that is rolled by summing all of the numbers and dividing by the total number of times that the die is rolled yields the
sample mean

𝒏𝟏 + 𝟐𝒏𝟐 + 𝟑𝒏𝟑 + 𝟒𝒏𝟒 + 𝟓𝒏𝟓 + 𝟔𝒏𝟔

𝒙 𝑵𝑻 =
𝑵𝑻

o If 𝑁𝑇 ≫ 1, then 𝑛𝑘 𝑁𝑇 ≈ 𝑃𝑟 {𝑥 = 𝑘} and above equation becomes

𝒙 𝑵𝑻 = 𝒌𝑷𝒓 {𝒙 = 𝒌}
𝒌=𝟏
o The Mean or Expected Value of a discrete random variable 𝒙 that assumes a value of 𝛼𝑘 with probability 𝑃𝑟 {𝑥 = 𝑘} is therefore defined as

𝑬𝒙 = 𝜶𝒌 𝑷𝒓 {𝒙 = 𝜶𝒌 }
𝒌
o The expected value of 𝒙𝟐 is an important statistical average referred to as the Mean Square Value.
o The mean square value is frequently used as a measure for the quality of an estimate.
o For example, in developing an estimator for a random variable 𝑥, it is common to find the estimator 𝑥 that minimized the
mean square error,
𝝃 = { 𝒙 − 𝒙 𝟐}
o A related statistical average is the Variance which is the mean square value of the random variable 𝒙 − 𝑬{𝒙}, denoted either
by 𝑽𝒂𝒓{𝒙} or 𝝈𝒙𝟐 , the variance is
o The square root of the variance, 𝝈𝒙, is the Standard Deviation. For complex random variables, the mean square value is
𝑬{𝒛𝒛∗ ) = 𝑬{|𝒛|𝟐} and the variance is
𝝈𝒙𝟐 = 𝑬 𝒛 − 𝑬{𝒛} 𝒛∗ − 𝑬{𝒛∗ } = 𝑬 𝒛 − 𝑬{𝒛} 𝟐

o It is clear from the above equation that the expectation is a linear operator since, for any two random variables 𝑥 and 𝑦 and
for any constants 𝑎 and 𝑏,
𝑬 𝒂𝒙 + 𝒃𝒚 = 𝒂𝑬 𝒙 + 𝒃𝑬{𝒚}
o Using the linearity of the expectation operator, the variance may be expressed as
𝟐
𝑽𝒂𝒓 𝒙 = 𝑬 𝒙 − 𝑬{𝒙} = 𝑬 𝒙𝟐 − 𝑬𝟐 {𝒙}

o Therefore, it follows that if the mean of 𝒙 is zero, then 𝑽𝒂𝒓{𝒙} = 𝑬{𝒙𝟐}.

Random Processes
o Just as a random variable may be thought of as a mapping from a sample space of a given experiment into the set of
real or complex numbers, a discrete-time random process is a mapping from the sample space Ω into the set of
discrete-time signals 𝑥(𝑛).
o Thus, a discrete-time random process is a collection, or Ensemble, of discrete-time signals.
Ensemble Averages
o Since a discrete time random process is an indexed sequence of random variables, we may calculate the mean of
each of these random variables and generate the (deterministic) sequence
𝒎𝒙 (𝒏) = 𝑬{𝒙(𝒏)}
o known as the Mean of the process. Similarly, computing the variance of each random variable in the sequence
𝝈𝒙𝟐 (𝒏) = 𝑬 𝒙 𝒏 − 𝑚𝑥 (𝑛) 𝟐

o defines the Variance of the process.

o These first order statistics represent ensemble averages and both, in general, depend upon 𝑛.
o Whereas the Mean defines the average value of the process as a function of 𝒏, the Variance represents the
average squared deviation of the process away from the mean.
o Two additional ensemble averages that are important in study of random processes are the Autocovariance
∗
𝒄𝒙 (𝒌, 𝒍) = 𝑬 𝒙 𝒌 − 𝒎𝒙 (𝒌) 𝒙 𝒍 − 𝒎𝒙 (𝒍)

o and the Autocorrelation

𝒓𝒙 (𝒌, 𝒍) = 𝑬 𝒙 𝒌 𝒙∗ (𝒍)

o relating the random variables 𝑥(𝑘) and 𝑥(𝑙). Note that 𝑘 = 𝑙 then the autocovariance function reduces to variance
𝒄𝒙 (𝒌, 𝒌) = 𝝈𝒙𝟐 (𝒌)

o Note also that if the product in autocovariance expression is expanded, then it follows that the autocovariance and
autocorrelation sequences and related by
𝒄𝒙 𝒌, 𝒍 = 𝒓𝒙 𝒌, 𝒍 − 𝒎𝒙 (𝒌)𝒎𝒙∗ (𝒍)

o Thus, for zero mean random process the autocovariance and autocorrelation are equal.

o The autocorrelation and autocovariance provide information about the degree of linear dependence between two
random variables.

o For example, if 𝒄𝒙 𝒌, 𝒍 = 𝟎 for 𝑘 ≠ 𝑙, then the random variables 𝑥(𝑘) and 𝑥(𝑙) are uncorrelated and knowledge of
one does not help in the estimation of the other using a linear estimator.
o The autocovariance and autocorrelation sequences provide information about the statistical relationship between the two random
variables that are derived from the same process, e.g., 𝑥(𝑘) and 𝑥(𝑙).
o In applications involving more than one random process it is often of interest to determine the covariance or the correlation between a
random variable in one process, 𝑥(𝑘), and a random variable in another, 𝑦(𝑙).
o Given two random processes, 𝑥(𝑛) and 𝑦(𝑛), the cross-covariance is defined by
𝒄𝒙𝒚 (𝒌, 𝒍) = 𝑬 𝒙 𝒌 − 𝒎𝒙 (𝒌) [𝒚 𝒍 − 𝒎𝒚 (𝒍)]∗

o And the cross-correlation by

𝒓𝒙𝒚 (𝒌, 𝒍) = 𝑬 𝒙 𝒌 𝒚∗ (𝒍)

o These two functions satisfy the relation

𝒄𝒙𝒚 𝒌, 𝒍 = 𝒓𝒙𝒚 𝒌, 𝒍 − 𝒎𝒙 (𝒌)𝒎𝒚∗ (𝒍)

o Just as a pair of random variables are said to be uncorrelated if 𝒄𝒙𝒚 = 𝟎, two random processes 𝑥(𝑛) and 𝑦(𝑙) are said to be
uncorrelated if 𝒄𝒙𝒚 𝒌, 𝒍 = 𝟎
o For all 𝑘 and 𝑙 or, equivalently, if
𝒓𝒙𝒚 (𝒌, 𝒍) = 𝒎𝒙 (𝒌)𝒎𝒚∗ (𝒍)

o Two random processes 𝑥(𝑛) and 𝑦(𝑛) are said to be orthogonal if their cross-correlation is zero 𝒓𝒙𝒚 (𝒌, 𝒍) = 𝟎
o Although orthogonal random processes are not necessarily uncorrelated, zero mean processes that are uncorrelated are orthogonal.
Property of Autocorrelation
o If two random processes 𝑥(𝑛) and 𝑦(𝑛) are uncorrelated, then the autocorrelation of the sum 𝑧(𝑛) = 𝑥(𝑛) + 𝑦(𝑛) is equal to the sum
of the autocorrelations of 𝑥(𝑛) and 𝑦(𝑛),
𝒓𝒛 (𝒌, 𝒍) = 𝒓𝒙 (𝒌, 𝒍) + 𝒓𝒚 (𝒌, 𝒍)
Stationary Random Processes
o In signal processing applications, the statistics or ensemble averages of a random process are often independent of time.
o For example, quantization noise that results from roundoff errors in a fixed point digital signal processor typically has a constant mean
and a constant variance whenever the input signal is “sufficiently complex.”
o In addition, it is often assumed that the quantization noise has first and second-order probability density functions that are independent
of time. These conditions are examples of “statistical time-invariance” or stationarity.
o A stationarity assumption is important for the estimation of ensemble averages.
o If the first-order density function of a random process 𝑥(𝑛) is independent of time, i.e.,
𝒇𝒙(𝒏) 𝜶 = 𝒇𝒙(𝒏+𝒌) 𝜶
o for all k, then the process is said to be First-order Stationary. For a first-order stationary process, the first-order statistics will be
independent of time.
o For example, the Mean of the process will be constant
𝒎𝒙 𝒏 = 𝒎𝒙
o and the same will be true of the Variance, 𝝈𝒙𝟐 𝒏 = 𝝈𝒙𝟐 .
o Similarly, a process is said to be Second-order stationary if the second-order joint density function
𝒇𝒙 𝒏 𝟏 ,𝒙 𝒏𝟐 𝜶𝟏 , 𝜶𝟐 = 𝒇𝒙 𝒏𝟏 +𝒌 ,𝒙 𝒏𝟐 +𝒌 𝜶𝟏 , 𝜶𝟐

o If a process is second-order stationary, then it will also be first-order stationary.

o In addition, second-order stationary processes have second-order statistics that are invariant to a time shift of the
process.
o For example, it follows that the autocorrelation sequence has the property

∞ ∞

𝒓𝒙 𝒌, 𝒍 = 𝜶𝜷𝒇𝒙 𝒌 ,𝒙 𝒍 (𝜶, 𝜷) 𝒅𝜶 𝒅𝜷 = 𝜶𝜷𝒇𝒙 𝒌+𝒏 ,𝒙 𝒍+𝒏 (𝜶, 𝜷) 𝒅𝜶 𝒅𝜷 = 𝒓𝒙 𝒌 + 𝒏, 𝒍 + 𝒏

−∞ −∞

o Therefore, the correlation between the random variables 𝑥(𝑘) and 𝑥(𝑙) depends only on the difference, 𝑘 − 𝑙,
separating the two random variables in time
𝒓𝒙 𝒌, 𝒍 = 𝒓𝒙 𝒌 − 𝒍, 𝟎
o This difference, k-l, is called lag, and with a slight abuse in notation, we will drop the second argument and write
𝑟𝑥 𝑘, 𝑙 simply as a function of the lag,
𝒓𝒙 𝒌, 𝒍 ≡ 𝒓𝒙 𝒌 − 𝒍
o Continuing to higher order joint density functions, a process is said to be stationary of order L if the processes 𝑥(𝑛) and 𝑥(𝑛 + 𝑘) have
the same 𝐿𝑡ℎ-order joint density functions.
o Finally, a process that is stationary for all orders 𝐿 > 0 is said to be stationary in the strict sense.
o Wide Sense Stationarity – A random process 𝑥(𝑛) is said to be wide-sense stationary if the following three conditions are satisfied:
o The mean of the process is a constant, 𝑚𝑥 (𝑛) = 𝑚𝑥
o The autocorrelation 𝑟𝑥 𝑘, 𝑙 depends only on the difference 𝑘 − 𝑙 .
o The variance of the process is finite, 𝑐𝑥 0 < ∞
o Since constraints are placed on ensemble averages rather than on density functions, wide-sense stationarity is a weaker constraint than
second-order stationarity.
o However, in the case of a Gaussian process, wide sense stationarity is equivalent to strict-sense stationarity.
o This is a consequence of the fact that a Gaussian random process is completely defined in terms of the mean and covariance.
o In the case of two or more processes, similar definitions exists for joint stationarity. For example, two processes 𝑥(𝑛) and 𝑦(𝑛) are said
to be jointly wide-sense stationary if 𝑥(𝑛) and 𝑦(𝑛) are wide-sense stationary and if the cross correlation 𝑟𝑥 𝑘, 𝑙 depends only on the
difference 𝑘 − 𝑙 ,
𝑟𝑥𝑦 𝑘, 𝑙 = 𝑟𝑥𝑦 𝑘 + 𝑛, 𝑙 + 𝑛

o Again, for jointly WSS processes, we will write the cross-correlation as function only of the lag, 𝑘 − 𝑙 , as follows,
𝑟𝑥𝑦 𝑘 − 𝑙 = 𝐸{𝑥 𝑘 𝑦 ∗ (𝑙)}
o The autocorrelation sequence of a WSS process has a number of useful and important properties.

o Property 1 – Symmetry – The autocorrelation sequence of a WSS random process is a conjugate symmetric function of 𝑘,
𝒓𝒙 (𝒌) = 𝒓𝒙∗ −𝒌

o For a real process, the autocorrelation sequence is symmetric

𝒓𝒙 (𝒌) = 𝒓𝒙 −𝒌

o Property 2 – Mean-square value – The autocorrelation sequence of a WSS process at lag 𝑘 = 0 is equal to the mean-square
value of the process
𝟐
𝒓𝒙 (𝟎) = 𝑬 𝒙(𝒏) ≥𝟎

o Property 3 – Maximum Value – The magnitude of the autocorrelation sequence of a WSS random process at lag 𝑘 is upper
bounded by its value at 𝑘 = 0
𝒓𝒙 (𝟎) ≥ 𝒓𝒙 (𝒌)

o Property 4 – Periodicity – If the autocorrelation sequence of a WSS random process is such that 𝑟𝑥 (𝑘0 ) = 𝑟𝑥 (0) for some 𝑘0 ,
then 𝑟𝑥 (𝑘) is periodic with period 𝑘0 . Furthermore
𝟐
𝑬 𝒙 𝒏 − 𝒙(𝒏 − 𝒌𝟎 ) =𝟎

o and 𝑥(𝑛) is said to be mean-square periodic.

Rational Power Spectra

o Fourier analysis is an important tool in the description and analysis of deterministic discrete time signals, it also plays
an important role in the study of random processes.

o However, since a random process is an ensemble of discrete time signals, we cannot compute the Fourier transform
of the process itself.

o It is possible to develop a frequency domain representation of the process if we express the Fourier transform in
terms of an ensemble average.

o The autocorrelation sequence of a WSS process provides a time domain description of the second order moment of
the process.

o Since 𝑟𝑥(𝑘) is a deterministic sequence we may compute its DFT,

𝑷𝒙 𝒆𝒋𝝎 = 𝒓𝒙 (𝒌)𝒆−𝒋𝒌𝝎 → ①
𝒌=−∞

o Which is called the Power Spectrum or Power Spectral Density of the process.
o Given the power spectrum, the autocorrelation sequence may be determined by taking the inverse DFT of 𝑃𝑥 𝑒 𝑗𝜔 ,

𝝅
𝟏
𝒓𝒙 𝒌 = 𝑷𝒙 𝒆𝒋𝝎 𝒆𝒋𝒌𝝎 𝒅𝝎 → ②
𝟐𝝅
−𝝅

o Thus, the power spectrum provided a frequency domain description of the second order moment of the process.
o In some cases it may be more convenient to use the z-transform instead of the DFT, in which case

𝑷𝒙 𝒛 = 𝒓𝒙 (𝒌)𝒛−𝒌 → ③
𝒌=−∞

o will also be referred to as the Power Spectrum of 𝑥(𝑛).

o Just as with the autocorrelation sequence, there are a number of properties of the power spectrum that will be
useful.
o Since the autocorrelation of a WSS random process is conjugate symmetric, it follows that the power spectrum will
be a real valued function of 𝜔. In the case of real valued random process, the autocorrelation sequence is real and
even, which implies that the power spectrum is real and even. Thus we have symmetry property of power spectrum.
Properties of Power Spectrum
∗
o Property 1 – Symmetry – The power spectrum of a WSS random process 𝑥(𝑛) is real valued, 𝑃𝑥 𝑒 𝑗𝜔 = 𝑃𝑥 𝑒 𝑗𝜔 , and 𝑃𝑥 𝑧
satisfies the symmetry condition
∗
𝑷𝒙 𝒛 = 𝑷𝒙 𝟏/𝒛∗

o In addition, if 𝑥(𝑛) is real then the power spectrum is even, 𝑃𝑥 𝑒 𝑗𝜔 = 𝑃𝑥 𝑒 −𝑗𝜔 , which implies that
∗
𝑷𝒙 𝒛 = 𝑷𝒙 𝒛∗
o Property 2 – Positivity – The power spectrum of a WSS random process is non-negative
𝑷𝒙 𝒆𝒋𝝎 ≥ 𝟎
o Property 3 – Total Power – The power in a zero mean WSS random process is proportional to the area under the power
spectral density curve

𝝅
𝟐
𝟏
𝑬 𝒙(𝒏) = 𝑷𝒙 𝒆𝒋𝝎 𝒅𝝎
𝟐𝝅
−𝝅
o Property 4 – Eigenvalue External Property – The eigenvalues of the n x n autocorrelation matrix of a zero mean WSS random
process are upper and lower bounded by the maximum and minimum values, respectively, of the powwr spectrum.
𝒎𝒊𝒏 𝒎𝒂𝒙
𝑷𝒙 𝒆𝒋𝝎 ≤ 𝝀𝒊 ≤ 𝑷 𝒆𝒋𝝎
𝝎 𝝎 𝒙
Filters for generating random Processes from white noise and inverse filter

o Some random processes may be generated by filtering white noise with a linear shift-invariant filter that has a
rational system function.

o These include the autoregressive, moving average, and autoregressive moving average processes.

o Of interest will be the form of the autocorrelation sequence and the power spectrum of these processes. We will
show, for example, that the autocorrelation sequences of these processes satisfy a set of equations, known as the
Yule-Walker equations, relating 𝒓𝒙(𝒌) to the parameters of the filter.

Autoregressive Moving Average Processes (ARMA)

o Suppose that we filter white noise 𝑣(𝑛) with a causal linear shift-invariant filter having a rational system function
with 𝑝 poles and 𝑞 zeros

𝒒 −𝒌
𝑩𝒒 (𝒛) 𝒌=𝟎 𝒃𝒒 (𝒌)𝒛
𝑯 𝒛 = = → ❶
𝑨𝒑 (𝒛) 𝟏 + 𝒑𝒌=𝟏 𝒂𝒑 (𝒌)𝒛−𝒌
o Assuming that the filter is stable, the output process 𝑥(𝑛) will be wide-sense stationary and, with 𝑃𝑣 (𝑧) = 𝜎𝑣 2 , the power
spectrum will be

𝟐
𝑩𝒒 𝒛 𝑩𝒒 ∗ (𝟏/𝒛∗ )
𝑷𝒙 (𝒛) = 𝝈𝒗
𝑨𝒑 (𝒛)𝑨𝒑 ∗ (𝟏/𝒛∗ )

o Or in terms of 𝜔,

𝟐
𝑩𝒒 𝒆𝒋𝝎
𝑷𝒙 𝒆𝒋𝝎 = 𝝈𝒗𝟐 𝟐
→ ❷
𝑨𝒑 (𝒆𝒋𝝎
o A process having a power spectrum of this form is known as an Autoregressive Moving Average Process of order (𝑝, 𝑞) and is
referred to as an 𝐴𝑅𝑀𝐴(𝑝, 𝑞) process.

o Since 𝑥(𝑛) and 𝑣(𝑛) are related by the linear constant coefficient difference equation

𝒑 𝒒

𝒙 𝒏 = 𝒂𝒑 (𝒍)𝒓𝒙 (𝒌 − 𝒍) = 𝒃𝒒 𝒍 𝒗(𝒏 − 𝒍) → ❸
𝒍=𝟏 𝒍=𝟎

o It follows that the autocorrelation of 𝑥(𝑛) and the cross correlation between 𝑥(𝑛) and 𝑣(𝑛) satisfy the same difference
equation.
o To see this, note that if we multiply both sides of ❸ by 𝒙∗ (𝒏 − 𝒌) and take the expected value, then we have

𝒑 𝒒

𝒓𝒙 𝒌 + 𝒂𝒑 𝒍 𝒓𝒙 𝒌 − 𝒍 = 𝒃𝒒 𝒍 𝑬 𝒗 𝒏 − 𝒍 𝒙∗ 𝒏 − 𝒌 → ❹
𝒍=𝟏 𝒍=𝟎

o Since v(n) is WSS, it follows that v(n) and x(n) are jointly WSS and
𝑬 𝒗 𝒏 − 𝒍 𝒙∗ 𝒏 − 𝒌 = 𝒓𝒗𝒙 𝒌 − 𝒍
o Therefore ❹ becomes

𝒑 𝒒

𝒓𝒙 𝒌 + 𝒂𝒑 𝒍 𝒓𝒙 𝒌 − 𝒍 = 𝒃𝒒 𝒍 𝒓𝒗𝒙 𝒌 − 𝒍 → ❺
𝒍=𝟏 𝒍=𝟎

o Which is a difference equation of the same form as ❸.

o In its present form, this difference equation is not very useful. However, by writing the cross correlation 𝒓𝒗𝒙 𝒌 in terms of
autocorrelation 𝒓𝒙 𝒌 and the unit sample response of the filter, we may derive a set of equations relating the
autocorrelation of 𝑥(𝑛) to the filter. Since

𝒙 𝒏 =𝒉 𝒏 ∗𝒗 𝒏 = 𝒗 𝒎 𝒉(𝒏 − 𝒎)
𝒎=−∞
o The cross correlation may be written as

∞ ∞

𝑬 𝒗 𝒏 − 𝒍 𝒙∗ 𝒏 − 𝒌 =𝑬 𝒗 𝒏 − 𝒍 𝒗∗ (𝒎)𝒉∗ (𝒏 − 𝒌 − 𝒎) = 𝑬 𝒗 𝒏 − 𝒍 𝒗∗ (𝒎) 𝒉∗ (𝒏 − 𝒌 − 𝒎)
𝒎=−∞ 𝒎=−∞
𝑬 𝒗 𝒏 − 𝒍 𝒙∗ 𝒏 − 𝒌 = 𝝈 𝒗 𝟐 𝒉∗ 𝒍 − 𝒌 → ❻

o Where the last equality follows from the fact that 𝑬 𝒗 𝒏 − 𝒍 𝒗∗ (𝒎) = 𝝈𝒗 𝟐𝜹 𝒏 − 𝒍 − 𝒎 , i.e., 𝑣(𝑛) is white noise.

o Substituting ❻ in ❺ gives

𝒑 𝒒

𝒓𝒙 𝒌 + 𝒂 𝒑 𝒍 𝒓𝒙 𝒌 − 𝒍 = 𝝈 𝒗 𝟐 𝒃𝒒 𝒍 𝒉∗ 𝒍 − 𝒌 → ❼
𝒍=𝟏 𝒍=𝟎

o Assuming that ℎ(𝑛) is causal, the sum on the right side of ❼, which we denote by 𝑐𝑞 (𝑘), may be written as

𝒒 𝒒−𝒌

𝒄𝒒 𝒌 = 𝒃𝒒 𝒍 𝒉∗ 𝒍 − 𝒌 = 𝒃𝒒 𝒍 + 𝒌 𝒉∗ 𝒍 → ❽
𝒍=𝒌 𝒍=𝟎
o Since 𝒄𝒒 𝒌 = 𝟎 for 𝑘 > 𝑞, we may write ❼ for 𝑘 ≥ 0, as follows

𝒑
𝝈𝒗 𝟐𝒄𝒒 𝒌 ; 𝟎≤𝒌≤𝒒
𝒓𝒙 𝒌 + 𝒂𝒑 𝒍 𝒓𝒙 𝒌 − 𝒍 = → ❾
𝟎 ; 𝒌>𝒒
𝒍=𝟏

o Which are Yule-Walker equations. Writing these equation for 𝑘 = 0,1, … , 𝑝 + 𝑞 in matrix form we have

→ ❿

o Note that ❾ defines a recursion for the autocorrelation sequence in terms of the filter coefficients,
𝑎𝑝 𝑘 𝑎𝑛𝑑 𝑏𝑞 (𝑘).

o Therefore, Yule-Walker equations may be used to extrapolate the autocorrelation sequence from a finite set of
values of 𝑟𝑥 (𝑘).
o For example, if 𝑝 ≥ 𝑞 and if 𝑟𝑥 0 , … , 𝑟𝑥 𝑝 − 1 are known, then 𝑟𝑥 𝑘 for k ≥ 𝑝 may be computed recursively using
the difference equation

𝒓𝒙 𝒌 = − 𝒂𝒑 (𝒍)𝒓𝒙 (𝒌 − 𝒍)
𝒍=𝟏

o Note that the Yule-Walker equations also provide a relationship between the filter coefficients and the
autocorrelation sequence.

o Thus, ❾ may be used to estimate the filer coefficients, 𝒂𝒑 𝒌 𝑎𝑛𝑑 𝒃𝒒 (𝒌), from the autocorrelation sequence 𝒓𝒙 𝒌 .

o However, due to the product 𝒉∗ (𝒍)𝒃𝒒 (𝒌 + 𝒍) that appears in ❽, the Yule-Walker equations are nonlinear in the filter
coefficients and solving them for the filter coefficients is, in general, difficult.

o Yule-Walker equations are important in problems such as signal modelling and spectrum estimation.

o There are two special cases of ARMA processes. The first is when 𝒒 = 𝟎 (Autoregressive Process), and the second is
when 𝒑 = 𝟎 (Moving Average Process).
Autoregressive Process

o A special type of 𝐴𝑅𝑀𝐴(𝑝, 𝑞) process results when 𝑞 = 0 in ❶, i.e., when 𝐵𝑞 𝑧 = 𝑏(0). In this case 𝑥(𝑛) is
generated by filtering white noise with an all-pole filter of the form

𝒃(𝟎)
𝑯 𝒛 = 𝒑 −𝒌
→ ⓫
𝟏+ 𝒌=𝟏 𝒂𝒑 (𝒌)𝒛

o An 𝐴𝑅𝑀𝐴(𝑝, 0) process is called an Autoregressive Process of order 𝑝 and will be referred to as an 𝐴𝑅(𝑝) process.
o If 𝑃𝑣 (𝑧) = 𝜎𝑣 2 , then the power spectrum of 𝑥(𝑛) is

𝟐
𝒃 𝟎
𝑷𝒙 𝒛 = 𝝈𝒗𝟐 → ⓬
∗ 𝟏
𝑨𝒑 𝒛 𝑨𝒑
𝒛∗
o Or in terms of frequency variable 𝜔,

𝟐
𝒋𝝎 𝟐
𝒃 𝟎
𝑷𝒙 𝒆 = 𝝈𝒗 𝟐
→ ⓭
𝑨𝒑 (𝒆𝒋𝝎 )
o Thus, power spectrum of 𝐴𝑅(𝑝) process contain 2𝑝 poles and no zeros (except those located at 𝑧 = 0 & 𝑧 = ∞).
o The Yule-Walker equations for an 𝐴𝑅(𝑝) process may be found from ❾ by setting 𝑞 = 0. With 𝑐0 (0) = 𝑏(0)ℎ∗ (0) =
𝑏(0) 2 these equations are

𝒓𝒙 𝒌 + 𝒂𝒑 𝒍 𝒓𝒙 𝒌 − 𝒍 = 𝝈𝒗𝟐 𝒃 𝟎 𝟐
𝜹 𝒌 ; 𝒌≥𝟎 → ⓮
𝒍=𝟏

o which, in matrix form become

𝒓𝒙 (𝟎) 𝒓𝒙 −𝟏 … 𝒓𝒙 (−𝒑) 𝟏 𝟏
𝒓𝒙 (𝟏) 𝒓𝒙 𝟎 … 𝒓𝒙 (−𝒑 + 𝟏) 𝒂𝒑 (𝟏) 𝟎
. . . . .
. = 𝝈𝒗𝟐 𝒃(𝟎) 𝟐
→ ⓯
. . . .
. . . . .
𝒓𝒙 (𝒑) 𝒓𝒙 𝒑 = 𝟏 … 𝒓𝒙 (𝟎) 𝒂𝒑 (𝒑) 𝟎

o Note that since the Yule-Walker equations are linear in the coefficients 𝑎𝑝 (𝑘), it is a simple matter to find the

coefficients 𝑎𝑝 (𝑘) from the autocorrelation sequence 𝑟𝑥 (𝑘).

o For example, suppose that we are given the first two autocorrelation values of a real-valued 𝐴𝑅(1) process. Assuming that
𝝈𝒗𝟐 = 𝟏, using the property that 𝑟𝑥 𝑘 = 𝑟𝑥 (−𝑘) for real processes the Yule-Walker equations for 𝑘 = 0, 1 are as follows:
𝒓𝒙 𝟎 + 𝒓𝒙 𝟏 a(1)=𝒃𝟐 𝟎
𝒓𝒙 𝟎 𝒂 𝟏 = −𝒓𝒙 𝟏
o From the second equation, we find for 𝑎(1)

𝒓𝒙 (𝟏)
𝒂 𝟏 =−
𝒓𝒙 (𝟎)
o And, from the first equation, we may solve for b(0),

𝟐
𝒓 𝒙 𝟐 𝟎 − 𝒓𝒙 𝟐 𝟏
𝒃 𝟎 = 𝒓𝒙 𝟎 + 𝒓𝒙 𝟏 𝒂 𝟏 =
𝒓𝒙 𝟎
o Expressed in terms of 𝑟𝑥 0 and 𝑟𝑥 1 , a first-order filter that generates an 𝐴𝑅(1) process with the given autocorrelation
values is

𝒓𝒙 𝟎 𝒓𝒙 𝟐 𝟎 − 𝒓𝒙 𝟐 𝟏
𝑯 𝒛 =
𝒓𝒙 𝟎 − 𝒓𝒙 𝟏 𝒛−𝟏
o In similar fashion, we may use the Yule-Walker equations to generate the autocorrelation sequence from a given set of filter
coefficients.
o For example, again let 𝑥(𝑛) be a first order autoregressive process. Writing the first two Yule-Walker equations in matrix form in terms of
the unknowns 𝒓𝒙 𝟎 and 𝒓𝒙 𝟏 we have

𝟏 𝒂(𝟏) 𝒓𝒙 (𝟎) 𝟐
= 𝒃 (𝟎)
𝒂(𝟏) 𝟏 𝒓𝒙 (𝟏) 𝟎
o Solving for 𝑟𝑥 (0) and 𝑟𝑥 (1) we find

𝒃𝟐 (𝟎)
𝒓𝒙 𝟎 =
𝟏 − 𝒂𝟐 (𝟏)

𝒃𝟐 (𝟎)
𝒓𝒙 𝟏 = −𝒂 𝟏 𝒓𝒙 𝟎 = −𝒂(𝟏)
𝟏 − 𝒂𝟐 (𝟏)

o Now observe from ⓮ that for all 𝑘 > 0

𝒓𝒙 𝒌 = −𝒂 𝟏 𝒓𝒙 𝒌 − 𝟏
o Therefore, the autocorrelation sequence for 𝑘 ≥ 0 may be written as

𝒃𝟐 (𝟎) 𝒌
𝒓𝒙 𝒌 = −𝒂(𝟏) ;𝒌 ≥ 𝟎
𝟏 − 𝒂𝟐 (𝟏)

o Finally using the symmetry of 𝑟𝑥 𝑘 gives

𝒃𝟐 (𝟎) 𝒌
𝒓𝒙 𝒌 = −𝒂(𝟏) → ⓰
𝟏 − 𝒂𝟐 (𝟏)
Moving Average Processes

o Another special case of 𝐴𝑅𝑀𝐴(𝑝, 𝑞) process results when 𝑝 = 0. with 𝑝 = 0, 𝑥(𝑛) is generated by filtering white
noise with an FIR filter that has a system function of the form

𝑯 𝒛 = 𝒃𝒒 (𝒌)𝒛−𝒌
𝒌=𝟎

o i.e., 𝐴𝑝(𝑧) = 1 in ❶. An 𝐴𝑅𝑀𝐴(0, 𝑞) process is known as Moving Average process of order 𝑞 and will be referred to
as an 𝑀𝐴(𝑞) process.

o We know that if 𝑷𝒗 (𝒛) = 𝝈𝒗𝟐 , then the power spectrum is

∗ 𝟏
𝑷𝒙 𝒛 = 𝝈𝒗𝟐 𝑩𝒒 𝒛 𝑩𝒒 → ⓱
𝒛∗

o Or in terms of 𝜔,
𝟐
𝑷𝒙 𝒆𝒋𝝎 = 𝝈𝒗𝟐 𝑩𝒒 𝒆𝒋𝝎 → ⓲

o Therefore, 𝑃𝑥 𝑧 contain 2𝑞 zeros and no poles (except those that lie at 𝑧 = 0 and 𝑧 = ∞).
o The Yule-Walker equations for an 𝑀𝐴(𝑞) process may be found by simply taking the inverse z-transform of ⓱.
Alternatively, they may be found from by ❾ setting 𝑎𝑝(𝑘) = 0 and noting that since ℎ(𝑛) = 𝑏(𝑛), then

𝒒−𝒌

𝒄𝒒 𝒌 = 𝒃𝒒 𝒍 + 𝒌 𝒃∗ 𝒍
𝒍=𝟎

o In either case we have

𝒒− 𝒌

𝒓𝒙 𝒌 = 𝝈𝒗𝟐 𝒃𝒒 𝒌 ∗ 𝒃𝒒∗ −𝒌 = 𝝈𝒗𝟐 𝒃𝒒 𝒍 + 𝒌 𝒃𝒒∗ 𝒍 → ⓲

𝒍=𝟎

o Where we have used the absolute value on k so that the expression for 𝒓𝒙 𝒌 is valid for all 𝑘.

o Thus, we see that the autocorrelation sequence of an 𝑀𝐴(𝑞) process is equal to zero for all values of 𝑘 that lie
outside the interval [−𝑞, 𝑞].

o In addition, note that 𝒓𝒙 𝒌 depends non-linearly on the moving average parameters, 𝒃𝒒 𝒌 .

o For example, for 𝑞 = 2
𝟐
𝒓𝒙 (𝟎) = 𝒃(𝟎) + 𝒃(𝟏) 𝟐 + 𝒃 𝟐 𝟐

𝒓𝒙 𝟏 = 𝒃∗ 𝟎 𝒃(𝟏) + 𝒃∗ 𝟏 𝒃(𝟐)
𝒓𝒙 𝟐 = 𝒃∗ 𝟎 𝒃 𝟐 → ⓳

o Therefore, unlike the case for an autoregressive process, estimating the moving average parameters for a moving
average process is nontrivial problem.

o Moving average processes are characterized by slowly changing functions of frequency that will have sharp nulls in
the spectrum if 𝑃𝑥(𝑧) contains zeros that are close to the unit circle.