Stationary Processes

We can classify random processes based on many different criteria. One of the important questions that we can ask about a random
process is whether it is a stationary process. Intuitively, a random process {X(t), t ∈ J } is stationary if its statistical properties do
not change by time. For example, for a stationary process, X(t) and X(t + Δ) have the same probability distributions. In
particular, we have

FX(t) (x) = FX(t+Δ) (x),  for all t, t + Δ ∈ J .

More generally, for a stationary process, the joint distribution of X(t1 ) and X(t2 ) is the same as the joint distribution of
X(t1 + Δ) and X(t2 + Δ). For example, if you have a stationary process X(t) , then

P ((X(t1 ), X(t2 )) ∈ A) = P ((X(t1 + Δ), X(t2 + Δ)) ∈ A),

2
for any set A ∈ R . In sum, a random process is stationary if a time shift does not change its statistical properties. Here is a formal
definition of stationarity of continuous-time processes.
A continuous-time random process {X(t), t ∈ R} is strict-sense stationary or simply stationary if, for all t1 , t2 , ⋯ , tr ∈ R
and all Δ ∈ R, the joint CDF of

X(t1 ), X(t2 ), ⋯ , X(tr )

is the same as the joint CDF of

X(t1 + Δ), X(t2 + Δ), ⋯ , X(tr + Δ).

That is, for all real numbers x 1 , x 2 , ⋯ , x r , we have

FX(t (x 1 , x 2 , ⋯ , x r ) = FX(t (x 1 , x 2 , ⋯ , x r ).
1 )X(t2 )⋯X(tr ) 1 +Δ)X(t2 +Δ)⋯X(tr +Δ)

We can provide similar definition for discrete-time processes.

strict-sense stationary or simply stationary, if for all n1 , n2 , ⋯ , nr ∈ Z and all D ∈ Z, the joint CDF of

X(n1 ), X(n2 ), ⋯ , X(nr )

is the same as the joint CDF of

X(n1 + D), X(n2 + D), ⋯ , X(nr + D).

That is, for all real numbers x 1 , x 2 , ⋯ , x r , we have

FX(n (x 1 , x 2 , ⋯ , x n ) = FX(n (x 1 , x 2 , ⋯ , x r ).
1 )X(n 2 )⋯X(n r ) 1 +D)X(n 2 +D)⋯X(n r +D)

Example 10.7
Consider the discrete-time random process {X(n), n ∈ Z ⋯} , in which the X(n) 's are i.i.d. with CDF FX(n) (x) = F (x).

Show that this is a (strict-sense) stationary process.

Solution
Intuitively, since X(n) 's are i.i.d., we expect that as time evolves the probabilistic behavior of the process does not
change. Therefore, this must be a stationary process. To show this rigorously, we can argue as follows. For all real
numbers x 1 , x 2 , ⋯ , x r and all distinct integers n1 , n2 ,⋯ , nr , we have
 FX(n (x 1 , x 2 , ⋯ , x r )
1 )X(n 2 )⋯X(n r )

= FX(n (x 1 )FX(n (x 2 ) ⋯ FX(n (x r )  (since the X(ni )'s are independent)

1) 2) r)

= F (x 1 )F (x 2 ) ⋯ F (x r )  (since FX(t )
(x) = F (x)).
i

We also have

FX(n (x 1 , x 2 , ⋯ , x r )
1 +D)X(n 2 +D)⋯X(n r +D)

= FX(n (x 1 )FX(n (x 2 ) ⋯ FX(n (x r )  (since the X(ni + D)'s are independent)

1 +D) 2 +D) r +D)

= F (x 1 )F (x 2 ) ⋯ F (x n )  (since FX(n (x) = F (x)).

i +D)

In practice, it is desirable if a random process X(t) is stationary. In particular, if a process is stationary, then its analysis is usually
simpler as the probabilistic properties do not change by time. For example, suppose that you need to do forecasting about the future
of a process X(t) . If you know the process is stationary, you can observe the past, which will normally give you a lot of information
about how the process will behave in the future.

However, it turns out that many real-life processes are not strict-sense stationary. Even if a process is strict-sense stationary, it might
be difficult to prove it. Fortunately, it is often enough to show a "weaker" form of stationarity than the one defined above.

Weak-Sense Stationary Processes:

Here, we define one of the most common forms of stationarity that is widely used in practice. A random process is called weak-sense
stationary or wide-sense stationary (WSS) if its mean function and its correlation function do not change by shifts in time. More
precisely, X(t) is WSS if, for all t1 , t2 ∈ R and all Δ ∈ R,

1. E [X(t1 )] = E [X(t2 )] ,
2. E [X(t1 )X(t2 )] = E [X(t1 + Δ)X(t2 + Δ)] .

Note that the first condition states that the mean function μX (t) is not a function of time, t, thus we can write μX (t) = μX . The
second condition states that the correlation function RX (t1 , t2 ) is only a function of τ = t1 − t2 , and not t1 and t2
individually. Thus, we can write RX (t1 , t2 ) = RX (t1 − t2 ) = RX (τ ) . Therefore, we can provide the following definition.
A continuous-time random process {X(t), t ∈ R} is weak-sense stationary or wide-sense stationary (WSS) if

1. μX (t) = μX , for all t ∈ R,

2. RX (t1 , t2 ) = RX (t1 − t2 ) , for all t1 , t2 ∈ R.

We can provide a similar definition for discrete-time WSS processes.

A discrete-time random process {X(n), n ∈ Z} is weak-sense stationary or wide-sense stationary (WSS) if

1. μX (n) = μX , for all n ∈ Z ,

2. RX (n1 , n2 ) = RX (n1 − n2 ) , for all n1 , n2 ∈ Z.

Example 10.8
Consider the random process {X(t), t ∈ R} defined as

X(t) = cos(t + U ),

where U ∼ U nif orm(0, 2π) . Show that X(t) is a WSS process.

Solution
We need to check two conditions:
1. μX (t) = μX , for all t ∈ R, and
2. RX (t1 , t2 ) = RX (t1 − t2 ) , for all t1 , t2 ∈ R.
We have
 μX (t) = E [X(t)]

= E [cos(t + U )]

2π
1
= ∫ cos(t + u) du
0
2π

= 0,  for all t ∈ R.

We can also find RX (t1 , t2 ) as follows

RX (t1 , t2 ) = E [X(t1 )X(t2 )]

= E [cos(t1 + U ) cos(t2 + U )]

1 1
= E [ cos(t1 + t2 + 2U ) + cos(t1 − t2 )]
2 2

1 1
= E [ cos(t1 + t2 + 2U )] + E [ cos(t1 − t2 )]
2 2
2π
1 1
= ∫ cos(t1 + t2 + u) du + cos(t1 − t2 )
0
2π 2

1
= 0+ cos(t1 − t2 )
2
1
= cos(t1 − t2 ),  for all t1 , t2 ∈ R.
2

As we see, both conditions are satisfied, thus X(t) is a WSS process.

Since for WSS random processes, RX (t1 , t2 ) = RX (t1 − t2 ) , we usually denote the correlation function by RX (τ ) , where
τ = t1 − t2 . Thus, for a WSS process, we can write

RX (τ ) = E [X(t)X(t − τ )] = E [X(t + τ )X(t)] (10.1)

As we will see in Section 10.2, RX (τ ) is a very useful tool when we do frequency domain analysis. Here, we would like to study
some properties of RX (τ ) for WSS signals. Let {X(t), t ∈ R} be a WSS process with correlation function RX (τ ) . Then, we
can write

2
RX (0) = E [X(t) ].

The quantity E [X(t)2 ] is called the expected (average) power in X(t) at time t. For a WSS process, the expected power is not a
function of time. Since X(t)2 ≥ 0 , we conclude that RX (0) ≥ 0.

2
RX (0) = E [X(t) ] ≥ 0

Next, let's consider RX (−τ ). We have

RX (−τ ) = E [X(t)X(t + τ )] (by definition (Equation 10.1))

= E [X(t + τ )X(t)]

= RX (τ ) (Equation 10.1)

Thus, we conclude that RX (τ ) is an even function.

RX (τ ) = RX (−τ ), for all τ ∈ R.

Finally, we would like to show that RX (τ ) takes its maximum value at τ = 0. That is, X(t) and X(t + τ ) have the highest
correlation when τ = 0 .

|RX (τ )| ≤ RX (0), for all τ ∈ R.

The proof can be done using the Cauchy-Schwarz inequality: For any two random variables X and Y , we have
−−−−−−−−−−
2 2
|E XY | ≤ √E [X ]E [Y ],

where equality holds if and only if X = αY for some constant α ∈ R. Now, if we choose X = X(t) and Y = X(t − τ ) ,
we obtain
−−−−−−−−−−−−−−−−−
2 2
|E [X(t)X(t − τ )]| ≤ √E [X(t) ]E [X(t − τ ) ]

−−−−−−−−−−
= √RX (0)RX (0)

= RX (0).

Therefore, we conclude that |RX (τ )| ≤ RX (0) . Considering these properties, Figure 10.4 shows some possible shapes for
RX (τ ) .

Figure 10.4 - Some possible shapes for RX (τ ) .

Jointly Wide-Sense Stationary Processes:

We often work with multiple random processes, so we extend the concept of wide-sense stationarity to more than one process. More
specifically, we can talk about jointly wide-sense stationary processes.
Two random processes {X(t), t ∈ R} and {Y (t), t ∈ R} are said to be jointly wide-sense stationary if

1. X(t) and Y (t) are each wide-sense stationary.

2. RXY (t1 , t2 ) = RXY (t1 − t2 ) .

Example 10.9
Let X(t) and Y (t) be two jointly WSS random processes. Consider the random process Z (t) defined as

Z (t) = X(t) + Y (t).

Show that Z (t) is WSS.

Solution
Since X(t) and Y (t) are jointly WSS, we conclude
1. μX (t) = μX , μY (t) = μY ,
2. RX (t1 , t2 ) = RX (t1 − t2 ) , RY (t1 , t2 ) = RY (t1 − t2 ) ,

3. RXY (t1 , t2 ) = RXY (t1 − t2 ) .

Therefore, we have
 μZ (t) = E [X(t) + Y (t)]

= E [X(t)] + E [Y (t)]

= μX + μY .

RZ (t1 , t2 ) = E [(X(t1 ) + Y (t1 ))(X(t2 ) + Y (t2 ))]

= E [X(t1 )X(t2 )] + E [X(t1 )Y (t2 )] + E [Y (t1 )X(t2 )]E [Y (t1 )Y (t2 )]

= RX (t1 − t2 ) + RXY (t1 − t2 ) + RY X (t1 − t2 ) + RY (t1 − t2 ).

Cyclostationary Processes:

Some practical random processes have a periodic structure. That is, the statistical properties are repeated every T units of time (e.g.,
every T seconds). In other words, the random variables

X(t1 ), X(t2 ), ⋯ , X(tr )

have the same joint CDF as the random variables

X(t1 + T ), X(t2 + T ), ⋯ , X(tr + T ).

Such random variables are called cyclostationary. For example, consider the random process {X(t), t ∈ R} defined as

X(t) = A cos(ωt),

where A is a random variable. Here, we have

2π
X (t + ) = A cos(ωt + 2π)
ω

= A cos(ωt) = X(t).

2π
We conclude X(t) is in fact a periodic signal with period T =
ω
. Therefore, the statistical properties of X(t) do not change by
shifting the time by T units, so X(t) is a cyclostationary random process with period T = 2π
ω
. Similarly, we can define wide-
sense cyclostationary random processes.
A continuous-time random process {X(t), t ∈ R} is cyclostationary if there exists a positive real number T such that, for all
t1 , t2 , ⋯ , tr ∈ R , the joint CDF of

X(t1 ), X(t2 ), ⋯ , X(tr )

is the same as the joint CDF of

X(t1 + T ), X(t2 + T ), ⋯ , X(tr + T ).

A continuous-time random process {X(t), t ∈ R} is weak-sense cyclostationary or wide-sense cyclostationary if there exists a
positive real number T such that

1. μX (t + T ) = μX (t), for all t ∈ R;

2. RX (t1 + T , t2 + T ) = RX (t1 , t2 ) , for all t1 , t2 ∈ R.

Similarly, you can define cyclostationary discrete-time processes. For example, a discrete-time random process {X(n), n ∈ Z} is
wide-sense cyclostationary if there exists M ∈ N such that

1. μX (n + M ) = μX , for all n ∈ Z ;
2. RX (n1 + M , n2 + M ) = RX (n1 , n2 ) , for all n1 , n2 ∈ Z.
Derivatives and Integrals of Random Processes:

Many real-life systems are described by differential equations. To analyze such systems when randomness is involved, we often need
to differentiate or integrate the random processes that are present in the system. You have seen concepts such as continuity,
differentiability, and integrability in calculus for deterministic signals (deterministic functions). Here, we need to extend those
concepts to random processes. Without going much into mathematical technicalities, here we would like to provide some guidelines
on how to deal with derivatives and integrals of random processes.

Let X(t) be a continuous-time random process. We say that X(t) is mean-square continuous at time t if

2
lim E [∣
∣ X(t + δ) − X(t)∣
∣ ] = 0.
δ→0

Note that mean-square continuity does not mean that every possible realization of X(t) is a continuous function. It roughly means
that the difference X(t + δ) − X(t) is small on average.

Example 10.10
The Poisson process is discussed in detail in Chapter 11. If X(t) is a Poisson process with intensity λ, then for all t > s ≥ 0, we
have

X(t) − X(s) ∼ P oisson(λ(t − s)).

Show that X(t) is mean-square continuous at any time t ≥ 0.

Solution
We have

X(t + δ) − X(t) ∼ P oisson(λδ).

Thus,
2 2
lim E [|X(t + δ) − X(t)| ] = lim λδ + (λδ)
δ→0 δ→0

= 0.

It is worth noting that there are jumps in a Poisson process; however, those jumps are not very "dense" in time, so the random process
is still continuous in the mean-square sense. Figure 10.5 shows a possible realization of a Poisson process.

Figure 10.5 - A possible sample function of a Poisson process.

We can similarly talk about mean-square differentiability and mean-square integrability. If X(t) is a random process, the derivative
of X(t) ,

d
Y (t) = X(t),
dt

is also a random process. For nice and smooth processes, the derivative can be obtained in a natural way. For example, if you have a
random process defined as

2
X(t) = A + Bt + C t ,  for all t ∈ [0, ∞),

where A, B, and C are random variables, then the derivative of X(t) can be written as

′
X (t) = B + 2C t,  for all t ∈ [0, ∞).

Without trying to go much into mathematical technicalities, here we would like to provide some guidelines on how to deal with
derivatives and integrals of random processes (assuming some mild regularity conditions are satisfied). A key point to note is that
differentiation and integration are linear operations. This, for example, means that you can often interchange integration and
expectation. More specifically, you can write
t t

E [∫ X(u)du] = ∫ E [X(u)]du.
0 0

Similarly, if the derivative of X(t) is well-defined, we can write

d d
E [ X(t)] = E [X(t)].
dt dt

Example 10.11
d
Consider a random process X(t) and its derivative, X ′ (t) = X(t) . Assuming that the derivatives are well-defined, show that
dt

∂
RXX ′ (t1 , t2 ) = RX (t1 , t2 ).
∂ t2

Solution
We have
′
RXX ′ (t1 , t2 ) = E [X(t1 )X (t2 )]

d
= E [X(t1 ) X(t2 )]
dt2

∂
= E [ (X(t1 )X(t2 ))]
∂ t2

∂
= E [X(t1 )X(t2 )]
∂ t2

∂
= RX (t1 , t2 ).
∂ t2