Communication Systems

(EEEg4172)

Chapter 2
Information Theory and Coding

1
 2.1 Information Theory Basics

Outline:
▪ Introduction
▪ Information Source
▪ Measure of Information
▪ Discrete Memoryless Channels
▪ Mutual Information
▪ Channel Capacity

2
 Introduction

▪ Information theory is a field of study which deals with

quantification, storage, and communication of information.
▪ It is originally proposed by Claude E. Shannon in his famous paper
entitled “A Mathematical Theory of Communication”.
▪ In general, information theory provides:
✓ a quantitative measure of information contained in message signals.
✓ a way to determine the capacity of a communication system to transfer
information from source to destination.

3
 Information Source

▪ An information source is an object that produces an event, the outcome of

which is selected at random according to a specific probability distribution.
▪ A practical information source in a communication system is a device that
produces messages and it can be either analog or discrete.
▪ A discrete information source is a source that has only a finite set of
symbols as possible outputs.
▪ The set of source symbols is called the source alphabet and the elements of
the set are called symbols or letters.

4
 Information Source…
▪ Information sources can be classified as:
➢ Sources with memory, or
➢ Memoryless sources.
▪ A source with memory is one for which a current symbol depends on the
previous symbols.
▪ A memoryless source is one for which each symbol produced is
independent of the previous symbols.
▪ A discrete memoryless source (DMS) can be characterized by the set of it’s
symbols, the probability assignment to the symbols, and the rate by which
the source generates symbols.

5
 Measure of Information

▪ The amount of information contained in an event is closely related to its

uncertainty.
▪ Messages containing knowledge of high probability of occurrence convey
relatively little information.
▪ If an event is certain, it conveys zero information.
▪ Thus, a mathematical measure of information should be a function of the
probability of the outcome and should satisfy the following axioms:
1. Information should be proportional to the uncertainty of an outcome.
2. Information contained in independent outcomes should add.

6
 Measure of Information…

Information Content of a Symbol:

▪ Consider a DMS, denoted by X, with alphabets {x1, x2, …, xm}.
▪ The information content of a symbol xi, i = 1, 2, …, m; denoted by I(xi), is
defined by:
1
I ( xi ) = log = − logb
P ( xi )
b P ( xi )

where P( xi ) is the probability of occurenceof symbol x i

▪ Units of I(xi) depends on the base b, can be defined as follows:

✓ If b = 2, the unit is bits/symbol (aka, Shannon)
✓ If b = 10, the unit is in decimal digit (or hartleys)
✓ If b = e, the unit is in “nats/symbol)

7
 Measure of Information…

Information Content of a Symbol:

▪ Note that I(xi) satisfies the following properties.

i. I ( xi ) = 0 for P( xi ) = 1
ii. I ( xi )  0

iii. I ( xi )  I ( x j ) if P ( xi )  P ( x j )
iv. I ( xi , x j ) = I ( xi ) + I ( x j )
if xi and x j are independent

8
 Measure of Information…
Average Information or Entropy of a DMS:
▪ In a practical communication system, we usually transmit long sequences
of symbols from an information source.
▪ Thus, we are more interested in the average information that a source
produces than the information content of a single symbol.
▪ The mean value of I(xi) over the alphabet of source X with m different
symbols is given by:
m
H ( X ) = E[ I ( xi )] =  P( xi ) I ( xi )
i =1
m
= − P( xi ) log2
P ( xi )
bits/symbol
i =1

9
 Measure of Information…
Average Information or Entropy of a DMS…
▪ The quantity H(X) is called the entropy of the source X.
▪ It a measure of the average information content per symbol for a DMS X.
▪ The source entropy H(X) can be considered the average amount of
uncertainty within source X that is resolved by use of the alphabet.
▪ The source entropy H(X) satisfies the following relations:
0  H ( X )  log2
m
where m is the size of the
alphabet of source X .
▪ The lower bound corresponds to no uncertainty, which occurs when one symbol has
probability p(xi)=1 while p(xj)= 0 for j≠i, so X emits xi at all times. The upper bound
corresponds to the maximum uncertainty which occurs when p(xj)= 1/m for all i (i.e: when
all symbols have equal probability to be generated by X.
10
 Measure of Information…
Average Information or Entropy of a DMS…

Information Rate:
▪ If the time rate at which the source X emits symbols is r (symbols/sec), the
information rate R of the source is given by:
𝑅 = 𝑟𝐻 𝑋 𝑏/𝑠

11
 Discrete Memoryless Channels
Channel Representation:
▪ A communication channel is the path or medium through which the
symbols flow from a source to a receiver.
▪ A discrete memoryless channel (DMC) is a statistical model with an input
X and output Y as shown in the figure below.

12
 Discrete Memoryless Channels…

Channel Representation…
▪ The channel is discrete when the alphabets of X and Y are both finite.
▪ It is memoryless when the current output depends on only the current input
and not on any of the previous inputs.
▪ In the DMC shown above, the input X consists of input symbols x1, x2, …,
xm and the output Y consists of output symbols y1, y2, …, yn.
▪ Each possible input-to-output path is indicated along with a conditional
probability P(yj/xi), which is known as channel transition probability.

13
 Discrete Memoryless Channels…
Channel Matrix:
▪ A channel is completely specified by the complete set of transition
probabilities.
▪ The channel matrix , denoted by [P(Y/X)], is given by:

▪ Since each input to the channel results in some output, each row of the
channel matrix must sum to unity, i.e:

14
 Discrete Memoryless Channels…
Special Channels:
1. Lossless Channel
▪ A channel described by a channel matrix with only one non-zero element in
each column is called a lossless channel.
▪ An example of a lossless channel is shown in the figure below.

15
 Discrete Memoryless Channels…
Special Channels:
2. Deterministic Channel
▪ A channel described by a channel matrix with only one non-zero unity
element in each row is called a deterministic channel.
▪ An example of a deterministic channel is shown in the figure below.

16
 Discrete Memoryless Channels…
Special Channels:
3. Noiseless Channel
▪ A channel is called noiseless if it is both lossless and deterministic.
▪ A noiseless channel is shown in the figure below.
▪ The channel matrix has only one element in each in each row and each
column and this element is unity.

17
 Discrete Memoryless Channels…
Special Channels:
4. Binary Symmetric Channel
▪ The binary symmetric channel (BSC) is defined by the channel matrix and
channel diagram given below.

18
 Mutual Information
Conditional and Joint Entropies:
▪ Using the input probabilities P(xi), output probabilities P(yj), transition
probabilities P(yj/xi), and joint probabilities P(xi, yj), we can define the
following various entropy functions for a channel with m inputs and n
outputs.

19
 Mutual Information…
Conditional and Joint Entropies:
▪ The above entropies can be interpreted as the average uncertainties of the
inputs and outputs.
▪ Two useful relationships among the above various entropies are:

Mutual Information of a Channel:

▪ The mutual information I(X;Y) of a channel is defined by:

20
 Mutual Information…
Mutual Information of a Channel:

21
Channel Capacity

22
Channel Capacity…

23
Channel Capacity…

24
 Additive White Gaussian Noise(AWGN) Channel

▪ In a continuous channel an information source produces a continuous signal

x(t).
▪ The set of possible signals is considered as an ensemble of waveforms
generated by some ergodic random process.

Differential Entropy:
▪ The average amount of information per sample value of x(t) is measured by

▪ The entropy H(X) defined by equation above is known as the differential

entropy of X.

25
 Additive White Gaussian Noise(AWGN) Channel…

▪ The average mutual information in a continuous channel is defined (by analogy

with the discrete case) as:

26
 Additive White Gaussian Noise(AWGN) Channel…

▪ In an additive white Gaussian noise (AWGN ) channel, the channel output Y is given
by:
Y= X + n
▪ where X is the channel input and n is an additive band-limited white Gaussian noise
with zero mean and variance σ2.
▪ The capacity Cs of an AWGN channel is given by:

where S/N is the signal-to-noise ratio at the channel output.

27
 Additive White Gaussian Noise(AWGN) Channel…

▪ If the channel bandwidth B Hz is fixed, then the output y(t) is also a band-limited
signal completely characterized by its periodic sample values taken at the Nyquist
rate 2B samples/s.
▪ Then the capacity C (b/s) of the AWGN channel is given by:

▪ This equation is known as the Shannon-Hartley law.

28
 Examples on Information Theory and Coding
Example-1:

A DMS X has four symbols x1 , x2 , x3 , x4 with probabilities

P( x1 ) = 0.4, P( x2 ) = 0.3, P( x3 ) = 0.2 and P( x4 ) = 0.1

a. Calculate H ( X )
b. Find the amount of information contained in the messages
x1 x2 x3 x4 and x4 x3 x3 x2

29
 Examples on Information Theory and Coding Cont’d…
Solution:
4
a. H ( X ) = − P( xi ) log2
[ P ( xi )]

i =1

= −0.4 log2 − 0.3 log2 − 0.2 log2 − 0.1log2

0.4 0.3 0.2 0.1

= 1.85 bits/symbol
0.0024
b. P( x1 x2 x3 x4 ) = (0.4)(0.3)(0.2)(0.1) = 0.0096
I ( x1 x2 x3 x4 ) = − log2 = 6.7 bits/symbol
0.0096
8.7 bits/symbol
Similarly,
P( x1 x2 x3 x4 ) = (0.1)(0.2) 2 (0.3) = 0.0012
I ( x1 x2 x3 x4 ) = − log2 = 9.7 bits/symbol
0.0012

30
 Examples on Information Theory and Coding Cont’d….
Example-2:
Consider a binary symmetric channel shown below.

a. Find the channel matrix of the channel

b. Find P( y1 ) and P( y2 ) when P( x1 ) = P( x2 ) = 0.5
c. Find the joint probabilities P ( x1 , y2 ) and P ( x2 , y1 )
when P( x1 ) = P( x2 ) = 0.5
31
 Examples on Information Theory and Coding Cont’d….
Solution:
a. The channel matrix is given by :
 P( y1 / x1 ) P( y2 / x1 )  0.9 0.1
P(Y / X ) =   = 
 P ( y1 / x 2 ) P ( y 2 / x 2 
)  0.2 0.8

b. P (Y ) = P ( X )P (Y / X )
0.9 0.1
= 0.5 0.5 
 0.2 0 . 8
= 0.55 0.45 = P ( y1 ) P ( y 2 )
 P ( y1 ) = 0.55 and P ( y 2 ) = 0.45

32
Examples on Information Theory and Coding Cont’d….
Solution:

c. P( X , Y ) = P( X ) d P(Y / X )

0.5 0 0.9 0.1
=  0.2 0.8
 0 0.5 
0.45 0.05  P( x1 , y1 ) P( x1 , y2 ) 
=  = 
 0.1 0 .4   P ( x 2 , y1 ) P ( x 2 , y 2 
)
 P( x1 , y2 ) = 0.05 and P( x2 , y1 ) = 0.1

33
 Examples on Information Theory and Coding Cont’d….
Example-3:
An information source can be modeled as a bandlimited process with a
bandwidth of 6kHz. This process is sampled at a rate higher than the
Nyquist rate to provide a guard band of 2kHz. It is observed that the
resulting samples take values in the set {-4, -3, -1, 2, 4, 7} with
probabilities 0.2, 0.1, 0.15, 0.05, 0.3 and 0.2 respectively. What is the
entropy of the discrete-time source in bits/sample? What is the entropy in
bits/sec?

34
 Examples on Information Theory and Coding Cont’d….
Solution:
The entropy of the source is
6
H ( X ) = − P( xi ) log2 = 2.4087 bits/sample
P ( xi )

i =1

The sampling rate is

f s = 2kHz + 2 * (6kHz) = 14kHz
This means that 14000samples are taken per each second.
 The entropy of the source in bits per second is given by :
H ( X ) = 2.4087 (bis/sample) *14000(samples/sec)
= 33721.8bits/sec = 33.7218Kbps

35
 References:
▪ Simon Haykin, Communication Systems, 4th edition. (chapter-9).
▪ B.P. Lathi, Modern Digital and Analog Communication Systems, 3rd
edition. (Chapter-15)
▪ Henk C.A van Tilborg, Coding Theory, April 03, 1993.

Additional Reading:
▪ https://en.wikipedia.org/wiki/Information_theory
▪ https://en.wikipedia.org/wiki/Coding_theory
▪ https://en.wikipedia.org/wiki/Noisy-channel_coding_theorem

36