Basic Information Theory: Thinh Nguyen Oregon State University

Lecture 2:
Basic Information
Theory
Thinh Nguyen
Oregon State University
What is information?
 Can we measure information?
 Consider the two following sentences:
1. There is a traffic jam on I5

2. There is a traffic jam on I5 near Exit 234
Sentence 2 seems to have more information than that of

sentence 1. From the semantic viewpoint, sentence 2 provides
more useful information.
 It is hard to measure the “semantic” information!
 Consider the following two sentences

It’s not clear whether sentence 1 or 2 would have more information!

 Let’s attempt at a different definition of
information.
 How about counting the number of letters in
the two sentences:
1. There is a traffic jam on I5 (22 letters)

2. There is a traffic jam on I5 near Exit 234 (33 letters)
Definitely something we can measure and compare!

It’s interesting to know that
log is the only functionf
that satisfies
What is information? f ( s l )  lf ( s)
 First attempt to quantify information by Hartley

(1928).
 Every symbol of the message has a choice of s possibilities.

l
 A message of length l , therefore can have s distinguishable
possibilities.
 Information measure is then the logarithm of sl

I  log(s )  l log(s )
l
Intuitively, this definition makes sense:

one symbol (letter) has the information of log(s ) then a sentence of length l
should have l times more information, i.e. l log s
How about we measure information as the number of
Yes/No questions one has to ask to get the correct
answer to a simple game below
1 2 How many questions?
3 4 2
1 2 3 4
How many questions?
5 6 7 8
4
9 10 11 12
13 14 15 16
Randomness due to uncerntainty of where the circle is!

Shannon’s Information Theory
The
Claude Shannon:
Shannon A Mathematical Theory of Communication
Bell System Technical Journal, 1948
 Shannon’s measure of information is the number of bits to

represent the amount of uncertainty (randomness) in a
data source, and is defined as entropy
n
H   pi log( pi )
i 1
Where there are n symbols 1, 2, … n, each with

probability of occurrence of pi
Shannon’s Entropy
 Consider the following string consisting of symbols a and b:
abaabaababbbaabbabab… ….
 On average, there are equal number of a and b.

 The string can be considered as an output of a below source
with equal probability of outputting symbol a or b:
0.5 a
We want to characterize the average

information generated by the source!
0.5 b
source
Intuition on Shannon’s Entropy
n
Why H   pi log( pi )
i 1
Suppose you have a long random string of two binary symbols 0 and 1, and the
probability of symbols 1 and 0 are p
0 and 1 p
Ex: 00100100101101001100001000100110001 ….
If any string is long enough say N, it is likely to contain Np0 0’s and Np1 1’s.
The probability of this string pattern occurs is equal to
p  p0Np0 p1Np1
 Np0  Np1
Hence, # of possible patterns is 1 / p  p p1
0
1
# bits to represent all possible patterns is log( p  Np0
0 p1 Np1 )   Npi log pi
i 0
The average # of bits to represent the symbol is therefore
1
  pi log pi
i 0
More Intuition on Entropy
 Assume a binary memoryless source, e.g., a flip of a coin. How
much information do we receive when we are told that the
outcome is heads?
 If it’s a fair coin, i.e., P(heads) = P (tails) = 0.5, we say that

the amount of information is 1 bit.
 If we already know that it will be (or was) heads, i.e.,

P(heads) = 1, the amount of information is zero!
 If the coin is not fair, e.g., P(heads) = 0.9, the amount of

information is more than zero but less than one bit!
 Intuitively, the amount of information received is the same if

P(heads) = 0.9 or P (heads) = 0.1.
Self Information
 So, let’s look at it the way Shannon did.
 Assume a memoryless source with
 alphabet A = (a1, …, an)
 symbol probabilities (p1, …, pn).
 How much information do we get when finding out that the
next symbol is ai?
 According to Shannon the self information of ai is
Why?
Assume two independent events A and B, with
probabilities P(A) = pA and P(B) = pB.
For both the events to happen, the probability is

pA ¢ pB. However, the amount of information
should be added, not multiplied.
Logarithms satisfy this!

No, we want the information to increase with
decreasing probabilities, so let’s use the negative
logarithm.
Self Information
Example 1:
Example 2:
Which logarithm? Pick the one you like! If you pick the natural log,
you’ll measure in nats, if you pick the 10-log, you’ll get Hartleys,
if you pick the 2-log (like everyone else), you’ll get bits.
Self Information
On average over all the symbols, we get:
H(X) is called the first order entropy of the source.
This can be regarded as the degree of uncertainty

about the following symbol.
Entropy
Example: Binary Memoryless Source
BMS 01101000…
Let
Ofte
n den
Then oted
1
The uncertainty (information) is greatest when
0 0.5 1
Example
Three symbols a, b, c with corresponding probabilities:
P = {0.5, 0.25, 0.25}
What is H(P)?
Three weather conditions in Corvallis: Rain, sunny, cloudy with

corresponding probabilities:
Q = {0.48, 0.32, 0.20}
What is H(Q)?
Entropy: Three properties
1. It can be shown that 0 · H · log N.
2. Maximum entropy (H = log N) is reached when

all symbols are equiprobable, i.e.,
pi = 1/N.
3. The difference log N – H is called the redundancy

of the source.

Basic Information Theory: Thinh Nguyen Oregon State University

Uploaded by

Copyright:

Available Formats

Basic Information Theory: Thinh Nguyen Oregon State University

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Basic Information Theory: Thinh Nguyen Oregon State University

Uploaded by

Copyright:

Available Formats

Lecture 2:

1. There is a traffic jam on I5

Sentence 2 seems to have more information than that of

1. There is a traffic jam on I5 near Exit 160

It’s not clear whether sentence 1 or 2 would have more information!

1. There is a traffic jam on I5 (22 letters)

Definitely something we can measure and compare!

 First attempt to quantify information by Hartley

 Every symbol of the message has a choice of s possibilities.

 Information measure is then the logarithm of sl

Intuitively, this definition makes sense:

Randomness due to uncerntainty of where the circle is!

 Shannon’s measure of information is the number of bits to

Where there are n symbols 1, 2, … n, each with

 On average, there are equal number of a and b.

We want to characterize the average

 If it’s a fair coin, i.e., P(heads) = P (tails) = 0.5, we say that

 If we already know that it will be (or was) heads, i.e.,

 If the coin is not fair, e.g., P(heads) = 0.9, the amount of

 Intuitively, the amount of information received is the same if

For both the events to happen, the probability is

Logarithms satisfy this!

On average over all the symbols, we get:

H(X) is called the first order entropy of the source.

This can be regarded as the degree of uncertainty

P = {0.5, 0.25, 0.25}

Three weather conditions in Corvallis: Rain, sunny, cloudy with

Q = {0.48, 0.32, 0.20}

2. Maximum entropy (H = log N) is reached when

3. The difference log N – H is called the redundancy

You might also like

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.