Announcements

§  Project 3: MDPs and Reinforcement §  Midterm 1

Learning
§  Monday 3/10, 6:30pm-9:30pm
§  Due Friday 3/7 at 5pm
§  [A-F] 120 Latimer
§  Optional contest §  [G-S] 1 Pimentel
§  Due Thursday 3/13 at 11:59pm §  [T-Z] 145 Dwinelle

§  Preparation page up
§  Topics: Lectures 1 through 11 (inclusive)
§  Past exams
§  Special midterm 1 office hours

§  Practice Midterm 1
§  Optional
§  One point of EC on Midterm 1 for completing
§  Due: Saturday 3/10 at 11:59pm
AI Outside of 188: Angry Birds Competition

§  http://www.aibirds.org
 CS 188: Artificial Intelligence

Markov Models

Instructors: Dan Klein and Pieter Abbeel --- University of California, Berkeley
[These slides were created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188 materials are available at http://ai.berkeley.edu.]
 Independence
§  Two variables are independent in a joint distribution if:

§  Says the joint distribution factors into a product of two simple ones
§  Usually variables aren’t independent!

§  Can use independence as a modeling assumption

§  Independence can be a simplifying assumption
§  Empirical joint distributions: at best “close” to independent
§  What could we assume for {Weather, Traffic, Cavity}?

§  Independence is like something from CSPs: what?

 Example: Independence?

T P
hot 0.5 The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart
your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

cold 0.5
T W P T W P
hot sun 0.4 hot sun 0.3
hot rain 0.1 hot rain 0.2
cold sun 0.2 cold sun 0.3
cold rain 0.3 cold rain 0.2
W P
sun 0.6
rain 0.4
 Example: Independence
§  N fair, independent coin flips:

H 0.5 H 0.5 H 0.5

T 0.5 T 0.5 T 0.5
Conditional Independence
 Conditional Independence
§  P(Toothache, Cavity, Catch)

§  If I have a cavity, the probability that the probe catches in it

doesn't depend on whether I have a toothache:
§  P(+catch | +toothache, +cavity) = P(+catch | +cavity)

§  The same independence holds if I don’t have a cavity:

§  P(+catch | +toothache, -cavity) = P(+catch| -cavity)

§  Catch is conditionally independent of Toothache given Cavity:

§  P(Catch | Toothache, Cavity) = P(Catch | Cavity)

§  Equivalent statements:
§  P(Toothache | Catch , Cavity) = P(Toothache | Cavity)
§  P(Toothache, Catch | Cavity) = P(Toothache | Cavity) P(Catch | Cavity)
§  One can be derived from the other easily
 Conditional Independence
§  Unconditional (absolute) independence very rare (why?)

§  Conditional independence is our most basic and robust form

of knowledge about uncertain environments.

§  X is conditionally independent of Y given Z

if and only if:

or, equivalently, if and only if
 Conditional Independence
§  What about this domain:
§  Traffic
§  Umbrella
§  Raining
 Conditional Independence
§  What about this domain:
§  Fire
§  Smoke
§  Alarm
 Probability Recap

§  Conditional probability

§  Product rule

§  Chain rule

§  X, Y independent if and only if:

§  X and Y are conditionally independent given Z if and only if:

 CS 188: Artificial Intelligence

Markov Models

Instructors: Dan Klein and Pieter Abbeel --- University of California, Berkeley
[These slides were created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188 materials are available at http://ai.berkeley.edu.]
 Reasoning over Time or Space

§  Often, we want to reason about a sequence of observations

§  Speech recognition
§  Robot localization
§  User attention
§  Medical monitoring

§  Need to introduce time (or space) into our models

 Markov Models
§  Value of X at a given time is called the state

X1 X2 X3 X4

§  Parameters: called transition probabilities or dynamics, specify how the state

evolves over time (also, initial state probabilities)
§  Stationarity assumption: transition probabilities the same at all times
§  Same as MDP transition model, but no choice of action
 Joint Distribution of a Markov Model
X1 X2 X3 X4

§  Joint distribution:
P (X1 , X2 , X3 , X4 ) = P (X1 )P (X2 |X1 )P (X3 |X2 )P (X4 |X3 )
§  More generally:
P (X1 , X2 , . . . , XT ) = P (X1 )P (X2 |X1 )P (X3 |X2 ) . . . P (XT |XT 1)
T
Y
= P (X1 ) P (Xt |Xt 1)
t=2
§  Questions to be resolved:
§  Does this indeed define a joint distribution?
§  Can every joint distribution be factored this way, or are we making some assumptions
about the joint distribution by using this factorization?
 Chain Rule and Markov Models
X1 X2 X3 X4

§  From the chain rule, every joint distribution over can be written as:
X 1 , X2 , X3 , X4

P (X1 , X2 , X3 , X4 ) = P (X1 )P (X2 |X1 )P (X3 |X1 , X2 )P (X4 |X1 , X2 , X3 )

§  Assuming that
and
X3 ? ? X1 | X2 X4 ?
? X 1 , X2 | X 3

results in the expression posited on the previous slide:

P (X1 , X2 , X3 , X4 ) = P (X1 )P (X2 |X1 )P (X3 |X2 )P (X4 |X3 )

 Chain Rule and Markov Models
X1 X2 X3 X4

§  From the chain rule, every joint distribution over can be written as:
X 1 , X2 , . . . , X T
T
Y
P (X1 , X2 , . . . , XT ) = P (X1 ) P (Xt |X1 , X2 , . . . , Xt 1)
t=2
§  Assuming that for all t:
Xt ?
? X1 , . . . , X t 2 | Xt 1

gives us the expression posited on the earlier slide:

T
Y
P (X1 , X2 , . . . , XT ) = P (X1 ) P (Xt |Xt 1)
t=2
 Implied Conditional Independencies
X1 X2 X3 X4

§  We assumed: and
X3 ? ? X1 | X2 X4 ?
? X 1 , X2 | X 3

§  Do we also have 1?
X ? X3 , X
4|X
2 ?
§  Yes!
P (X1 , X2 , X3 , X4 )
§  Proof: P (X1 | X2 , X3 , X4 ) =
P (X2 , X3 , X4 )
P (X1 )P (X2 | X1 )P (X3 | X2 )P (X4 | X3 )
=P
x1 P (x1 )P (X2 | x1 )P (X3 | X2 )P (X4 | X3 )
P (X1 , X2 )
=
P (X2 )
= P (X1 | X2 )
 Markov Models Recap
§  Explicit assumption for all t : Xt ?
? X1 , . . . , X t 2 | Xt 1

§  Consequence, joint distribution can be written as:

P (X1 , X2 , . . . , XT ) = P (X1 )P (X2 |X1 )P (X3 |X2 ) . . . P (XT |XT 1)
T
Y
= P (X1 ) P (Xt |Xt 1)
t=2

§  Implied conditional independencies: (try to prove this!)

§  Past variables independent of future variables given the present
i.e., if or then:
t1 < t2 < t3 t1 > t2 > t3 ? Xt3 | Xt2
Xt1 ?
§  Additional explicit assumption: is the same for all t
P (Xt | Xt 1 )
 Conditional Independence

§  Basic conditional independence:

§  Past and future independent of the present
§  Each time step only depends on the previous
§  This is called the (first order) Markov property

§  Note that the chain is just a (growable) BN

§  We can always use generic BN reasoning on it if we
truncate the chain at a fixed length
 Example Markov Chain: Weather
§  States: X = {rain, sun}

§  Initial distribution: 1.0 sun

§  CPT P(Xt | Xt-1): Two new ways of representing the same CPT

Xt-1 Xt P(Xt|Xt-1) 0.9

0.3
sun sun 0.9 0.9
sun sun
sun rain 0.1 rain sun 0.1
rain sun 0.3 0.3
rain rain
rain rain 0.7 0.7 0.7
0.1
 Example Markov Chain: Weather
0.9
§  Initial distribution: 1.0 sun 0.3

rain sun

0.7
0.1

§  What is the probability distribution after one step?

 Mini-Forward Algorithm

§  Question: What’s P(X) on some day t?

X1 X2 X3 X4

X
P (xt ) = P (xt 1 , xt )
x
X
t 1

= P (xt | xt 1 )P (xt 1 )
xt 1
Forward simulation
 Example Run of Mini-Forward Algorithm
§  From initial observation of sun

1)
P(X P(X2) P(X3) P(X4) P(X∞)
§  From initial observation of rain

P(X1) P(X2) P(X3) P(X4) P(X∞)

§  From yet another initial distribution P(X1):

…
P(X1) P(X∞) [Demo: L13D1,2,3]
Video of Demo Ghostbusters Basic Dynamics
Video of Demo Ghostbusters Circular Dynamics
Video of Demo Ghostbusters Whirlpool Dynamics
 Stationary Distributions

§  For most chains: §  Stationary distribution:

§  Influence of the initial distribution §  The distribution we end up with is called
gets less and less over time. the stationary distribution P1 of the
§  The distribution we end up in is chain
independent of the initial distribution §  It satisfies
X
P1 (X) = P1+1 (X) = P (X|x)P1 (x)
x
 Example: Stationary Distributions
§  Question: What’s P(X) at time t = infinity?
X1 X2 X3 X4

P1 (sun) = P (sun|sun)P1 (sun) + P (sun|rain)P1 (rain)

P1 (rain) = P (rain|sun)P1 (sun) + P (rain|rain)P1 (rain)

P1 (sun) = 0.9P1 (sun) + 0.3P1 (rain) Xt-1 Xt P(Xt|Xt-1)

P1 (rain) = 0.1P1 (sun) + 0.7P1 (rain) sun sun 0.9

P1 (sun) = 3P1 (rain) sun rain 0.1

rain sun 0.3
P1 (rain) = 1/3P1 (sun)
P1 (sun) = 3/4 rain rain 0.7

Also: P1 (sun) + P1 (rain) = 1 P1 (rain) = 1/4

Application of Stationary Distribution: Web Link Analysis
§  PageRank over a web graph
§  Each web page is a state
§  Initial distribution: uniform over pages
§  Transitions:
§  With prob. c, uniform jump to a
random page (dotted lines, not all shown)
§  With prob. 1-c, follow a random
outlink (solid lines)

§  Stationary distribution
§  Will spend more time on highly reachable pages
§  E.g. many ways to get to the Acrobat Reader download page
§  Somewhat robust to link spam
§  Google 1.0 returned the set of pages containing all your
keywords in decreasing rank, now all search engines use link
analysis along with many other factors (rank actually getting
less important over time)
Application of Stationary Distributions: Gibbs Sampling*

§  Each joint instantiation over all hidden and query

variables is a state: {X1, …, Xn} = H U Q

§  Transitions:
§  With probability 1/n resample variable Xj according to

P(Xj | x1, x2, …, xj-1, xj+1, …, xn, e1, …, em)

§  Stationary distribution:
§  Conditional distribution P(X1, X2 , … , Xn|e1, …, em)
§  Means that when running Gibbs sampling long enough
we get a sample from the desired distribution
§  Requires some proof to show this is true!
Next Time: Hidden Markov Models!