Scribd
Upload
6 views13 pages

PMRslides 02

The document discusses two fundamental assumptions that can be made to efficiently represent probabilistic models: independence assumptions and assumptions on the form of interaction between variables. Independence assumptions involve factorizing the joint probability distribution using conditional independence between variables. Assumptions on the form of interaction further restrict how variables may interact with each other, such as using parametric families like logistic regression.

Uploaded by

zeliawillscumberg
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
6 views13 pages

PMRslides 02

The document discusses two fundamental assumptions that can be made to efficiently represent probabilistic models: independence assumptions and assumptions on the form of interaction between variables. Independence assumptions involve factorizing the joint probability distribution using conditional independence between variables. Assumptions on the form of interaction further restrict how variables may interact with each other, such as using parametric families like logistic regression.

Uploaded by

zeliawillscumberg
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 13

Basic Assumptions

for Efficient Model Representation

Michael Gutmann

Probabilistic Modelling and Reasoning (INFR11134)


School of Informatics, University of Edinburgh

Spring Semester 2020


Recap

P
p(x,yo ,z)
p(x|yo ) = P z p(x,y ,z)
o
x,z

Assume that x, y, z each are d = 500 dimensional, and that each


element of the vectors can take K = 10 values.
I Issue 1: To specify p(x, y, z), we need to specify
K 3d − 1 = 101500 − 1 non-negative numbers, which is
impossible.
Topic 1: Representation What reasonably weak assumptions
can we make to efficiently represent p(x, y, z)?

Michael Gutmann Assumptions for Model Representation 2 / 13


Two fundamental assumptions

Consider two assumptions:


1. only a limited number of variables may directly interact with
each other (independence assumptions)
2. for any number of interacting variables, the form of interaction
is limited or restricted (often: parametric family assumptions)
The two assumptions can be used together or separately.

Michael Gutmann Assumptions for Model Representation 3 / 13


Program

1. Independence assumptions

2. Assumptions on form of interaction

Michael Gutmann Assumptions for Model Representation 4 / 13


Program

1. Independence assumptions
Definition and properties of statistical independence
Factorisation of the pdf and reduction in the number of
directly interacting variables

2. Assumptions on form of interaction

Michael Gutmann Assumptions for Model Representation 5 / 13


Statistical independence
I Let x and y be two disjoint subsets of random variables. Then x and
y are independent of each other if and only if (iff)
p(x, y) = p(x)p(y)
for all possible values of x and y; otherwise they are said to be
dependent.
I We say that the joint factorises into a product of p(x) and p(y).
I Equivalent definition by the product rule (or by definition of
conditional probability)
p(x|y) = p(x)
for all values of x and y where p(y) > 0.
I Notation: x ⊥
⊥y
I Variables x1 , . . . , xn are independent iff
n
Y
p(x1 , . . . , xn ) = p(xi )
i=1

Michael Gutmann Assumptions for Model Representation 6 / 13


Conditional statistical independence

I The characterisation of statistical independence extends to


conditional pdfs (pmfs) p(x, y|z).
I The condition p(x, y) = p(x)p(y) becomes
p(x, y|z) = p(x|z)p(y|z)
I The equivalent condition p(x|y) = p(x) becomes
p(x|y, z) = p(x|z)
I We say that x and y are conditionally independent given z iff,
for all possible values of x, y, and z with p(z) > 0:

p(x, y|z) = p(x|z)p(y|z) or

p(x|y, z) = p(x|z) (for p(y, z) > 0)


I Notation: x ⊥
⊥ y|z

Michael Gutmann Assumptions for Model Representation 7 / 13


The impact of independence assumptions

I The key is that the independence assumption leads to a


partial factorisation of the pdf (pmf).
I For example, if x, y, z are independent of each other, then

p(x, y, z) = p(x)p(y)p(z)

I Independence assumption forces p(x, y, z) to take on a


particular form.

Michael Gutmann Assumptions for Model Representation 8 / 13


The impact of independence assumptions

Assume p(x, y, z) = p(x)p(y)p(z)


I If dim(x) = dim(y) = dim(z) = d, and each element of the
vectors can take K values, factorisation reduces the numbers
that need to be specified (“parameters”) from K 3d − 1 to
3(K d − 1).
I If all variables were independent: 3d(K − 1) numbers needed.

For example: 101500 − 1 vs. 3(10500 − 1) vs. 1500(10 − 1) = 13500

I But full independence (factorisation) assumption is often too


strong and does not hold.

Michael Gutmann Assumptions for Model Representation 9 / 13


The impact of independence assumptions

I Conditional independence assumptions are a powerful


middle-ground.
I For p(x) = p(x1 , . . . , xd ), we have by the product rule:

p(x) = p(xd |x1 , . . . xd−1 )p(x1 , . . . , xd−1 )

I If, for example, xd ⊥


⊥ x1 , . . . , xd−4 | xd−3 , xd−2 , xd−1 , we have

p(xd |x1 , . . . , xd−1 ) = p(xd |xd−3 , xd−2 , xd−1 )

I If the xi can take K different values:


p(xd |x1 , . . . , xd−1 ) specified by K d−1 · (K − 1) numbers
p(xd |xd−3 , xd−2 , xd−1 ) specified by K 3 · (K − 1) numbers

For d = 500, K = 10: 10499 · 9 ≈ 10500 vs 9000 ≈ 104 .

Michael Gutmann Assumptions for Model Representation 10 / 13


Program

1. Independence assumptions

2. Assumptions on form of interaction


Parametric model to restrict how a given number of variables
may interact

Michael Gutmann Assumptions for Model Representation 11 / 13


Assumption 2: limiting the form of the interaction
I The (conditional) independence assumption limits the number
of variables that may directly interact with each other, e.g.
xd only directly interacted with xd−3 , xd−2 , xd−1 .
I How xd interacts with the three variables, however, was not
restricted.
I Assumption 2: We restrict how a given number of variables
may interact with each other.
I For example, for xi ∈ {0, 1}, we may assume that
p(xd |x1 , . . . , xd−1 ) is specified as
1
p(xd = 1|x1 , . . . , xd−1 ) =  Pd−1 
1 + exp −w0 − i=1 wi xi

with d free numbers (“parameters”) w0 , . . . , wd−1 .


I d vs 2d−1 parameters (for d = 500: 500 vs. 2499 ≈ 10150 )
Michael Gutmann Assumptions for Model Representation 12 / 13
Program recap

We asked: What reasonably weak assumptions can we make to


efficiently represent a probabilistic model?

1. Independence assumptions
Definition and properties of statistical independence
Factorisation of the pdf and reduction in the number of
directly interacting variables

2. Assumptions on form of interaction


Parametric model to restrict how a given number of variables
may interact

Michael Gutmann Assumptions for Model Representation 13 / 13

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy