Machine Learning – 18CS71

Module 2

Introduction
 Module 2
Concept Learning • Inductive Bias

• Concept learning task

• Concept learning as search

• Find-S algorithm

• Version space

• Candidate Elimination
algorithm

Machine Learning – 18CS71 2

 • Task T
• Performance Measure P
Well-Posed Learning Problems • Training Experience E

Learning is broadly defined as any computer program that improves its

performance at some task through experience.

Definition: Machine Learning

•A computer program is said to learn from experience E with respect to some

class of tasks T and performance measure P, if its performance at tasks in T,

as measured by P, improves with experience E.

Machine Learning – 18CS71 3
 Ex1: A checkers learning problem

• Task T - Playing checkers

• Performance Measure P - Percentage of games won against
opponent
• Training Experience E - Playing practice games against
itself

Machine Learning – 18CS71 4

Ex 2: A handwriting recognition learning
problem
Task T: recognizing and classifying
handwritten words within images

Performance measure P: percent

of words correctly classified
Training experience E: a database
of handwritten words with given
classifications
Machine Learning – 18CS71 5
Ex3: A robot driving learning problem

T: driving on public 4-lane highways

using vision sensors

P: average distance traveled before an

error (as judged by human overseer)

E: a sequence of images and

steering commands recorded by
observing a human driver
Machine Learning – 18CS71 6
 Concept Learning
Things

Smart
Phones Tablet
Phones

Machine Learning – 18CS71 7

 • Big Space -> reduced to small space
Concept Learning • Target function – Boolean – True/False

Inducing general functions from specific training examples is a main issue of machine learning.

A task of acquiring a potential hypothesis (solution) that best fits the training examples

It is the process of acquiring the definition of a general category from given sample positive
and negative training examples of the category.

Concept Learning can seen as a problem of searching through a predefined space of potential
hypotheses for the hypothesis that best fits the training examples.

Concept learning: Inferring a boolean-valued function from training examples of its input and
output.

Machine Learning – 18CS71 8

Terminologies
• Concept
• Concept Learning
• Concept Learning Task
• Hypothesis
• Target Concept
• Instances
• Training Examples

Machine Learning – 18CS71 9

enjoysport.csv

Machine Learning – 18CS71 10

Concept Learning Task

Problem is to learning the target concept

• "Days on which my friend Sachin enjoys his favorite water
sport”.
Given

Machine Learning – 18CS71 11

Hypothesis representation Concept
For each attribute, the hypothesis will either
• Hypothesis:
• ? – any value is acceptable
• Φ – not acceptable
• Single value – specific attribute

If some instance x satisfies all the constraints of hypothesis h, then h classifies x
as a positive example (h(x) = 1).
Most general hypothesis - (?, ?, ?, ?, ?, ?)
Most specific possible hypothesis- (Φ, Φ, Φ, Φ, Φ, Φ)

Example:
(?, Cold, High, ?, ?, ?)

Machine Learning – 18CS71 12

“EnjoySport” Concept Learning Task
• Instances – All X
• Hypothesis – conjunction of
constraints
• Target Concept – True/ False
• Training Example – Both +ve and -
ve

• Conjunction – combination of
two or more attributes

Machine Learning – 18CS71 13

Terminologies
• Concept Boolean – Valued Function

• Concept Learning Inferring a Boolean – Valued Function from Training Examples

• Concept Learning Task
• Hypothesis Conjunction of constraints on the instance attribute
• Hypothesis Representation
• Target Concept Boolean Valued Function defined over X
• Instances
• Training Examples

Machine Learning – 18CS71 14

Hypothesis
• A set of constraints on our attributes
• Best Hypothesis -> fits the training set best

Machine Learning – 18CS71 15

Inductive learning hypothesis Best
Fit
Our assumption is that the Fundamental assumption of inductive
learning.
• “Best hypothesis regarding unseen instances is the hypothesis
that best fits the observed training data.”

The inductive learning hypothesis.

• Any hypothesis found to approximate the target function well over a
sufficiently large set of training examples will also approximate the
target function well over other unobserved examples.
Machine Learning – 18CS71 16
 • Hypothesis:
• ? – general
Concept learning as search • Single value – specific
• Φ – more specific

• Concept learning can be viewed as the task of searching through

a large space of hypotheses
• The goal is to find the hypothesis that best fits training examples.
• General-to-Specific Ordering of Hypotheses

Machine Learning – 18CS71 17

Concept learning as search

Machine Learning – 18CS71 18

Module 2- Outline
Chapter 2: Concept Learning

• Concept learning task

• Concept learning as search

• Find-S algorithm

• Version space

• Candidate Elimination algorithm

• Inductive Bias

• Summary

Machine Learning – 18CS71 19

Find-S: Finding A Maximally Specific
Hypothesis
• How can we use the more-general-than partial ordering to
organize the search for a hypothesis consistent with the observed
training examples?
• One way is to begin with the most specific possible hypothesis in
H, then generalize this hypothesis each time it fails to cover an
observed positive training example.
• FIND-S algorithm is used for this purpose.

Machine Learning – 18CS71 20

Find-S Algorithm
To Find Maximally Specific Hypothesis
1. Initialize h to the most specific hypothesis in H
2. For each positive training instance x
For each attribute constrain ai in h
If the constraint ai is satisfied by x then do nothing
Else replace ai in h by the next more general constraint that is satisfied
by x
3. Output Hypothesis h

Machine Learning – 18CS71 21

Illustration

Machine Learning – 18CS71 22

Key Property
• The key property of the Find-S algorithm is that for hypothesis
spaces described by conjunctions of attribute constraints
• Find-S is guaranteed to output the most specific hypothesis
within H that is consistent with the positive training examples.

Machine Learning – 18CS71 23

Machine Learning – 18CS71 24
Find S - Drawback
• Has the learner converged to the correct target concept?
• Why prefer the most specific hypothesis?
• If multiple hypotheses consistent with the training examples, FIND-S will find
the most specific. It is unclear whether we should prefer this hypothesis
• Are the training examples consistent?
• Training examples may contain at least some errors or noise. Such inconsistent
sets of training examples can severely mislead FIND-S, since it ignores negative
examples.
• What if there are several maximally specific consistent
hypotheses?
• There can be several maximally specific hypotheses consistent with the data.
Find S finds only one.
Machine Learning – 18CS71 25
Module 2- Outline
Chapter 2: Concept Learning

• Concept learning task

• Concept learning as search

• Find-S algorithm

• Version space

• Candidate Elimination algorithm

• Inductive Bias

• Summary

Machine Learning – 18CS71 26

Definition: Version Space

This subset of all hypotheses is called the version space with respect to the
hypothesis space H and the training examples D, because it contains all
plausible versions of the target concept.

Machine Learning – 18CS71 27

Example

Machine Learning – 18CS71 28

The List-Then-Eliminate algorithm
• One obvious way to represent the version space is simply to list all
of its members.
• This leads to a simple learning algorithm, which we might call the
List-Then-Eliminate algorithm.

Machine Learning – 18CS71 29

Definition

Machine Learning – 18CS71 30

Module 2- Outline
Chapter 2: Concept Learning

• Concept learning task

• Concept learning as search

• Find-S algorithm

• Version space

• Candidate Elimination algorithm

• Inductive Bias

• Summary

Machine Learning – 18CS71 31

Candidate Elimination Algorithm
• The CEA computes the version space containing all hypotheses
from H that are consistent with an observed sequence of training
examples.
• It begins by
• initializing the version space to the set of all hypotheses in H; that is, G-
most general hypothesis in H

• and initializing the S boundary set to contain the most specific

(least general) hypothesis

Machine Learning – 18CS71 32

Candidate Elimination Algorithm

Machine Learning – 18CS71 33

Example

Machine Learning – 18CS71 34

Example

Machine Learning – 18CS71 35

 +ve instance:

Cont...
Change only
S
-ve instance:
Change only
G

Machine Learning – 18CS71 36

 +ve instance:
Change only
S

Machine Learning – 18CS71 37

 -ve instance:
Change only
G

Machine Learning – 18CS71 38

 +ve instance:
Change only
S

Cancel the
inconsistent

Machine Learning – 18CS71 39

Cont...

Machine Learning – 18CS71 40

Illustration
• The final version space for the EnjoySport concept learning
problem

Machine Learning – 18CS71 41

Module 1- Outline
Chapter 1: Introduction Chapter 2: Concept Learning

• Well posed learning problems • Concept learning task

• Designing a Learning system • Concept learning as search

• Perspective and Issues in Machine Learning • Find-S algorithm

• Summary • Version space

• Candidate Elimination algorithm

• Inductive Bias

• Summary

Machine Learning – 18CS71 42

Inductive Bias
• The CEA will converge toward the true target concept provided it
is given accurate training examples
• The fundamental questions for inductive inference in general.
• What if the target concept is not contained in the hypothesis space?
• How does the size of this hypothesis space influence the ability of the
algorithm to generalize to unobserved instances?
• How does the size of the hypothesis space influence the number of
training examples that must be observed?
• Here we examine them in the context of the CEA.
• The conclusions can be extended to any concept learning system.
Machine Learning – 18CS71 43
Wikipedia Definition
• The inductive bias (also known as learning bias) of a learning
algorithm is the set of assumptions that the learner uses to predict
outputs given inputs that it has not encountered.

Machine Learning – 18CS71 44

English Meaning
Inductive: using a particular set of facts or
ideas to form a general principle

Machine Learning – 18CS71 45

1. A Biased Hypothesis Space
• Suppose we wish to assure that the hypothesis space contains the
unknown target concept.
• The obvious solution is to enrich the hypothesis space to include
every possible hypothesis.
• Consider EnjoySport example in which we restricted the
hypothesis space to include only conjunctions of attribute values.

Machine Learning – 18CS71 46

1. A Biased Hypothesis Space
(Continued)
• Most specific hypothesis consistent with the first two examples

• It incorrectly covers the third (negative) training example

• The problem is that we have Clickbiased the learner to consider only
to add text
conjunctive hypotheses.
• In this case we require a more expressive hypothesis space.
Conjuctive -
Disjunctive -
Negation -

Machine Learning – 18CS71 47

 H – hypothesis space
2. An unbiased learner
• The obvious solution to be an unbiased learner– design hypothesis
space H to represent every teachable concept;
• It should capable of representing every possible subset of the instances
X. In general, the set of all subsets of a set X is called the power-set of
X.
• In general, number of distinct subsets is 2|X|.
• Thus, there are 296, or approximately distinct target concepts that could
be defined over this instance space and that our learner might be called
upon to learn.
• Our conjunctive hypothesis space is able to represent only 973 of
these-a very biased hypothesis space indeed!
Machine Learning – 18CS71 48
 H – hypothesis space
H' - the power set of X.
2. An unbiased learner (Continued)
• Let us reformulate the Enjoysport learning task
• Let H’ represent every subset of instances; that is, let H'
correspond to the power set of X.
• One way to define such an H' is to allow arbitrary disjunctions,
conjunctions, and negations of our earlier hypotheses.
• For instance, the target concept "Sky = Sunny or Sky = Cloudy"
could then be described as

Machine Learning – 18CS71 49

 (xl, x2, x3) - +ve examples
(x4, x5) -ve examples
2. An unbiased learner (Continued)
• Now new problem: we are completely unable to generalize
beyond the observed examples!
• To see why, suppose we present three positive examples (xl, x2, x3)
and two negative examples (x4, x5) to the learner.
• At this point, the S and G boundary of the version space will be

• Here in order to converge to a single, final target concept, we will

have to present every single instance in X as a training example!

Machine Learning – 18CS71 50

3. The Futility of Bias-Free Learning
• The fundamental property of inductive inference:
• A learner that makes no a priori assumptions regarding the
identity of the target concept has no rational basis for classifying
any unseen instances.
• CEA generalizes observed training examples because it was
biased by the implicit assumption that the target concept could
be represented by a conjunction of attribute values.
• If this assumption is correct (and the training examples are error-free),
its classification of new sample will also be correct.
• If this assumption is incorrect, however, it is certain that the CEA will
mis-classify at least some instances from X.
Machine Learning – 18CS71 51
 L – Learning Algorithm
c arbitrary concept
Dc - arbitrary set of training
examples
L(xi, Dc) classification after L

• Consider a concept learning algorithm L for the set of instances

X.
• Let c be an arbitrary concept defined over X, and let Dc =
• { x, c(x) } be an arbitrary set of training examples of c.
• Let L(xi, Dc) denote the classification assigned to the instance xi
by L after training on the data Dc.
• The inductive bias of L is any minimal set of assertions B such
that for any target concept c and corresponding training examples
Dc
Machine Learning – 18CS71 52
Machine Learning – 18CS71 53
Algorithms listed from weakest to
strongest bias
• Rote-Learner: Learning corresponds simply to storing each
observed training example in memory.
• CEA: New instances are classified only in the case where all
members of the current version space agree on the classification.
Otherwise, the system refuses to classify the new instance.
• FIND-S: This algorithm, described earlier, finds the most specific
hypothesis consistent with the training examples. It then uses
this hypothesis to classify all subsequent instances.

Machine Learning – 18CS71 54