Int cti to chi Le i

LzÎuctionw

= *ț '

Lior Rokach
Department of Information Systems Engineering
Ben-Gurion University of the Negev
 bo
Prof. Lior Rokach
Department of Information Systems Engineering
Faculty of Engineering Sciences
Head of the Machine Learning Lab
Ben-Gurion University of the Negev ”,;

Email: Iiorrk@bgu.ac.ii
htt www.ise.b u.ac.ii facult

PhD (2004) from Tel Aviv University

 Machine Learning
• Herbert Alexander Simon:
“Learning is any process by
which a system improves
performance from experience.” ’•
• “Machine Learning is concerned
with computer programs that ’
automatically improve their ”
performance through
Herbert Simon
experience. “ Turing Award 1975
Nobel Prize in Economics 1978
 Why Machine Learning?
• Develop systems that can automatically adapt and customize
themselves to individual users.
— Personalized news or mail filter
• Discover new knowledge from large databases (data mining).
— Market basket analysis (e.g. diapers and beer)
• Ability to mimic human and replace certain monotonous tasks -
which require some intelligence.
• like recognizing handwritten characters
• Develop systems that are too difficult/expensive to construct
manually because they require specific detailed skills or
knowledge tuned to a specific task (knowledge engineering
bottleneck).

'1
• ood of available data (especial with the
advent of the Internet)
• Increasi computational power
• rowing progress in availa e algorithms and
theory developed by researchers
• Increasing support from industries
 lica i

ess
Nat a¿;g age p
ce SIS
BioinformaticsE mmece SS•eee nit
Genneeeexxprreessssiioonn •° Med icine 'r
, ‹ „ y›Lf,•Collaborative filtering object recognition
Th co ce lea ing " syste
• Learn —Im with ex erience at some
task
Improve over task 7,
—With respect to performance measure, P
Based on experience, E.
 Motivating Example
Learning to Filter Spam
Example: Spam Filtering
Spam - is all email the user does not
want to receive and has not asked to
receive
T: Identify Spam Emails

% of spam emails that were filtered

% of ham/ (non-spam) emails that
were incorrectly filtered-out
E: adatabase of emails that were
labelled by users
6enłłnuous
 Th Le ing rocess

Measuring Dimensionality Model

devices
Preprocessin reduction Model Learning
Tl3sting

The
‘real world"
 Th Le rning rocess i Exa I

Measuring Dimensionality ModI2I

devices Preprocessin reduction Model Learning
TI=s1ing

The łys
"real world' results

ensozs

y Ąumber of recipients
y Size of message
t hjumbęr of ąttąchments
y h|umDer of "re's ' in the
subject line
Email Server
 ata Set
Target
Input Attributes
Attribute

ii, 0 2 Germany Gold Ham

1 4 Germany Silver Ham
5 2 Nigeria Bronze Spam
I nstances

2 4 Russia Bronze Spam

3 4 Germany Bronze Ham
0 1 USA Silver Ham
4 2 USA Silver Spam

Numeric Nominal o dinal

 Ste I Le i

Learner Classifier
Database
lnducer Classification Model
Training Set
Induction Algorithm
 Ste I Testi

Database Learner
Training Set lnducer
Classifier
Induction Algorithm
Classification Model
Learning Algorithms

Apñori

eguarizat
,•Committee
{ Ensemble agatioo
Activatiixi function
ans :
Nor Spam (Ham)

Spam

2 3 4 5
Num ber of New Recipients
Nor Spam (Ham)

Spam

Num ber of New Recipients

Non Spam (Ham)
Error

Number of New Recipients

Nor Spam (Ham)

Spam

Num er of New Recipients

Non Spam

Nură er of New Recipients

2 3.2 4.4
 How would you
classify this data?

New Recipients
 How would you
classify this data?

New Recipients
 ew il i sent
1. We first place the new email in the space
2. Classify it according to the subspace in which it resides

New Recipients
 How would you
classify this data?

New Recipients
 How would you
classify this data?

New Recipients
 How would you
classify this data?

New Recipients
 Any of these would
be fine..

.but which is best?

New Recipients
rgi

Define the margin of

a linear classifier as
the width that the
boundary could be
increased by before
hitting a datapoint.

ew Recipients
 The maximum
margin linear
classifier is the
linear classifier with
the, maximum
margin.
This is the simplest
kind of SVM (Called
an LSVM)
New Recipients
Linear SVM
 I ca cove II
i sta ces

How would you

classify this data?

New Recipients
• Ideal the best decision bounda shou be the
one which provides an optimal performance such as
in the followi figure
 I ca cove II
i sta ces

New Recipients
• owever, our satisfaction is premature
because the central aim of esigning a
classifier is to correctly classify novel input

Issue of generalization!
New Recipients New Recipients
2 Errors 0 Errors
Simple model Complicated model
Eval i t’s Le
1. We randomly select a portion of the data to be used for training (the
training set)
2. Train the model on the training set.
3. Once the model is trained, we run the model on the remaining instances
(the test set) to see how it performs

Confusion Matrix

Classified As

Blue Red
Blue 7 1
Red 0 5

New Recipients
 S$U î* ! d ! * * b /*\ * N

as ¡ eda !
 S$U î* ! d ! * * b /*\ * N

as ¡ eda !
 S$U î* ! d ! * * b /*\ * N

as ¡ eda !
 s\ ua!d!*ãg MãN

as ¡ edas i
 La Le rs
• eneralization beyond e training ata is
elayed until a new instance is provi ed to e
system

Training Set Learner Classifier

 La Le rs
Instance-based learning

Training Set
 s\ua !d!*ãg u ã N

¿pasn aq p¡noqsavnseauuasuP p V>!P A •

paqpnoqs eqn

țSãJ N-1 •
 i i re
A flow-chart-like tree structure
Internal node denotes a test on an attribute
Branch represents an outcome of the test
Leaf nodes represent class labels or class distribution
W c/! i‹› i t i’‹ rlJvirlr f.I r fi.'.ttt jim sț›n‹'r! /ii›t.n ‹sxi țxzratlel i’ix:I iiglx„ ai rl Înî›rl ‹•: c:h/rer't»iigIr

. .0. . ? . J . .e xI
To Down I cti of i Trees
Email Len

<1.8 1.8

1 Error 8 Errors

New Recipients
A single level decision tree is also known as
Decision Stump
To Down I cti of i Trees
Email Len

<1.8 1.8

1 Error 8 Errors

New Recip

New Recipients ‹2.3 2.3

4 Errors 11 Errors
To Down I cti of i Trees
Email Len

<1.8 1.8

1 Error 8 Errors

New Recip

New Recipients ‹2.3 2.3

4 Errors 11 Errors
To Down I cti of i Trees
Email Len

<1.8 1.8

Emaíì Len

New Recipients 3 Errors 1 Error

To Down I cti of i Trees
Email Len

<1.8 1.8

Emaíì Len

New Recip
New Recipients

0 Errors 1 Error
•., . „,

New Recipients New Recipients

 Overfitting rfitting

Tree Size
Overtraining: means that it learns the training set too well —it overfits to the
training set such that it performs poorly on the test set.
Underfitting: when model is too simple, both training and test errors are large
 I etwo
Inputs

Output
Age .6
0.6
Gender
of
beingAlive
Stage

Dependent
Independent Weights HiddenLaye Weights variable
variables r
Prediction
 “TO I I ISU I
Inputs

Output
Age .6
0.6
Gender
of
beingAlive
Stage

Dependent
Independent Weights HiddenLaye Weights variable
variables
Prediction
 Inputs

Age

0.6
Gender
“Probability of
beingAlive
Stage

Dependent
Independent Weights HiddenLave Weights variable
variables r
Prediction
 Inputs

Output
Age .6
0.6
Gender
of
beingAlive
Stage

Dependent
Independent Weights HiddenLaye Weights variable
variables r
Prediction
 Age .6
0.6
Gender
of
beingAlive
Stage

Dependent
Independent Weights HiddenLaye Weights variable
variables r
Prediction
 Ensemble Learning
• The idea is to use multiple models to obtain
better predictive performance than could be
obtained from any of the constituent models.

• Boosting involves incrementally building an

ensemble by training each new model instance to
emphasize the training instances that previous
models misclassified.
Exam nsem Wea assifie

Training Combined clossifier

i ri ci les
 Occam’s razor
(14th-century)

• In Latin “lex parsimoniae”, translating to “law

of parsimony”
• The explanation of any phenomenon should
make as few assumptions as
possible, eliminating those that make no
difference in the observable predictions of
the explanatory hypothesis or theory
• The Occam Dilemma: Unfortunately, in
ML, accuracy and simplicity (interpretability)
are in conflict.
 ree Lunch Theo m in ac ine
Lea ing Wolpert, 200
• any two learning there are
just as many situations (appropriately
weighted) in which one is superior
to two as vice versa, according to
any the measures “superiority
 So why developing new algorithms?
• Practitioner are mostly concerned with choosing the most
appropriate algorithm for the problem at hand
• This requires some a priori knowledge —data distribution,
prior probabilities, complexity of the problem, the physics of
the underlying phenomenon, etc.
• The No Free Lunch theorem tells us that —unless we have
some a priori knowledge —simple classifiers (or complex ones
for that matter) are not necessarily better than others.
However, given some a priori information, certain classifiers
may better MATCH the characteristics of certain type of
problems.
• The main challenge of the practitioner is then, to identify the
correct match between the problem and the classifier!
...which is yet another reason to arm yourself with a diverse
set of learner arsenal !
 Less is re
Th Curse i sio I
Ben 19

D=1
 Less is More
The Curse of Dimensionality
• Learning from a high-dimensional feature space requires an
enormous amount of training to ensure that there are
several samples with each combination of values.
• With a fixed number of training instances, the predictive
power reduces as the dimensionality increases.
• As a counter-measure, many dimensionality reduction
techniques have been proposed, and it has been shown
that when done properly, the properties or structures of
the objects can be well preserved even in the lower
dimensions.
• Nevertheless, naively applying dimensionality reduction
can lead to pathological results.
While dimensionality reduction is an important tool in machine learning/data mining, we
must always be aware that it can distort the data in misleading ways.

Above is a two dimensional projection of an intrinsically three dimensional world....

Screen dumps of a short video from www.cs. mu.edu essica DimReducDan er.htm
I recommend you imbed the original video instead

A cloud of points in 3D

Can be projected into 2D

XY or ' or Y2'

In 2D ”‘. we see
a triangle

In 2D YZ we see
a square

In 2D XY we see
a circle
 Less is More?
• In the past the published advice was that high
dimensionality is dangerous.
• But, Reducing dimensionality reduces the amount
of information available for prediction.
• Today: try going in the opposite direction: Instead
of reducing dimensionality, increase it by adding
many functions of the predictor variables.
• The higher the dimensionality of the set of
features, the more likely it is that separation
occurs.
 Meaningfulness of Answers

• A big data-mining risk is that you will

“discover” patterns that are meaningless.
• Statisticians call it Bonferroni’s principle:
(roughly) if you look in more places for
interesting patterns than your amount of
data will support, you are bound to find
crap.
 I of fe i i i I
Track Terrorists
The ine Paradox: a great example of how
not to con uct scientific research.
 Why Tracking Terrorists Is (Almost)
Impossible!
• Suppose we believe that certain groups of evil-
doers are meeting occasionally in hotels to plot
doing evil.
• We want to find (unrelated) people who at
least twice have stayed at the same hotel on
the same day.
 The Details
• 109 people being tracked.
• 1000 days.
• Each person stays in a hotel 1o/oof the time
(10 days out of 1000).
• Hotels hold 100 people (so 105 hotels).
• If everyone behaves randomly (I.e., no evil-
doers) will the data mining detect anything
suspicious?
 it
at
some
Calculations (1)
hotel Same
some
hotel hotel

• P obabili that given perso y and q’ will

be t the sa e hotel given day r :
—1/100 1/100 05 = 109 .
• Probability that y and y will be at the same
hotel on given days d, and :
10 9 X 10 9 —10 1
.
• Pairs of days:
5x10’.
 Cal I i

• roba I that and q will be at the same

hotel on some two ays:
s X 10' X 10 1 - s X 10- 13.

• Pairs of people:
5x 10 17

• Expected number of “suspicious” pairs of

people:
5x 1017 x 5x10 " = 250,000.
 OGCIUSIOG

• Suppose there are (say) 10 pairs of evil-doers

who definitely stayed at the same hotel twice.
• Analysts have to sift through 250,010
candidates to find the 10 real cases.
Not gonna happen.
But how can we improve the scheme?
 ral
• When looking for a property (e “two
rcp Ie r›ayccJ nt !itc saiyr l«otel t›r •.r”), make
sure that e property oes not allow so many
possi lities at ran om ata will surely
produce facts “of interest.”
 Rhine Paradox

• Joseph Rhine was a parapsychologist in the

1950’s who hypothesized that some people
had Extra-Sensory Perception.
• He devised (something like) an experiment
where subjects were asked to guess 10 hidden
cards red or blue.
• He discovered that almost 1 in 1000 had ESP
they were able to get all 10 right!
 Rhine Paradox —
• He told these people they had ESP and called
them in for another test of the same type.
• Alas, he discovered that almost all of them
had lost their ESP.
• What did he conclude?
Answer on next slide.
 i Pa
• e concluded that shoul n’t tell people
they have ESP; it causes them to lose it.
 ral
• nderstanding Bonferroni’s Principle will el
you look a little less stu than a
parapsychologist.
 Insta ty and the ,
Rashomon Effect
• Rashomon is a Japanese movie in which four
people, from different vantage points, witness
a criminal incident. When they come to testify
in court, they all report the same facts, but
their stories of what happened are very
different.
• The Rashomon effect is the effect of the
subjectivity of perception onrecollection.
• The Rashomon Effect in ML is that there is
often a multitude of classifiers of giving about
the same minimum error rate.
• For example in decition trees, if the training set
is perturbed only slightly, I can get a tree quite
different from the original but with almost the
same test set error.
 The Wisdom of Crowds
Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes

Business, Economies, Societies and Nations

• Under certain controlled conditions, the

T HE \YISD 0III
aggregation of information in groups,
0 F C R0 MDS resulting in decisions that are often
SUR0WlEGfil
superior to those that can been made by
any single - even experts.
• Imitates our second nature to seek
several opinions before making any
crucial decision. We weigh the individual
opinions, and combine them to reach a
final decision
 Committees of Experts
“ . a medical school that has the objective that all
students, given a problem, come up with an identical
solution”

• There is not much point in setting up a committee of

experts from such a group - such a committee will not
improve on the judgment of an individual.

• Consider:
 Ot Le ing Ta ks

silk z on rules Kernel

Exploitation Supervised learning
Unsupervised Li ea
’
Texh classification Lazg lerner Linearly separable
Exploration ° '
ise Le rni i I

New Recipients
 is Le rni i I
Multi-label learning refers to the classification problem where each example can be
assigned to multiple class labels simultaneously

ew Recipients
 ise rning Regre i
Find a relationship between a numeric dependent variable and one or more
independent variables

New Recipients
 ise ing I steri
Clustering is the assignment of a set of observations into subsets (called
clusters) so that observations in the same cluster are similar in some sense

New Recipients
 ise ing—An etecti
Detecting patterns in a given data set that do not conform to an established
normal behavior.

New Recipients
 Source of Training Data
• Provided random examples outside of the learner’s control.
— Passive Learning
— Negative examples available or only positive? Semi-Supervised Learning
— Imbalanced
• Good training examples selected by a "benevolent teacher."
— "Near miss" examples
Learner can query an oracle about class of an unlabeled example in the
environment.
— Active Learning
• Learner can construct an arbitrary example and query an oracle for its label.
Learner can run directly in the environment without any human guidance and
obtain feedback.
— Reinforcement Learning
• There is no existing class concept
— A form of discovery
— Unsupervised Learning
• Clustering
• Association Rules
 Other Learning Tasks
• Other Supervised Learning Settings
— Multi-Class Classification
— Multi-Label Classification
— Semi-supervised classification —make use of labeled and unlabeled data
— One Class Classification —only instances from one label are given
• Ranking and Preference Learning
• Sequence labeling
• Cost-sensitive Learning
• Online learning and Incremental Learning- Learns one instance at a
time.
• Concept Drift
• Multi-Task and Transfer Learning
• Collective classification —When instances are dependent!
 Software

RapidMiner
Want to Le re
 Want to Learn More?

• Thomas Mitchell (1997), Machine Learning, Mcgraw-Hill.

• R. Duda, P. Hart, and D. Stork (2000), Pattern

Classification, Second Edition.

• Ian H. Witten, Eibe Frank, Mark A. Hall (2011), Data

Mining: Practical Machine Learning Tools and
Techniques, Third Edition, The Morgan Kaufmann

• Oded Maimon, Lior Rokach (2010), Data Mining and

Knowledge Discovery Handbook, Second Edition, Springer.