Peer Instruction

 I pose a challenge question (usually multiple choice), which will

help solidify understanding of topics we have studied
– Might not just be one correct answer
 You each get some time (1-2 minutes) to come up with your
answer and vote – use Mentimeter
 Then you get some time to convince your group (neighbors)
why you think you are right (2-3 minutes)
– Learn from and teach each other!
 You vote again. May change your vote if you want.
 We discuss together the different responses, show the votes,
give you opportunity to justify your thinking, and give you
further insights

CS 478 – Peer Instruction 1

 Peer Instruction (PI) Why

 Studies show this approach improves learning

 Learn by doing, discussing, and teaching each other
– Curse of knowledge/expert blind-spot
– Compared to talking with a peer who just figured it out and who can
explain it in your own jargon
– You never really know something until you can teach it to someone
else – More improved learning!
 Just as important to understand why wrong answers are
wrong, or how they could be changed to be right.
– Learn to reason about your thinking and answers
 More enjoyable - You are involved and active in the learning

CS 478 – Peer Instruction 2

 How Groups Interact
 Best if group members have different initial answers
 3 is the “magic” group number
– You can self-organize "on-the-fly" or sit together specifically to be a
group
– Can go 2-4 on a given day to make sure everyone is involved
 Teach and learn from each other: Discuss, reason, articulate
 If you know the answer, listen to where colleagues are coming
from first, then be a great humble teacher, you will also learn by
doing that, and you’ll be on the other side in the future
– I can’t do that as well because every small group has different
misunderstandings and you get to focus on your particular questions
 Be ready to justify to the class your vote and justifications!

CS 478 – Peer Instruction 3

CS 478 - Perceptrons 4
Basic Neuron

CS 478 - Perceptrons 5
Expanded Neuron

CS 478 - Perceptrons 6
 Perceptron Learning Algorithm

 First neural network learning model in the 1960’s

 Simple and limited (single layer models)
 Basic concepts are similar for multi-layer models so this is
a good learning tool
 Still used in many current applications (modems, etc.)

CS 478 - Perceptrons 7
Perceptron Node – Threshold Logic Unit

x1 w1

x2 w2 𝜃q z

xn wn

n
1 if åx w ³q
i =1
i i

z= n
0 if åx w <q
i =1
i i

CS 478 - Perceptrons 8
 Perceptron Node – Threshold Logic Unit

x1 w1

x2 w2 𝜃 z

xn wn

• Learn weights such that an objective 1 if åx w ³q

i =1
i i
function is maximized. z= n
• What objective function should we use?
• What learning algorithm should we use?
0 if åx w <q
i =1
i i

CS 478 - Perceptrons 9
 Perceptron Learning Algorithm

x1 .4

.1 z

x2 -.2

åx w ³q
x1 x2 t
1 if i i
i =1
.8 .3 1 z= n
.4 .1 0 0 if åx w <q
i =1
i i

CS 478 - Perceptrons 10
 First Training Instance

.8 .4

.1 z =1

.3 -.2 net = .8*.4 + .3*-.2 = .26

åx w ³q
x1 x2 t
1 if i i
i =1
.8 .3 1 z= n
.4 .1 0 0 if åx w <q
i =1
i i

CS 478 - Perceptrons 11
 Second Training Instance

.4 .4

.1 z =1

.1 -.2 net = .4*.4 + .1*-.2 = .14

åx w ³q
x1 x2 t
1 if i i
.8 .3 1 z=
i =1 Dwi = (t - z) * c * xi
n
.4 .1 0 0 if åx w <q
i =1
i i

CS 478 - Perceptrons 12
 Perceptron Rule Learning
Dwi = c(t – z) xi
 Where wi is the weight from input i to perceptron node, c is the learning
rate, t is the target for the current instance, z is the current output, and xi is
ith input
 Least perturbation principle
– Only change weights if there is an error
– small c rather than changing weights sufficient to make current pattern correct
– Scale by xi
 Create a perceptron node with n inputs
 Iteratively apply a pattern from the training set and apply the perceptron
rule
 Each iteration through the training set is an epoch
 Continue training until total training set error ceases to improve
 Perceptron Convergence Theorem: Guaranteed to find a solution in finite
time if a solution exists

CS 478 - Perceptrons 13
CS 478 - Perceptrons 14
 Augmented Pattern Vectors
1 0 1 -> 0
1 0 0 -> 1
Augmented Version
1 0 1 1 -> 0
1 0 0 1 -> 1
 Treat threshold like any other weight. No special case.
Call it a bias since it biases the output up or down.
 Since we start with random weights anyways, can ignore
the - notion, and just think of the bias as an extra
available weight. (note the author uses a -1 input)
 Always use a bias weight

CS 478 - Perceptrons 15
 Perceptron Rule Example
 Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
 Assume a learning rate c of 1 and initial weights all 0: Dwi = c(t – z) xi
 Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output DW

001 1 0 0000

CS 478 - Perceptrons 16
 Example
 Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
 Assume a learning rate c of 1 and initial weights all 0: Dwi = c(t – z) xi
 Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output DW

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000

CS 478 - Perceptrons 17
 Example
 Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
 Assume a learning rate c of 1 and initial weights all 0: Dwi = c(t – z) xi
 Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output DW

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000 0 0 1 1 1 1
101 1 1 1111

CS 478 - Perceptrons 18
 **Challenge Question** - Perceptron
 Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
 Assume a learning rate c of 1 and initial weights all 0: Dwi = c(t – z) xi
 Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output DW

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000 0 0 1 1 1 1
101 1 1 1111

 Once this converges the final weight vector will be

A. 1111
B. -1 0 1 0
C. 0000
D. 1000
E. None of the above
CS 478 - Perceptrons 19
 Example
 Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
 Assume a learning rate c of 1 and initial weights all 0: Dwi = c(t – z) xi
 Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output DW

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000 0 0 1 1 1 1
101 1 1 1111 3 1 0 0 0 0
011 1 0 1111

CS 478 - Perceptrons 20
 Example
 Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
 Assume a learning rate c of 1 and initial weights all 0: Dwi = c(t – z) xi
 Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output DW

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000 0 0 1 1 1 1
101 1 1 1111 3 1 0 0 0 0
011 1 0 1111 3 1 0 -1 -1 -1
001 1 0 1000

CS 478 - Perceptrons 21
 Example
 Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
 Assume a learning rate c of 1 and initial weights all 0: Dwi = c(t – z) xi
 Training set 0 0 1 -> 0
1 1 1 -> 1
1 0 1 -> 1
0 1 1 -> 0

Pattern Target Weight Vector Net Output DW

001 1 0 0000 0 0 0 0 0 0
111 1 1 0000 0 0 1 1 1 1
101 1 1 1111 3 1 0 0 0 0
011 1 0 1111 3 1 0 -1 -1 -1
001 1 0 1000 0 0 0 0 0 0
111 1 1 1000 1 1 0 0 0 0
101 1 1 1000 1 1 0 0 0 0
011 1 0 1000 0 0 0 0 0 0

CS 478 - Perceptrons 22
 Perceptron Homework
 Assume a 3 input perceptron plus bias (it outputs 1 if net > 0, else 0)
 Assume a learning rate c of 1 and initial weights all 1: Dwi = c(t – z) xi
 Show weights after each pattern for just one epoch
 Training set 1 0 1 -> 0
1 1 0 -> 0
1 0 1 -> 1
0 1 1 -> 1

Pattern Target Weight Vector Net Output DW

1 1 1 1

CS 478 - Perceptrons 23
 Training Sets and Noise

 Assume a Probability of Error at each bit

 0 0 1 0 1 1 0 0 1 1 0 -> 0 1 1 0
 i.e. P(error) = .05

 Or a probability that the algorithm is applied wrong

(opposite) occasionally

 Averages out over learning

CS 478 - Perceptrons 24
 If no bias weight then
the hyperplane must go
through the origin
CS 478 - Perceptrons 25
Linear Separability

CS 478 - Perceptrons 26
Linear Separability and Generalization

When is data noise vs. a legitimate exception

CS 478 - Perceptrons 27
Limited Functionality of Hyperplane

CS 478 - Perceptrons 28
 How to Handle Multi-Class Output
 This is an issue with any learning model which only
supports binary classification (perceptron, SVM, etc.)
 Create 1 perceptron for each output class, where the
training set considers all other classes to be negative
examples
– Run all perceptrons on novel data and set the output to the class of
the perceptron which outputs high
– If there is a tie, choose the perceptron with the highest net value
 Create 1 perceptron for each pair of output classes, where
the training set only contains examples from the 2 classes
– Run all perceptrons on novel data and set the output to be the class
with the most wins (votes) from the perceptrons
– In case of a tie, use the net values to decide
– Number of models grows by the square of the output classes

CS 478 - Perceptrons 29
UC Irvine Machine Learning Data Base
Iris Data Set
4.8,3.0,1.4,0.3, Iris-setosa
5.1,3.8,1.6,0.2, Iris-setosa
4.6,3.2,1.4,0.2, Iris-setosa
5.3,3.7,1.5,0.2, Iris-setosa
5.0,3.3,1.4,0.2, Iris-setosa
7.0,3.2,4.7,1.4, Iris-versicolor
6.4,3.2,4.5,1.5, Iris-versicolor
6.9,3.1,4.9,1.5, Iris-versicolor
5.5,2.3,4.0,1.3, Iris-versicolor
6.5,2.8,4.6,1.5, Iris-versicolor
6.0,2.2,5.0,1.5, Iris-viginica
6.9,3.2,5.7,2.3, Iris-viginica
5.6,2.8,4.9,2.0, Iris-viginica
7.7,2.8,6.7,2.0, Iris-viginica
6.3,2.7,4.9,1.8, Iris-viginica

CS 478 – Introduction 30
 Objective Functions: Accuracy/Error
 How do we judge the quality of a particular model (e.g.
Perceptron with a particular setting of weights)
 Consider how accurate the model is on the data set
– Classification accuracy = # Correct/Total instances
– Classification error = # Misclassified/Total instances (= 1 – acc)
 Usually minimize a Loss function (aka cost, error)
 For real valued outputs and/or targets
– Pattern error = Target – output: Errors could cancel each other
 S|ti – zi| (L1 loss)

 Common approach is Squared Error = S(ti – zi)2 (L2 loss)

– Total sum squared error = S pattern sqaured errors = S S (ti – zi)2

 For nominal data, pattern error is typically 1 for a mismatch and
0 for a match
– For nominal (including binary) output and targets, SSE and
classification error are equivalent

CS 478 - Perceptrons 31
 Mean Squared Error

 Mean Squared Error (MSE) – SSE/n where n is the number

of instances in the data set
– This can be nice because it normalizes the error for data sets of
different sizes
– MSE is the average squared error per pattern
 Root Mean Squared Error (RMSE) – is the square root of
the MSE
– This puts the error value back into the same units as the features
and can thus be more intuitive
 Since we squared the error on the SSE
– RMSE is the average distance (error) of targets from the outputs in
the same scale as the features

CS 478 - Perceptrons 32
 **Challenge Question** - Error
 Given the following data set, what is the L1 (S|ti – zi|), SSE/L2
(S(ti – zi)2), MSE, and RMSE error for the entire data set?
x y Output Target Data Set
2 -3 1 1
0 1 0 1
.5 .6 .8 .2
L1 x
SSE x
MSE x
RMSE x

A. .4 1 1 1
B. 1.6 2.36 1 1
C. .4 .64 .21 0.46
D. 1.6 1.36 .68 .82
E. None of the above
CS 478 - Perceptrons 33
 **Challenge Question** - Error
 Given the following data set, what is the L1 (S|ti – zi|), SSE/L2
(S(ti – zi)2), MSE, and RMSE error for the entire data set?
x y Output Target Data Set
2 -3 1 1
0 1 0 1
.5 .6 .8 .2
L1 1.6
SSE 1.36
MSE 1.36/3 = .45
RMSE .45^.5 = .67

A. .4 1 1 1
B. 1.6 2.36 1 1
C. .4 .64 .21 0.46
D. 1.6 1.36 .68 .82
E. None of the above
CS 478 - Perceptrons 34
 SSE Homework
 Given the following data set, what is the L1, SSE (L2),
MSE, and RMSE error of Output1, Output2, and the entire
data set? Fill in cells that have an x.

x y Output1 Target1 Output2 Target 2 Data Set

-1 -1 0 1 .6 1.0
-1 1 1 1 -.3 0
1 -1 1 0 1.2 .5
1 1 0 0 0 -.2
L1 x x x
SSE x x x
MSE x x x
RMSE x x x

CS 478 - Perceptrons 35
 Gradient Descent Learning: Minimize
(Maximze) the Objective Function

Error Landscape
SSE:
Sum
Squared
Error
S (ti – zi)2

0
Weight Values

CS 478 - Perceptrons 36
 Deriving a Gradient Descent Learning
Algorithm
 Goal is to decrease overall error (or other objective
function) each time a weight is changed
 Total Sum Squared error one possible objective function E:
S (ti – zi)2
¶E
 Seek a weight changing algorithm such that is
¶wij
negative
 If a formula can be found then we have a gradient descent
learning algorithm
 Delta rule is a variant of the perceptron rule which gives a
gradient descent learning algorithm with perceptron nodes

CS 478 - Perceptrons 37
 Delta rule algorithm
 Delta rule uses (target - net) before the net value goes through the
threshold in the learning rule to decide weight update

Dwi = c(t - net)xi

 Weights are updated even when the output would be correct
 Because this model is single layer and because of the SSE objective
function, the error surface is guaranteed to be parabolic with only one
minima
 Learning rate
– If learning rate is too large can jump around global minimum
– If too small, will work, but will take a longer time
– Can decrease learning rate over time to give higher speed and still
attain the global minimum (although exact minimum is still just for
training set and thus…)

CS 478 - Perceptrons 38
 Batch vs Stochastic Update
 To get the true gradient with the delta rule, we need to sum errors over
the entire training set and only update weights at the end of each epoch
 Batch (gradient) vs stochastic (on-line, incremental)
– With the stochastic delta rule algorithm, you update after every pattern, just like
with the perceptron algorithm (even though that means each change may not be
exactly along the true gradient)
– Stochastic is more efficient and best to use in almost all cases, though not all have
figured it out yet
 We’ll take about this a little bit when we get to Backpropagation

CS 478 - Perceptrons 39
 Perceptron rule vs Delta rule
 Perceptron rule (target - thresholded output) guaranteed to
converge to a separating hyperplane if the problem is
linearly separable. Otherwise may not converge – could
get in cycle
 Singe layer Delta rule guaranteed to have only one global
minimum. Thus it will converge to the best SSE solution
whether the problem is linearly separable or not.
– Could have a higher misclassification rate than with the perceptron
rule and a less intuitive decision surface – we will discuss this later
with regression
 Stopping Criteria – For these models stop when no longer
making progress
– When you have gone a few epochs with no significant
improvement/change between epochs (including oscillations)

CS 478 - Perceptrons 40
 Exclusive Or

1 0

X2

0 1
X1

Is there a dividing hyperplane?

CS 478 - Perceptrons 41
 Linearly Separable Boolean Functions
 d = # of dimensions

CS 478 - Perceptrons 42
 Linearly Separable Boolean Functions
 d = # of dimensions
 P = 2d = # of Patterns

CS 478 - Perceptrons 43
 Linearly Separable Boolean Functions
 d = # of dimensions
 P = 2d = # of Patterns
 2P = 22d= # of Functions
n Total Functions Linearly Separable Functions
0 2 2
1 4 4
2 16 14

CS 478 - Perceptrons 44
 Linearly Separable Boolean Functions
 d = # of dimensions
 P = 2d = # of Patterns
 2P = 22d= # of Functions
n Total Functions Linearly Separable Functions
0 2 2
1 4 4
2 16 14
3 256 104
4 65536 1882
5 4.3 × 109 94572
6 1.8 × 1019 1.5 × 107
7 3.4 × 1038 8.4 × 109

CS 478 - Perceptrons 45
 Linearly Separable Functions

d
å
(P-1)!
LS(P,d) = 2 (P-1-i)!i! for P > d
i=0

= 2P for P ≤ d

(All patterns for d=P)

i.e. all 8 ways of dividing 3 vertices of a
cube for d=P=3

Where P is the # of patterns for training and

d is the # of inputs

lim
d -> ∞ (# of LS functions) = ∞
CS 478 - Perceptrons 46
 Linear Models which are Non-Linear in the
Input Space
n
 So far we have used f (x,w) = sign(å wi x i )
1=1

 We could preprocess the inputs in a non-linear way and do

m
f (x,w) = sign(å wif i (x ))
1=1
 To the perceptron algorithm it looks just the same and can use
the same learning algorithm, it just has different inputs - SVM
 For example, for a problem with two inputs x and y (plus the
bias), we could also add the inputs x2, y2, and x·y
 The perceptron would just think it is a 5 dimensional task, and it
is linear in those 5 dimensions
– But what kind of decision surfaces would it allow for the original 2-d
input space?

CS 478 - Perceptrons 47
 Quadric Machine

 All quadratic surfaces (2nd order)

– ellipsoid
– parabola
– etc.
 That significantly increases the number of problems that
can be solved
 But still many problem which are not quadrically separable
 Could go to 3rd and higher order features, but number of
possible features grows exponentially
 Multi-layer neural networks will allow us to discover high-
order features automatically from the input space
CS 478 - Perceptrons 48
 Simple Quadric Example

-3 -2 -1 0 1 2 3
f1
 Perceptron with just feature f1 cannot separate the data
 Could we add a transformed feature to our perceptron?

CS 478 - Perceptrons 49
 Simple Quadric Example

-3 -2 -1 0 1 2 3
f1
 Perceptron with just feature f1 cannot separate the data
 Could we add a transformed feature to our perceptron?
 f2 = f12

CS 478 - Perceptrons 50
 Simple Quadric Example

f2

-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3
f1 f1
 Perceptron with just feature f1 cannot separate the data
 Could we add another feature to our perceptron f2 = f12
 Note could also think of this as just using feature f1 but
now allowing a quadric surface to separate the data
CS 478 - Perceptrons 51
 Quadric Machine Homework
 Assume a 2 input perceptron expanded to be a quadric perceptron (it outputs 1 if
net > 0, else 0). Note that with binary inputs of -1, 1, that x2 and y2 would always
be 1 and thus do not add info and are not needed (they would just act like two more
bias weights)
 Assume a learning rate c of .4 and initial weights all 0: Dwi = c(t – z) xi
 Show weights after each pattern for one epoch with the following non-linearly
separable training set (XOR).
 Has it learned to solve the problem after just one epoch?
 Which of the quadric features are actually needed to solve this training set?

x y Target
-1 -1 0
-1 1 1
1 -1 1
1 1 0

CS 478 - Regression 52