Scribd
Upload
21 views

10606_Fa21_Lecture_4_Linear_Regression

Maths for ML

Uploaded by

Cin Naw
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
21 views

10606_Fa21_Lecture_4_Linear_Regression

Maths for ML

Uploaded by

Cin Naw
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 21

Announcements

HW1
▪ Due Mon 9/20, 11:59 pm
▪ Online + Written components
▪ Reach out for help (probability, LaTeX, anything!)

Drop deadline
▪ Fri 9/17

Quizzes

Vocab and Notation doc


▪ Notes section added!
Mathematical
Foundations for
Machine Learning

Linear Regression

Instructor: Pat Virtue


Today
Linear regression
▪ Linear algebra formulation
▪ Linear algebra properties
▪ Calculus
▪ Multivariate calculus
▪ Calculus with linear algebra
▪ Optimization
▪ Convex functions (briefly)
▪ Closed-form solutions
Linear Regression
As a reason to learn linear algebra, calculus, and optimization
Last time: Fitting a linear model

𝑖 𝑖 4
𝒟= 𝑥 ,𝑦 𝑖=1
= { −1, 2.5 ,
(7, −1.5),
(−5, 4.5),
(1.5, 1.25)}
Last time

𝑖 𝑖 4
𝒟= 𝑥 ,𝑦 𝑖=1
= { −1, 2.5 ,
(7, −1.5),
(−5, 4.5),
(1.5, 1.25)}
Linear Regression
Last recitation
1) Found 𝑚 that minimized MSE with model 𝑦 = 𝑚𝑥, 𝑥∈ℝ
2) Found 𝑚, 𝑏 that minimized MSE with model 𝑦 = 𝑚𝑥 + 𝑏, 𝑥 ∈ ℝ
Tons of summations!

Today
Linear algebra formulation
1) Find 𝑚 that minimizes MSE with model 𝑦 = 𝑚𝑥, 𝑥∈ℝ
2) Find 𝐰 that minimizes MSE with model 𝑦 = 𝐰 𝑇 𝐱, 𝐱 ∈ ℝ2
3) Find 𝐰, 𝑏 that minimizes MSE with model 𝑦 = 𝐰 𝑇 𝐱 + 𝑏, 𝐱 ∈ ℝ𝑀
Poll 1
True or False
For any matrices 𝐴, 𝐵 ∈ ℝ𝑀×𝑁 , the following always holds:
𝐴+𝐵 𝑇 = 𝐴𝑇 + 𝐵𝑇
Poll 2
True or False
For any matrices 𝐴, 𝐵 ∈ ℝ𝑀×𝑁 , 𝐶 ∈ ℝ𝐾×𝑀 , the following always holds:
𝐶𝐴 + 𝐶𝐵 = 𝐴 + 𝐵 𝐶
Poll 3
Select ALL that apply
2
Which of the following are equal to 𝑧 2 for any vector 𝑧 ∈ ℝ𝑁 ?

A. 𝑧 𝑇 𝑧
B. 𝑧𝑧 𝑇
𝑧12
C. ⋮
𝑧𝑁2
D. None of the above
E. I have no idea
Poll 4
Select ALL that apply
Which of the following hold for any 𝐮, 𝐯 ∈ ℝ𝑁 and 𝐴, 𝐵 ∈ ℝ𝑀×𝑁 ?

A. 𝐮𝑇 𝐯 = 𝐯 𝑇 𝐮
B. 𝐴𝑇 𝐵 = 𝐵𝑇 𝐴
C. None of the above
D. I have no idea
Linear Algebra
Properties
The following hold for any 𝐴, 𝐵 ∈ ℝ𝑀×𝑁

▪ 𝐴𝐵 𝑇 = 𝐵𝑇 𝐴𝑇
𝑇 𝑇
▪ 𝐴 𝐵 = 𝐵𝑇 𝐴
𝑇 𝑇
▪ 𝐴𝐵 = 𝐵𝐴𝑇

The following hold for any 𝐮, 𝐯 ∈ ℝ𝑁


▪ 𝐮𝑇 𝐯 = 𝐯 𝑇 𝐮
(Why is this ok if we can’t say 𝐴𝑇 𝐵 = 𝐵𝑇 𝐴)
Linear Regression
Linear algebra formation
Find 𝑚 that minimizes MSE with model 𝑦ො = 𝑚𝑥, 𝑥 ∈ ℝ

𝑖 𝑖 4
𝒟= 𝑥 ,𝑦 𝑖=1

𝑥 𝑖 𝑦 (𝑖)
2 0
0 -1
-3 0
1 2
Linear Regression
Linear algebra formation
Find 𝑚 that minimizes MSE with model 𝑦ො = 𝑚𝑥, 𝑥 ∈ ℝ
1 2
𝐽 𝑚; 𝐱, 𝐲 = 𝐲 − 𝑚𝐱 2
𝑁

𝑖 𝑖 4
𝒟= 𝑥 ,𝑦 𝑖=1

𝑥 𝑖 𝑦 (𝑖)
2 0
0 -1
-3 0
1 2
1 input feature
Derive the following expansion 𝐽(𝑚; 𝒟) No bias term
𝑦ො = 𝑚𝑥 𝑚, 𝑥 ∈ ℝ
Figure out each step, given the provided justification All data
𝐲, 𝐱 ∈ ℝ𝑁

Justification
1 2
𝐽 𝑚; 𝐱, 𝐲 = 𝐲 − 𝑚𝐱 2
𝑁

2
= 𝐳 2 = 𝐳𝑇 𝐳
𝑇
= 𝐴+𝐵 = 𝐴𝑇 + 𝐵𝑇
𝐴 + 𝐵 𝐶 = 𝐴𝐶 + 𝐵𝐶 and
= 𝐶 𝐴 + 𝐵 = 𝐶𝐴 + 𝐶𝐵
1
= 𝐲 𝑇 𝐲 − 2𝑚𝐲 𝑇 𝐱 + 𝑚2 𝐱 𝑇 𝐱 𝐮𝑇 𝐯 = 𝐯 𝑇 𝐮
𝑁
Linear Regression
Linear algebra formation
Find 𝑚 that minimizes MSE with model 𝑦ො = 𝑚𝑥, 𝑥 ∈ ℝ
1 2
𝐽 𝑚; 𝐱, 𝐲 = 𝐲 − 𝑚𝐱 2
𝑁
1
= 𝐲 𝑇 𝐲 − 2𝑚𝐲 𝑇 𝐱 + 𝑚2 𝐱 𝑇 𝐱
𝑁

𝑖 𝑖 4
What shape is 𝐽(𝑚)? 𝒟= 𝑥 ,𝑦 𝑖=1

𝑥 𝑖 𝑦 (𝑖)
2 0
0 -1
-3 0
1 2
Poll 5
True or False
The following MSE objective function is always a parabola going up (a U
shape, rather than ∩) regardless of the data in 𝐱, 𝐲:
1 2
𝐽 𝑚; 𝐱, 𝐲 = 𝐲 − 𝑚𝐱 2
𝑁
1
= 𝐲 𝑇 𝐲 − 2𝑚𝐲 𝑇 𝐱 + 𝑚2 𝐱 𝑇 𝐱
𝑁
Linear Regression
1
𝐽 𝑚; 𝐱, 𝐲 = 𝐲 𝑇 𝐲 − 2𝑚𝐲 𝑇 𝐱 + 𝑚2 𝐱 𝑇 𝐱
𝑁
Linear and Convex Functions
Optimization
Linear function
If 𝑓(𝒙) is linear, then:
▪ 𝑓 𝒙+𝒛 =𝑓 𝒙 +𝑓 𝒛
▪ 𝑓 𝛼𝒙 = 𝛼𝑓 𝒙 ∀𝛼
▪ 𝑓 𝛼𝒙 + 1 − 𝛼 𝒛 = 𝛼𝑓 𝒙 + 1 − 𝛼 𝑓 𝒛 ∀𝛼
Optimization
Convex function
If 𝑓(𝒙) is convex, then:
▪ 𝑓 𝛼𝒙 + 1 − 𝛼 𝒛 ≤ 𝛼𝑓 𝒙 + 1 − 𝛼 𝑓 𝒛 ∀0≤𝛼 ≤1

Convex optimization
If 𝑓(𝒙) is convex, then:
▪ Every local minimum is also a
global minimum ☺

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy