CS5670: Intro to Computer Vision

Noah Snavely

Lecture 1: Images and image filtering

Hybrid Images, Oliva et al., http://olivalab.mit.edu/hybridimage.htm

CS5670: Intro to Computer Vision
Noah Snavely

Lecture 1: Images and image filtering

Hybrid Images, Oliva et al., http://olivalab.mit.edu/hybridimage.htm

CS5670: Intro to Computer Vision
Noah Snavely

Lecture 1: Images and image filtering

Hybrid Images, Oliva et al., http://olivalab.mit.edu/hybridimage.htm

Reading
• Szeliski, Chapter 3.1-3.3
Announcements
• Project 1 (Hybrid Images) will be released early next week
– Code due Friday, Feb 10
– This project will be done solo
– Other projects planned to be done in groups of 2

• Project is in Python – we will provide skeleton code and

instructions for setting up a Python environment for the
project
What is an image?
What is an image?

Digital Camera

We’ll focus on these in this class

(More on this process later) The Eye

Source: A. Efros
What is an image?
• A grid (matrix) of intensity values
255 255 255 255 255 255 255 255 255 255 255 255

255 255 255 255 255 255 255 255 255 255 255 255

255 255 255 20 0 255 255 255 255 255 255 255

=
255 255 255 75 75 75 255 255 255 255 255 255

255 255 75 95 95 75 255 255 255 255 255 255

255 255 96 127 145 175 255 255 255 255 255 255

255 255 127 145 175 175 175 255 255 255 255 255

255 255 127 145 200 200 175 175 95 255 255 255

255 255 127 145 200 200 175 175 95 47 255 255

255 255 127 145 145 175 127 127 95 47 255 255

255 255 74 127 127 127 95 95 95 47 255 255

255 255 255 74 74 74 74 74 74 255 255 255

255 255 255 255 255 255 255 255 255 255 255 255

255 255 255 255 255 255 255 255 255 255 255 255

(common to use one byte per value: 0 = black, 255 = white)

What is an image?
• Can think of a (grayscale) image as a function f from R2 to R:
– f (x,y) gives the intensity at position (x,y)

f (x, y)

snoop
3D view

– A digital image is a discrete (sampled, quantized) version of this function

Image transformations
• As with any function, we can apply operators to an image

g (x,y) = f (x,y) + 20 g (x,y) = f (-x,y)

• Today we’ll talk about a special kind of operator,

convolution (linear filtering)
Filters
• Filtering
– Form a new image whose pixel values are a combination of the
original pixel values
• Why?
– To get useful information from images
• E.g., extract edges or contours (to understand shape)
– To enhance the image
• E.g., to remove noise
• E.g., to sharpen and “enhance image” a la CSI
– A key operator in Convolutional Neural Networks
Canonical Image Processing problems
• Image Restoration
– denoising
– deblurring
• Image Compression
– JPEG, HEIF, MPEG, …
• Locating Structural Features
– corners
– edges
Question: Noise reduction
• Given a camera and a still scene, how can you reduce
noise?

Take lots of images and average them!

What’s the next best thing?
Source: S. Seitz
Image filtering
• Modify the pixels in an image based on some function of
a local neighborhood of each pixel

10 5 3 Some function
4 5 1 7
1 1 7
Local image data Modified image data

Source: L. Zhang
Linear filtering
• One simple version of filtering: linear filtering (cross-correlation,
convolution)
– Replace each pixel by a linear combination (a weighted sum) of its
neighbors
• The prescription for the linear combination is called the “kernel”
(or “mask”, “filter”)
10 5 3 0 0 0
4 6 1 0 0.5 0 8
1 1 8 0 1 0.5
Local image data kernel Modified image data

Source: L. Zhang
Cross-correlation
Let be the image, be the kernel (of size
2k+1 x 2k+1), and be the output image

This is called a cross-correlation operation:

• Can think of as a “dot product” between local

neighborhood and kernel for each pixel
Convolution
• Same as cross-correlation, except that the kernel is
“flipped” (horizontally and vertically)

This is called a convolution operation:

• Convolution is commutative and associative

Convolution

Adapted from F. Durand

Mean filtering

0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 20 10

=
0 0 0 90 90 90 90 90 0 0 0 20 40 60 60 60 40 20

*
1 1 1 0 0 0 90 90 90 90 90 0 0 0 30 60 90 90 90 60 30
0 0 0 90 90 90 90 90 0 0 0 30 50 80 80 90 60 30
1 1 1
0 0 0 90 0 90 90 90 0 0 0 30 50 80 80 90 60 30
1 1 1
0 0 0 90 90 90 90 90 0 0 0 20 30 50 50 60 40 20
0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 30 20 10
0 0 90 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Mean filtering/Moving
average
Mean filtering/Moving
average
Mean filtering/Moving
average
Mean filtering/Moving
average
Mean filtering/Moving
average
Mean filtering/Moving
average
Linear filters: examples

0 0 0

* 0
0
1
0
0
0

Original

Source: D. Lowe
What image operation does filtering with this
kernel perform?
([0 0 0; 0 1 0; 0 0 0])

ⓘ Start presenting to display the poll results on this slide.

Linear filters: examples

=
0 0 0

* 0
0
1
0
0
0

Original Identical image

Source: D. Lowe
Linear filters: examples

0 0 0

* 1
0
0
0
0
0

Original

Source: D. Lowe
What image operation does filtering with this
kernel perform? ([0 0 0; 1 0 0; 0 0 0])

ⓘ Start presenting to display the poll results on this slide.

Linear filters: examples

=
0 0 0

* 1
0
0
0
0
0

Original Shifted left by 1 pixel

Source: D. Lowe
Linear filters: examples

=
1 1 1

* 1
1
1
1
1
1

Original Blur (with a mean filter)

Source: D. Lowe
Linear filters: examples

=
0 0 0 1 1 1
-
* 0
0
2
0
0
0
1
1
1
1
1
1

Sharpening filter
Original
(accentuates edges)

Source: D. Lowe
Sharpening

Source: D. Lowe
Smoothing with box filter revisited

Source: D. Forsyth
Gaussian kernel

Source: C. Rasmussen
Gaussian filters

= 1 pixel = 5 pixels = 10 pixels = 30 pixels

Mean vs. Gaussian filtering
Gaussian filter
• Removes “high-frequency” components from the image
(low-pass filter)
• Convolution with self is another Gaussian

* =
– Convolving twice with Gaussian kernel of width
= convolving once with kernel of width

Source: K. Grauman
Sharpening revisited
• What does blurring take away?

– =
original smoothed (5x5) detail
(This “detail extraction” operation is also
Let’s add it back: called a high-pass filter)

+α =
original detail sharpened
Photo credit: https://www.flickr.com/photos/geezaweezer/16089096376/
Sharpen filter
+
blurred
image unit impulse
image
(identity kernel
with single 1 in
center, zeros
elsewhere)

scaled impulse Gaussian Sharpen filter

Sharpen filter

unfiltered

filtered
“Optical” convolution
Camera shake

= *
Source: Fergus, et al. “Removing Camera Shake from a Single Photograph”, SIGGRAPH 2006

Bokeh: Blur in out-of-focus regions of an image.

Source: https://www.diyphotography.net/diy_create_your_own_bokeh/
Filters: Thresholding
Linear filters
• Can thresholding be implemented with a linear filter?
Can thresholding be implemented
with a linear filter?

ⓘ Start presenting to display the poll results on this slide.

Questions?