Photo Stitching

10/25/11
Image Stitching
Computational Photography
Derek Hoiem, University of Illinois
Photos by Russ Hewett
Last Class: Keypoint Matching

1. Find a set of
distinctive keypoints
A1
A2
2. Define a region
around each
keypoint
A3
fA
fB
d( f A, fB ) T
3. Extract and
normalize the
region content
4. Compute a local
descriptor from the
normalized region
5. Match local
descriptors
K. Grauman, B. Leibe
Last Class: Summary

Keypoint detection: repeatable
and distinctive
Corners, blobs
Harris, DoG
Descriptors: robust and selective

SIFT: spatial histograms of gradient
orientation
Today: Image Stitching

Combine two or more overlapping images to
make one larger image
Add example
Slide credit: Vaibhav Vaish
Problem basics
Do on board
Basic problem
x = K [R t] X
x = K [R t] X
t=t=0
.X
x
x'
f
f'
x=Hx where
H = K R R-1 K-1
Typically only R and f will change (4 parameters),
but, in general, H has 8 parameters
Views from rotating camera
Camera Center
Image Stitching Algorithm Overview

1. Detect keypoints
2. Match keypoints
3. Estimate homography with four matched
keypoints (using RANSAC)
4. Project onto a surface and blend
Image Stitching Algorithm Overview

1. Detect/extract keypoints (e.g., DoG/SIFT)
2. Match keypoints (most similar features,
compared to 2nd most similar)
Computing homography
Assume we have four matched points: How do
we compute homography H?
Direct Linear Transformation (DLT)
x' Hx
w' u'
x ' w' v'
w'
h1
H h4
h7
h2
h5
h8
h3
h6
h9
0
0 uu vu u
u v 1 0
h0
0
0
0 u v 1 uv vv v
h1
h
2
h3

h4
h h5

h6
h7

h8
h
9
Direct Linear Transform
0
0 u1u1 v1u1 u1
u1 v1 1 0
0
0
0 u1 v1 1 u1v1 v1v1 v1
h 0 Ah 0
0
0
0
1
u
v
v
v
v
n
n
n n
n n
n
Apply SVD: UDVT = A

h = Vsmallest (column of V corr. to smallest singular value)
h1
h1
h
h 2 H h4
h7

h9
h2
h5
h8
h3
h6
h9
Matlab
[U, S, V] = svd(A);
h = V(:, end);
Assume we have four matched points: How do
Normalized DLT
1. Normalize coordinates for each image
a) Translate for zero mean
b) Scale so that u and v are ~=1 on average
~
x Tx
~x Tx
This makes problem better behaved numerically (see

Hartley and Zisserman p. 107-108)
2. Compute H using DLT in normalized coordinates

1 ~
3. Unnormalize: H T HT
xi Hx i
Assume we have matched points with outliers:
How do we compute homography H?
Automatic Homography Estimation with RANSAC
RANSAC: RANdom SAmple Consensus

Scenario: Weve got way more matched points
than needed to fit the parameters, but were not
sure which are correct
RANSAC Algorithm
Repeat N times
1. Randomly select a sample
Select just enough points to recover the parameters
2. Fit the model with random sample
3. See how many other points agree

Best estimate is one with most agreement
can use agreeing points to refine estimate
Assume we have matched points with outliers: How do
Automatic Homography Estimation with RANSAC
1. Choose number of samples N
2. Choose 4 random potential matches
3. Compute H using normalized DLT
4. Project points from x to x for each potentially
matching pair: xi Hx i
5. Count points with projected distance < t
E.g., t = 3 pixels
6. Repeat steps 2-5 N times
Choose H with most inliers

HZ Tutorial 99
Automatic Image Stitching

1. Compute interest points on each image
2. Find candidate matches
3. Estimate homography H using matched points
and RANSAC with normalized DLT
4. Project each image onto the same surface and
blend
Choosing a Projection Surface

Many to choose: planar, cylindrical, spherical,
cubic, etc.
Planar Mapping
x
x
1) For red image: pixels are already on the planar surface

2) For green image: map to first image plane
Planar vs. Cylindrical Projection
Planar
Photos by Russ Hewett

Planar
Cylindrical Mapping
x
x
f
1) For red image: compute h, theta on cylindrical surface from (u, v)

2) For green image: map to first image plane, than map to cylindrical surface

Cylindrical

Cylindrical
Planar
Cylindrical
Simple gain adjustment
Automatically choosing images to stitch
Recognizing Panoramas
Some of following material from Brown and Lowe 2003 talk
Brown and Lowe 2003, 2007
Input: N images
1. Extract SIFT points, descriptors from all
images
2. Find K-nearest neighbors for each point (K=4)
3. For each image
a) Select M candidate matching images by counting
matched keypoints (M=6)
b) Solve homography Hij for each matched image
Input: N images
1. Extract SIFT points, descriptors from all
images
2. Find K-nearest neighbors for each point (K=4)
3. For each image
a) Select M candidate matching images by counting
matched keypoints (M=6)
b) Solve homography Hij for each matched image
c) Decide if match is valid (ni > 8 + 0.3 nf )
# inliers
# keypoints in
overlapping area
RANSAC for Homography
Initial Matched Points
Final Matched Points
Verification
Recognizing Panoramas (cont.)

(now we have matched pairs of images)
4. Find connected components
Finding the panoramas
Recognizing Panoramas (cont.)

(now we have matched pairs of images)
4. Find connected components
5. For each connected component
a) Perform bundle adjustment to solve for rotation
(1, 2, 3) and focal length f of all cameras
b) Project to a surface (plane, cylinder, or sphere)
c) Render with multiband blending
Bundle adjustment for stitching

Non-linear minimization of re-projection error
x Hx where H = K R R-1 K-1

N
Mi
error dist (x , x )
k
Solve non-linear least squares (LevenbergMarquardt algorithm)

See paper for details
Bundle Adjustment
New images initialized with rotation, focal
length of the best matching image
Bundle Adjustment
New images initialized with rotation, focal
length of the best matching image
Blending
Gain compensation: minimize intensity
difference of overlapping pixels
Blending
Pixels near center of image get more weight
Multiband blending to prevent blurring
Multi-band Blending (Laplacian Pyramid)

Burt & Adelson 1983
Blend frequency bands over range l
Multiband blending
Blending comparison (IJCV 2007)
Blending Comparison
Straightening
Rectify images so that up is vertical
Further reading
Harley and Zisserman: Multi-view Geometry book
DLT algorithm: HZ p. 91 (alg 4.2), p. 585

Normalization: HZ p. 107-109 (alg 4.2)
RANSAC: HZ Sec 4.7, p. 123, alg 4.6
Tutorial:
http://users.cecs.anu.edu.au/~hartley/Papers/CVPR99tutorial/tut_4up.pdf
Recognising Panoramas: Brown and Lowe, IJCV

2007 (also bundle adjustment)
Tips and Photos from Russ Hewett
Capturing Panoramic Images

Tripod vs Handheld
Help from modern cameras
Leveling tripod
Gigapan
Or wing it
Image Sequence
Requires a reasonable amount of overlap (at least 15-30%)
Enough to overcome lens distortion
Exposure
Consistent exposure between frames
Gives smooth transitions
Manual exposure
Makes consistent exposure of dynamic scenes easier
But scenes dont have constant intensity everywhere
Caution
Distortion in lens (Pin Cushion, Barrel, and Fisheye)
Polarizing filters
Sharpness in image edge / overlap region
Pikes Peak Highway, CO
Photo: Russell J. Hewett
Nikon D70s, Tokina 12-24mm @ 16mm, f/22, 1/40s
Pikes Peak Highway, CO
(See Photo On Web)
360 Degrees, Tripod Leveled
Nikon D70, Tokina 12-24mm @ 12mm, f/8, 1/125s
Howth, Ireland
(See Photo On Web)
Handheld Camera
Nikon D70s, Nikon 18-70mm @ 70mm, f/6.3, 1/200s
Handheld Camera
Les Diablerets, Switzerland
(See Photo On Web)
Macro
Photo: Russell J. Hewett & Bowen Lee
Nikon D70s, Tamron 90mm Micro @ 90mm, f/10, 15s
Side of Laptop
Photo: Russell J. Hewett & Bowen Lee
Considerations For Stitching

Variable intensity across the total scene
Variable intensity and contrast between frames
Lens distortion
Pin Cushion, Barrel, and Fisheye
Profile your lens at the chosen focal length (read from EXIF)
Or get a profile from LensFun
Dynamics/Motion in the scene

Causes ghosting
Once images are aligned, simply choose from one or the other
Misalignment
Also causes ghosting
Pick better control points
Visually pleasing result
Super wide panoramas are not always pleasant to look at
Crop to golden ratio, 10:3, or something else visually pleasing
Ghosting and Variable Intensity
Ghosting From Motion
Photo: Bowen Lee
Nikon e4100 P&S
Motion Between Frames
Nikon D70, Nikon 70-210mm @ 135mm, f/11, 1/320s
Gibson City, IL
(See Photo On Web)
Mount Blanca, CO
Mount Blanca, CO
(See Photo On Web)
Things to remember
Homography relates rotating cameras
Recover homography using RANSAC and
normalized DLT
Can choose surface of projection: cylinder,
plane, and sphere are most common
Lots of room for tweaking (blending,
straightening, etc.)
Next class
Using interest points to find objects in
datasets

Photo Stitching

Uploaded by

Copyright:

Available Formats

Photo Stitching

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Photo Stitching

Uploaded by

Copyright:

Available Formats

10/25/11

Last Class: Keypoint Matching

Last Class: Summary

Descriptors: robust and selective

Today: Image Stitching

Slide credit: Vaibhav Vaish

Views from rotating camera

Image Stitching Algorithm Overview

Image Stitching Algorithm Overview

Apply SVD: UDVT = A

This makes problem better behaved numerically (see

2. Compute H using DLT in normalized coordinates

RANSAC: RANdom SAmple Consensus

3. See how many other points agree

can use agreeing points to refine estimate

6. Repeat steps 2-5 N times

Choose H with most inliers

Automatic Image Stitching

Choosing a Projection Surface

1) For red image: pixels are already on the planar surface

Planar vs. Cylindrical Projection

Planar vs. Cylindrical Projection

1) For red image: compute h, theta on cylindrical surface from (u, v)

Planar vs. Cylindrical Projection

Planar vs. Cylindrical Projection

Simple gain adjustment

Automatically choosing images to stitch

Some of following material from Brown and Lowe 2003 talk

Brown and Lowe 2003, 2007

RANSAC for Homography

Initial Matched Points

RANSAC for Homography

Final Matched Points

RANSAC for Homography

Recognizing Panoramas (cont.)

Finding the panoramas

Finding the panoramas

Finding the panoramas

Recognizing Panoramas (cont.)

Bundle adjustment for stitching

x Hx where H = K R R-1 K-1

Solve non-linear least squares (LevenbergMarquardt algorithm)

Multi-band Blending (Laplacian Pyramid)

Blending comparison (IJCV 2007)

DLT algorithm: HZ p. 91 (alg 4.2), p. 585

Recognising Panoramas: Brown and Lowe, IJCV

Tips and Photos from Russ Hewett

Capturing Panoramic Images

Pikes Peak Highway, CO

Photo: Russell J. Hewett

Nikon D70s, Tokina 12-24mm @ 16mm, f/22, 1/40s

Pikes Peak Highway, CO

Photo: Russell J. Hewett

(See Photo On Web)

360 Degrees, Tripod Leveled

Photo: Russell J. Hewett

Nikon D70, Tokina 12-24mm @ 12mm, f/8, 1/125s

Photo: Russell J. Hewett

(See Photo On Web)