3D Computer Vision

Camera Model & Calibration

Prof. Didier Stricker

Kaiserslautern University
http://ags.cs.uni-kl.de/
DFKI – Deutsches Forschungszentrum für Künstliche Intelligenz
http://av.dfki.de

1
Appetizer: Scene Flow
View of one camera of the stereo camera

Exemplary stereo camera

slow

Scene reconstructed as dynamic 3D point cloud 2

 L1: The Camera L4: Parameter Estimation
Solving Linear
Pinhole Thin Lens Singular Value Lagrange Multiplier Non-linear Estimation:
Lenses Systems of Equations: Robust Estimation (RANSAC)
Model Formula Decomposition (SVD) How to triangulate?

Mathematical How to solve the How to How to How to estimate How to minimize
Description optimization estimate estimate multiple camera the ICP error?
& How to find the problem Homographies? F and E Matrix? parameters?
Camera Parameters of calibration? How to triangulate?
Get 3D points
from Align multiple
L2: Camera Calibration L6/L7: Structure from Motion (SfM) L10: Point Cloud Registration
Multiple Views Point Clouds
Estimation of
Mapping 3D-2D Point Multiple Bundle Align multiple Iterative Closest
Intrinsic and Extrinsic Triangulation Drift
3D to 2D Correspondences Views Adjustment Point Clouds Point (ICP)
Parameters

Special Case: How to get better features

2D to 2D mapping for matching?
Alternative way to get 3D Points

L3: Homographies L8: Dense 3D Reconstruction L11: Depth Cameras

Camera Pose
2D-2D Point Mapping Stereo Window-based Scene Time of Flight Structured Kinect
Estimation
Correspondences 2D to 2D Reconstruction Matching Representation Camera Light Fusion
(in special cases)

How to constrain
the correspondence Introduce artificial features
search to a line?
L5: Epipolar Geometry L9: Structured Light L12/13/14: Selected Topics
Artificial Features
Mapping 2D point to Fundamental Essential Binary Phase Various additional
through an
line in second view Matrix Matrix Coding Shifting applications of 3DCV
active device

3
 Overview

• Homogenous coordinates and central linear projection

• Camera parameters
• Intrinsic parameters
• Extrinsic parameters

• Calibration procedure
• Computing camera matrix 𝑃
• Extracting extrinsic parameters 𝑅, 𝑡
• Extracting calibration matrix 𝐾
• Advanced calibration

• Representation of 3D rotation

4
Recall: Modeling projection

•Projection equations
• Compute intersection with Π’ of ray from P = (x,y,z) to O
• Derived using similar triangles

• We get the projection by throwing out the last coordinate:

x y
( x, y , z ) → ( f ' , f ' ) 5
z z Source: J. Ponce, S. Seitz
Homogeneous coordinates
x y
( x, y , z ) → ( f ' , f ' )
z z

• Is this a linear transformation?

• no—division by z is nonlinear
Trick: add one more coordinate:

homogeneous image homogeneous scene

coordinates coordinates
Converting from homogeneous coordinates

6
Slide by Steve Seitz
Perspective Projection Matrix

• Projection is a matrix multiplication using

homogeneous coordinates:

 x
1 0 0 0    x 
0 1 0 0   y  =  y  x y
(f' , f' )
  z    z z
0 0 1 / f ' 0    z / f ' divide by the third
1  coordinate

7
Perspective Projection Matrix

• Projection is a matrix multiplication using homogeneous

coordinates:
 x
1 0 0 0    x 
0 1 0 0   = y 
y x y
(f' , f' )
  z    z z
0 0 1 / f ' 0    z / f ' divide by the third
1  coordinate

In practice: lots of coordinate transformations…

Camera to World to 3D
2D Perspective
pixel coord.
point = trans. matrix
projection matrix camera coord.
trans. matrix
point
(3x1) (3x4) (4x1)
(3x3) (4x4)
8
 Perspective Projection
For simplicity, we use the virtual image plane in front of the camera.
- Same results in the image plane (same image) but
- All variables/values are positive, ie. in front of the camera center and the horizontal flip of the image (what is simpler)

Using homogeneous coordinates, the perspective projection becomes linear!

Definition used in
Computer Vision
(equivalent to the
previous formula) 9
 Overview
• Homogenous coordinates and central linear projection

• Camera parameters
• Intrinsic parameters
• Extrinsic parameters

• Calibration procedure
• Computing camera matrix 𝑃
• Extracting extrinsic parameters 𝑅, 𝑡
• Extracting calibration matrix 𝐾
• Advanced calibration

• Representation of 3D rotation
10
Intrinsic Parameters – Pixel Coordinates

• Transformation uses:
▪ Principal point (𝑥0, 𝑦0)
▪ Scaling factors 𝑘𝑥 and 𝑘𝑦

And in homogeneous coordinates with α𝑥 = 𝑓𝑘𝑥 and α𝑦 = 𝑓𝑘𝑦 :

11
Intrinsic Parameters - Calibration Matrix 𝐾

• Intrinsic parameters:
▪ α𝑥 = 𝑓𝑘𝑥 and α𝑦 = 𝑓𝑘𝑦 : “Focal lengths” in pixels
▪ 𝑥0 and 𝑦0 : Coordinates of principal point in pixels
▪ Added parameter 𝑠 is skew parameter

• 𝑲 is called calibration matrix:

▪ Five degrees of freedom
▪ 𝐾 is a 3x3 upper triangular matrix 12
Intrinsic Parameters – Intrinsic Calibration
• Finding camera’s internal / intrinsic parameters that effect the
imaging process

▪ Position of the principal point on the image (𝑥0, 𝑦0), in the optimal case
this is the image center (width/2, height/2).
▪ Focal length 𝑓.
▪ Scaling factors for row and column pixels 𝑘𝑥 and 𝑘𝑦 .
▪ Skew parameter 𝑠.
▪ Lens distortion.

13
Extrinsic Parameters
– World Coordinates and Camera Coordinates

• In general world coordinates and camera coordinates do not coincide.

→ We need to find a coordinate transformation.

14
Extrinsic Parameters – World Frame to Camera Frame

▪ 𝑀 = (𝑋𝑠 , 𝑌𝑠 , 𝑍𝑠 ): point in world coordinates

▪ 𝑀 = (𝑥𝑠 , 𝑦𝑠 , 𝑧𝑠 ): point in camera coordinates
▪ 𝑡 = 𝐶𝑂: translation vector between origins

Rotation matrix

15
Extrinsic Parameters – Coordinate Transformation

Using homogeneous coordinates: 𝐶ሚ

And with 𝐶ሚ = −𝑅 𝑇 𝑡 (𝐶 is the camera center):

16
Camera Matrix 𝑃 – Putting everything together

17
 Overview

• Homogenous coordinates and central linear projection

• Camera parameters
• Intrinsic parameters
• Extrinsic parameters

• Calibration procedure
• Computing camera matrix 𝑃
• Extracting extrinsic parameters 𝑅, 𝑡
• Extracting calibration matrix 𝐾
• Advanced calibration

• Representation of 3D rotation

18
Calibration – Procedure
• Calibration means finding the function (defined by the camera)
that maps 3D points to 2D image plane.

▪ First thing required is to obtain pairs of corresponding 3D and 2D points:

(𝑋1, 𝑌1, 𝑍1 ), (𝑥1, 𝑦1), (𝑋2 , 𝑌2 , 𝑍2), (𝑥2 , 𝑦2 ), …

19
Calibration – Calibration Target
• Calibration target: Two perpendicular planes with chessboard pattern.

1. We know the 3D positions of the corners with respect to a coordinates system defined
on the target.

2. Place a camera in front of the target and we can locate the corresponding corners on
the image. This gives us the correspondences.

3. Recover the equation that describes imaging projection and camera’s internal
parameters. At the same time, also recover the relative orientation between the camera
and the target (pose).

20
Calibration – Finding Correspondences
• Multiple ways to find correspondences:

▪ Corner detector.
▪ Canny Edge detector plus fitting lines to the detected edges. Find the
intersections of the lines
▪ Manual input.

• Matching 3D and 2D points (we know the

number of corners) by counting gives
corresponding point pairs.

▪ (𝑋1 , 𝑌1 , 𝑍1 ), (𝑥1 , 𝑦1 ), (𝑋2 , 𝑌2 , 𝑍2 ), (𝑥2 , 𝑦2 ), …

21
Calibration – Computing the Camera Matrix 𝑃

• Set up equations for all corresponding point pairs:

▪ 𝐱 i = 𝑃𝐗 i , 𝐱 i = (𝑥i , 𝑦i ), 𝐗 i = (𝑋i , 𝑌i , 𝑍i ), i = 1, …

▪ Involving homogeneous coordinates 𝐱 i and 𝑃𝐗 i just have to be proportional:

𝐱 i × 𝑃𝐗 i = 0, i = 1, … (× being the cross product)

22
Calibration – Computing the Camera Matrix P

▪ Let be the row vectors of matrix 𝑃.

▪ We get the linear system of equations:

23
Calibration – Computing the Camera Matrix P

▪ The third row can be obtained from the sum of 𝑢′i times the first row and
−𝑣′i times the second row.
Rank 2

▪ So we get 2 independent equations in 11 unknowns (ignoring scale).

▪ With at 6 point correspondences we have enough equations to compute

matrix 𝑃 by solving:

24
Calibration – Computing the Camera Matrix 𝑃

▪ Linear system.

▪ If possible, have at least 5 times as many equations as unknowns (28 points).

▪ Minimize with the constraint to avoid the trivial solution.

→ The minimizer p of this kind of minimization problem is given by the unit

singular vector of 𝐴 corresponding to the smallest singular value (≠0).

Usually, it is the last column of 𝑉, if the SVD is given by:

25
Calibration – Extracting Extrinsics

• Having computed 𝑃 we can extract the extrinsic parameters from

▪ The left 3x3 sub-matrix is a product of an upper triangular matrix 𝐾 and an

orthogonal matrix 𝑅.

26
Calibration – Extracting Translation

• First, we want to find homogeneous coordinates of 𝐶 in the scene:

▪ 𝐶 is the null-vector of matrix 𝑃:

▪ Find null-vector 𝐶 of 𝑃 using SVD again.

27
Calibration – Extracting Rotation and Calibration Matrix

• Next we want to find the camera orientation and the internal parameters of the camera:

▪ 𝐾 and an upper triangular matrix

▪ 𝑅 is an orthogonal matrix.

→ From the first 3x3 sub-matrix of 𝑃 = 𝐾𝑅 we can extract 𝐾 and 𝑅 by QR-Decomposition.

▪ A QR-decomposition (also called QR-factorization) of a matrix is a decomposition into an orthogonal

(Q) and a right triangular matrix (R).
▪ In our case we use RQ decomposition, which is similar to QR (apply QR decomposition on inverted
input and invert output).

KR =(( KR )-1)-1=( Q R )-1= R -1 Q -1

28
Calibration – Degenerate Case
Calibration points 𝐗 i , i = 1, … can not lie on the same plane!

Xi=(Xi, Yi, Zi)

In this case there would exist a mapping with only 8 degrees of freedom (called homography
→ next lecture).
Fitting a 𝑃-Matrix with 11 DOF is not uniquely defined!
29
Calibration – Back-Projection Error

• How should we estimate the accuracy of a calibration algorithm?

▪ Project known 3D points on the image.

▪ Compare their projections with the corresponding pixel coordinates of the points.

▪ Repeat for many points and estimate back-projection / re-projection” error!

30
Advanced Calibration – Further Improvement

• How can we further improve the calibration?

▪ Use received linear calibration as initialization for nonlinear methods for refinement.

▪ Minimize the nonlinear error function (usually back-projection error):

2

▪ Most popular non-linear minimization algorithm is the Levenberg-Marquart (LM)

minimization

▪ LM is more robust to local minima than e.g. the Gauss–Newton algorithm and the
method of gradient descent.

31
Advanced Calibration – Nonlinear Distortions
▪ Radial distortions (Barrel) Tangential distortions

wideangle (barrel distortion) tele (pincursion distortion)

32
Advanced Calibration - Nonlinear Distortions
• Dealing with (radial) distortion parameters
▪ Usually the points of a captured checkerboad are radially distorted.
→ Alternatingly estimate
▪ nter a enter
Intrinsic textit
a e textit
parameters , ) nter
, ) a entertextit a e x i,textit
i ) x i, i )
▪ Distortion parameters
ho ho o nload o nload ho erif ho oerif
nload o nload erif erif
▪ Notations for projected points in normalized image coordinates and pixel coordinates:

nter a e ongrightarro
nter a e begin align hat x xi xi nter a e begin align hat x i
xi xi
i

eriferif
ho o nload
ho o nload erif ho o nload

▪ Usually Brown‘s distortion model is used to model radial distortions.

33
 nter a e
nter textit
a e textit
, ) nter
, ) a entertextit
a e x i,textit
i ) x i, i )

Advanced
nter a e Calibration
begin align begin-pma
Nonlinear Distortions
nter a e begin align begin
ho ho o nload o nload ho erif ho oerif
nload o nload eri

ho o nload erif ho o nload erif

• Dealing with (radial) distortion parameters
▪ Calibration matrix K, assuming zero-skew
nter a e begin align begin pmatrix
nter a e begin align begin pmatrix
ho o nload erif ho o nload erif

▪ Relation between pixel and normalized image coordinates (assuming image plane with f=1)
▪ (Same relation for the distorted case)

34
 nter a e
nter textit
a e textit
, ) nter
, ) a entertextit
a e x i,textit
i ) x i, i )

Advanced Calibration - Nonlinear

nter
nter a e begin align begin pmatrix a e Distortions
ongrightarro
ho ho o nload o nload ho erif ho oerif
nload o nload eri

nter a e begin align

nter a e alpha xxi
begin align hat alpha h
• ho o nload erif ho
Dealing with (radial) distortion parameters o nload erif x
ho o nload ho oerif
nload erif
▪ Relation between pixel and normalized image coordinates (assuming image plane with f=1)
▪ (Same
nter a e relation
begin inalign frac hat
the distorted case) al
0

ho o nload erif
nter a e begin align hat 0
▪ hoInserting the above relations into Brown’s distortion model, we get the following equation in case of 𝑥𝑖 :
o nload erif
nter a e begin align hat 0
ho o nload erif
▪ Simplified:

▪ The same in the case of 𝑦𝑖 :

35
 nter a e
nter textit
a e textit
, ) nter
, ) a entertextit
a e x i,textit
i ) x i, i )
nter a e begin align hat 0
Advanced Calibrationerif- Nonlinear Distortions
ho o nload
ho ho o nload o nload ho erif ho oerif
nload o nload eri

• nter adistortion
Dealing with (radial) e beginparameters
align begin pmatrix

ho o nload erif

▪ Stacking the above equations for each point in each image, we get the following system of equations:

Distorted points Undistorted points

(detected) (projected from
camera model)

▪ We can estimate the radial distortion coefficients, by solving:

Pseudo-inverse (see lecture
on „Parameter Estimation“)

36
Advanced Calibration – Image Rectification

Distorted image. Rectified image.

37
Calibration – Exemplarily Calibration Steps

38
Calibration – Direct Linear Calibration

• Advantage
▪ Very simple to formulate and solve.

• Disadvantages
▪ oesn’t tell o the camera parameters.
▪ oesn’t model radial distortion.
▪ Hard to impose constraints (e.g., known focal length).
▪ oesn’t minimize the right error f nction.

For those reasons nonlinear methods are preferred.

Nevertheless, the direct linear approaches can provide very good
initializations.

39
Calibration – Other Approach

• Calibration with a single plane:

▪ Only requires a plane.
▪ on’t ha e to no positions/orientations.
▪ Good code available online!
▪ Intel’s OpenCV library: https://opencv.org/
▪ Matlab version by Jean-Yves Bouget:
http://www.vision.caltech.edu/bouguetj/calib_doc/index.html
▪ Zhengyou Zhang’s eb site: http://research.microsoft.com/~zhang/Calib/
40
Application – Overlay of a virtual CAD-Cockpit onto a
“real” image (Augmented Reality)

• Compute the 𝑃-matrix of the camera.

• Draw the picture as background.
Collect 2D and 3D corresponding • Use 𝑃 to render the CAD object.
41
points (manual selection).
Important addendum: Ways to Represent Rotation

• There are several options:

▪ 3D-Rotation matrix (3x3)

▪ Euler angles
▪ Axis angle (rotation vector)
▪ Quaternions
▪ …

42
Rotation – 3D Rotation Matrix

▪ A matrix 𝑅 is a rotation matrix if and only if it is an orthogonal

matrix with determinant +1.

▪ Orthogonal matrix is a square matrix with real entries, where

columns and rows are orthogonal vectors with length 1.

This is equivalent to:

or

43
Rotation – Euler Angle

• Euler’s Theorem: Any two independent orthonormal coordinate frames can be related by a sequence of rotations (not
more than three) about coordinate axes, where no two successive rotations may be about the same axis.

• A sequence of rotations around the principle axes is called an Euler Angle Sequence.

• We just multiply a sequence of rotation matrices to build a matrix from a set of Euler angles.

44
Rotation – Euler Angle
• Euler angles for rotation 𝑹 = 𝑹𝒛𝑹𝒚𝑹𝒙:

▪ Rotation by 𝜓 around 𝑧 axis:

▪ Rotation by 𝜃 around 𝑦 axis:

▪ Rotation by 𝜙 around 𝑥 axis:

45
Rotation – Euler Angle

• Properties of Euler angle representation:

▪ As matrix multiplication is not commutative, the order of operations is

important.

▪ Minimal parametrization, nice geometric interpretation.

▪ Periodicity, Gimbal lock.

46
Rotation – Euler Angle

▪ Rotation angles like roll (around x-axis), pitch (around y-axis) and yaw
(around z-axis) are often used in literature and in a lot of applications
(vehicles, aircrafts, robotics, etc).

X
Y
47
Rotation – Axis Angle

• Represents rotation by a 3D vector:

▪ Direction is rotation axis

▪ Length is rotation angle

▪ Con ersion to rotation matrix b Rodrig es’ form la:

▪ Singularity at 𝜃 = 0.

48
Rotation – Quaternions

• Represents rotation by a 4D vector:

▪ Quaternion is a 4D vector:

▪ Compromise between rotation matrix and minimal representation.

▪ Quaternions are actually an extension to complex numbers of the 4

components, one is a ‘real’ scalar n mber, and the other 3 form a ector
in imaginary 𝑖 − 𝑗 − 𝑘 space.

▪ Singularity-free, can be interpolated.

49
Rotation – Quaternions

▪ Conversion to rotation vector (axis-angle):

▪ Conversion to rotation matrix:

50
THANK YOU!

51