US007889905B2

(12) United States Patent

(10) Patent N0.:

(45) Date of Patent:

Higgins et al.
(54)

FAST 3D-2D IMAGE REGISTRATION

Feb. 15, 2011

(Continued)

METHOD WITH APPLICATION TO

CONTINUOUSLY GUIDED ENDOSCOPY
75

US 7,889,905 B2

OTHER PUBLICATIONS

( ) Inventors' Wllham E Hlggmsistate College PA

(Us); scottA' Memtt State College

Deli g ianni ,

F. ,

A.

Chun g,

and

G.

Yan g .

Patient-s P eci?c

bronchoscope simulation with pq-space-based 2D/3D registration.

Com uter Aided Sur e

PA (US); Lav Rai, State College, PA

vol 9 N0 5

g ry

'

(Us)

215-226 (2004)

' p'

'

(Continued)

(73) Assignee: The Penn State Research Foundation,

University Park, PA (US)

Primary ExamineriDavid P Rashid

(74) Attorney, Agent, or Firm4Gifford, Krass, Sprinkle,

Anderson & Citkowski, P.C.

(*)

Notice:

Subject to any disclaimer, the term of this

patent is extended or adjusted under 35

(57)

ABSTRACT

U.S.C. 154(b) by 863 days.

(21) Appl' NO: 11/437230

(22) Filed:

A novel framework for fast and continuous registration

May 19, 2006

(65)

between two imaging modalities 1s dlsclosed. The approach

makes 1t possible to completely determlne the rigid transfor

Prior Publication Data

Us 2007/0013710 A1

mation between multiple sources at real-time or near real

time frame-rates in order to localize the cameras and register
the two sources. A disclosed example includes computing or

Jan 18 2007

Related U-s- Application Data

' ~

capturing a set of reference images within a known envlron

'

ment, complete with correspondlng depth maps and image

(60) PFOVlSlOnal aPPllCaUOI1 N0~ 60/683,588, ?led 011 May

23, 2005

(5 1) Int Cl

feed. Given one frame from this video feed, and starting from

G06K 9/00

(52)

(200601)

an initial guess of viewpoint, the real-time video frame is

G06K 9/32

(2006.01)

A61B 5/05

(200601)

6 0 0/4 2 4

.
.
.
warped to the nearest v1ew1ng
s1te
of the reference source. A11
ima e difference is com uted between the w ed video
g
. p
.
.
. mp
.
frame and the reference image. The viewpoint 18 updated v1a

382/130

repeated for each frame until the viewpoint converges or the

us. Cl. ..................... .. 382/130; 382/151; 382/294;

a Gauss-Newton parameter update and certain of the steps are

(58) lsrleld 0f1_C lassl?glatiqon searclh """"

ee app lcanon

(56)

gradients. The collection of these images and depth maps

constitutes the reference source. The second source is a real
time or near-real time source which may include a live video

e or Comp ete Seam

lstory'

next video frame becomes available. The ?nal viewpoint

References Cited

gives an estimate of the relative rotation and translation

between the camera at that particular video frame and the
reference source. The invention has far-reaching applica

us PATENT DOCUMENTS
4,791,934 A
12/1988 Brunnett
.

tions,particularly inthe ?eld of assisted endoscopy, including

bronchoscopy and colonoscopy. Other applications include

(Commued)

aerial and ground-based navigation.

FOREIGN PATENT DOCUMENTS

KR

20020041577

6/2002

17 Claims, 2 Drawing Sheets

Flafelen-ie Images

interior Airway
Tree Surtazes
Ralaranca De plhs

[3T ?tites :7,

F
Flalsrgma Gradients

Re 31

WE! ml?! El

1111:1154
5

rf/
\

=1 *Jiclen \#_ DIEIEYBHKB E}

|I|

Frame I l ,

F >;

r 12:, I,

mage

_/

ii

Parameter

Updma

US 7,889,905 B2
Page 2
US. PATENT DOCUMENTS

5740802
5,748,767
5,765,561
5,776,050
5,782,762
5,792,147
5,830,145
5,891,034
5,901,199
5,920,319
5,956,418
5 963 612

A
4/1998
A
5/1998
A
6/1998
A
7/1998
A
7/ 1998
A
8/1998
A
11/1998
A
4/1999
A
5/1999
A
7/1999
A * 9/1999
A
10/1999

2008/0262297 A1*

Na?s er a1~
Raab
Chen et al.
Chen et al.

10/2008 Gilboa et al. ............. .. 600/109

FOREIGN PATENT DOCUMENTS

WO

WO-2006076789

Vining

7/2006

OTHER PUBLICATIONS

Evans et al.
Tenhoff
Bucholz

Navab

Lee, P.Y and 1B. Moore. Pose Estimation via Gauss-Newton-on

manifold. 16th International Symposium on Mathematical Theory
of Network and System (MTNS), Leuven, 2004.
Hamadeh, A., S. Lavallee, and P. Cinquin. Automated 3-Dimen
Sional Computed Tomographic and ?uoroscopic Image Registra
tion Computer Aided Surgery 1998, vol, 3: p. 11-19.

539633613 A

10/1999 Navab

Roberson, RE. and PW. Likins. A Linear-iZation Tool for Use with

5 971 767 A
5999840 A

10/1999 Kaufman et a1
0/1999 Grimson et a1

Matrix Formalisms of Rotational Dynamics Archive of Applied

Mathematics, vol. 37, No. 6: p. 388-392. Nov. 1969.

'

Asano, F., Y. Matsuno, T. Matsushita, H. Kondo, Yoshio Saito, A.

Murphy et al
Vining et a1
Aigeretal. ............... .. 382/154

6006126 A
60 l6439 A

12/1999 Cosman
V2000 Acker

Seko, and Y. Ishihara. Transbronchial Diagnosis of a Pulmonary

Peripheral Small Lesion Using an Ultrathin Bronchoscope with Vir

6,019,725 A *

2/2000 Vesely et al. .............. .. 600/447

mal Bronchoscopic Navigation? Journal of Bronchology (2002),

6,049,582 A

4/2000 Navab

V01 9, N0 2,1) 108411

6,083,162 A
6,190,395 B1

7/2000 Vining
2/2001 Williams

Geiger, B., G.M. Weiner, K. SchulZe, J. Bilger, P. Krebs, K. Wolf,

T.Albrecht. Virtual Bronchoscopy Guidance System for
Transbronchial Needle Aspiration. Proceedings of SPIE vol. 5746
(2005).
Grimson, W.E.L., G.J. Ettinger, S.J. White, T. LoZano-PereZ, W.M.
Wells III, and R. Kikinis. An Automatic Registration Method for
Frameless Stereotaxy, Image Guided Surgery, and Enhanced Reality
Visualization. IEEE Transactions on Medical Imaging, Apr. 1996,

6,201,543
6,236,743
6,241,657
6,272,366
6,311,116

B1
B1
B1
B1
B1

3/2001
5/2001
6/2001
8/2001
10/2001
1/2002

6,334,847 B1

ODonnell et al.
Pratt
Chen et al.
Vining
Lee et a1,
Fenster et a1

6,343,936 B1

2/2002 Kaufman et al.

VOL 15, N~ 2, P 129-149

6,351,573 B1

2/2002 Schneider

Helferty, J.P., A.J. Sherbondy, A.P. Kiraly, and WE. Higgins. Com

6 363 163 B1 *

3/2002 Xu et al. ................... .. 382/130

6,366,800 B1

4/2002 Vinin

g e

t l
3

puter'based System for the Virtual Endoscopic Guidance of

.

600/426

used, and/or published prior to the ?ling date ofthis application).

Hopper, K.D., T.A. Lucas, K. Gleeson, J.L. Stauffer, R. Bascom, D.

' """"""" "

Tanaka et al. ............. .. 600/425

T. Mauger, R. Mahraj. Transbronchial Biopsy with Virtual CT

B
h
dN d 1H- hl,,
-

6,480,732 B1 *

11/2002

6,491,702 B2

12/2002 Heilbrun et al.

SlIIlOIl et

ronc oscopy an

221 , N0 ~ 2 ,

2/2003 Kaufman et 31'

T

Bronchoscopy (believed to have been offered for sale, publicly

6 442 417 B1
8/2002 Shahidi et al
6470207 Bl * 10/2002 Simon et 31

65l4082 B2

P~

o a

1g ighting. Radiology Nov. 2001, vol.

531636 ~

Maurer, C.R., J.M. Fitzpatrick, M.Y Wang, R.L. Galloway, Jr., R].

. . . . . . . . . . . . . . ..

Maciunas, and

Allen Registration of Head Volume Images

6,546,279 B1

4/2003 Bova et 31'

Using Implantable Fiducial Markers IEEE Transactions on Medical

6,556,695 B1 *

4/2003 Packer et al. .............. .. 382/128

Imaging, Aug 1997, V01 16,N04,p447_462

6,593,884
6,674,879
6,675,032
6,694,163
6,714,668

7/2003
1/2004
V2004
2/2004
3/2004

McAdams, H.P., P.C. Goodman, and P. Kussin. Virtual

Bronchoscopy for Directing Transbronchial Needle Aspiration of
Hilar and Mediastinal Lymph Nodes: A Pilot StudyAJR May 1998,
vol. 170, p. 1361-1364.
Merritt, S.A., L. Rai, and WE. Higgins. Real-Time CT-Video Reg

B1
B1
B2
B1
B1

Gilboa et a1~
Weisman et a1~
Chen et a1
Vining
Kerrien et al.

6,725,080 B2 *

4/2004 Melkent et al. ........... .. 600/424

istration for Continuous Endoscopic Guidance (believed to have

6,771,262 B2
6,785,410 B2

8/2004 Krishnan
8/2004 Vining et al.

been offered for sale, publicly used, and/or published prior to the
?ling date of this application).

6,816,607 B2 *

11/2004 ODonnell et al. ........ .. 382/ 131

Mori, K, T. Enjoji, D. Deguchi, T. Kitasaka,Y Suenaga, J . Toriwaki,

6,819,785
6,859,203
6,909,913
6,920,347

11/2004
2/200 5
6/2005
700%

Vining et 31,
Van Muiswinkel et a1
Vining
Simon et a1

H. Takabatake, and H. Natori. New image similarity measures for

bronchoscope tracking based on image registration between virtual

Johnson et al.

aPP11at1n)~

B1
B2
B2
B2

6,928,314 B1 *

8/2005

6,947,584 B1

9/2005 Avila et a1

6 980 682 B1

........... .. 600/407

12/2005 Avinash et a1

3/2006 GOtO et al.

7,343,036 B2

3/2008

6/2003 Wes hal et al. .......... .. 356/479

_ tp
8/2003 GGIgeI
10/2004 Geiger
12/2004 Reeves et 31'

Guidance Method for Lung-Cancer Assessment (believed to have

ublicl

used and/or ublished rior to the

?ling date ofthis appiigatiomy

Kleen et al.

2002/0057345 A1* 5/2002 Tamaki et al. ............. .. 348/207

2003/0103212 A1*

Rai, L., S.A. Merritt, and WE. Higgins. Real-time Image-based

been offered for sale

7,019,745 B2

2003/0152897 A1
2004/0209234 A1
2004/0252870 A1

and real bronchoscopic images (believed to have been offered for

sale, publicly used, and/or published prior to the ?ling date of this

'

P
.

..

52mg" 134' Nakjmotoy Tamag? T 8382111135111 sakgaY' Nakajlma

'

on eI-l an

S Tamra

Image Gm

Ce 0 Breast- Carper

Surgery Using 3-D Ultrasound Images and Augmented RealityVisu

aliZation IEEE Transactions on Medical Imaging, Oct. 1998, vol.
17, NO 5,1) 68l_693
SchwarZ,Y, A.C. Mehta, A. Ernst, F. Herth, A. Engel, D. Besser, and

2005/0074151 A1 *

4/2005 Chen et al. ................ .. 382/128

H D Becker Electromagnetic Navigation during Flexible

2005/0078858 A1
2005/0096526 A1

4/2005 Yao et a1~

5/2005 Reinschke

Bronchoscopy Respiration 2003, vol. 70, p. 516-522.

Shinagawa, N., K. YamaZaki, Y. Onodera, K. Miyasaka, E. Kikuchi,

2005/0182295 A1<

2005/0272999 A1
2006/0084860 A1

8/2005 Soper et al. ............... .. 600/117

12/2005 Guendel
4/2006 Geiger et al.

H, Dosaka-Akita, and M, Nishimufa, CT-Guided Transbronchial

Biopsy Using an Ultrathin Bronchoscope with Virtual

Bronchoscopic NavigationChest, Mar. 2004, vol. 25, p. 1138-1143.

US 7,889,905 B2
Page 3
Shoji, H., K. Mori, J. Sugiyama, Y. Suenaga, J. Toriwaki, H.
Takabatake, and H. Natori. Camera motion tracking of real

Lucas, et al., An iterative image registration technique with an appli

cation to stereo vision, Proceedings of the 7th International Joint

endoscope by using virtual endoscopy system and texture informa

Conference on Arti?cial Intelligence (IJCAI 81), Apr. 1981, pp.

tion Proceedings of SPIE vol. 4321, p. 122-133 (2001).

Stefansic, J.D., A.J. Herline, Y. Shyr, W.C. Chapman, J.M.
Fitzpatrick, B.M. Dawant, and R.L. Galloway, Jr. Registration of

674-679.

Physical Space to Laparoscopic Image Space for Use in Minimally

Invasive Hepatic Surgery. IEEE Transactions on Medical Imaging,
Oct. 2000, vol. 19, No. 10, p. 1012-1023.
Turcza, P. and M. Duplaga. Navigation Systems Based on Registra
tion of Endoscopic and CT-derived Virtual Images for
Broncho?beroscopic Procedures. Studies in Health Technology and
Informatics, vol. 105, p. 253-263 (2004).
White, C.S., E.A. Weiner, P. Patel, and EJ Britt. Transbronchial
Needle Aspiration: Guidance with CT Fluoro scopy. Chest 2000, vol.
118, No. 6, p. 1630-1638.

Higgins, W.,W.J.T. Spyra, R.A. Karwoski, andE.L. Ritman. System

for Analyzing High-Resolution Three-Dimensional Coronary

Minami, et al., lnterbronchoscopist variability in the diagnosis of

lung cancer by ?exible bronchoscopy, Chest, 105: 1658-1662, 1994.
Okatani, et al., Shape reconstruction from an endoscope image by
shape from shading technique for a point light source at the projection
center, Computer J/lsion and Image Understanding, 66: 1 19-131,
1997.

Shum, et al., Systems and experiment paper: construction of pan

oramic image mosaics with global and local alignment, International

Journal ofComputer J/lsion, 36(2): 101-130, 2000.

Shum, et al, Panoramic image mosaics, Microsoft Research, Techni
cal Report MSR-TR-97-23, 1997.
Swift, et al., Automatic axis generation for virtual bronchoscopic
assessment of major airway obstructions, Computerized Medical
Imaging and Graphics, 26: 103-118, 2002.

Angiograms IEEE Transactions on Medical Imaging, Jun. 1996,

vol. 15, No. 3, p. 377-385.

Asano, et al., Transbronchial diagnosis of a pulmonary peripheral

Brady, M.L., W.E. Higgins, K. Ramaswamy. Interactive navigation

bronchoscopic navigation, Journal of Bronchology, 9: 108-11, 2002.

Helferty, et al., Technique for registering 3D virtual CT images to
endoscopic video, IEEE Image Processing, 893-896, 2001.

inside 3D radiological images IEEE 1995, p. 33-40.

Higgins, W.E. and K. Ramaswamy. Toward dynamic visualization
for endoscopy simulation IEEE 1994, p. 700-701.
Bricault, I., G. Ferretti, and P. Cinquin. Registration of Real and
CT-Derived Virtual Bronchoscopic Images to Assist Transbronchial
Biopsy IEEE Transactions on Medical Imaging, Oct. 1998, vol. 17,
No. 5, p. 703-714.

Sherbondy, A.J., A.P Kiraly, A.L. Austin, J.P. Helferty, S. Wan, J.Z.
Turlington, T. Yang, C. Zhang, E.A. Hoffman, and G. McLennan.
Virtual Bronchoscopic approach for combining 3D CT and
Endoscopic Video Proceedings of SPIE 2000, vol. 3978, No. 104.
Helferty, J .P., A.J . Sherbondy, A.P. Kiraly, J .Z. Turlington, E.A. Hoff
man, G. McLennan, W.E. Higgins. Experiments in virtual

endoscopic guidance of bronchoscopy Proceedings of SPIE 2001,

vol. 4321, No. 111.

Helferty, J .P andW.E. Higgins. Combined endscopic video tracking

and virtual 3D CT registration for surgical guidance Proceedings of
the 2002 International Conference on Image Processing, vol. 2, pp.

small lesion using an ultrthin bronchoschope with virtual

Baker, et al., Equivalence and ef?ciency of image alignment algo

rithms, Proceedings of the 2001 Conference on ComputerVision and
Pattern Recognition, 1090-1097.
Bricault, et al., Registration of real and CT-derived virtual
bronchoscopic images to assist transbronchial biopsy, IEEE Trans
actions of Medical Imaging, 17(5):703-714, 1998.

Deguchi, et al., Camera motion tracking of real bronchoscope using

epipolar geometry analysis and CT derived bronchoscopic images,
Proceedings of SPIE, 4683: 30-41, 2002.
Haralick, et al., Analysis and solutions of the three point perspective
pose estimation problem, Proceedings Computer vision and pattern
recognition, IEEE Computer Society Conference, 592-598, 1991.
Harris, et al., A combined corner and edge detector, Proceedings of
the Fourth Alvey Vision Conference, 147-151, 1988.
Horaud, et al., A analytic solution for the perspective 4-point prob

lem, Computer Vision, Graphics, and Image Processing, 47:33-44,

961-964.

1989.

Higgins, W.E., J .P. Helferty, and DR. Pad?eld. Integrated

bronchoscopic video tracking and 3D CT registration for virtual
bronchoscopy Proceedings of SPIE 2003, vol. 5031, No. 80.
Kiraly, A.P., J .P. Helferty, E.A. Hoffman, G. McLennan, W.E. Hig

Fischler, et al., Random sample consensus: a paradigm for model

gins. Three-dimensional path planning for virtual bronchoscopy

?tting with application to image analysis and automated cartography,

Communications of the ACM, 24(6): 381-395, 1981.
Zuliani, et al., A mathematical comparison of point detectors, Pro
ceedings of the 2004 IEEE Computer Society Conference on Com

IEEE Transactions on Medical Imaging 2004, vol. 23, No. 11, pp.

puter Vision and Pattern Recognition Workshops.

1365-1379.

Ansar, et al., Linear pose estimation from points or lines, IEEE

Transactions on Pattern Analysis and Machine Intelligence,

Helferty, J.P., E.A. Hoffman, G. McLennan, W.E. Higgins. CT

video registration accuracy for virtual guidance of bronchoscopy
Proceedings of SPIE 2004, vol. 5369, pp. 150-164.
Higgins, W.E., L. Rai, S.A. Merritt, K. Lu, N. T. Linger, and K.C.Yu.
3D image fusion and guidance for computer-assisted
bronchoscopy Proceedings of SPIE 2005, vol. 6016.
Asano, F., Y Matsuno, N. Shinagawa, K. Yamazaki, T. Suzuki, T.

tem, Lecture Notes in Computer Science, 2208: 1-8, 2001.

Shi, et al., Good features to track, IEEE Conference on Computer
Vision and Pattern Recognition, 593-600, 1994.

Ishida, and H. Moriya. AVirtual Bronchoscopic Navigation System

Triggs, B., Detecting keypoints with stable position, orientation, and

for Pulmonary Peripheral Lesions. Chest 2006, vol. 130, No. 2, pp.

scale under illumination changes, Lecture Notes on Computer Sci

ence, 3024: 100-113, 2004.

559-566.

Kukuk, M. Modeling the Internal and External Constraints of a

Flexible Endoscope for Calculating its Workspace: Application in

Transbronchial Needle Aspiration Guidance Proceedings of SPIE
2002, vol. 4681, pp. 539-550.
Gibbs, JD. and WE. Higgins. 3D Path Planning and Extension for
Endoscopic Guidance Proceedings of SPIE 2007, vol. 6509.
Mori, K., S. Ema, T. Kitasaka, Y. Mekada, I. Ide, H. Murase, Y
Suenaga, H. Takabatake, M. Mori, and H. Natori. Automated
Nomenclature of Bronchial Branches Extracted from CT Images and

Its Application to Biopsy Path Planning in Virtual Bronchoscopy

Medical Image Computing and Computer-Assisted Intervention
2005, Lecture Notes in Computer Science 3750, pp. 854-861.

Kiraly, et al., 3D human airway segmentation for virtual

bronchoscopy, Proceedings ofSPIE, 4683: 16-29, 2002.

25(5):578:589, 2003.
Mori, et al., A method for tracking the camera motion of real

endoscope by epipolar geometry analysis and virtual endoscopy sys

Quan, et al., Linear N-point camera pose determination, IEEE Trans

actions on Pattern Analysis and Machine Intelligence, 21(8): 774
780, 1999.

Haralick, et al., Pose estimation from corresponding point data, IEEE

Transaction on Systems, Man, and Cybernetics, 19(6): 1426-1446,
1989.

Lowe, D., Fitting parameterized three-dimensional models to

images, IEEE Transaction on Pattern Analysis and Machine Intelli

gence, 13(5):441-450, 1991.

Hager, et al., Fast and globally convergent pose estimation from
video images, IEEE Transactions on Pattern Analysis and Machine

Intelligence, 22(6):610-622, 2000.

Hartley, et al., Multiple View Geometry in Computer Vision, Cam
bridge University Press, ISBN:0521623049, pp. 238, 239, 281, 2000.

US 7,889,905 B2
Page 4
Higgins, et al., Integrated bronchoscopic video tracking and 3D CT
registration for virtual bronchoscopy, Medical Imaging 2003: Physi
ology and Function: Methods, Systems, and Applications.

tion, and Structure from Medical Images, 165-176.

2001: Physiology and function from Multidimensional Images.

Helferty, et al., Combined endoscopic video tracking and virtual 3D
CT registration for surgical guidance, IEEE International Conference
on Image Processing, 2002.
Helferty, J ., Image-guided endoscopy and its application to pulmo
nary medicine, A Thesis in Electrical Engineering, The Pennsylvania
State University, 2002.

Shoji, et al., Camera motion tracking of real endoscope by using

virtual endoscopy system and texture information, Medical Imaging

* cited by examiner

Mori, et al., New image similarity measures for bronchoscope track

ing based on image registration between virtual and real

bronchoscopic images, Medical Imaging 2004: Physiology, Func

US. Patent

Feb. 15, 2011

Sheet 1 012

US 7,889,905 B2

Rare-rams [ms-132s

lme rim Amway

TJEE EEUEfH-{i-EE
'

Ralarama [I19 plha

13? SEices =31

\\\.

Helaranz-a Gradianls

if, [
He '11

1 {1

liliffe IEEHEI-B

hale 41

Frame
I

[Image

_ :}

ll
Pam ma'tar
Lind-m9

I
FIG. 1

FIG. 3A

FIG. 3B

FIG. 3C

US. Patent

FIG. 2A

FIG. 21)

Feb. 15, 2011

Sheet 2 of2

FIG. 213

FIG. 2B

US 7,889,905 B2

FIG. 2C

FIG. 2F

US 7,889,905 B2
1

FAST 3D-2D IMAGE REGISTRATION

METHOD WITH APPLICATION TO
CONTINUOUSLY GUIDED ENDOSCOPY

lead to over?tting of the image. A more recent choice of

parameters attempts to match tWo images obtained from a
camera that can have arbitrary 3D rotations around its focal

REFERENCE TO RELATED APPLICATION

point [8]. This algorithm succeeds in extracting the physically

relevant parameters (rotation angles about the focal point).
HoWever, While it is able to handle small translations, it can

This application claims priority to US. Provisional Patent

not handle general translation and treats it as a source of error.

Application Ser. No. 60/683,588, ?led May 23, 2005, the

Little has been done to tackle the problem of registration of

entire content of Which is incorporated herein by reference.

tWo images generated by cameras related by a general rigid

transformation (i.e., 3D rotation and translation). The main

STATEMENT OF GOVERNMENT
SPONSORSHIP

reason for this is that the accurate visualiZation of a reference

image as seen from a different camera location ideally

This invention Was made With government support under

Grant No. R01 CA074325, awarded by the National Institutes
of Health. The Government has certain rights in the invention.

requires that the depth map associated With that image be

knoWnisomething Which is not generally true. In certain
situations, such as a robot operating in a knoWn man-made
environment, or during bronchoscopy Where 3D scans are

FIELD OF THE INVENTION

typically performed before the procedure, this information is

This invention relates generally to image correlation and,
in particular, to a fast image registration method applicable to
guided endoscopy and other ?elds.

20

knoWn. Indeed, even in situations Where the depth map is

unknown, it can often be estimated from the images them
selves.

An example of this is the aforementioned shape-from

BACKGROUND OF THE INVENTION

25

shading problem in bronchoscopy guidance [9]. Current

Registration and alignment of images taken by cameras at

practice requires a physician to guide a bronchoscope from

different spatial locations and orientations Within the same

environment is a task Which is vital to many applications in

the trachea to some predetermined location in the airWay tree

With little more than a 3D mental image of the airWay struc
ture, Which must be constructed based on the physicians

computer vision and medical imaging. For example, registra

tion betWeen images taken by a mobile camera and those from

30

a ?xed surveillance camera can assist in robot navigation.

Other applications include the ability to construct image

mosaics and panoramas, high dynamic range images, or
super-resolution images, or the fusion of information
betWeen the tWo sources.

lost Within the airWay during navigation [1]. Such navigation

errors result in missed diagnoses, or cause undue stress to the

patient as the physician may take multiple biopsies at incor

35

HoWever, because the structure of a scene is inherently lost

by the 2D imaging of a 3D scene, only partial registration

information can typically be recovered. In many applications,
depth maps can be generated or estimated to accompany the
images in order to reintroduce the structure to the registration

interpretation of a set of computed tomography (CT) ?lms.

This complex task can often result in the physician getting

rect locations, or the physician may need to spend extra time

returning to knoWn locations in order to reorient themselves.
In order to alleviate this problem and increase the success

rate of bronchoscopic biopsy, thereby improving patient care,

40

some method of locating the camera Within the airWay tree

problem.

must be employed. Fluoroscopy can provide intraoperative

Most currently available 2D alignment algorithms use a

gradient descent approach Which relies on three things: a

vieWs Which can help determine the location of the endo

scope. HoWever, as the images created are 2D projections of

parameteriZation of the spatial relationship betWeen tWo

images (e.g., the 2D rotation and translation betWeen tWo 2D
images), the ability to visualiZe these images under any value
of the parameters (e.g., vieWing a 2D reference image rotated
by 30 degrees), and a cost function With associated image

45

algorithm includes the inverse compositional alignment algo

rithm Which greatly speeds the computation of the parameter
update by recasting the problem, alloWing all gradient and

radiation dose to the patient.

A feW techniques also exist that determine the broncho
50

55

every iteration [6]. Several other improvements have centered

around the choice of parameters and the corresponding image
60

data and attempting to match these to the real bronchoscopic

video using mutual information [3] or image difference [4].
While these methods can register the video to the CT With

varying degrees of success, all operate very sloWly and only

by an af?ne transformation or an 8-parameter homography

involve single-frame registrationinone of them are fast

[7].
eters of the system, and, in the case of the homography, can

bronchoscope images in order to do 3D-to-3D alignment of

the CT airWay surface. This method requires many assump
tions to be made regarding the lighting model and the airWay
surface properties and results in large surface errors When
these assumptions are violated. A second method of doing

this is by iteratively rendering virtual images from the CT

obtained from tWo identical cameras observing the same

scene from a different location can be approximately related

The main problem With these types of parameteriZations is

that they do not truly capture the physically relevant param

scopes location by attempting to match the bronchoscopes

video to the preoperative CT data. One method uses shape
from-shading, as in [2], to estimate 3D surfaces from the

Hessian information to be calculated one time instead of

Warps these parameteriZations induce. For example, images

endoscope position. Additionally, ?uoroscopy is not alWays

available and comes With the added cost of an increased

gradient information Which alloWs an estimate of the param

eter updates to be calculated. Among the most straightfor

Ward and earliest of these algorithms is the Lucas-Kanade
algorithm, Which casts image alignment as a Gauss-Newton
minimiZation problem [5]. A subsequent re?nement to this

the 3D airWays, they can only give limited information of the

65

enough to provide continuous registration betWeen the real

video and the CT volume. They rely on optimiZation methods
Which make no use of either the gradient information nor the

US 7,889,905 B2
3

known depth of the CT-derived images, and thus require very

frame, FIG. 2B shows a warped real image at initial view

point, FIG. 2C shows edges from ?nal reference image over
laid, FIG. 2D shows a reference virtual image corresponding
to ?nal registration, FIG. 2E shows a warped real image at

computationally intensive searches of a parameter space.

SUMMARY OF THE INVENTION

?nal viewpoint, and FIG. 2F shows edges of corresponding

virtual image overlaid; and

This invention resides in a novel framework for fast and

continuous registration between two imaging modalities. A

method of registering an image according to the invention

FIGS. 3A-3C show source images and sample results for

virtual-to-virtual registration; speci?cally, FIG. 3A shows a
real image, FIG. 3B shows a reference image, and FIG. 3C
shows a warped real image.

comprises the steps of providing a set of one or more refer

ence images with depth maps, and registering the image to at

least one of the reference images of the set using the depth
map for that reference image. The image and the reference set

DETAILED DESCRIPTION OF THE INVENTION

may both be real, virtual, or one real with the other virtual.
The set of reference images may endoscopic, derived from a

Broadly, this invention is a 2D image alignment algorithm

which is augmented to three dimensions by introducing the
depth maps of the images. The method provides an ideal way
to extend the existing matching framework to handle general

bronchoscope, colonoscope, laparoscope or other instrument.

The registration preferably occurs in real-time or near real
time, and one or more of the images in the set of reference

images can be updated before, during, or after registration.

According to a robust implementation, the set of reference

images represents viewpoints with depth maps and image

20

gradients, and the image to be registered is derived from a

video feed having a plurality of consecutive frames. The
method includes the steps of:
a) warping a frame of the video to the nearest viewpoint of
the reference source;

used in guided bronchoscopy. In a typical bronchoscopic

procedure, a CT scan is initially performed and can subse
25

b) computing an image difference between the warped

into the airways and a camera mounted on the tip transmits in

real-time a sequence of real bronchoscopic (RB) video

30

viewpoint converges or the next video frame becomes avail

able.
The invention makes it possible to completely determine
the rigid transformation between multiple sources at real
time or near real-time frame-rates in order to register the two

quently be processed to extract the airway tree surfaces. The

interior of the hollow airway tree constitutes the known envi

ronment. During bronchoscopy, the bronchoscope is inserted

video frame and the reference image;

c) updating the viewpoint using a Gauss-Newton param
eter update; and

d) repeating steps a) through c) for each frame until the

3D camera motion, allowing one to directly solve for the

extrinsic parameters of the camera and localiZe it within its
environment.
For the purpose of explaining the method in a very concrete
fashion, discussion shall focus on a situation similar to that

35

images.Assuming that the calibrationparameters of the endo

scope are known, virtual bronchoscopic (VB) images (en
doluminal renderings) can be rendered at arbitrary view
points within the airway tree. It is also clear that the depths
corresponding to each pixel of the VB image can be immedi
ately calculated and form a virtual depth map (VDM).

sources. A disclosed embodiment involving guided broncho

scopy includes the following steps:

The problem is that we have a ?xed real-time RB image

from an unknown location within the interior of an airway, but

1. In the off-line phase, a set of reference images is com

puted or captured within a known environment, com

we also have a known VB source with known location and 3D

information that enables us to create manifestations of the

plete with corresponding depth maps and image gradi

40

same hollow airway structure from arbitrary viewpoint.

ents. The collection of these images and depth maps

Given the above setup, the goal is to locate the source of the

constitutes the reference source.

2. The second source is a real-time source from a live video

RB image by attempting to ?nd the best match between that

?xed RB image and any possible VB endoluminal rendering.

feed. Given one frame from this video feed, and starting
from an initial guess of viewpoint, the real-time video
frame is warped to the nearest viewing site of the refer

A fairly straightforward approach to accomplish this is by

45

image and the VB image with respect to the viewing param

eters (i.e., viewpoint) of the virtual image. The method for
doing this is similar to the Lucas-Kanade image alignment

ence source.

3. An image difference is computed between the warped

video frame and the reference image.
4. The viewpoint is updated via a Gauss-Newton parameter

employing a Gauss-Newton gradient descent algorithm that

attempts to minimize a difference measure between the RB

50

update.

algorithm [5].
The objective function used in [5, 6] is the sum squared
difference (SSD) between the pixel intensities of the two

5. Steps 2-4 are repeated for each frame until the viewpoint
converges or the next video frame becomes available.

images, although weighted SSD is equally viable, and

The ?nal viewpoint gives an estimate of the relative

weighted or unweighted normalized cross-correlation (CC)

rotation and translation between the camera at that par

ticular video frame and the reference source.

The invention has far-reaching applications, particularly in

the ?eld of assisted endoscopy, including bronchoscopy and
colonoscopy. Other applications include aerial and ground
based navigation.

55

can be used if some additional weak assumptions are made.

Using the SSD, the objective function can thus be written as

E = 2 [1m v; p + Ap) - w. m2

(1)

60

BRIEF DESCRIPTION OF THE DRAWINGS

where p is the vector of viewing parameters, Iv (u, v; p+Ap) is

the virtual VB image rendered from the viewpoint p+Ap, u

FIG. 1 shows a block diagram of registration algorithm for

guided bronchoscopy;
FIGS. 2A-2F show source images and results for virtual
to-real registration; speci?cally, FIG. 2A shows a real video

65

and v are the row and column indices, and I, is the real RB

image. Following the procedure of [5], it is shown that that the

Gauss-Newton parameter update Ap can be found as

US 7,889,905 B2
6
depth map of the real image Zr. Solving for the Gauss-New
ton parameter update associated With 4 yields
Ap = W12 [glam v; p) - w. w]

(2)

Where the Hessian H is approximated per Gauss-Newton as

While this may seem to add unnecessary complexity and error

(3)

to the problem, it actually serves to greatly speed the iteration

and has the additional side bene?t of eliminating the need to
render arbitrary vieWpoints on the ?y if you instead have a

MW

collection of pre-rendered (or pre-captured) images and cor

responding depth maps. The reason for this signi?cant
increase in speed is that the VB image and VB image gradi
ents are alWays evaluated at p:0, the reference vieWing site,

Where

and as such alloWs all of the folloWing operations to be pre

computed before iteration begins:

20

is a vector that gives the change in the intensity of a pixel (u,

V) in a VB image lv rendered at vieWpoint p With respect to
each of the components of the parameter vector Ap.

1. The knoWn environment is sampled as a set of vieWing

sites.

2. Virtual images Iv are pre-rendered at each vieWing site.

3. Virtual depth maps Zv are computed at each site.
25

4. Steepest descent images

30

can also be interpreted as a vector of steepest descent images,

are computed With respect to each of the vieWing parameters

Where each component of the vector is actually an image that

describes the variation of the image intensities With respect a
component of the parameter vector. Because the steepest

in vector p.

descent images

5. The inverse Hessian H-1 is Gauss-NeWton estimated

35

from the steepest descent images

01,
5P

40

via equation (14).

change at every vieWpoint p, they, and the Hessian must be
recomputed every iteration, leading to a very computationally

costly algorithm.

The iterative portion of the algorithm may then be carried out

in the folloWing steps:

45

To speed up the iteration, the inverse compositional algo

ing the virtual vieWpoint toWards the real vieWpoint using the

2. Compute the error image: I,(W(u,v, Z,;p))lv(u, v;0).

3. Compute the parameter update Ap via equation (5).

parameter update, We instead move the real vieWpoint toWard

the virtual vieWpoint using the inverse of the parameter

50

update. Since the computer obviously has no control over the

location of the bronchoscope tip, this may seem to be an

These steps are illustrated in FIG. 1. Ignoring the Warp

function, all the equations presented thus far are general and

the RB image can be Warped to simulate its appearance from

can apply equally Well to 2D transformations, such as af?ne

55

60

tant also to note that the Warp in this case is dependent on the

or homography, or 3D rotations. The focus is noW narroWed,

hoWever, to the full 3D motion case With our choice of coor

dinate system and parameters. One may realiZe from inspec

tion of the Warps in (4) that the problem is de?ned in terms of

is:

The Warping function W ()warps the image coordinates of

the RB image I, and hence Warps the image itself. It is impor

4. Find the neW values of p by incrementing the old param

eters by the inverse of the update (Ap)_l.

unfeasible strategy. HoWever, using a depth-based Warping,

other vieWpoints. This strategy results in comparing a Warped
version of the real image to a stationary virtual image. Under
this formulation, the objective function We seek to minimiZe

1 . Warp the real image from pose p to the nearest reference

site.

rithm Was proposed [6]. Under this strategy, instead of mov

65

several local coordinate systems as each reference vieW is

de?ned to be at p:01, yielding a different coordinate system
for each vieWing site used. It is, hoWever, a trivial matter to
relate each of these coordinate systems to a global coordinate
frame in order to perform parameter conversions betWeen
frames. Therefore, given a camera pose With respect to the
global camera frame, We can de?ne our parameter vector as

US 7,889,905 B2
7
With three Euler rotation angles and three translations With
respect to the nearest reference vieW.
BI

With this parameteriZation, the Warping W(u,v,Z;p) is gov

erned by the matrix equation

Where VHI and Val are the image gradients With respect to the
roWs and columns of the image, and JP is the Jacobian of the
Warped coordinates With respect to p and thus can be found by
differentiating u' and v' from (7) With respect to each of the
Warp parameters and evaluating it at a particular current value
of p. In the case of the inverse compositional algorithm, the
image derivatives are alWays evaluated at p:() and thus the
Jacobian is constant for each reference vieWing site:
Where R is the rotation matrix de?ned by the Euler angles (6 ,,
6P, 6y), u and V are the columns and roWs of the image, f is the
focal length, and Z is the entry on the depth map Z corre

sponding to the point (u,v). Here (u',v') gives the Warped

image coordinate of interest, and Z' gives the Warped depth

(9)

20

corresponding to that point. Note that in the problem state

ment, We assume only that the virtual depth map Zv is knoWn.

However, When using the inverse compositional algorithm,

the Warp is applied to the real image I, and the real depth map
Z, must ?rst be calculated by Warping the virtual depth map

We noW have all the necessary information to calculate the

25

Zv to the current estimated pose of the real camera via p. This

can also be performed using (7) and then interpolating the

resulting Warped depth map onto the coordinate system of the
real image. In doing so, We are implicitly assuming that our
estimate of p is relatively close to its actual value. If this is not
the case, the parameter error can lead to large errors in the real

depth map Zr, and therefore large errors in the image Warping.
Under such circumstances, the forWard gradient descent
method governed by (1-2) may be better suited to the prob

iterated parameter update Ap. The ?nal step is to invert this

update, and compose it With the current estimate of p. The
Euler angles can be found from the rotation matrix resulting
from

R'IRRf

Where Rd is the incremental rotation matrix associated With

the rotation angles in Ap. The updated translations can be

found from
35

(11)

lem.
In order to apply the Warping function, at each pixel coor

dinate (u,v), With intensity I(u,v) and depth Z(u,v), a neW

coordinate (u',v') and depth Z' (u',v') are found via (7). The
original intensities and depths may then be mapped onto the

(10)

30

40

neW image array I(u',v'). Some special care must be taken

Where Atl- are the translation elements of the parameter update

When performing the Warping. Firstly, the image difference in

Ap.

(4) requires that the coordinate locations be the same for both
images. The resultant array must therefore be interpolated

45

Operations performed on full-resolution images can be very

computationally intensive. Therefore, a resolution pyramid is
used Wherein all images, depth maps, and gradients are doWn

onto the same coordinate grid as the original arrays. Because

of this interpolation, and because the depth-based Warping

may result in occlusion, it can be dif?cult to choose the proper

sampled, preferably by a factor of 4, at each level. As We are

intensity corresponding to an output pixel. This can be miti

gated someWhat if the intensities corresponding to larger

depths are discarded When they overlap With those of smaller

50

not particularly concerned With computation time regarding

the precomputed virtual vieWs and gradients, and most video

capture hardWare provides real-time hardWare subsampling

for the real image, the computational cost of this subsampling
is inconsequential and provides much quicker iteration times.

depths.
Finally, We turn to the calculation of the steepest-descent

images

In order to improve the performance When applying the

above approach, several optimiZing techniques are used.

55

When implementing the above registration algorithm

using pyramid decomposition, the algorithm is begun at the
loWest resolution level of the pyramid (experimental results in

61

13p
60

this paper Were performed starting at level 3; i.e., a factor of

64 reduction in resolution) and run until a reasonable stop
ping criteria Was met before proceeding to a higher resolution

level. This pyramidal approach not only speeds computation,

There are several Ways to generate the steepest descent

it also serves to prevent convergence to local optima, because

images. They may be generated numerically by taking the

only the largest features are present in the highly subsampled

difference of the reference images Warped to small positive

and negative values of each parameter. They may also be

generated analytically by expanding the derivative via the

images, While sharper features are introduced in higher reso

lution levels to aid in ?ne adjustment.
A second optimiZation that is used in practice is the use of

chain rule:

the Weighted normaliZed cross-correlation objective function

65

US 7,889,905 B2
10
in order to emphasize dark areas, which tend to have more

information in bronchoscopic video. The video frame, taken

from a bronchoscopic procedure performed on h005 was ?rst
processed to remove the geometric barrel distortion from the
lens to obtain the real image I,. In the virtual-to-real registra
tion case, it is dif?cult to give ground truth locations as the
location of the scope tip is in practice unknown. Without

external localization, the quality of a good registration is

somewhat qualitative in nature. FIG. 2 shows a sample of the

that allows images of different mean intensities and intensity

ranges to be compared and also allows weighting of indi
vidual pixel values. It should be noted that in order to use this

registration results, with edges from the virtual image over

laid on the unregistered and registered real views. The results

show that the alignment is qualitatively very satisfying.

Virtual-to-Virtual Registration

objective function under the inverse compositional algorithm,

the weights must be constant and they must be chosen prior to

the computation of the steepest descent images (i.e. they must

be based off features of the virtual images). Taking advantage

In the virtual-to-virtual registration case, the real image

is actually a rendering generated at a speci?ed location in the

of the equivalence of normalized SSD and normalized cross

correlation, the update can be found as:

(13)
F0

airway, but with all depth information discarded. The algo

rithm uses pyramid decomposition starting from level 3 and
ending at level 1, and the weighted SSD objective function
20

was used where the weights WM, were chosen as in (15) as

before.

FIG. 3 shows the real image I, prior to registration, the

virtual image I, at the nearest reference site and the warped

real image I, (W(u,v,Z;p)) after registration is complete.

25

where the Hessian in this case is

mi

#21124 13%],

(14)

Viewpoint

60L (deg)

@[5 (deg)

6y
(deg)

30

Initial
Reference
Site
Registered
Ground
Truth
35
Error

is the set of mean-subtracted steepest descent images divided

by the variance of the virtual image Iv, and Iii are the normal

position position Z position

(mm)
(mm)
(mm)
147.5
146.7

149.2
149.4

71.1
73.3

20.2
7.3

1.7
5.1

0
19.9

147.6
147.1

149.0
148.9

73.9
73.8

20.9
20.24

1.2
1.8

3.2
0.4

0.6

0.1

0.5

0.7

3.0

2.8

At least four different alternatives are available for regis

tering the real and virtual sources in the case of bronchoscopy.
40

ized images.

These scenarios are outlined below:

1. Virtual-to-real registration: real-time or pre-recorded

video images I, from a bronchoscope at an unknown

Examples

location are registered to a set of endoluminal CT ren

To validate the algorithm, sample results for the virtual-to

derings I, and depth maps Z,.

45

real and virtual-to-virtual registration cases are given. In both

of the cases outlined below, the virtual environment is a CT
chest scan of a human patient designated h005. The airway

I, with unknown location and with or without an associ

ated depth map Z, is registered to a set of endoluminal

surfaces were automatically generated using the methods of

Kiraly et al. [10] . Airway centerlines were extracted using the
methods of Swift et al. and the virtual viewing sites were

50

CT renderings Iv and depth maps Zv.

3. Real-to-real registration: real-time video images I, from
an endoscope at an unknown location is registered to a

set of previously recorded video images I, with known or

chosen along these airway centerlines at intervals varying

estimated depth maps Z,.

between 0.3 mm and 1 mm, with the viewing direction chosen

parallel to the airway centerline [l 1]. Virtual images and

depth maps were generated by an OpenGL renderer assuming

2. Virtual-to -virtual registration: an endoluminal rendering

4. Real-to -virtual registration: an endoluminal rendering I,

55

with unknown position and with or without an associ

ated depth map Z, is registered to a set of previously

recorded video images I, with known or estimated depth

a spot light source at the camera focal point, a ?eld of view of

78.2 degrees and a 264x264 image size to match the calibra

tion parameters of the bronchoscope camera.

maps Zv.

The application has far-reaching applications, particularly

Virtual-to-Real Registration
The virtual-to-real registration was performed using pyra
mid decomposition starting from level 3 and ending at level 1 .
To account for the difference in intensity characteristics
between the imaging sources, the weighted normalized cross
correlation (12) was used as the objective function, with
weights WW chosen as

(15)

60

in the ?eld of assisted endoscopy. The registration between a

CT volume and real-time bronchoscopic video allows the
fusion of information between the CT realm and the broncho

scope. This allows regions of interest (ROIs) de?ned only in

the CT volume to be superimposed on the real video frame to
65

assist the physician in navigating to these ROIs. Likewise,

airway centerlines, branch labels and metric information such
as distances to walls can be displayed on the video frame.

US 7,889,905 B2
11

12

A natural extension of this concept is to other forms of

endoscopy such as colonoscopy, Where similar guidance

information could be displayed on the registered colono

11. R. SWift, A. Kiraly, A. Sherbondy, A. L. Austin, E. A.

Hoffman, G. McLennan, and W. E. Higgins, Automatic
axes-generation for virtual bronchoscopic assessment of

scopic image. Virtual-to-real registration can also be applied

major airWay obstructions, Computerized Medical Imag

to pre-recorded endoscopic video, and opens the door to

many post-processing options, such as mapping textural and
color information available only in the endoscopic video onto

ing and Graphics 26, pp. 103-118, March-April 2002.

We claim:
1. A method of registering video frames of a body lumen
received from an endoscope inserted into the body lumen to

the CT-derived surfaces to enable their visualiZation from

vieWpoints not available in the video alone.
previously acquired images of the body lumen, the method
An application of the real-to-real registration scenario that 10 comprising the steps of:
can be envisioned for this approach, is for aerial navigation.
computing a plurality of reference images of a body lumen,
Satellite imagery, combined With topographic terrain infor
each reference image including a vieWpoint With a depth
mation provides the knoWn 3D environment, While real-time
map and image gradients;
images from a mobile camera aboard an aircraft can be reg

receiving video frames from the endoscope Wherein said

istered to this environment to give the aircrafts location and

orientation Without GPS or radar information. Similarly, this

receiving step comprises receiving a live bronchoscopic

video frame associated With an airWay; and
providing a processor to register each video frame received
from the endoscope to at least one of the reference

method also assists in ground-based robotic navigation

Within a knoWn environment. Reference images and depth
maps can be captured at knoWn locations throughout the
robots Working environment using a stereo camera setup,

images, said processor being programmed to perform

20

and a camera mounted on the robot can be registered to this set

of images and depth maps.

the steps of:

a) Warping the video frame to the vieWpoint of one

reference image of the plurality of reference images

or Warping one reference image of the plurality of
reference images at a vieWpoint to the video frame;

REFERENCES
25

1. H. Minami, Y. Ando, F. Nomura, S. Sakai, and K.

betWeen (i) the Warped video frame and the reference

image, or (ii) the Warped reference image and the
video frame;
c) updating a vieWpoint of the video frame using an

Shimokata, Interbronchoscopist variability in the diagno

sis of lung cancer by ?exible bronchoscopy, Chest 105(2),
pp. 1658-1662, June 1994.

2. I. Bricault, G. Ferretti, and P. Cinquin, Registration of real

and CT-derived virtual bronchoscopic images to assist
transbronchial biopsy, IEEE Transactions On Medical
Imaging, Vol. 17, No. 5, pp. 703-714, October 1998.

3. J. Helferty, Image-Guided Endoscopy and its Application

To Pulmonary Medicine. PhD thesis, The Pennsylvania

30

35

State University, 2002.

4. D. Deguchi, K. Mori, J. HasegaWa, J. ToriWaki, and H.
Natori et al., Camera motion tracking of real broncho
scope using epipolar geometry analysis and CT derived

bronchoscopic images, SPIE Medical Imaging 2002:

Physiol. Func. from Multidim. Images A Clough and C.
Chen (ed.), v. 4683, pp. 30-41, 2002.
5. B. Lucas and T. Kanade, An iterative image registration
technique With an application to stereo vision, Proceed
ings of the International Joint Conference on Arti?cial

40

45

50

Technical Report MSR-TR-97-23, Microsoft Research.

8. H.-Y. Shum and R. SZeliski, Systems and Experiment

1997.

10.A. P. Kiraly, E. A. Hoffman, G. McLennan, W. E. Higgins,

and J. M. Reinhardt, 3D human airWay segmentation for
virtual bronchoscopy, SPIE Medical Imaging 2002:
Physiology and Funct. from Multidim. Images, A. V.
Clough and C. T. Chen, eds. 4683, pp. 16-29, 2002.

3. The method of claim 1, including the step of guessing an

initial vieWpoint of the video frame.
4. The method of claim 1, Wherein the ?nal vieWpoint gives
an estimate of the relative rotation and translation betWeen the

using a resolution pyramid Wherein all images, depth maps,

and gradients are doWn-sampled.
6. The method of claim 1, further including the step of
using a Weighted normalized cross-correlation objective
function.
7. The method of claim 1, Wherein the optimiZation tech
nique is based on a Gauss-NeWton parameter update.
8. The method of claim 1 Wherein the computed set of
reference images of a body lumen comprises a 3-D model of
a bronchial tree.

9. The method of claim 1 Wherein the body lumen is an

55

airWay.
10. A method of registering video frames of a body lumen
received from an endoscope inserted into the body lumen to

previously acquired images of the body lumen, the method

comprising the steps of:

9. T. Okatani and K. Deguchi, Shape reconstruction from an

endoscope image by shape from shading technique for a

point light source at the projection center, Computer
Vision and Image Understanding 66, pp. 119-131, May

until the vieWpoint of the video frame converges on a

?nal vieWpoint or a next video frame becomes avail
able.
2. The method of claim 1, Wherein at least one reference

video frame of the video feed and the reference image.

5. The method of claim 1, further including the step of

image alignment algorithms, Proceedings of the IEEE

Paper: Construction of panoramic image mosaics With

global and local alignment, International Journal of Com
puter Vision 36(2), pp 101-130 (2000).

optimiZation technique; and

d) repeating steps a) through c) for each video frame

image is updated before, during, or after registration.

Intelligence, pp. 674-679, 1981.

6. S. Baker and I. MattheWs, Equivalence and e?iciency of
Conference on Computer Vision and Pattern Recognition
1, pp. 1090-1097, 2001.
7. H.-Y. Shum and R. SZelisld, Panoramic Image Mosaics.

b) automatically computing an image difference

60

computing a plurality of reference images of a body lumen,

each reference image including a vieWpoint With a depth

map and image gradients;

receiving at least one video frame from the endoscope; and
providing a processor to register each video frame received
from the endoscope to at least one of the reference

images using of the plurality, said processor being pro

grammed to perform the steps of:

US 7,889,905 B2
14

13

12. The method of claim 10, including the step of guessing

a) Warping the video frame to the vieWpoint of one

reference image of the plurality of reference images

an initial vieWpoint of the video frame.

or Warping one reference image of the plurality of

reference images at a vieWpoint to the video frame;

13. The method of claim 10, further including the step of

using a resolution pyramid Wherein all images, depth maps,
and gradients are doWn-sampled.
14. The method of claim 10, further including the step of
using a Weighted normalized cross-correlation objective

b) automatically computing an image difference

betWeen (i) the Warped video frame and the reference
image, or (ii) the Warped reference image and the
video frame;
c) updating a vieWpoint of the video frame using an

optimiZation technique; and

d) repeating steps a) through c) for each video frame
until the vieWpoint of the video frame converges on a
?nal vieWpoint or a next video frame becomes avail

able Wherein the ?nal vieWpoint gives an estimate of

relative rotation and translation betWeen the video

frame and the reference image.

11. The method of claim 10, Wherein at least one reference

image is updated before, during, or after registration.

function.
15. The method of claim 10 Wherein the optimiZation tech
nique is based on a Gauss-Newton parameter update.
16. The method of claim 10 Wherein the step of receiving at
least one video frame from an endoscope comprises receiving
a live bronchoscopic video frame.
17. The method of claim 16 Wherein the computed set of
reference images of the body lumen comprises a 3-D model of
a bronchial tree.