0 AUGMENTED REALITY

ABSTRACT

The basic idea of Augmented Reality is to superimposing graphics,

audio and other sense enhancements over a real world environment
in real time. Augmented Reality is still in an early stage of research
and development at various universities and high-tech companies.
Eventually, possible by the end this decade, we will see first mass-
marketed Augmented Reality system, which one researcher calls “the
Walkman of 21st century”. What Augmented Reality attempts to do is
not only superimposing graphics over a real environment in real
time, but also change those graphics to accommodate a users head
and eye movements, so that the graphics always fit the perspective.

C.A.S PUTHUPPALLY 0
 1 AUGMENTED REALITY

CONTENTS

INTRODUCTION 2

a. Definition 3
b. Components Required 4

REVIEW 5

a. Head Mounted Display 5

1. Optical Vs Video 8
2. Comparison 11
b. Tracking & Orientation 14

ERRORS 16

a. Static Errors 16
b. Dynamic Errors 17

LIMITATIONS 20

a. Technological Limitations 20
b. User Interface Limitations 21
c. Social Acceptance 21

APPICATIONS 22

CONCLUSIONS 24

REFERANCES 25

C.A.S PUTHUPPALLY 1
 2 AUGMENTED REALITY

INTRODUCTION

Augmented Reality supplements the real world with

virtual (computer –generated) objects that appear to coexist in
the same space as real world.

The beginnings of AR date back to Sutherland’s

work in the 1960’s, which used a see through HMD to present 3D
graphics . However , only over the past decade has there been
enough work to refer to AR as a research field. In 1997, Azuma
published survey that defined the field, described many problems,
and summarized the developments up to that point. Since then,
Augmented Reality’s growth and progress have been remarkable.
In the late 1990’s, several conferences on augmented Reality began,
including the international Workshop and Symposium on
Augmented Reality, the international Symposium on Mixed
Reality , and the designing Augmented Reality Environments
Workshop . Some well-funded organizations formed that focused
on Augmented Reality, notably the Mixed Reality Systems Lab in
Japan and the Arvika consortium in Germany. A software toolkit
(the A toolkit ) for rapidly building Augmented Reality
applications is now freely available at

http://www.hitl.washington.edu/research/shared space/.

Because of the wealth of new developments , this field needs

an updated survey to guide and encourage further
research in this exciting area.

C.A.S PUTHUPPALLY 2
 3 AUGMENTED REALITY

a. Definition

Augmented Reality (AR) is a variation of

Virtual Environments (VE) , or Virtual Reality as it is more
commonly called . VE technology completely Immerses a
user inside a synthetic environment . while immersed , the
user cannot see the real world around him . In contrast , AR
allows the user to see the real world , with virtual objects
superimposed upon or composite with the real world .
Therefore , AR supple- ments reality , rather than completely
replacing it . Ideally , it would appear to the user that the
virtual and real objects coexist in the same space .

AR as system has the following three characteristics:

1) Combines real and virtual

2) Interactive in real time

3) Registered in 3-D

Augmented Reality enhances a user’s

perception of and interaction between the real world . The
virtual objects display information that the user cannot
directly detect with his own senses . The information
conveyed by the virtual objects helps a user perform real
world tasks .

C.A.S PUTHUPPALLY 3
 4 AUGMENTED REALITY

b. Components Required :

The three components needed to make an augmented - reality

System work :

a. Head –mounted display

b. Tracking system

The goal of augmented – reality developers is to

incorporate these three components into one unit , housed
in a belt - worn device that wirelessly relays information
to a display that resembles an ordinary pair of eye glasses .
Besides adding objects to a real environment , Augmented
Reality also have the potential to remove them . Current
work has focused on adding virtual objects to a real
environment . However , graphic overlays might also be
used to remove or hide parts of the real environment from
a user

AR could be extended to include sound .

The user would wear headphones equipped with
microphones on the outside . The headphone would add
synthetic , directional 3-D sound , while the external micro-
phones would detect incoming sounds from the environment.
This would give the system a chance mask or cover up
selected real sounds from the environment by generating

C.A.S PUTHUPPALLY 4
 5 AUGMENTED REALITY

a masking signal that exactly cancelled real sound . While

this would not be very easy to do , it might be possible .

Review

Augmented Reality will further blur the line

between what’s real and what’s computer generated by
enhancing what we see , hear , feel and smell . On the
spectrum between virtual reality, which creates immiscible,
computer generated environments and the real world ,
augmented reality is closer to the real world . Augmented
Reality adds graphics , sound , hap tics and smell to the
natural world , as it exists . You can expect video games to
drive the developments of augmented reality , but this
technology will have countless applications . Everyone from
tourists to military troops will benefit from the ability to
place the computer - generated in their field of vision .

a. Head-mounted displays:

Just as the monitors allow us to see text and

graphics generated by computers , head - mounted displays
(HMDs) will enable us to view graphics and text created by
augmented reality systems .

So far , there haven’t been many HMD’s

created specifically augmented reality in mind . Most of
the displays , which resemble some type of skiing goggles,
were originally created for virtual reality.

C.A.S PUTHUPPALLY 5
 6 AUGMENTED REALITY

There are two types of HMD’s

1. Video see – through

2. Optical see – through

Video see through displays block out the wearer’s

surrounding environment , using small video cameras attached
to the out side of the goggles to capture images . On the
inside of the display , the video image is played in real -
time and the graphics are superimposed on the video . One
problem with the use of video cameras is that there are
more legs , meaning that there is a delay in image adjustment
when the viewer moves his or her head .

Augmented Reality displays are still pretty bulky; but

developers believe that they can create a display that
resembles a pair of eye glasses . Most companies who have
made optical - see through displays have gone out of bussi-
ness . Sony makes a optical see - through display that some
researchers use called the Glasstron . Blair Macintyre , director
of the Augmented Environments Lab at Georgia Tech
believes that the Micro vision’s Virtual Retinal Display
holds the most promise for an augmented reality system.
This device actually uses light to paint images onto retina
by rapidly moving the light source across and down the
retina . The problem with the Micro vision displays is that it
currently costs about $ 10,000

C.A.S PUTHUPPALLY 6
 7 AUGMENTED REALITY

Windows Over the Real World

Augmented Reality Display will Overlay

Computer Generated
Graphics on to the Real - World

C.A.S PUTHUPPALLY 7
 8 AUGMENTED REALITY

1. Optical vs Video

A basic design decision in building

an AR system is how to accomplish the combining of
Real and virtual . Two basic chooses are available : optical
and video technologies . each has particular advantages and
disadvantages . A see - through HMD is one device used to
combine real and virtual . Standard closed - view HMDs do
not allow any direct view of real world . In contrast, a see
through HMD lets the user see the real world , with virtual
objects superimposed by optical or video technologies .

Optical see - through HMDs work by placing , optical

combiners in front of the user’s eye . These combiners are
partially Tran missive , so that the user can look directly
through them to see the real world . The combiners are also
Partially reflective , so that the user sees virtual images
bounced off the combiners from head mounted monitors .
this approach is similar in nature to Head – Up Displays
(HUDs) commonly used in military aircraft , except that the
combiners are attached to the head. Thus, optical see-through
HMDs have sometimes been described as a “HUD on a
head” . However , most existing optical see - through HMDs
do reduce the amount of light from the real world , so
they act as pair of sunglasses when power is cutoff .

C.A.S PUTHUPPALLY 8
 9 AUGMENTED REALITY

Hea
d Mounted Display

In contrast, video see-through HMDs work by combining a

closed view HMD with one or two head mounted video cameras.
The video cameras provide the user’s view of the real world. Video
from these cameras is combined with the graphic images. Created by
the scene generator, Blending the real and virtual. The result is
sending to the monitors in front of the user’s eye in the closed view
HMD. Figure shows an actual video see-through HMD, with two
video cameras mounted on top of a Flight Helmet.

C.A.S PUTHUPPALLY 9
 10 AUGMENTED REALITY

C.A.S PUTHUPPALLY 10
 11 AUGMENTED REALITY

An optical approach has the following advantages over a

video approach :

2.Comparison:

This section compares the relative advantages and

disadvantages of optical and video approaches, starting with optical.

1) Simplicity : Optical blending is simpler and cheaper than

video blending . Optical approach has only one “stream”
of video to worry about - the graphic images . The real
world is seen directly through the combiners , and that time
delay is generally a few nanoseconds . Video blending
on the other hand, must deal with separate wide streams
for the real and virtual images .

2) Resolution : Video blending limits the resolution of what

the user sees , both real and virtual , to resolution of the
display devices . Optical see – through also shows the
graphic images at the resolution of the display device, but
the user’s view of the real world is not degraded. Thus
video reduces the resolution of the real world , while
optical see – through does not .

3) Safety : Video see – through HMD’s are essentially

modified closed view HMD’s . If the power is cutoff, the
user is effectively blind . This is a safety concern in some
applications . in contrast , when power is removed from
an optical see – through HMD , the user still has a direct
view of the real world . THE HMD then becomes a
pair of heavy sunglasses , but the user can still see .

C.A.S PUTHUPPALLY 11
 12 AUGMENTED REALITY

4) No eye offset : with video see – through , the user’s view

of real world is provided by the video cameras . In
essence , this puts his “eyes” where the video cameras
are . In most configuration , the cameras are not located
exactly where user’s eye are , creating an offset between
the cameras and real eyes . The distance separating the
cameras may also not be exactly the same as the user’s
inter pupilary distance ( IPD ) . This difference between
camera locations and eye locations displacements from
what the user sees compared to what he expects to see .

Video blending offers the following advantages over

Optical blending :

1) flexibility in comparison strategies : A basic problem with

optical see through is that the virtual objects do not
completely obscure the real world objects , because the
optical combiners allow light from both virtual and real
sources . Building an optical see – through HMD that can
selectively shut down the light from the real world is
difficult . In contrast , video see–through is far more flexible
about how it merges the real and virtual images . Since
both the real and virtual are available in digital form,
video see – through compositors can , on a pixel –by –pixel
basis , take the real , or the virtual , or some blend between
the two to simulate transparency . Because of this flexibility,
video see – through approaches .

C.A.S PUTHUPPALLY 12
 13 AUGMENTED REALITY

2) Wide field - of – view : Distortion in systems are a function

of the radial distance away from the optical axis. The further
one looks away from the center of the view, the larger the
distortion get . A digitized image taken through a distorted
optical system can be undistorted by applying image-
processing techniques to unwrap the image , provided that
the optical distortion is well characterized . This requires
significant amounts of computations , but this constraint will
be less important in the future as computers become faster. It
is harder to build wide field – of –view displays with
optical see - through techniques . Any distortion of the user’s
view of the real world must be corrected optically , rather
than digitally , because the system has no digitized image
of the real world to manipulate . Complex optics are
expensive and adds weight to the HMD. Wide field of view
systems are an exception to the general trend of optical
approaches being simpler and cheaper than video appr-
oaches .

3) Real and virtual view delays can be matched : Video offers

an approach for reducing or avoiding problems caused by
temporal mismatches between the real and virtual images .
Optical see - through HMD’s offer an almost instantaneous
view of the real world but a delayed view of the virtual .
This temporal mismatch can cause problems . With video
approaches , it is possible to delay the video of the real
world to match the delay from the virtual image stream.

4) Additional registration strategies : in optical see – through ,

the only information the system has about the user’s head
location comes from the head tracker . Video blending provides
another source of information : the digitized image of the real
scene .This digitized image means that video approaches can
employ additional registration strategies unavailable to optical
approaches . Both optical and video technologies have their

C.A.S PUTHUPPALLY 13
 14 AUGMENTED REALITY

roles , and the choice of technology depends on the application

requirements . Many of the mechanical assembly and repair
prototypes use optical approaches , possibly because of the
coast and safety issues . If successful ,the equipment would
have to be replicated in large numbers to equip workers
on a factory floor . in contrast , most of the prototypes for
medical applications use video approaches , probably for
the flexibility in blending real and virtual and for
additional registration strategies offered . We can classify
displays for viewing the merged virtual and real
environments in to the following categories : head
mounted , handheld and projective .

b. TRACKING AND ORIENTATION :

The biggest challenge facing developers of

augmented reality is the need to know where the
user is located in reference to his or her surroundings .
There’s also the additional problem of tracking the
movements of user’s eye and heads .

C.A.S PUTHUPPALLY 14
 15 AUGMENTED REALITY

A tracking system has to recognize these movements and

projected the graphics related to the real - world
environment the user is seeing at any given moment .
Currently , both video see – through and optical see –
through
displays typically have lag in the overlaid material due to
the tracking technologies currently available .

For augmented reality to reach it’s full potential , it

must be useful both outdoors and indoors . Currently ,
the best tracking technologies currently available for large
open areas is the Global Positioning System .
However , GPS receivers have an accuracy of about 10
to 30 meters , which is not bad in the grand scheme of
things , but isn’t good enough for augmented reality ,
which needs accuracy measured in millimeters or
smaller . An augmented reality system would be
worthless if the graphics projected were of something
10 to 30 meters away from what you were actually
looking at .

There ways to increase tracking accuracy . For

instance , the military use multiple GPS signals . There
are also differential GPS , which involves using an
area that has already been survived . Then the system
would use a GPS receiver with an antenna that’s
location is known very precisely to track your location
within that area . This will allow users to know
exactly how inaccurate their GPS receivers are , and can
adjust an augmented – reality system accordingly .
Differential GPS allows for sub meter accuracy . A more
accurate system being developed , known as real – time
kinematics GPS , can achieve centimeter – level accuracy .

C.A.S PUTHUPPALLY 15
 16 AUGMENTED REALITY

Tracking is easier in small spaces than in large

spaces . Researchers at the University of North
Carolina – Chapel Hill have developed a very precise
system that works within 500 square feet . The Hi Ball
Tracking System is an optoelectronics tracking system
made two parts :

* Six user – mounted , optical sensors

* Infrared – light – emitting diodes (LED’s) embedded in
special ceiling panels

The Hi Ball tracking system uses an optical sensing

device and LED embedded ceiling tiles to track movements
over a short range .

Errors
Just like any system in universe Augmented Reality
systems are also prone to errors . Registration errors are
difficult to adequately control because of the high accuracy
requirements and the numerous source of error . These
sources of error can be divided in to two types : Static and
Dynamic . Static errors are the ones that cause registration
errors even when the user’s viewpoint and the objects in
the environment remain completely still . Dynamic errors are
the ones that have no effect until either the view point
begin moving . For current HMD – based systems , dynamic
errors are by far the largest contributors to registration
errors , but static errors cannot be ignored either .

a. Static Errors
The three main sources of static errors are :

C.A.S PUTHUPPALLY 16
 17 AUGMENTED REALITY

*Optical distortion

*Errors in the tracking system

*Mechanical misalignments

1. Distortion in the optics : Optical distortion exist in

most of the camera and lens systems , both in the
cameras that record the real environment and in the
optics uses for the display . Distortions are usually a
function of the radial distance away from the optical axis
. Near the center of the field - of - view , images are
relatively undistorted , but far away from the center ,
image distortion can be large . For example , straight
lines may appear curved . In a see- through HMD with
narrow field – of – view displays , the optical combiners add
virtually no distortion , so the user’s view of the real world
is not warped . However , the optics used to focus and
magnify the graphic images from the display monitors
can introduce distortion . This mapping of distorted
virtual images on top of the undistorted view of the real
world causes static registration errors . Optical distortions
are usually systematic errors , so they can be mapped
and compensated .

2. Errors in the tracking system : Errors in the reported

outputs from the tracking and sensing systems are often
the most serious type pf static registration errors .
These distortions are not easy to measure and eliminate .
Almost all commercially available tracking systems are not
accurate enough to satisfy the requirements of AR
systems .

3. Mechanical misalignments : Mechanical misalignments are

discrepancies between the model or specification of the
hardware and the actual physical properties of the
real system . For example , the combiners , optics , and

C.A.S PUTHUPPALLY 17
 18 AUGMENTED REALITY

monitors in an optical see – through HMD may not at

the expected distance or orientation with respect to each
other . These misalignments can cause subtle changes in
the position and orientation of the projected virtual
images that are difficult to compensate .

b. Dynamic Errors

Dynamic errors occur because of system delays , or

lags . The end – to – end system delay is defined as the time
difference between the moment that the tracking system
measure the position and orientation of the view point to
the moment when the generated images corresponding to
that position and orientation appear in the displays . These
delays exist because each component in an Augmented
Reality system requires some time to do it’s job . The
delay in the tracking subsystem , the communication delays ,
the time it takes the scene generator to draw the appropriate
images in the frame buffers , and the scan out time from the
frame buffer to the displays all contribute to end – to – end
lag .

End –to – end delays of 100 ms are fairly typical on

existing systems . Simpler systems can have less delay ,

but other systems have more . Delays of 250 ms or more

can exist on slow , heavily loaded , or networked systems .
End –to – end system delay cause registration errors only
when motion occurs . Assume that the view point and all
objects remain still . Then the lag does not cause registration
errors . No matter how long the delay is , the images generated
are appropriate , since nothing has moved since the time the
tracker measurement was taken .

C.A.S PUTHUPPALLY 18
 19 AUGMENTED REALITY

Compare this to the case with motion . For example , assume

a user wears a see – through HMD and moves her head .

The tracker measures the head at an initial time t . The

images corresponding to time t will not appear until some
future time t2 , because of the end –to –end system delays .
During this delay , the user’s head remains in motion , so
when the images captured at time t finally appear , the user
sees them at different location than the one they were
computed for . thus , the images are incorrect for the time
they are actually viewed . To the user , the virtual objects
appear to “ swim around “ and “lag behind” the real objects .

Methods used to reduce registration fall under four

main categories :

* Reduce system lag

* Reduce apparent lag

*Match temporal streams (with video – based system)

*Predict future locations

1. Reduce system lag : the most direct approach is simply

to reduce , or ideally eliminate , the system delays . if
there are no delays , there are no dynamic errors . System
delays are not likely to completely disappear anytime
soon. Unfortunately , it is difficult to reduce system delays
to the point where they are no longer an issue .

C.A.S PUTHUPPALLY 19
 20 AUGMENTED REALITY

2. Reduce apparent lag : Image deflection is a clever technique

for reducing the amount of apparent system delay for
systems that only use head orientation . It is way to
incorporate more recent orientation measurements in to
the late stages of the rendering pipelining . it is a feed
-forward technique .

3. Match temporal streams : In video –based AR systems ,

the video camera and digitization hardware impose
inherent delays on the users view of the real world . This
is potentially a blessing when reducing dynamic errors ,
because it allows the temporal streams of the real and
virtual images to be matched . Additional delay is added
to the video from real world to match the scene
generator delays in generating the virtual images . This
additional delay to the video stream will probably not
remain constant , since the scene generator delay will
vary with the complexity of the rendered scene .
Therefore , the system must dynamically synchronize the
two streams .

LIMITATIONS

1. Technological limitations :

Although we’ve seen much progress in the

basic enabling technologies , they still primarily prevent the
development of many AR applications . Displays , trackers ,
and AR systems in general need to become more accurate ,
lighter , cheaper , and less power consuming . By describing
problems from our common experiences in building
outdoor AR systems , we hope to impart a sense of the
many areas that still need improvement . The equipment
isn’t nearly as portable as desired .

C.A.S PUTHUPPALLY 20
 21 AUGMENTED REALITY

Tracking in unprepared environments

remains an enormous challenge . Outdoor demonstrations
today have shone good tracking only with significant
restrictions in operating range , often with sensor suits
that are too bulky and expensive for practical use .
Today’s systems generally require extensive research in
AR can reduce these difficulties through improved tracking
in unprepared environments and calibration – free or auto
calibration approaches to minimize set-up requirements .

2. User interface limitations :

We need a better understanding of how to display

data to a user and how the user should interact with data.
Most existing research concentrates on low – level perceptual
issues . However , AR also introduces many high level
tasks , such as the need to identify what information
should be provided , what’s the appropriate representation
for that data , and how the user should make queries and
reports . For example , a user might want to walk down a
street , look in a shop window , and query the inventory
of that shop . To date , few have studied such issues .
However, we expect significant growth in this area because
research AR systems with sufficient capabilities are now
more commonly available.

3. Social acceptance

Another important challenge is social acceptance. Given a

system with idea hardware and an intuitive interface, how can AR
become an accepted part of a user’s everyday life, just like a mobile

C.A.S PUTHUPPALLY 21
 22 AUGMENTED REALITY

phone or a personal digital assistant (PDA)? Through films and

television, many people are familiar with images of simulated AR. In
the movie Terminator-2, the hero uses a augmented reality enabled
headset to display information about criminals. However persuading
a user to wear a system means addressing a number of issues. These
range from fashion (will users wear a system if they feel it detracts
from their appearance?) to privacy concerns ( we can also use the
tracking required for displaying information for monitoring and
recording). To date, little attention has been placed on these
fundamental issues. However, these must be addressed before AR
becomes widely accepted.

APPLICATIONS
1. Medical

Doctors could use augmented Reality as a visualization

and training aid for surgery. It may be possible to collect 3-D
datasets of a patient in real time, using non-invasive sensors like
Magnetic resonance Imaging (MRI), Computed Tomography scans
(CT), or ultrasound imaging. These datasets could then be rendered
and combined in real time with a view of the real patient. In effect,
this would give a doctor “X-ray vision inside of patient. This would
be very useful during minimally invasive surgery, which reduces the
trauma of an operation by using small incisions or no incisions art all.
A problem with minimally invasive techniques is that they reduce
the doctor’s ability to see inside the patient, making surgery more
difficult. AR technology could provide an internal view without the
need for larger incisions.

2. Military

The military has been devising uses for augmented reality

for decades. In fact, the Office of Naval Research has sponsored
some augmented-reality research. And the Defense Advanced

C.A.S PUTHUPPALLY 22
 23 AUGMENTED REALITY

Research Projects Agency (DARPA) has funded an HMD project to

develop a display that can be coupled with a portable information
system. The idea here is that an augmented-reality system could
provide troops with vital information about their surroundings, such
as showing where entrances are on the opposite end of building,
somewhat like X-ray vision. Augmented reality displays could also
highlight troop movements, and give soldiers the ability to move to
where the enemy can’t see them.

3. Instant information

Tourists and students could use these systems to learn

more about a certain historical even. Imagine walking onto a Civil
war battlefield and seeing a re-creation of historical events on a head-
mounted, augmented-reality display. It would immerse you in the
event, and the view would be panoramic.

4. Gaming

How cool would it be to take video games outside? The

game could be projected onto the real wold around you, and you
could, literally, be in int as one of the characters. One Australian
researcher has created a prototype game that combines Quake, a
popular video game, with augmented reality. He put a model of a
university campus into the game’s software. Now, when he uses this
system, the game surrounds him as he walks across campus.

5. Maintenance and repair

C.A.S PUTHUPPALLY 23
 24 AUGMENTED REALITY

Another category of Augmented Reality applications is the

assembly, maintenance, and repair of complex machinery.
Instructions might be easier to understand if they were available, not
as manuals with text and pictures, but rather as 3-D drawings
superimposed upon the actual equipment, showing step-by-step the
tasks that need to be done and how to do them. These superimposed
3-D drawings can be animated, making the directions even more
explicit.

6. Consumer Design

Virtual reality systems are already used for consumer

design. Using perhaps more of a graphics system than virtual reality
when you go to the typical home store wanting to add a new deck to
your house. They will show you a graphical picture of what the deck
will look like. It is conceivable that a future system would allow you
to bring a video tape of your house shot from various viewpoints
from your backyard and in real time it would augment that view to
show the new deck in its finished form attached to your house.

CONCLUSION

Augmented Reality has immense potential as the

technology of tomorrow, a tomorrow where wearable computing is
not a luxury but necessity, a tomorrow where instant access to
information will be taken for granted and a tomorrow where the
concept of a “Real Virtuality” will be used pervasively in almost all

C.A.S PUTHUPPALLY 24
 25 AUGMENTED REALITY

walks of life. Despite the tremendous promise that this concept

holds, Augmented Reality as it is now has to undergo a lot of
refinements before it can be seamlessly integrated into the real world.
Augmented Reality system would be developed only by around
2005.

C.A.S PUTHUPPALLY 25