Augmented Reality Projects in Automotive and Aerospace Industry
Augmented Reality Projects in Automotive and Aerospace Industry
Augmented Reality Projects in Automotive and Aerospace Industry
Holger Regenbrecht*
University of Otago, Dunedin, New Zealand
Gregory Baratoff*
Siemens VDO Automotive AG, Regensburg, Germany
Wilhelm Wilke
DaimlerChrysler AG, Ulm, Germany
Keywords:
H.5.1.b Artificial, augmented, and virtual realities, J.6 Computer-Aided Engineering,
H.5.2.k Prototyping
Abstract
In 2003 the International Symposium on Mixed and Augmented Reality (ISMAR)
was accompanied by a workshop on Potential Industrial Applications (PIA). The
organizers wisely called it "potential" because the real use of augmented reality (AR)
in an industrial context is still in its infancy. Our own experience in this field clearly
supports this viewpoint.
We have been actively involved in the research, development and deployment of AR
systems in the automotive, aviation and astronautics industries for more than five
years and have developed and implemented AR systems in a wide variety of
environments. In this paper we have selected ten AR projects from those we have
managed and implemented in the past to examine the main challenges faced and to
share some of the lessons learned.
We will conclude with some guidelines for successfully deploying AR in an industrial
context.
Page 1 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
applications are still rare. In some cases, AR technology was applied successfully in
certain use cases. For instance in supporting welding processes (Echtler et al., 2003)
or in some training scenarios (see Doerner et al., 2002).
To date there have been two major initiatives for AR innovation. The Mixed Reality
Systems Laboratory in Japan, with its focus set on the development of mixed reality
prototype applications comprising hardware and software, has demonstrated the
potential for the real-world use of AR (see Tamura, Yamamoto, & Katayama, 2001).
The successes of this project lead to the release of the mixed reality platform, a
comprehensive toolkit consisting of display, tracking, and AR software technology.
The other initiative being the German project “ARVIKA” lead by Siemens which
included the majority of the manufacturing industry in the country as well as selected
partners in academia and small and medium enterprises (see Friedrich, 2004). The
focus here was on the application of AR in the fields of design, production, and
servicing.
All these initiatives brought forward various prototypes and demonstrated
applications and have therefore been valuable in progressing the field of AR. The
lessons learned in these projects have had a strong influence on the direction of AR
R&D worldwide.
As part of this international community we have developed prototypes of AR
applications in the realm of automotive and aerospace industry. A majority of our
projects are presented here.
Page 2 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
Figure 1: Servicing projects from left to right: Space station filter change and engine maintenance
Engine Maintenance
For the diagnosis of maintenance and repair tasks, modern cars provide a system
interface (mostly via a plug-like connector). While this interface allows for very fast
and precise analysis of the state of the engine, the accompanying information is still
found on a dedicated PC or on print-outs. Hence, the object to which the diagnosis is
applied (part of the engine) and the resulting data yielded by the diagnosis are
spatially separated. AR has the potential to close this gap, enabling the diagnosis
results to be displayed right in immediate proximity to the engine.
Page 3 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
There are many important questions that must be considered, however: What kind of
information is useful, and should it be represented? What are the technological
alternatives available for solving this? If the data is very complex, as it often is, where
and how do you place the information at the engine?
These were the issues we had to address when implementing a prototype application
for a real Mercedes-Benz (8 cyl. SL) engine. Again, an HMD-based solution
connected to a portable PC (alternatively, a notebook computer) was chosen. The
tracking of the user’s position and orientation was done using a marker-based
approach. In this case markers were attached to a U-shaped object which was placed
into a certain location at the engine. The use of multiple markers at well-defined
positions provided us with reasonably precise tracking.
The following data types were presented (see figure 1 center): (1) maintenance and
repair instructions taken from the garage information system, represented as textual
and pictorial information in space, (2) pre-recorded video instructions in the form of a
“virtual TV set” placed at a fixed position in space, (3) 3D models with predetermined
animated sequences as overlays, and (4) a video/audio link to an expert technician as
an example of remote technical assistance displayed with the TV set approach, too.
While the choice of computer and display technology is straightforward, taking into
account such matters as cost, quality of design and reliability, the information
provision is a bigger challenge due to various key factors: (a) The appropriate
information has to be selected automatically out of the existing information system
(normally text and graphics with references to 3D models), (b) the user interaction has
to be supported in an easy-to-use way, (c) new multi-media content (esp. video and
3D models) has to be created and edited, and finally (d) the multi-media information
has to be brought into a spatial relationship with the object (engine). This entire
authoring process is currently subject to research and development (see Haringer &
Regenbrecht, 2002) and clearly deserves stronger attention.
Tram diagnosis
Trams (street cars) as well as other forms of public transportation are very big objects
with very widely distributed technological components, such as wire connectors,
relays, fuses and electronic units. The interplay of these components is very complex
and the maintenance of the entire system needs skilled technicians and highly
developed diagnosis systems. In addition to the car maintenance example mentioned
above, there are two main challenges: (1) An AR system needs to be wearable
because of the distance between the components and (2) an information system
linkage to the existing diagnosis systems is mandatory, because maintenance here
cannot be done without such a system.
A back-worn wearable AR system incorporating a modified notebook computer and
all peripheral components for a self-containing video see-through system was
developed and tested in a laboratory setup simulating the real tram environment. A
handheld computer with a touch screen display served as the main interaction
interface to the diagnosis system. Both system parts (AR and handheld) are connected
via wireless network technology to a dedicated server. The software running on the
server links the systems together. The augmented view shows arrows pointing at the
current diagnosis object, as well as explanatory texts.
While the system as a whole works very well in our laboratory setup, a real-world
application is very unlikely in the near future. The main shortcomings of the approach
are: (a) The instrumentation of the environment for tracking purposes is unacceptable.
Neither fiducials nor other mounted sensors are practical. (b) The wearable unit
Page 4 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
(approx. 6 kg) is too heavy to use for extended periods. (c) The wireless networking
technology is not robust enough for a real-world environment (incomplete coverage).
The network access point cannot be brought into the tram for every diagnosis due to
practical workflow reasons.
Page 5 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
designed, the update rate and robustness of the tracking system has to be improved,
and the access to CFD and 3D model data has to be improved.
Figure 2: Overlay of CFD data, clockwise: cabin, CFD only, CFD and phantom model, final augmentation
Page 6 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
Cockpit layout
The huge number of displays and controls in an airplane cockpit needs to be designed
very carefully considering many issues including functionality, ergonomics and
safety, to name but a few. It is helpful in the early design phases to be able to discuss
a rough layout of the cockpit in a collaborative setting and this can be achieved using
AR technology. In this instance we provided a metal whiteboard with magnetic
markers to which the cockpit elements were attached as virtual representations. This
allowed for the designers to experiment with different arrangements and to annotate
each of them. Figure 4 shows an inside view of the system, which is described in
detail in Poupyrev et al. (2002).
When lighting conditions are set carefully, the system is reasonably robust and easy to
use due to the tangibility of the interface. The combination of HMD and monitor
display options makes the interface very flexible. The main restrictions lie in the
limitations of available space for the elements on the whiteboard, the 2D arrangement
of elements on the whiteboard and the lack of access to dynamic data. Therefore our
solution is suitable only for very early design discussions and could be extended to
other tasks or phases only with significant further research and development.
Production support
Whoever comes into a large manufacturing or production shop floor will immediately
realize the difficulty in introducing any kind of sensitive equipment to such an
environment. By their very nature these environments are crowded with workers
and/or robots, are noisy, mostly tidy but dirty, have very little wasted space (no room
for extras), and are a scene of endless activity that appears to border on organized
chaos.
Page 7 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
Fuse placement
In the truck-manufacturing example considered here, almost no two trucks are really
identical. The variety of options available to customers is wide-ranging. Because of
the many possible combinations of these options, virtually every truck that leaves the
plant is unique. This can be observed, for instance, with the placement of fuses and
relays for the cockpit of the truck. According to a black and white schematic printed
on paper (figure 5 left), a worker inserts fuses manually into a board. The various
types of fuses are represented by numbers on the schematic (e.g. 15 means a red
colored fuse). Figure 5 (center) shows the working environment. The board is brought
into a fixed position on the table (and is connected to a control unit which sets the
internal parameters of the board’s electronics not considered here) where the operator
manually installs the fuses.
Because of the fixed position of the board, an AR solution without any tracking is
possible. In this example we placed a camera above the board and displayed the
overlay on a computer screen which was already present in the environment. The
overlaid image is shown in figure 5 (right).
This is an example of how a simple solution could be found because of the AR-
friendly nature of the environment. It leads to a very robust, value-adding solution.
This prototype is about to be introduced into the production process.
Figure 5: Fuse plan (left), working place (center), augmented view (right)
Picking
The process of picking can be described as manually seeking for a particular item in
storage according to a list, (see figure 6 left) taking that object and putting it into the
appropriate transport vehicle and bringing the objects to the required location for
processing. Picking occurs in many manufacturing and dispatch businesses. In our
case, car and truck manufacturing, picking is a necessary part of just-in-time assembly
line production and is an entirely manual task. We have identified two main
categories of possible improvement through the use of AR or similar technologies: (1)
error reduction i.e. reducing the number of errors in reading from the list and reading
the labels of the items in storage and (2) savings in the time taken to travel between
different storage locations and to and from the transport cart with the list attached to
Page 8 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
it. After assessing the merits of several AR approaches we eventually decided on the
implementation of an untracked, optical see-through head-worn display. The list
information was brought to the worker in a more robust and failure-minimizing way
(see figure 6 right for a list example).
To allow for comparison and usability testing we implemented two versions of the
system: (1) a Microvision Nomad Expert Technician System and (2) a MicroOptical
viewer connected to a Personal Digital Assistant. Both systems can be applied to the
environment and offer different advantages. While system option #1 allows for an
almost unobstructed view of the environment with a bright information display,
system #2 is more lightweight and is capable of displaying in color (compared to
monochrome red). The main disadvantages of both systems are the psychological
barrier of introducing a device with a laser beam going directly into the eye (system
#1) and partially blocking the view of sight (system #2). Figure 7 shows both systems
with display examples inset.
The main challenges with both system approaches are: (a) the failure-free and instant
provision of the data to be displayed, (b) the form of interaction (changing list items,
changing lists, confirmation and abortion of actions), and (3) the seamless
introduction and integration into existing work processes.
Wiring
As previously mentioned, some complex products offer so many build-to-order
options that the number of possible combinations of these options is almost infinite. In
our application for the truck assembly line, this was also true for the wiring inside the
vehicle. Bundles composed of wires have to be configured and cut according to the
aggregates and components to be controlled or powered. Because of the pace of the
Page 9 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
assembly line this cannot be done in one single step. The paths for the wire bundles
have to be measured at the start of the line. Then in a parallel, semi-automated
process, the wire bundles are put together and finally installed at a later stage on the
assembly line. The manual measuring step can be substituted by an augmented reality
approach where virtual wire bundles are laid into the real geometry (in this case the
girder of the truck) with on-the-fly data transmission to where the wiring bundles are
constructed.
We decided to introduce and compare different hardware technologies involving
combined active and passive optical tracking, HMD, large monitor, projection
technology in combination with a software solution developed especially for this
process. Figure 8 (left) shows a section of a truck chassis with wire bundles and figure
8 (right) the overlay of a wire bundle (in yellow) with mounting points.
While the software was well received by the users, the main shortcomings are: (1) the
tracking systems used require too much instrumentation of the environment and are
much too sensitive for the harsh environment (lighting conditions, vibrations, possible
collisions with objects or persons). (2) The display technology used is neither robust
enough nor ergonomically designed for extended use. A rugged (large) monitor
solution does not give the impression of working “inside” the girder. The HMD
solution is too obtrusive and a projection approach is impossible due to the black
girder surface or too difficult to integrate into the working environment (e.g. Laser).
Figure 8: Girder without (left) and with (right) augmented wire bundles
Although Mizell (2001) applied AR in a slightly different context (wiring on boards
distant from the actual production line) he came to similar conclusions. Eventually the
introduction process was too complex and the available technology too immature to
be entirely successful.
Training
One main advantage of using AR technology for training compared with virtual
reality or other multi-media technologies is the possibility of on-site experiences for
the trainee. Instead of teaching in a more or less abstract environment, the didactic
information can be brought to where the final application will occur. We assume that
one can imagine many possible training scenarios in different fields.
We have decided to present a more “exotic” scenario here to illustrate the possible
scope of AR applications.
Page 10 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
always very challenging and sometimes create dangerous situations for the driver.
This is as true today as it ever was even though today’s cars are equipped with
innovative assistance systems. ABS, ESP, ABC for instance, are abbreviations for
safety systems found in modern car specifications. Even if the driver is familiar with
the underlying principles of these systems, he or she may not always know how these
systems work in extreme situations. Many automotive clubs, car manufacturers, and
public and private agencies therefore offer driver safety programs. In a carefully
controlled environment, the trainee learns to cope with adverse road and weather
conditions and simulated accident situations rarely found in everyday life.
Unfortunately the possibilities are limited nevertheless. For instance, a situation
simulating a child suddenly jumping onto the street can be done only with very
abstract substitutes (e.g. throwing an object onto the street manually). Augmented
reality technology however, offers an almost limitless array of possibilities for
experiencing extreme situations.
We developed a system that the driver and trainer can use within the car while it is
being driven. Both, the trainee and the trainer wear head-mounted displays with video
see-through capabilities (see figure 9 left). The virtually overlaid content is delivered
by notebook computers located on the back seat (see figure 9 right). The current
tracking uses a combination of fiducial and inertial systems. A number of driving
scenarios have been developed together with the customer (the safety program
agency) and implemented using standard virtual reality and animation tools.
From our customer’s perspective this application is clearly seen as a value-adding
addition to the existing programs. The trainees react in a very similar way compared
to real scenes illustrating the effectiveness of the augmented content. The system is
going to be re-engineered for quality improvements and multiple deployments.
Figure 9: Driver and trainer wearing HMD’s (left) and augmented scene (right)
Lessons Learned
We have briefly introduced ten of our augmented reality projects applied in the
automotive and aerospace industries. As shown above there are many technical and
organizational issues to be resolved before one can apply AR in the field. Besides the
application-specific issues addressed, some general guidelines can already be drawn
from the experience we have gained. Even those may not represent empirical
evidence; we think some humble suggestions can be made here that have value for
anyone who wants to apply AR in an industrial context.
Data integration
Firstly, the effort required to incorporate real-world data into the AR application are
often seriously underestimated. Most demonstration scenarios work pretty well with
(manually) pre-configured and specially prepared data sets. When it comes to the first
Page 11 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
real data trial, however, the systems mostly tend to fail. This is usually either because
of the quantity of data needed, or the complexity and historic diversity of the data.
One could probably argue that the data delivery and preparation falls outside the
realm of AR research, but nobody else better understands the data interface than the
team of end-users and AR researchers involved in the project. The early consideration
of real-world data sets is crucial for the successful final deployment of the system.
If the existing data cannot be used right away, a dedicated workflow and tools
(especially authoring tools) need to be developed for successful process integration.
Acceptance
AR technology and research has not yet reached a level of maturity that allows for a
widespread deployment “from scratch”. The initial application fields need to be
identified very carefully with key persons in innovator roles. These persons should
work together with the researcher as close as possible, should know the application
field very well, and should be widely accepted among their colleagues to serve as a
point of multiplication for later dissemination. If one cannot find such a person who
fully accepts the approach and is willing and able to drive it to success, the entire
project will probably fail. Furthermore, the integration of many parties in the early
process of the project (managers, company physician, union representatives etc.) is
laborious but worthwhile. Additionally, usability studies with representative subjects
should be a part of every application project.
Finally, if one has the opportunity to choose between different application scenarios,
the preference should be given to single-user, single-location, and single-task settings.
There is a far greater chance of success if your AR system is setup in an “island”
environment compared with, for example, trying to equip hundreds of workers with
wearable AR systems.
Simplicity
Albert Einstein once said: “Keep it simple, but not simpler!” This is very true for
industrial AR projects. From the researcher’s point of view the best solution found
might not be the one with the highest level of originality or novelty, but imagine if the
users realize later on that there was a simpler, more elegant solution for their problem.
The disappointment will probably put an end to future cooperation. It is advisable
therefore to provide a simple, but accepted solution first and to build on it for
advanced versions at a later time.
The maturity of AR display and tracking technology in particular, seldom allows for
the use of the most advanced and most recent systems available. One has to consider
all alternatives available. Choosing the most accepted and robust one is always better
than offering the latest “bleeding-edge” technology.
Added Value
At the beginning of a planned project, consideration of factors like cost, quality, time
and knowledge gain, helps and often enables the project to get started. Even if the
figures are “educated guesses”, estimates of the value added and sometimes even a
return on investment appraisal are widely expected. Indeed, this is not the core
competence of an AR researcher, but it is to be expected that the researcher will be
concerned with these issues. Preferably one can find experts in the field of industrial
economics to provide appropriate data or estimates.
Page 12 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
Acknowledgements
We would like to thank T. Alt, M. Buhl, M. Dittmann, M. Duthweiler, S. Jacobsen, B.
Kounovsky, W. Krauss, B. Luehr, U. Munzert, C. Ott, M. Wagner, B. Westerburg, H.
Schmidt, C. Schön-Schmidt, R. Specht for their contributions to these projects, and
Graham Copson for his help with the manuscript.
References
Baratoff, G. & Regenbrecht, H. (2004). Developing and Applying AR Technology in
Design, Production, Service, and Training. in Ong, S.K. & Nee, A.Y.C. (eds.) Virtual
and Augmented Reality Applications in Manufacturing. London: Springer, 207-236.
Echtler, F., Sturm, F., Kindermann, K., Klinker, G., Stilla, J., Trilk, J., Najafi, H.
(2003). The Intelligent Welding Gun: Augmented Reality for Experimental Vehicle
Construction. in Ong, S.K. & Nee, A.Y.C. (eds.) Virtual and Augmented Reality
Applications in Manufacturing. London: Springer.
Doerner, R., Geiger, C., Grimm, P., and Haller, M. (2002). Campfire Stories:
Production Process of 3D Computer Graphics Applications. ACM Computer
Graphics, Vol. 36, No. 4, 22-24.
Navab, N. (2004). Developing Killer Apps for Industrial Augmented Reality. IEEE
Computer Graphics and Applications, May/June 2004, 16-20.
Mizell, D. (2001). Boeing’s Wire Bundle Assembly Project. In Barfield and Caudell,
ed., Fundamentals of Wearable Computers and Augmented Reality, Lawrence
Erlbaum & Associates, New Jersey, 447-467.
Poupyrev, I., Tan, D.S., Billinghurst M., Kato, H., Regenbrecht, H., & Tetsutani, N.
(2002). Developing a Generic Augmented-Reality Interface. IEEE Computer, Vol. 35,
Number 3, March 2002, pp. 44-50.
Tamura, H., Yamamoto, H., & Katayama, A. (2001). Mixed Reality: Future Dreams
Seen at the Border between Real and Virtual Worlds. Computer Graphics and
Applications Vol. 21 No. 6, IEEE, 64 – 70.
Page 13 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
IEEE COMPUTER GRAPHICS AND APPLICATIONS FINAL MANUSCRIPT November / December 2005
Footnote:
* This work was done while the authors were at DaimlerChrysler AG, Ulm, Germany
Page 14 of 14 Regenbrecht_etal_AR_in_AutomotiveAerospace.doc
Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.
Alternative Proxies: