Available online at www.sciencedirect.

com

ScienceDirect
Procedia CIRP 61 (2017) 299 – 304

The 24th CIRP Conference on Life Cycle Engineering

AR-guided Product Disassembly for Maintenance and Remanufacturing

M. M. L. Changb, S. K. Onga,b, A. Y. C. Neea,b,*
a
Mechanical Engineering Department, National University of Singapore, 9 Engineering Drive 1, Singapore 117576
b
NUS Graduate School for Integrative Sciences and Engineering, National University of Singapore, 28 Medical Drive, Singapore 117456, Singapore

* Corresponding author. E-mail address: mpeneeyc@nus.edu.sg

Abstract

Product disassembly is an inevitable operation during the two essential modules in realizing a sustainable product life cycle: routine
maintenance and end-of-life strategies particularly remanufacturing. In this paper, an AR-guided product disassembly (ARDIS) framework
with an automatic content generation module is proposed to improve the efficiency of the disassembly process performed by human operator
without expert intervention. Through intuitive AR-guidance in disassembling a product, the quality of maintenance services and effective core-
retrieval during remanufacturing can be improved. A disassembly sequence table is first generated based on product ontology. Next, an
automatic content generation domain will link the generated sequence with appropriate virtual information based on the taxonomy of visual
cues. The final virtual content rendered on the real scene serves as straightforward step-by-step guide to the human operator. A case study of
Nespresso machine is presented to demonstrate the preliminary workflow of the proposed framework.

©©2017
2017Published by Elsevier
The Authors. B.V. This
Published byisElsevier
an open B.V.
access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the scientific committee of the 24th CIRP Conference on Life Cycle Engineering.
Peer-review under responsibility of the scientific committee of the 24th CIRP Conference on Life Cycle Engineering
Keywords: Product life cycle; Disassembly; AR guidance; Maintenance; Remanufacturing

1. Introduction Despite active research in disassembly planning, there is a

lack of an intuitive platform to translate the resulting
Over recent decades, tremendous growth in consumerism disassembly guidance information effectively to the human
as well as increasing environmental awareness have led to operators who will eventually dismantle the products. In most
stricter recycling regulations and an urgent demand for cases, these human operators rely on hardcopy manuals or past
remanufacturing to reduce resources depletion. As shown in experience to perform the disassembly tasks. This is non-ideal
Fig. 1, two important modules, namely, maintenance services because the operators have to shift their attention from the task
to prolong product life cycle and end-of-life (EOL) strategies, on hand to the instruction manuals, which could cause the
such as remanufacturing to retrieve usable parts before final disassembly process to be error-prone and time-consuming.
disposal of the products, have been introduced to achieve a
closed-loop product “industrial” life cycle [1]. In both Notably, an evaluation study by Havard et al. that compares
modules, disassembly operation is essential. Unfortunately, maintenance efficiency across different media support, such as
disassembly operation is often regarded as a non-value added paper maintenance manuals, video instructions, augmented
operation to the companies considering that it requires reality (AR) tablets and AR Smart Glasses, has concluded that
additional effort and cost. Hence, disassembly operation has to the AR technology can reduce maintenance time with
be optimized such that it can be cost-efficient [2]. Numerous improved quality as compared to using hardcopy manual or
algorithms, such as optimization techniques [3,4], intelligent video instructions [8]. Drawing inspiration from AR
genetic algorithm [5] and CAD-based disassembly planning applications in maintenance, this paper thus proposes an AR-
[6,7], have been proposed to increase the overall efficiency of guided product disassembly (ARDIS) framework with an
the disassembly process. automatic content generation module to convey disassembly
guidance information intuitively to the human operators.

2212-8271 © 2017 Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the scientific committee of the 24th CIRP Conference on Life Cycle Engineering
doi:10.1016/j.procir.2016.11.194
300 M.M.L. Chang et al. / Procedia CIRP 61 (2017) 299 – 304

For remanufacturing, maintenance and repair, partial

disassembly is conducted since only certain parts require
servicing or retrieval. Generally, non-destructive disassembly
is preferred especially in the latter because destructive
disassembly is an irreversible operation.

Over the years, research development of disassembly

sequence generation has advanced from general disassembly
process planning to optimal disassembly sequence generation
Fig. 1. Maintenance and EOL Strategies: Close-loop Product Life Cycle [1] from all possible solutions using optimization techniques. The
most common disassembly sequence generation methods are
2. Related Work based on graphical representation such as AND/OR graph
[12], process graph [13], and precedence constraints graph
2.1. Augmented Reality [14] that reveal all feasible sequences and form the search
basis for an optimal sequence. Meanwhile, Moore et al. [15]
Augmented reality (AR) is an emerging human computer proposes Disassembly Petri Net (DPN) methods that modeled
interaction (HCI) technology that renders virtual information uncertainty of product conditions using transition probability
on a real scene. An AR system is formally defined as an for product recycling or remanufacturing. Other methods such
application that fulfills the following three properties, namely, as dynamic disassembly sequence planning based on product
(a) able to blend real and virtual content in a real environment, information model [16] and optimization techniques [3,4]
(b) is real time and interactive, and (c) can register virtual have also been reported recently. In this paper, an AR
content in 3D environment [9]. As shown in Fig. 2, a typical platform is proposed to effectively convey the disassembly
AR application consists of five modules, namely, registration, sequence information generated from these algorithms to the
tracking, rendering, interaction, and content generation. In human operators intuitively.
short, computer-generated information, such as annotations,
graphics and 3D models, should be rendered and registered on 2.3. Related AR applications
the real scene with accurate tracking and alignment, followed
by user-friendly interaction modes, such as gesture-based There is a long history of applying advanced HCI
input, speech input or with the help of external input devices, techniques in the industry to enhance manufacturing
such as data gloves, ray casting using mouse, etc. Lastly, operations. As early as the 90’s, virtual reality (VR), which is
relevant content in response to specific request or task should more immersive as compared to AR, has been applied in
be generated and displayed to the users [10]. different manufacturing domains, such as prototype design,
simulation and virtual machining [17]. Since AR enables the
overlay of virtual components with the real life scene, it is
more favorable in facilitating real life manufacturing tasks
Registration Tracking Rendering and is often applied in virtual assembly design or training.
According to a recent survey paper, reported research work in
AR for assembly till date can be categorized into three areas
Content depending on the goals of proposed system, namely, (a)
Generation Assembly guidance in AR environment, (b) Assembly design
and planning in AR environment, and (c) Assembly training
in AR environment [18].
Interaction
In contrast, there is less research work reported in AR
Fig. 2. Overview of an AR System
applications for disassembly and this is intriguing because
disassembly is often required in many stages of a product life
2.2. Product Disassembly cycle, such as maintenance, repair, and remanufacturing.
Some early work on augmented disassembly guidance
Disassembly is formally defined as the systematic includes a disassembly operation support system with motion
separation of parts from a system. A system can refer to monitoring of a human operator using 3D position sensors for
anything, such as machinery, equipment or even space the disassembly of photocopy machines [19]. For disassembly
stations. In particular, this paper will focus on product training, Farkhatdinov and Ryu developed an AR educational
disassembly. Products refer to physical objects that are system for automotive engineering that guides users to
created and assembled to perform specific tasks, such a disassemble mechanical parts of a transmission through
ballpoint pen, photocopy machine or a car. Depending on the animating the 3D virtual models. Notably, both virtual models
task requirements, product disassembly can be divided into and disassembly sequences were modeled and defined
different types, namely, complete or partial, destructive or manually by an expert in this system [20]. In the research
non-destructive [11]. For recycling, complete disassembly is reported in this paper, the disassembly sequence is generated
often required since the product has reached its end of life. based on product contact and translation functions and
 M.M.L. Chang et al. / Procedia CIRP 61 (2017) 299 – 304 301

automatically linked to the corresponding virtual information have one set of C and T functions that indicate the presence
for AR guidance. (1) or absence (0) of contact and collision-free path in six
directions. To construct the assembly sequence table (AST), C
3. ARDIS Framework and T functions are represented in a truth table and Boolean
algebra principles, such as “and” (operator “ š ”), “or”
The proposed AR-guided product disassembly (ARDIS) (operator “ › ”), are applied to verify the feasibilities of
framework consists of three major modules, namely, subassemblies in each level. In this example, the product will
disassembly sequence planning, automatic content generation have a 4-level AST from individual parts at level I to the full
and AR interface as shown in Fig. 3. assembly with all parts at level IV. Assembly states in the first
and last levels are known as primary states and will remain
unchanged regardless of the constraints for the same set of

CandT Product parts. To obtain all feasible assembly states from level II and
Functions Information upwards, a Cartesian product of the subassemblies from
previous levels and the part being added has to be verified.
Disassembly
Sequence

Visualcue
For instance, adding part#3 to subassembly {1, 2} will results
in a set of pairs [{1, 3}, {2, 3}] due to the Cartesian product
AR
 operation {1, 2} x {3}. The feasibility of assembling {1, 2}
Visualization and {3} together to achieve the subassembly of {1, 2, 3} is
Animations andGuidance
Optimal
Annotations
valid if both RTC and RTT are true (1) according to the
Sequence following rules:
Disassembly Planning Content Generation AR Interface
RTC TC1 ›TC2 ›TC3 ›TC4 ›TC5 ›TC6
Fig. 3. System Architecture of ARDIS
RTT TT1 ›TT2 ›TT3 ›TT4 ›TT5 ›TT6
3.1. Product Disassembly Sequence Planning
Mathematically, the resultant truth values of C functions
The algorithm for generating disassembly sequences is (TC) and T functions (TT) are obtained based on
adapted from the approach described by Gottipolu and Ghosh. TCi Ci (1,3)›Ci (2, 3) and TTi Ti (1,3)šTi (2,3) for
In particular, all possible pairs for all parts in a product will i 1 6 respectively as shown in Table 1 and Table 2.
have their respective contact (C) and translation (T) functions,
which consist of six values representing three axes (x, y, z) in
Table 1. RTC of adding {3} to {1, 2}
two directions (+, -) of a tri-orthogonal Cartesian coordinate
system. In particular, directions 1, 2 and 3 represent the Pair C1 C2 C3 C4 C5 C6
positive x, y, and z axes whereas directions 4, 5 and 6 indicate {1, 3} 0 0 0 0 1 0
the negative x, y, and z axes [21]. For instance, the C function {2, 3} 0 0 0 0 0 0
of a pair (1, 3) is [0, 0, 0, 0, 1, 0] since part#3 has only one TC 0 0 0 0 1 0
contact with part#1 in direction 5 (y-). Similarly, the T
function of a pair (1, 3) is [1, 1, 1, 1, 0, 1] since part#3 has Table 2. RTT of adding {3} to {1, 2}
collision-free paths with respect to part#1 in all directions
except 5 (y-). Pair T1 T2 T3 T4 T5 T6
{1, 3} 1 1 1 1 0 1
{2, 3} 1 1 1 1 0 1
4 TT 1 1 1 1 0 1

+Y Since both RTC and RTT are valid, subassembly {1, 2, 3}

3 +Z
2 is feasible and can be obtained by adding part#3 to {1, 2} in
3
-X any of the five directions as specified in TT. By repeating the
1 same logic on all subsequent states, all feasible assembly
4 +X
sequences can be identified.
6 -Z
5
-Y Notably, any secondary state that is not a constituent
2 element of a subsequent assembly task is considered as a dead
state and any task which constituent assembly states are
1 absent in the previous states is considered as a dead task [21].
Therefore, they should not be used to generate any further
Fig. 4. A Simple Product in Three-dimensional Space
higher order subassemblies. The final assembly sequence
table after eliminating all the dead states and tasks is
As shown in Fig. 4, a simple product with four parts will presented in Table 3.
have 12 possible pairs based on permutation. Each pair will
302 M.M.L. Chang et al. / Procedia CIRP 61 (2017) 299 – 304

Lastly, the AST is converted into a disassembly sequence AR content automatically depending on the representation
table (DST) based on additional constraints, such as removing styles and computational complexity.
infeasible states and tasks due to irreversible assembly
operations, such as welding. Assuming that part#3 and part#4
are inseparable, the resulting DST is as shown in Table 4.

Table 3. Assembly Sequence Table

Level States Tasks

I {1}, {2}, {3}, {4} -
II {1, 2} {(1), (2)}
{1, 3} {(1), (3)}
{3, 4} {(3), (4)}
III {1, 2, 3} {(1, 2), (3)}, {(1, 3), (2)}
IV {1, 2, 3, 4} {(1, 2, 3), (4)}, {(1, 2), (3, 4)}

Table 4. Disassembly Sequence Table

Level States Tasks

IV {1, 2, 3, 4} {(1, 2), (3, 4)}
III - -
II {1, 2} {(1), (2)}
I {1}, {2}, {3, 4} - Fig. 6. Taxonomy of Visual Cue

After the retrieval target either for maintenance or As shown in Fig. 6, the simplest visual cue refers to text-
remanufacturing has been specified, the shortest disassembly based instructions, such as label, messages or prompts
sequence can be obtained from the resulting DST. For followed by 2D images or graphics, such as arrow to indicate
example, a sequence of [(3, 4); (2); (1)] is obtained to retrieve disassembly direction. The most complex yet intuitive visual
part#1 from the DST. Fig. 5 summarizes the overall flowchart cue includes 3D models of the parts with appropriate
of the disassembly sequence planning module described in animation. All types of visual cues are rendered on the scene
this section. in real time using OpenGL and GLSL with libraries, such as
FreeType for text rendering and ASSIMP for 3D model
loading.
Productinformation

3.3. AR Guidance
BooleanAlgebra
AssemblySequence Given the disassembly sequence and product information,
PrinciplesonCand
Table
Tfunctions a series of disassembly tasks can be identified, which consists
of an action and direction, and linked to the corresponding
Disassembly Disassembly visual cues, such as arrows or 3D models for AR visualization
SequenceTable Constraints and guidance. The methodology used in generating sequence-
specific disassembly task is illustrated in Fig. 7.
SpecifiedRetrievalTarget OptimalDisassembly
Sequence
Fig. 5. Overview of Disassembly Sequence Planning Module

3.2. Automatic Content Generation

AR applications involve different kinds of augmentation

including visual augmentation, audio and haptic feedback, etc.
In the proposed framework, visual augmentation is applied.
Generally, there are two types of visual information, namely,
static information (general and prompts) and dynamic
information (guiding scene and instructions) [22].
Particularly, static information can be pre-defined beforehand
while dynamic information requires constant update during
the disassembly process according to the disassembly
sequence generated earlier. As shown in Fig. 6, the taxonomy
of visual information is proposed to generate corresponding Fig. 7. Generation of Disassembly Task
 M.M.L. Chang et al. / Procedia CIRP 61 (2017) 299 – 304 303

instructions and animated arrows. As shown in Fig. 9, the

Based on the shortest disassembly sequence, a part to be operator without prior knowledge of the product was able to
handled or removed as well as the corresponding actions and retrieve part#6 successfully in three steps.
directions are translated into AR instructions in the form of
<action>_<direction>_<part>, e.g., Remove_Leftward_PartX.
For example, a disassembly sequence of [(3, 4); (2); (1)] is Table 5. Simplified Disassembly Sequence Table To Retrieve Part 6
obtained from the DST in Table 4 in order to retrieve part#1. Level States Tasks
Based on this sequence and truth values of T functions for the X {1, 2, 3, 4, 5, 6, 7, 8, 9, {(1, 2, 3, 4, 5, 6, 7, 10),
disassembly direction, the resulting AR instructions are: 10} (8, 9)}
[Remove_Upward_Part3-4] and [Remove_Forward_Part2]. VIII {1, 2, 3, 4, 5, 6, 7, 10} {(1, 2, 3, 4, 5, 10), (6, 7)}
Once the disassembly direction has been identified, the II {6, 7} {(6), (7)}
corresponding 2D visual cue, such as an upward-pointing
arrow, can be retrieved and linked to the sequence for AR 5. Discussion
guidance.
An AR-based platform linking a final disassembly
4. Demonstration sequence to AR guidance information, such as disassembly
text instructions and 2D arrows is achieved. The disassembly
The general workflow of the proposed AR-guided sequence generation algorithm described in this paper is
disassembly framework is illustrated using a simple 10-parts inspired by a simple yet efficient assembly sequence table
coffee machine as shown in Fig. 8. The disassembly sequence method that considers both local constraints and global
planning module is implemented in Matlab and the AR feasibility of different parts in a product. The optimal
guidance is implemented using OpenCV with ARUCO disassembly sequence is assumed to be the shortest sequence
module for marker-based tracking and OpenGL with GLSL from the table. However, this might not be true considering
for rendering. that some tasks are more difficult to perform and may require
greater force. Hence, weights factors that indicate varying
levels of difficulty can be assigned to each task. To obtain
these weight values, additional information, such as part
conditions (rusty, damaged or dangerous) and associated
complexity metrics are required. Advanced searching
algorithms or optimization techniques that have been reported
in disassembly sequence planning literature can be used to
obtain a final disassembly sequence with minimum cost.

Currently, the proposed framework is a one-way interface

that conveys step-by-step disassembly guidance to the
operators when prompted. To further enhance AR-guided
disassembly planning, real time feedback, such as users’ input
is helpful because they are aware of the exact part conditions
during the time of disassembly. In future, advanced
disassembly sequence planning algorithm that takes into
account uncertain EOL conditions of the products and
dynamic feedback from the human operator during AR-
guidance will be explored.
Fig. 8. A 10-parts Product for Case Study
6. Conclusion
First of all, geometric and mobility constraints of the
Nespresso machine, represented in the form of contact and A preliminary AR-guided product disassembly (ARDIS)
translational functions, serve as the input. As described framework has been described in this paper. The main
earlier, all the feasible disassembly sequences can be obtained contribution of proposed framework is automatic content
based on a series of Boolean operations with additional generation that translates a disassembly sequence into AR-
constraints and are stored in the form of a disassembly based instructions for the human operators. An efficient and
sequence table (DST). Next, the used capsule container is error-free disassembly process will greatly benefit both the
identified as the target for maintenance or remanufacturing. industry and the operators considering that it is an
Finally, the shortest disassembly sequence of [(8,9); (6,7); (6)] unavoidable operation in a product life cycle. Overall, the
to retrieve part#6 selectively can be obtained as presented in proposed framework is useful for full or/and selective
Table 5. disassembly, which are vital in both maintenance and
remanufacturing.
The shortest disassembly sequence is linked to the
corresponding AR content, including step-by-step text
304 M.M.L. Chang et al. / Procedia CIRP 61 (2017) 299 – 304

Fig. 9. Step-by-step AR Guidance

References [12] Mello L, Sanderson A. Automatic Generation of Mechanical Assembly

Sequences. The Robotics Institute, Carnegie Mellon University
[1] Ulrich KT, Eppinger SD. Product design and development, McGraw-Hill, Pittsburgh, Pennsylvania, 1988.
2011. [13] Lambert AJD. Optimal disassembly of complex products. International
[2] Ilgin MA, Gupta SM. Remanufacturing modeling and analysis, CRC Journal of Production Research, 35, 2509-2524, 1997.
Press, 2011. [14] Wakamatsu H, Matshuishi M, Morinaga E, Arai E. Disassembly Support
[3] Xia K, Gao L, Li W, Chao KM. Disassembly sequence planning using a System for Used Products Considering Destruction of Their Parts. In:
Simplified Teaching–Learning-Based Optimization algorithm. Advanced Shirase K, Aoyagi S. (eds.) Service Robotics and Mechatronics: Selected
Engineering Informatics, 28, 518-527, 2014. Papers of the International Conference on Machine Automation
[4] Pornsing C, Watanasungsuit A. Discrete particle swarm optimization for ICMA2008. London: Springer London, 2010.
disassembly sequence planning. Management of Innovation and [15] Moore KE, Gungor A, Gupta SM. Petri net approach to disassembly
Technology (ICMIT), 2014 IEEE International Conference on, 23-25 process planning for products with complex AND/OR precedence
Sept. 2014 2014. 480-485, 2014. relationships. European Journal of Operational Research, 135, 428-449,
[5] Wang H, Xiang D, Duan G. A Genetic Algorithm for Product 2001.
Disassembly Sequence Planning. Engineering of Intelligent Systems, [16] Zhu B, Sarigecili MI, Roy U. Disassembly information model
2006 IEEE International Conference on, 1-5, 2006. incorporating dynamic capabilities for disassembly sequence generation.
[6] Hu D, Hu Y, Li C. Mechanical Product Disassembly Sequence and Path Robotics and Computer-Integrated Manufacturing, 29, 396-409, 2013.
Planning Based on Knowledge and Geometric Reasoning. The [17] Ong SK, Nee AYC. A Brief Introduction of VR and AR Applications in
International Journal of Advanced Manufacturing Technology, 19, 688- Manufacturing. In: Ong SK, Nee AYC. (eds.) Virtual and Augmented
696, 2002. Reality Applications in Manufacturing. London: Springer London, 2004.
[7] Briceno JJ, Pochiraju K. Automatic Disassembly Plan Generation from [18] Wang X, Ong SK, Nee AYC. A comprehensive survey of augmented
CAD Assembly Models. Assembly and Manufacturing, 2007. ISAM '07. reality assembly research. Advances in Manufacturing, 4, 1-22, 2016.
IEEE International Symposium on, 22-25 July 2007, 64-69. 2007. [19] Takata S, Isobe H, Fujii H. Disassembly Operation Support System with
[8] Havard V, Baudry D, Savatier X, Jeanne B, Louis A, Mazari B. Motion Monitoring of a Human Operator. CIRP Annals - Manufacturing
Augmented Industrial Maintenance (AIM): A case study for evaluating Technology, 50, 305-308, 2001.
and comparing with paper and video media supports, In: De Paolis LT, [20] Farkhatdinov I, Ryu JH. Development of educational system for
Mongelli A. (eds.) Augmented Reality, Virtual Reality, and Computer automotive engineering based on augmented reality. International
Graphics: Third International Conference, AVR 2016, Lecce, Italy, June conference on engineering education and research, 2009.
15-18, 2016. Proceedings, Part I. Cham: Springer International Publishing [21] Gottipolu RB, Ghosh K. A simplified and efficient representation for
[9] Azuma RT. A Survey of Augmented Reality. In Presence: Teleoperators evaluation and selection of assembly sequences. Computers in Industry,
and Virtual Environments, 6, 355-385, 1997. 50, 251-264, 2003.
[10] Baratoff G, Regenbrecht H. Virtual and Augmented Reality Applications [22] Chang MML, Ong SK, Nee AYC. Automatic Information Positioning
in Manufacturing. In: London, S. (ed.) Part II: 12 Developing and Scheme in AR-assisted Maintenance Based on Visual Saliency. In: De
Applying AR Technology in Design, Production, Service and Training, Paolis TL, Mongelli A. (eds.) Augmented Reality, Virtual Reality, and
2004. Computer Graphics: Third International Conference, AVR 2016, Lecce,
[11] Gungor A, Gupta SM. Disassembly sequence planning for products with Italy, June 15-18, 2016. Proceedings, Part I. Cham: Springer International
defective parts in product recovery. Computers & Industrial Engineering, Publishing
35, 161-164, 1998.