See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/327897040

DigiClay: An Interactive Installation for Virtual Pottery using Motion Sensing

Technology

Conference Paper · February 2018

DOI: 10.1145/3198910.3234659

CITATIONS READS
10 406

4 authors, including:

Zihan Gao Huiqiang Wang

Communication University of China Chinese Academy of Medical Sciences
13 PUBLICATIONS   27 CITATIONS    144 PUBLICATIONS   561 CITATIONS   

SEE PROFILE SEE PROFILE

Guangsheng Feng
Harbin Engineering University
73 PUBLICATIONS   282 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

National Natural Science Foundation of China View project

VR Fidelity View project

All content following this page was uploaded by Zihan Gao on 31 October 2018.

The user has requested enhancement of the downloaded file.

 DigiClay: An Interactive Installation for Virtual Pottery using
Motion Sensing Technology
Zihan Gao Jingwen Li∗
Department of Computer Science Department of Computer Science
Harbin Engineering University Harbin Engineering University
Harbin, China Harbin, China
gao_zihan@126.com ljw717297@163.com

Huiqiang Wang Guangsheng Feng

Department of Computer Science Department of Computer Science
Harbin Engineering University Harbin Engineering University
Harbin, China Harbin, China
wanghuiqiang@hrbeu.edu.cn ica@hrbeu.edu.cn

ABSTRACT Technology. In ICVR 2018: 2018 4th International Conference on Virtual Re-
In this paper we present an interactive installation for virtual pot- ality, February 24–26, 2018, Hong Kong, Hong Kong. ACM, New York, NY,
USA, 7 pages. https://doi.org/10.1145/3198910.3234659
tery, which adopts intuitive in-air hand interactions with the help
of motion sensing technology. A Microsoft Kinect depth camera is
1 INTRODUCTION
used for motion capture in this installation, and a real-time defor-
mation algorithm is applied to the mesh in Unity3D. Once a virtual As a symbol of civilization, pottery has played an important role
pottery work is completed, it can be shared to a social network throughout the development of heterogeneous cultures. Nowadays,
automatically. In this prototype, an automated publishing process pottery creation is regarded not only as manufacture process but
will upload the screenshot of a user’s work to Facebook, and the also as an artistic activity. However, the traditional workflow of
mesh data will be saved for future use, such as 3D printing. User pottery creation is not easily accessible to individuals in everyday
behaviors are observed during an exhibition, and some issues are life, especially to amateurs, which requires certain kind of material
identified from the observation for further improvement. Our work, (clay) and specialized equipments. With emerging technologies
which eliminates the gap between the traditional ceramic art and such as motion sensing technology, a novel way to create pottery
everyday life, offers a way to the common audience to express becomes possible.
themselves as artists. Our prototype can be further developed as Technologies that utilize the user motion data from a variety
commercial products in various fields, such as arts education, home of sensors have been widely adopted in interactive installations.
entertainment and public space exhibitions. Foot-turistic multimedia [2], an interactive installation, has used
RFID technology along with motion detection algorithms. Through
CCS CONCEPTS the interaction with the shop’s products, the retail experience is
enhanced in an entertainment way. Another interactive installa-
• Human-centered computing → User interface design; • Com-
tion [4], which is built in the context of museum visiting, provides
puting methodologies → Virtual reality; • Software and its
users with personalized experience based on individual movement
engineering → Virtual worlds training simulations;
data collected from computer vision technology. BilliArT [9] is an
installation based on billiard game, which uses OptiTrack motion
KEYWORDS capture system to track the movement of the balls. Specific visual
virtual pottery, interactive installation, Kinect, mesh deformation and auditory textures are applied to augment the billiard game, in
ACM Reference Format: which aesthetic motivation is emphasized as well. CoBoard flowers
Zihan Gao, Jingwen Li, Huiqiang Wang, and Guangsheng Feng. 2018. Digi- [5] uses face recognition technology to display various flower visu-
Clay: An Interactive Installation for Virtual Pottery using Motion Sensing alization with adaptive music based on individual age and gender.
The motion of user, specifically the face distance from the screen, is
∗ Corresponding author. detected by the camera and used as a parameter in the visualization.
There are several projects related to virtual pottery as well.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed Handy Potter [6–8] uses mid-air gestures for virtual pottery. The
for profit or commercial advantage and that copies bear this notice and the full citation point-cloud data from depth sensors has been used, which allows
on the first page. Copyrights for components of this work owned by others than ACM not only hands but also physical and virtual tools can be used to
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a deform the mesh. Nevertheless, the functionalities add complexity
fee. Request permissions from permissions@acm.org. to user interactions, which requires more training time. In Virtual
ICVR 2018, February 24–26, 2018, Hong Kong, Hong Kong Pottery [3], an asymmetric way of pottery is explored, which al-
© 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-6408-9/18/02. . . $15.00 lows more freedom to users. The work also converts the shape of
https://doi.org/10.1145/3198910.3234659 pottery to diverse sound, making the installation become a virtual
ICVR 2018, February 24–26, 2018, Hong Kong, Hong Kong Z. Gao et al.

musical instrument. Compared with these references above, inter- 3 TECHNICAL SOLUTIONS
active installation DigiClay is unique in following aspects: (1) In The technical realization is based on the game engine Unity3D1 ,
terms of technical method, we use user skeleton based technique which provides us a handy development environment, with a great
instead of point-cloud based method, which improves the accuracy number of built-in components and third-party plugins. We use
of user hand gesture input. (2) The user interaction is optimized for NGUI2 for our UI system and Zigfu3 as the middleware for the data
simplicity and become more realistic to the actual hand movement from Kinect.
in real pottery making process, which makes DigiClay more user- Figure 3 depicts the data flow in DigiClay. The hand motion
friendly and closely related to traditional pottery art. (3) We grant from a user is captured by the Kinect camera and the raw data is
the user the ability to show off their artworks by developing an parsed by OpenNI and then passed into Unity3D through Zigfu.
automated social network sharing pipeline: Once a pottery work is In Unity3D, we can access the skeleton data from OpenNI, which
done, the screenshot will be taken and uploaded automatically to a will be used in UI element position mapping and mesh deformation.
social network platform for everyone in the world to see. Hence, Whenever the automated publishing process is triggered by the
the effect and influence from DigiClay is prolonged and magnified user, a screenshot will be captured and uploaded to Facebook, and
in this way. the mesh data will be saved as well.
The paper is structured as follows: In Section 2 we focus on
the the concept and design of the installation, and the technical 3.1 UI Element Position Mapping
solutions will be discussed in section 3. In section 4 the evaluation
will be examined by user studies, and in the final section we draw The first challenge needs to be solved is to map the skeleton data
our conclusions and discuss the future work of our research. to UI element position, so a user could clearly see where her hands
are on the screen. As mentioned earlier, Zigfu, which is a Kinect
2 INSTALLATION DESIGN wrapper for Unity3D, has been used to get the data from Kinect
sensor. From Zigfu, a list of node data could be collected, which
2.1 Concept and Setup contains the information of user joints including positions and rota-
The aura of a work of art is devalued by mechanical reproduction tions. Since both hands are needed to interact with the installation,
[1]. The concept aura, according to Benjamin, means the uniqueness the node information of Left Hand and Right Hand are needed.
within a work of art. Overwhelmed by superfluous products in the Moreover, as the height of users is varied, the node data of Torso
consumer society, people tend to take design features for granted is required as well, which could be used to as a reference node
and have ignored the beauty of creation for a long time. In this for calculating the relative hand positions. Let pR , pL , pT be the
project, we aim to let the user feel the aura (i.e. uniqueness) within right hand, left hand and torso position data from Kinect camera
her own artwork through the process of creating a virtual pottery, respectively, the hand positions can be defined as:
which help the user to regain the sensitivity of beauty.
The environment setup for DigiClay is shown in Figure 1. A p0 = pR − pT (1)
Kinect camera is installed in front of the user and is about 0.7m
above the ground. The vertical angle of Kinect camera should be
adjusted manually to get an ideal view. A marker is set on the p1 = pL − pT (2)
ground to indicate where the player should stand. A white curtain
is set behind the user to reduce the background disturbance (such where p0 , p1 ∈ R3 and they are position vectors describing the
as people passing by), which could improve the accuracy of user right and left hand positions respectively. However, since these
detection. data are in 3D space, a conversion is needed for these 3D vectors,
mapping them to the screen 2D coordination. The position of hand
2.2 User Flow nodes can be mapped to the UI hand cursors using inverse linear
When a user enters the stage and is recognized by the Kinect camera, interpolation. Let pi =< x i , yi , zi >, i = 0, 1 and we have the cursor
there will be a zoom-in effect on the screen, enabling two hand position:
cursors which will follow the hand movement of the user (See
Figure 2). Similar to the pottery making process, the user can use pc i =< Sw ∗ f I nver se Ler p (kl ef t , kr iдht , x i ),
the hands to deform the 3D cylinder mesh on the screen: (1) the Sh ∗ f I nver se Ler p (kbot tom , ktop , yi ), (3)
vertical position of hands will determine which part of the mesh 0>
will be influenced, and (2) the horizontal distance between hands
will decide whether the influenced part of mesh will extend or where kl ef t , kr iдht , ktop , kbot tom are parameters to define a
shrink. In this intuitive way, the user can relish the creating process rectangle zone around the torso node. Sw and Sh are the screen
in an entertaining way with ease. width and height respectively. The function f I nver se Ler p (a, b, v)
Once the user feels pleased with what has been made, she could calculates the linear parameter that produces the interpolant v
raise right hand to touch the finish icon on the top-right corner of within the range [a, b].
the screen. With a particle system triggered and camera zoomed
out, a blossom effect can be seen on the screen. The system will 1 Unity3D. http://unity3d.com
upload the screenshot automatically, and the user can see her work 2 NGUI. http://www.tasharen.com
3 Zigfu.
immediately on Facebook. http://developkinect.com/groups/zigfu
DigiClay: An Interactive Installation for Virtual Pottery ICVR 2018, February 24–26, 2018, Hong Kong, Hong Kong

Screen

White Curtain

2.5m
2.5m
User Kinect

0.7m

2.0m

2.5m Marker
3.0m

Figure 1: The environment setup for the installation.

Figure 2: The UI in DigiClay (Large Picture) and user interaction (Small Picture).

3.2 Real-Time Mesh Deformation

1
As mentioned in section 2.2, the hand data will be used in order to ȳ = (y0 + y1 ) (4)
calculate the mesh deformation. First, we need to get the average 2
Also, the horizontal distance between two hands:
vertical position from two hands:

dhands = |x 0 − x 1 | (5)
ICVR 2018, February 24–26, 2018, Hong Kong, Hong Kong Z. Gao et al.

Hand Motion Kinect Camera

Zigfu OpenNI

Unity3D

NGUI Mesh Deformation

Screenshot Processing

Facebook

Figure 3: The data flow in DigiClay.

Let V = {v 1 , . . . , vn } be the set of vertices which consist the (1) the function should be monotonically decreasing on [0, 1], (2)
mesh, and the original position of ith vertex vi in the set V can f F allof f (0) ≤ 1, and (3) f F allof f (1) = 0. Table 1 demonstrates
be denoted as pvi =< xvi , yvi , zvi >. The updated position pv′ i in two examples of possible falloff functions, which demonstrates
every frame is computed as: the graphs and effects for different functions. From the compari-
son it can be clearly seen that the effects from different functions:
|yvi − ȳ| while the linear function creates a rigid deformation effect, the
pv′ i = pvi + kst r enдt h · ûvi · s · f F allof f ( ) (6) bell-shape function which is similar to Gaussian function offers a
kr anдe
much smoother deformation.
where kst r enдt h and kr anдe are predefined values, controlling
the deformation strength and influence range respectively. ûvi is 3.3 Automated Social Network Publishing
the unit vector presenting the radius direction of vi through its
In our project we use Facebook as the publishing platform, which
horizontal section:
provides multi-platform support for developers including Unity3D.
< xvi , 0, zvi > To use the service, we have registered as a Facebook developer and
ûvi = (7) have created a new app. Whenever a user finishes a pottery and
∥ < xvi , 0, zvi > ∥
The value of s, which presents the sign, could be 1 or -1 depending touches the finish button, an automated process will start: (1) the
on the relation between dhands and kdist ance : on-screen UI elements will be hidden, (2) a screenshot will be taken
by the system, and (3) the screenshot will be uploaded to social
network to share the work just done by the user (See Figure 4).
(
1 if dhands ≥ kdist ance ,
s= (8)
−1 otherwise. 4 EVALUATION
The falloff function maps the vertical distance between a vertex DigiClay was exhibited at the interactive work show MEDUIM:
and the average hand vertical position (AHVP) to a value between PLAY in Centre for Digital Media, Vancouver on April 10th, 2014.
0 and 1. The further a vertex is from the AHVP, the value should We used a large canvas as the screen, and two laptops are used
be closer to 0, which means the vertex is less influenced. As demon- in the exhibition. Connected to the projector, Laptop #1 works as
strated in (6), |yvi − ȳ|/kr anдe will be the argument of the function. the main computer, which runs the DigiClay program and uploads
Hence, we have: images to Facebook; Laptop #2 works as a review window, which
displays all the images from Facebook, allowing users to see what
f F allof f : [0, ∞) → [0, 1] (9) they have created with DigiClay and to leave comments under
There are a variety of choices for the falloff function. The ba- their artworks. Predefined values for parameters have been used
sic idea is that the function should be smoothly decreasing on in Table 2 to define the boundary for hand motion zone, which
[0, 1]. Hence, there are several criteria for the falloff functions: creates a virtual rectangle interactive area around a user. During
DigiClay: An Interactive Installation for Virtual Pottery ICVR 2018, February 24–26, 2018, Hong Kong, Hong Kong

Table 1: Two examples of falloff functions and deformation effects.

Function Expressions Graph on [0, 1] Deformation Effect

1

0.9

0.8

0.7
(
−x + 1 if 0 ≤ x ≤ 1
0.6

f F allof f (x) =
0.5

if x > 1
0.4

0 0.3

0.2

0.1

0
0 0.2 0.4 0.6 0.8 1

0.9

0.8

0.7

2
(
100−x − 0.01 if 0 ≤ x ≤ 1
0.6

f F allof f (x) =
0.5

if x > 1
0.4

0 0.3

0.2

0.1

0
0 0.2 0.4 0.6 0.8 1

the surrounding lighting condition and clothes a user wear. In this

case, the user is instructed to leave and re-enter the zone again,
which solves the issue.
User Interface. The cursors which indicate the hand position fol-
low the hands of users well. There are two main issues here: (1)
the concept of the width between two hands determines pull/push
action and the functionalities of two buttons (restart and upload)
need to be verbally conveyed to users before or during the process.
(2) the buttons on top corners are difficult to reach for some users,
which is due to the length of arm.
Accidents. There were a few cases when a user start the interac-
tion, the hand accidentally touched the upload button, causing an
unwanted upload. And some people did not touch the upload but-
Figure 4: A sample work uploaded from DigiClay by a user. ton and left the area when he/she finished, which lost the pottery
had been created.
Table 2: The boundary for hand motion zone. User Heights. Although in the algorithm we tried to address the
variance in user heights, in some cases adjusting the angle of Kinect
Parameter Name Predefined Value [m] manually is required. For instance, during the exhibition a child
who is about 1.2m tall has not been detected due to the height. In
kl e f t -0.5 this case, we lower the angle of Kinect in order to make the child
kr iдht 0.5 within the range of Kinect. This should be taken into consideration
kt op 0.75 for further development.
kbot t om -0.25

5 CONCLUSION
the 2-hour exhibition, 53 users tried the installation, and the age
In this paper, we present our approach towards designing and im-
range is between 7 and 60 (mean = 31.7, SD = 11.8). 96 images
plementing an interactive installation creating virtual pottery with
were uploaded to Facebook (some users have created more than
hand motions in real time. Our ultimate goal is to provide the users
one pottery work during their sessions), and the duration of each
with a simple, elegant interface for art creation in the research,
pottery work is depicted in Figure 6. Total of 60 pottery works (62.5
which combines modern technology and traditional art. DigiClay
%) are completed between 40 to 60 seconds, which matches our
has been exhibited at Centre for Digital Media (Vancouver, 2014)
initial estimation. Several findings have been identified during the
and Communication University of China (Beijing, 2015), which
exhibition, and the following ones were the most notable:
allows the audience, especially who are not familiar with ceramics
User Detection. For most of the time, when a user enters the zone,
art before, enjoying creating pottery artworks. We have identified
she can be detected by Kinect within 1 second. However, 2 out of
some issues from our observation in the exhibition, and several
53 users fails to be detected by Kinect. This might be attributed to
ICVR 2018, February 24–26, 2018, Hong Kong, Hong Kong Z. Gao et al.

Figure 5: Audience work from DigiClay.

40
the novice understand ceramics art better. In the market of home
entertainment, game console could be an ideal platform running
35
DigiClay for families. The installation can be displayed in pub-
30
lic space such as art museums, which could attract more visitors
to traditional pottery art. With 3D printing technology become
25
more mature in terms in cost and time consumption, DigiClay will
Work Count

20
provide users with more joy and excitement.

15

ACKNOWLEDGMENTS
10
This work is supported by the National Natural Science Foundation
5 of China (No. 61502118, 61402127 and 61370212), the Fundamental
Research Fund for the Central Universities (No.HEUCF170602) and
0
0 10 20 30 40
Duration [s]
50 60 70 80
the Natural Science Foundation of Heilongjiang Province in China
(No. F2016028, F2016009, F2015029).

Figure 6: Duration of each pottery work.

REFERENCES
[1] Walter Benjamin and James Amery Underwood. 2008. The work of art in the age
of mechanical reproduction. Vol. 10. Penguin London.
improvements can be made in the future to perfect the user experi- [2] Pedro Campos, Miguel Campos, Paulo Freitas, and Joaquim Jorge. 2012. Foot-
ence. turistic multimedia: designing interactive multimedia installations for shoe shops.
Multimedia tools and applications 61, 2 (2012), 471–487.
We are planning to continue the research and explore more po- [3] Yoon Chung Han and Byeong-jun Han. 2014. Virtual pottery: a virtual 3D audio-
tential from DigiClay. It can be applied in arts education, where visual interface using natural hand motions. Multimedia tools and applications 73,
some potential features, such as interactive tutorials, could help 2 (2014), 917–933.
 DigiClay: An Interactive Installation for Virtual Pottery ICVR 2018, February 24–26, 2018, Hong Kong, Hong Kong

[4] Svebor Karaman, Andrew D Bagdanov, Lea Landucci, Gianpaolo D’Amico, Andrea
Ferracani, Daniele Pezzatini, and Alberto Del Bimbo. 2016. Personalized multi-
media content delivery on an interactive table by passive observation of museum
visitors. Multimedia Tools and Applications 75, 7 (2016), 3787–3811.
[5] Miroslav Macik, Anna Kutikova, Adam J Sporka, and Zdenek Mikovec. 2016.
CoBoard flowers: An interactive installation for Prague spring international music
festival. In Cognitive Infocommunications (CogInfoCom), 2016 7th IEEE International
Conference on. IEEE, 000121–000126.
[6] Sundar Murugappan, Cecil Piya, Karthik Ramani, et al. 2013. Handy-potter:
Rapid exploration of rotationally symmetric shapes through natural hand motions.
Journal of Computing and Information Science in Engineering 13, 2 (2013), 021008.
[7] Karthik Ramani et al. 2015. A gesture-free geometric approach for mid-air ex-
pression of design intent in 3D virtual pottery. Computer-Aided Design 69 (2015),
11–24.
[8] Karthik Ramani et al. 2016. Extracting hand grasp and motion for intent expression
in mid-air shape deformation: A concrete and iterative exploration through a
virtual pottery application. Computers & Graphics 55 (2016), 143–156.
[9] Tim Vets, Luc Nijs, Micheline Lesaffre, Bart Moens, Federica Bressan, Pieter Col-
paert, Peter Lambert, Rik Van de Walle, and Marc Leman. 2017. Gamified music
improvisation with BilliArT: a multimodal installation with balls. Journal on
Multimodal User Interfaces 11, 1 (2017), 25–38.

View publication stats