Avatar Navigation in Second Life Using Brain Signal
Avatar Navigation in Second Life Using Brain Signal
Abstract—This paper presents an interdisciplinary project Employing scientific, yet game-based approaches and
conducted in a sophisticated approach to set up a connection technologies constructs also another aspect of new BCI
between the human brain and a virtual world illustration. The generation. Impairment in motor function leads to forseeable
main purpose of this research is to develop an interaction
difficulties in social interactions and a reduction in quality of
life for most patients. Paralyzed people not only suffer from
technique for entertainment and to investigate the virtual reality
aspects for physical rehabilitation. Users could freely explore
Second Life’s virtual world and steer their avatar to the target physical inabilities but also psychological aspects worsening
point using their brain signals. The experiment was carried out over time. Providing a fun and attractive interface might
on three subjects. The control can be accomplished by online provoke the user’s intention in a more effective way.
processing of single-trial EEG signals recorded from EPOC Improving the quality of life for paralyzed people due to spinal
headset, Emotiv, and passing the parameters to the main cord damage, stroke, or peripheral nervous system disabilities
interface. An adaptive filter-based approach was used to reduce are the major clinical motivation for developing a brain-
the eye blink distortions. Time–frequency features with computer interface which may help to restore their mobility
combination of an ensemble LDA classifier led to a maximum through prosthetic devices or software interfaces [11]. Elderly
87.2% of accuracy.
people with stroke [6] and children with sensorimotor
Keywords—Virtual World Environment, Brain Computer disorders may also gain from interactive computer play for
Interface, Second life, Emotiv. rehabilitation purposes [11] [8]. In [5] the effectiveness of the
Wii system was tested for balance rehabilitation in patients
I. INTRODUCTION with acquired brain injury.
A recent research study extends the usage of the well-known
Virtual social worlds (e. g Second Life) are becoming gait rehabilitation robot “Lokomat®” from a boring and
increasingly interesting for people from different generations continuous physical practice sessions to an exciting game
and social groups. Among all these, people with handicaps controller for children [7].
could also benefit broadly from this new technology. In this project a heuristic interaction technique was
Forfeiture of movement control in human is one of the developed that allows the user to freely control an avatar in the
common diseases worldwide with a deeply negative effect on Second Life virtual environment by using mental activities.
the patients’ quality of life and their communication ability [8, The system is trained for some predefined movements. The
9]. Brain-computer interfacing (BCI) promises a direct control can be accomplished by online processing of single-
communication solution from the brain to the external world trial EEG signals which is recorded using the EPOC headset
without employing the peripheral nervous system [1, 3]. The from Emotiv.
use of BCI, however, is still very limited to some serious
clinical applications [2]. In the past few years, emerging II. EXPERIMENTAL SETUP
simple and cheap BCI systems opened up the market and made Experimental runs were carried out with three able-bodied
BCI appealing for the new and large group of people [8] who volunteer subjects (1 female, 2 males). Only S1 had previous
initially started to use BCI for fun and entertainment purposes experience with BCI. In preparation for each run, subjects were
[10]. It expands the yet limited BCI application to more seated 0.5 meters from a computer monitor with arms in front
applicable scenarios in everyday life, even though noninvasive and palms flat and facing down on a desk. The setup consists of
methods will probably have limited use in fine tuned controle hardware and software parts.
[2, 4]. The goal of using BCI in the life of healthy people is to
render it a useful tool in situations where the normal and A. Hardware: Emotiv EPOC headset
conventional interfacing means such mouse, keyboard or The EEG acquisition machine utilized was an Emotiv
joystick cannot be an option, i.e. in space or marine projects EPOC headset. This headset is a low-cost, commercial
[1] or to improve attention, concentration or motivation [7]. To EEG cap with dry electrodes. It is easy to wear and use and is
date prototype BCI applications are available in the domain of equipped with Bluetooth connectivity to the computer. Fig. 1
gaming and entertainment using easy to wear hardware [10]. shows the headset and positions of electrodes covered by it.
Authorized licensed use limited to: Escuela Superior de Ingeneria Mecanica. Downloaded on April 29,2021 at 18:50:45 UTC from IEEE Xplore. Restrictions apply.
III. MATERIALS AND METHODS
Grand average-AF3
Grand av erage-AF4
10
15
forward
8 right
left
forward
right A. Preprocessing
6 10 left
stop stop
Preprocessing consisted of filtering and artifact removal.
Amplitude (uV)
Amplitude (uV)
2 5
The signals were filtered using an eleventh-order zero-phase
0
-2
0 forward and reverse Chebyshev lowpass filter with cutoff
-4
-5
frequency of 100 Hz. A notch filter at 50Hz was applied to
-6 remove power line noise. In each trial the mean values over
-8
100 200 300 400 500 600 700 800 900
-10
100 200 300 400 500 600 700 800 900 time of the imagination and relaxation signals were calculated
Time (ms) Time (ms)
Grand average-F3 Grand av erage-F4 and subtracted from each respective signal. In order to
8 15
forward forward implement an artifact removal system online, we used an
6 right right
left 10 left adaptive filter approach in the time domain. The data of the
4 stop stop
offline experiment was used to initialize the classifier and to
Amplitude (uV)
Amplitude (uV)
5
2
calculate the normalization and artifact reduction parameters.
0
0
-2
-5
B. Feature extraction and selection
-4
As a first step, EEG recordings from each channel are
-6 -10
100 200 300 400 500 600 700 800 900
Time (ms)
100 200 300 400 500 600 700 800 900
Time (ms)
decomposed into a sum of atoms, each being the product of
8
Grand average-FC6
4
Grand av erage-FC5 spatial (topographic) signature and waveforms having
forward
right 3
forward
right
determined time–frequency localization. Matching pursuit was
6
left
stop
2
left
stop
exploited to find the best matching projections of
4
multidimensional data onto an over-complete dictionary of
Amplitude (uV)
Amplitude (uV)
1
2 0
Gabor functions. The five first weighting coefficients extracted
-1
0
-2
from each channel were used as features in BCI.
-2
-3
-4
-4 1) Matching pursuit
-6 -5
100 200 300 400 500 600 700 800 900
Time (ms)
100 200 300 400 500 600 700 800 900
Time (ms)
Matching pursuit (MP) was initially introduced by Mallat
5
Grand average-P7
6
Grand average-P8 and Zhang and to date there are several extensions to the
forward
right
forward
right
original formulation [16] [20]. MP provides an intuitive and
4
left
stop
left
stop
rather simple algorithm which refers to a greedy
Amplitude (uV)
3300 ms 4300 ms
3.1
-1.6
-3.1
where n indexes the atoms that have been chosen, and is
at1
the weighting factor (an amplitude) for each atom. Given a
Fig. 4. The average ERP distribution 300ms after the onset of experiments fixed dictionary, matching pursuit will first find the atom that
and every second after that time point.
has the biggest inner product with the signal, and then subtract
Authorized licensed use limited to: Escuela Superior de Ingeneria Mecanica. Downloaded on April 29,2021 at 18:50:45 UTC from IEEE Xplore. Restrictions apply.
the contribution due to that atom; this process is repeated until Basis
the signal is satisfactorily decomposed [16]. 1
0
TABLE I. ALGORITHM MATCHING PURSUIT
-1
0 50 100 150 200 250 300 350 400 450
1.Initialization:
10 signal
approximation
3. Let -10
0 200 400 600 800 1000 1200 1400
Authorized licensed use limited to: Escuela Superior de Ingeneria Mecanica. Downloaded on April 29,2021 at 18:50:45 UTC from IEEE Xplore. Restrictions apply.
is minimized. It is then calculated as the eigenvector FP and FN are also called error type I and II respectively [17].
corresponding to the maximum eigenvalue of [19]. FP is incorrect rejection of a positive result and FN is the
failure to reject a false result [17].
0.6
Sensitivity
0.5
Feature Classifier 0.4
ext. Ch.1 .1
0.3
0.2
Feature Classifier
Decision
ext. Ch.2 .2 0.1
making
0
0 0.2 0.4 0.6 0.8 1
Specificity
Feature Classifier
.n Fig. 8. ROC curve for the relaxed and movement accuracy during the
ext. Ch.n experiment for all subjects. By changing the classifier parameters the
sensitivity and specificity change. The optimum classifier is selected by
Fig. 7. The idea of ensemble classification for different channel and decision looking at the right upper corner of the ROC curve
making based voting
Authorized licensed use limited to: Escuela Superior de Ingeneria Mecanica. Downloaded on April 29,2021 at 18:50:45 UTC from IEEE Xplore. Restrictions apply.
online experiment for S1, S2 and S3. S1 with previous experiment. Each feature extracts an important but selective
experience in BCI showed the most stable results. aspect of mental tasks from the dataset. In our experiments the
best classification accuracy was obtained when a subset of 10
100 features were combined with ensemble LDA classifer.
90
70
NeuroEngineering and Rehabilitation, vol. 8 no. 30, 2011.
[7] Schuler T., Brutsch K., Muller R., Hvan Hedel U. J., Meyer-Heim A.,
60 Virtual realities as motivational tools for robotic assisted gait training in
50
children: A surface electromyography study, Neuro Rehabilitation, vol.
28, no. 4, pp: 401-411, 2011
40 [8] Brooks Jr., F. P. , “What's Real About Virtual Reality?”, IEEE
30
Computer Graphics And Applications, vol. 19, no. 6, 16, 1999.
[9] Kempe L., “Serious Games für die Gesundheit: Spielen für eine ernste
20 Sache“, Dtsch Arztebl, vol. 107, no. 38, 2010
Ralax
10 Event [10] Roman Krepkiy, Brain-Computer interface: Design and implementation
Total of an online BCI system for the control in gaming application an virtual
0 limb, VDM Verlag Dr Muller, 2008
1 2 3 4 5 6 7 8 9 10
Run [11] Keshner Emily A, Virtual reality and physical rehabilitation: a new toy
or a new research and rehabilitation tool?, Journal of Neuroengineering
Fig. 9. Performance of S1, S2 and S3 in several runs of the online and rehabilitation, vol. 1, no. 8, 2004.
experiment. The red line corresponds to the stop command and the green line
shows the classification accuracy in average for move commands (Right, Left [12] Gonzalez-Sanchez, J., Chavez-Echeagaray, M. E., Atkinson, R., and
Burleson, W., An Agent-Based Software Architecture for a Multimodal
or Forward). The Black line demonstrates the average performance for all four
Emotion Recognition Framework. In Proceedings of 9th Working
classes.
IEEE/IFIP Conference on Software Architecture (WICSA’11), 2011.
[13] Stefan Debener et al, How about taking a low-cost, small, and wireless
On average there was a decrease in performance when EEG for a walk?”, Psychophysiology, vol. 49, no. 11, pp: 1617-21,
subjects encountered the 3D interface after the training PMID 23013047, 2012
session. The reason could be that we have selected the best [14] Au, Wagner James. The Making of Second Life, pg. 19. New York:
Collins. ISBN978-0-06-135320-8.
features in training data and applied the same subset during the
online
Authorized licensed use limited to: Escuela Superior de Ingeneria Mecanica. Downloaded on April 29,2021 at 18:50:45 UTC from IEEE Xplore. Restrictions apply.
[15] Alexander Y. Kaplan et al., Adapting the P300-based brain-computer [20] Mallat S. G. and Zhang, Z. Matching Pursuits with Time-Frequency
interface for gaming: a review, IEEE Transactions on Computational Dictionaries. IEEE Transactions on Signal Processing. December 1993,
Intelligence and AI in Games (Special Issue on Brain/Neuronal- pp. 3397–3415.
Computer Games Interfaces and Interaction), 2013, in press. [21] Peng H, Long F., and Din Ch. Feature selection based on mutual
[16] Durka P. J., Dobieslaw, I. and and Blinowska, K. J. Stochastic time– information: criteria of max-dependency, max-relevance, and min-
frequency dictionaries for matching pursuit. IEEE Trans. Signal redundancy. IEEE Transactions on Pattern Analysis and Machine
Process., vol. 49, no. 3, March 2001 Intelligence, vol. 27, no. 8, pp. 1226-1238, 2005.
[17] Hastie T., Tibshirani R., Friedman J. The Elements of Statistical [22] Koprinska, Irena Comparison of Feature Selection Methods for
Learning. 2008 Classification of Brain-Computer Interface Data. Workshop on Advances
[18] Saeid Sanei and Chambers J. A. EEG signal processing. s.l. : John and Issues in Biomedical Data Mining, pp. 41-50, 2009.
Wiley & Sons Ltd., 2007
[19] Fukunaga K. Statistical Pattern Recognition. New York: : Academic,
2nd edition, 1990
.
Authorized licensed use limited to: Escuela Superior de Ingeneria Mecanica. Downloaded on April 29,2021 at 18:50:45 UTC from IEEE Xplore. Restrictions apply.