Neurol Sci

DOI 10.1007/s10072-016-2631-9

ORIGINAL ARTICLE

Utility of Boston Qualitative Scoring System for Rey-Osterrieth

Complex Figure: evidence from a Parkinson’s Diseases sample
Federica Scarpina1,2 • Erika Ambiel1 • Giovanni Albani3 • Luca Guglielmo Pradotto3 •

Alessandro Mauro2,3

Received: 11 April 2016 / Accepted: 6 June 2016

Ó Springer-Verlag Italia 2016

Abstract This study examined the ability of the Boston Keywords Neurodegenerative disease

Qualitative Scoring System (BQSS) in comparison to the Neuropsychology
Cognition
Executive function

Osterrieth scoring method to separate Parkinson’s Disease Visuo-constructional ability
patients without dementia from healthy controls at the Rey-
Osterrieth Complex Figure (ROCF) copy. 30 PD partici-
pants and 30 healthy participants completed ROCF copy. Introduction
The performance was scored according to both methods.
The results indicated that PD patients performed signifi- Among the nonverbal instruments, the Rey-Osterrieth
cantly worse on ROCF. According to ROC analyses, BQSS Complex Figure (ROCF) [1, 2] stands out as one of the
Copy Total score represented the most suitable index to most widely used neuropsychological test in both clinical
distinguish between the two groups: a score below or equal and experimental settings to evaluate visuo-constructional
to 16 indicates an impaired performance. Moreover, PD abilities and nonverbal memory [3], also in the presence of
participants reported lower performance in the BQSS motor symptoms [4, 5]. However, because of the com-
scores of Planning and Neatness. PD patients’ poor per- plexity of the task, performance to ROCF copy also reflects
formance in ROCF copy was related to executive diffi- executive functions [6], specifically in terms of organiza-
culties, specifically in terms of planning and impulsivity, tional and planning abilities [7, 8]. This aspect appeared to
instead of global visuo-constructional impairments. An be emphasized in Parkinson’s Disease (PD) population:
extensive evaluation of copy drawings allowing to disen- problem-solving difficulty in PD patients would became
tangle between different involved cognitive domains would evident only in complex patterns [9], as ROCF; moreover,
be suitable, specifically in those clinical conditions like PD, PD patients’ poor performance in ROCF copy [4, 5] was
in which motor impairments affect drawing performance. related to executive deficits, in particular in planning [5],
problem solving, working memory, verbal fluency and set-
shifting [10].
The quantitative Osterrieth scoring algorithm [2] does
not assess the executive aspects of patients’ drawings [11];
according to this system, the figure is split into 18 identi-
& Federica Scarpina fiable areas, each of which is considered separately and
f.scarpina@auxologico.it marked on the accuracy of its position and the distortions
1
Psychology Research Laboratory, IRCCS Istituto Auxologico
exhibited [2]. Limitations of this scoring system include the
Italiano, Ospedale San Giuseppe, Piancavallo, VCO, Italy lack of organizational information (such as whether the
2 drawing was produced in a piecemeal or a logical fashion)
‘‘Rita Levi Montalcini’’ Department of Neuroscience,
University of Turin, Turin, Italy and the failure to differentiate the diagnostic importance of
3 different sections [12]. To solve these limitations, the
Division of Neurology and Neuro-Rehabilitation, IRCCS
Istituto Auxologico Italiano, Ospedale San Giuseppe, Boston Qualitative Scoring System (BQSS) [13] has been
Piancavallo, VCO, Italy developed: it makes use of guides and templates to produce

123
 Neurol Sci

a comprehensive score, in which also executive aspects Neuropsychological assessment

[7, 13, 14] are accounted. BQSS application in different
clinical population, such as schizophrenia [15], depression The following neuropsychological tests were administered
[14], obsessive–compulsive disorder [16], traumatic brain to describe cognitive functioning. The Clock Drawing Test
injury [17] and detoxified alcoholics [18], were yet repor- is considered a global measure of cognitive functioning
ted in literature. [23]; it is an adequately sensitive and specific measurement
This study contrasted the ability of the BQSS scoring for the detection of dementia in primary care [23]. To
system in comparison to the Osterrieth scoring method [2] assess the executive domain, we administered the Italian
to separate PD patients without dementia from healthy version of Frontal Assessment Battery—FAB [24]: it is
controls. considered a reliable measure of frontal lobe dysfunction
and recommended for detecting first symptoms of cogni-
tive decline among patients diagnosed with neurodegen-
Methods erative diseases [24]. In addition, the ‘phonological loop of
working memory was assessed using through the Digit
Participants Span Backward [25] and the inhibitory control using the
Stroop Test—Error Index [26].
30 PD patients and 30 healthy subjects took part in the
study. All participants were right-handed. ROCF
PD patients (m = 10, f = 20; mean age = 65,
SD = 10; mean education in years m = 9, SD = 3) were All participants were required to copy the ROCF [2]. They
consecutively recruited during their hospitalization at the were provided with a blank sheet of paper on which they
Neurology and Neurorehabilitation division of the were asked to copy the complex figure as best as they
I.R.C.C.S. Istituto Auxologico Italiano-Ospedale San Giu- could. There was no time limit set for the copy.
seppe (Piancavallo, Oggebbio, VCO, Italy). They had been The copy drawing was scored according with both the
diagnosed as having PD (mean years from diagnosis Osterrieth scoring [2, 27] and the BQSS [13].
m = 7, SD = 5) according to the Hoehn and Yahr’s According to Osterrieth scoring method [2, 27], the
classification [19]. The PD group reported a mean score of ROCF figure is divided in 18 elements (Fig. 1a). For each
35 (SD = 16.28) to the unified Parkinson’s disease rating element, a score was assigned according to the following
scale (UPDRS) [20]. All patients were evaluated during procedure: 2 points when the element is correct and placed
subjective on-phase, in the absence of significant fluctua- properly; 1 point when the element is (1) correct, but
tions or motor blocks. Clinical details are reported in placed poorly, or (2) distorted or incomplete but recog-
Table 1. nizable, placed properly; 0.5 point when the element is
Exclusion criteria were evidence of others neurological distorted or incomplete, but recognizable, placed poorly;
disorders (ictus, traumatic brain injury) or pathological finally, 0 when the element is absent or not recognizable.
conditions (psychiatric syndromes, potus). Moreover, a Instead, according to BQSS [13], the ROCF figure is
threshold of 24 for Mini Mental State Examination divided into configural elements, clusters and details
(MMSE) [21] was adopted as an inclusion criterion [22]. (Fig. 1b). Configural elements are scored as present or
Not hospitalized healthy subjects (m = 9, f = 21; mean accurate; clusters are scored as present, accurate, and
age = 60, SD = 10; mean education in years m = 9, placed correctly; and details are scored as present and
SD = 3) were recruited as control group. People reporting placed correctly. Scores range from 0 (absent or very poor)
neurological impairments and/or other diseases with any to 4 (present or well done) and their sum represents the
implications on their health status were excluded. Copy Total score. Moreover, the executive scores of
The two groups are comparable in terms of Age Planning, Fragmentation, Perseveration and Neatness are
[t(50) = 1.16; p = 0.25] and Education [t(50) = 0.001; calculated, ranging from 0 (absent or very poor) to 4
p = 0.99]. (present or well done). The sum of Fragmentation and
This study was approved by the ethical committee of Planning results in global Organization score: it ranges
IRCSS Istituto Auxologico Italiano and it was performed in from 0 (poor) to 8 (good). Finally, the visuo-constructional
compliance with Declaration of Helsinki’s ethical princi- indexes of Confabulation, Vertical Expansion, Horizontal
ples. The administered tests were part of the routine clinic Expansion, Reduction and Rotation were scored from 0
visit for PD patients; PD and healthy participants gave their (poor) to 4 (good), while Asymmetry is a categorial rating
written consensus for their participation in this research. referring to the presence of less details or more distortions

123
Neurol Sci

Table 1 Clinical details

Sex Age Education Duration of disease Y&H stage UPDRS score LED
relative to PD participants
m 59 8 1 1 15 0
f 71 8 4 2 19 200
f 76 15 16 3 38 400
m 75 13 9 – 23 400
f 62 18 8 – 18 300
f 66 8 6 2 24 400
m 79 8 11 3 45 600
m 59 13 15 3 51 200
f 71 5 18 4 64 1200
f 72 12 1 2 48 400
f 80 8 4 3 – –
f 47 8 3 2 47 400
m 73 8 5 2 24 0
m 75 10 11 1 14 0
f 63 8 10 1 26 0
m 73 10 1 3 46 600
m 72 5 3 4 48 950
m 73 8 11 1 19 200
f 53 13 13 3 58 500
f 67 13 5 3 63 100
f 69 8 6 2 18 600
m 58 5 13 – 26 0
f 33 18 8 4 32 500
m 55 13 5 – 26 400
f 65 8 1 3 42 600
f 72 5 1 – 65 300
f 74 8 17 2 22 400
f 49 8 4 3 32 200
f 60 13 7 – 14 200
f 74 3 20 3 48 500
Mean 65 9 7 2.5 35 363.7
SD 10 3 5 0.93 16.2 280.5
Education and duration of disease expressed in years. Levodopa equivalent dose (LED) expressed in
mg/day. Mean and standard deviations (SD) were reported in the lower part of the table
m male, f female

in one side (i.e., left) respect to the other side (i.e., right). neuropsychological measures of Clock Drawing Test, FAB,
More details are reported in the original manual [13]. In Digit Span Backward, Stroop Test—index error to verify
Table 2 an example of the two methods application was possible differences in cognitive performance.
shown in relation to a healthy participant’s drawing
(Fig. 1c) and two PD participants’ drawings (Fig. 1d, e). ROCF

Statistical analyses Difference between the two groups in relation to the ROCF
copy score according to Osterrieth method and the BQSS
Neuropsychological assessment indexes of Copy Total score and Organization was asses-
sed with non-parametric Mann–Whitney test. The same
Non-parametric Mann–Whitney test were run to test dif- analyses was used to compared the executive score of
ference between groups about the scores relative to the Planning, Perseveration, Fragmentation and Neatness and

123
 Neurol Sci

123
 Neurol Sci

b Fig. 1 a Osterrieth scoring method: the 18 elements of the ROCF Table 2 examples of the application of Osterrieth score and BQSS
were indicated. b BQSS: the configural elements, clusters and details on two PD participants’ and one healthy participant’s ROCF copy
were shown. c–e Examples of ROCF copy performance of a healthy drawings
participant (c) and two PD patients (d, e); refers to Table 2 for score
details Elements Healthy participant I PD patient II PD patient
Score Score Score

Osterrieth score
the other visuo-constructional scores of Confabulation, 1 2 2 1
2 2 1 0.5
Vertical and Horizontal expansion, Reduction, Rotation
3 2 2 0.5
and Asymmetry.
4 2 2 0.5
5 2 1 1
ROC analyses
6 1 1 0
7 2 2 0
A comparison of ROC curves relative to the ROCF Copy
8 2 2 0
Total Score, ROCF Organization and the Osterrieth score 9 2 2 0.5
method was run using a nonparametric approach to the 10 2 2 0
analysis of areas under correlated ROC curves [28], to 11 1 2 0.5
compare the accuracy of these three scores in discrimi- 12 2 2 1
nating PD group from healthy group. 13 2 1 0.5
14 2 1 0.5
15 2 2 0.5
Results 16 2 2 0.5
17 2 1 0.5
Neuropsychological assessment 18 2 2 2
Sum (max 36) 34 30 10
A significant difference emerged at the Clock Drawing Indexes Healthy I PD II PD
Test: PD group had lower performance with respect to participant patient patient
control group (Table 3). In addition, PD participants had Score Score Score
lower performance then the healthy subjects in the global BQSS
executive cognitive functioning measured by the FAB Configural elements
tests; specifically, they showed a significant impairment in Presence 4 4 3
executive domains of inhibitory control, measured by the Accuracy 2 4 2
Stroop’s Test—index error, and of the verbal working Clusters
memory, measured by the Digit Span Backward. Presence 4 4 2
Accuracy 3 4 0
ROCF Placement 3 4 2
Details
PD group showed a worse performance in ROCF scored Presence 4 4 1
according to the Osterrieth score (Table 3). The same result Placement 4 4 2
emerged about BQSS Copy Total Score and Organization. Fragmentation: measures integration of 3 3 3
information, i.e., whether or not the
focusing on BQSS subscales, PD group showed signifi- individual elements are drawn as
cantly worse performance relative to executive indexes of whole units
Planning and Neatness and in the visuo-constructive skill Planning: overall planning ability 3 2 1
of Rotation. based on the order in which elements
are drawn, placement on the page,
placement within the figure, integrity
ROC analyses of the production
Neatness: indicated how neatly the 3 4 1
In Table 4 and Fig. 2, the results relative to the three dif- figure was drawn as evidenced by the
number of wavy lines, gaps and
ferent ROC curves comparisons were reported. BQSS Copy overshoots, cross-outs, rounded
Total Score significantly discriminated the two group corners
respect to the BQSS Organization (p = 0.04) and respect Perseveration: measures the extent of 3 4 1
recognizably inappropriate repetition
to the Osterrieth score (p = 0.02), while no difference of components within a cluster or
emerged between BQSS Organization and Osterrieth score elements of the figure

123
 Neurol Sci

Table 2 continued the prevalence according to Youden index (J = 0.66) [29];

Indexes Healthy I PD II PD
a cut off B16 showed a sensitivity of 70 and a specificity of
participant patient patient 96.67.
Score Score Score

Vertical expansion: size distortion on 4 3 4

the vertical axys Discussion
Horizontal expansion: size distortion 4 3 4
on the horizontal axys
Reduction of the drawing dimension 4 4 4
In this work, we verified the applicability of BQSS to dis-
Rotation of the production on the page 4 4 4
tinguish PD patients from healthy subjects respect to the
Confabulation: indicated the intrusion 4 4 4 Osterrieth scoring method in relation to ROCF performance.
of elements not presented in the According to both scoring systems, PD participants had
original figure a lower ROCF copy performance then healthy participants,
Asymmetry: comparison of the 0 0 2 confirming previous results [4, 5]. However, BQSS Copy
distortion and/or lack of details in the
right and left sides of the figure; in Total Score appeared to be more suitable to distinguish the
this work 0 = asymmetry PD performance from healthy participants’ performance
Copy Total Score: arithmetic sum of 12 13 6 respect to the drawing traditional Osterrieth score method
Fragmentation, Planning, Neatness
and Perseveration, providing a more
[1, 2] and respect to BQSS organization index. A BQSS
omnibus measure of completeness Copy Total score below or equal to 16 would be indicative
and accuracy of the production (Max of global cognitive impairment, in which also executive
16)
domain acts a critical role. Indeed, an acceptable level of
Organization: arithmetic sum of 6 5 4
Fragmentation and Planning scores, accuracy would be related to preserved executive moni-
providing a more omnibus measure toring: people have to monitor their behavior and sustain a
of organizational skills (Max 8) complex mental set while performing mental manipula-
For healthy participant, refer to Fig. 1c; for I PD patient, to Fig. 1d; tions [30], to avoid perseveration or omission. Our results
for II PD patient, to Fig. 1e contrast with Sommerville et al. [7], according which
Organization, and not the Copy Total Score, is a better
(p = 0.5). Thus, BQSS Copy Total Score is best able to index of executive functioning since it appeared to differ-
distinguish PD patients performance from healthy subjects. entiate patients of a variety of diagnostic categories with
Thus, we calculated the cut-off point independently from either no, mild, or severe executive dysfunction.

Table 3 Means an standard

Neuropsychological variable PD group Control group Mann–Whitney test
deviation (in brackets) for each
neuropsychological measures BQSS
and relative subscales
Fragmentation 2.7 (0.98) 3.06 (0.69) U = 632; p = 0.16
Planning 2.17 (0.9) 3.46 (0.79) U = 137; p \ 0.001*
Neatness 2.8 (0.88) 3.73 (0.52) U = 190; p \ 0.001*
Perseveration 3.53 (0.68) 3.86 (0.34) U = 328; p = 0.19
Vertical expansion 3.93 (0.25) 4 (0) U = 420; p = 0.15
Horizontal expansion 3.73 (0.58) 3.76 (0.43) U = 458; p = 0.86
Reduction 3.56 (0.67) 3.83 (0.37) U = 367.5; p = 0.1
Rotation 3.66 (0.8) 4 (0) U = 345; p = 0.005*
Confabulation 3.86 (0.34) 3.96 (0.18) U = 405; p = 0.16
Asymmetry 0.13 (0.8) 0 (0) U = 480; p = 0.15
Copy Total Score 14.83 (3.29) 18.73 (1.25) U = 122.5; p \ 0.001*
Organization 4.8 (1.58) 6.16 (1.5) U = 239; p = 0.002*
Osterrieth score (max = 36) 27.68 (7.3) 33.71 (2.98) U = 204; p \ 0.001*
Clock Drawing test (max = 18) 7.68 (3.02) 9.35 (1.39) U = 272; p = 0.005*
Frontal assessment battery (max = 18) 14.63 (2.78) 15.93 (2.18) U = 307; p = 0.032*
Digit Span Backward 3.53 (1) 4.3 (1.72) U = 308.5; p = 0.03*
Stroop test–error index 4.31 (5.22) 0.93 (1.27) U = 621.5; p = 0.009*
* p \ 0.05

123
Neurol Sci

Table 4 The different ROC

Factors Area under curve Standard error 95 % confidence interval
curves (upper part) followed by
the result of pairwise BQSS Copy Total score 0.86 0.04 0.75–0.93
comparison of all ROC curves
(lower part) were reported: BQSS Organization 0.73 0.06 0.6–0.84
specifically the difference Osterrieth score 0.77 0.06 0.64–0.87
between the areas, the standard
Comparison Difference between Standard 95 % confidence z p value
error, the 95 % confidence
areas error interval

BQSS Copy Total 0.12 0.06 0.002–0.25 1.99 0.04*

score
BQSS Organization
BQSS Copy Total 0.09 0.04 0.009–0.17 2.18 0.02*
score
Osterrieth score
BQSS Organization 0.03 0.07 -0.1 to 0.18 0.53 0.59
Osterrieth score
* p \ 0.05

patients did not differ from controls. However, planning

difficulties appeared to be due to dysfunctional frontal-
basal-ganglia network subserving goal directed and vol-
untary actions, since the role of the subthalamic nucleus in
the implementation of optimal planning strategies in PD
[31, 32]. About Neatness, it indicates impulsivity in
drawing [13]. Impulsivity is a key feature of PD, and it is
due to a poorer inhibitory control [33, 34], as suggested
also by the performance in Stroop Test—error index [26] in
our sample. As far as the visuo-constructional abilities are
concerned, PD patients’ low score in Rotation would mir-
ror visuo-spatial deficit [5, 7, 35] and specifically in
judgement of lines orientation [36].
Overall, the PD patients’ poor performance in ROCF
copy appeared to be related more to executive difficulties
then global visuo-constructional impairments, in line with
Hanes et al. [9]. An extensive evaluation of copy drawings
allowing to disentangle between different involved cogni-
tive domains would be suitable, specifically in those clin-
ical conditions like PD, in which motor impairments would
Fig. 2 ROC curves relative to the BQSS Copy Total Score, BQSS
affected the performance. As previously suggested by
Organization and the Osterrieth scoring method. The true positive rate
(Sensitivity) is plotted in function of the false positive rate (100- Elderkin-Thompson et al. [14], we consider BQSS method
Specificity) for different cut-off points [13] for the ROCF as a useful tool for this purpose in
clinical setting. However, considering the extra time and
training required for the BQSS scoring protocol, further
According to our results, PD patients’ poor performance
studies are recommended to enhance the qualitative inter-
was related more to executive difficulties, in terms of
pretation and efficacy of BQSS indexes.
Planning and Neatness, rather than visuo-constructional
impairments. Indeed, PD participants reported lower score Acknowledgments The authors thank Guido E. d’Aniello to help in
in terms of Rotation score, without other difference respect recruiting patients; moreover, the authors thank all subjects for their
to the healthy performance. Stern et al. [13] suggested that kind participation.
difficulties in Planning would represent an index of pre-
Compliance with ethical standards
frontal systems dysfunctions, corresponding to the tradi-
tional pathological PD cognitive description [10], even Conflict of interest The authors declare the absence of any conflict
though in their manual [13] the authors reported that PD of interest.

123
 Neurol Sci

References Clin Neuropsychol 14:789–790. doi:10.1016/S0887-6177(99)

80330-8
1. Rey A, Osterrieth PA (1993) Translations of excerpts from Rey’s 18. Dawson LK, Grant I (2000) Alcoholics’ initial organizational and
‘Psychological Examination of Traumatic Encephalopathy’ and problem-solving skills predict learning and memory performance
on the Rey-Osterrieth Complex Figure. J Int Neuropsychol Soc
Osterrieth’s ‘The Complex Figure Test’. Clin Neuropsychol
7:2–21 6:12–19
2. Osterrieth PA (1994) Le test de copie d’une figure complexe. 19. Hoehn MM, Yahr MD (1967) Parkinsonism: onset, progression
Archives de Psychologie 30:206–356 and mortality. Neurology 17:427–442
20. Fahn S, Elton RL (1987) UPDRS program members, Unified
3. Spreen O, Strauss E (1998) A compendium of neuropsychologi-
cal tests: administration, norms, and commentary, 2nd edn. Parkinsons Disease Rating Scale. In: Fahn S, Marsden CD,
Oxford University Press, London Goldstein M, Calne DB (eds) Recent developments in Parkinsons
4. Grossman M, Carvell S, Peltzer L, Stern MB, Gollomp S, Hurtig disease, vol 2. Macmillan Healthcare Information, Florham Park,
HI (1993) Visual construction impairments in Parkinson’s dis- pp 153–163
ease. Neuropsychologia 7:536–547. doi:10.1037/0894-4105.7.4. 21. Folstein MF, Folstein SE, McHugh PR (1975) ‘‘Mini-mental
536 state’’. A practical method for grading the cognitive state of
5. Mckinlay A, Grace RC, Dalrymple-Alford JCV, Roger D (2010) patients for the clinician. J Psychiatr Res 12:189–198. doi:10.
Characteristics of executive function impairment in Parkinson’s 1016/0022-3956(75)90026-6
disease patients without dementia. J Int Neuropsychol Soc 22. Lezak MD, Howieson DB, Loring DW (2004) Neuropsycholog-
16:268–277. doi:10.1017/S1355617709991299 ical assessment, 4th edn. Oxford University Press, Oxford
23. Tuokko H, Hadjistavropoulos T, Miller A, Beattie BL (1992) The
6. Shorr JS, Delis DC, Massman PJ (1992) Memory for the Rey–
Osterrieth figure: perceptual clustering, encoding, and storage. clock test: a sensitive measure to differentiate normal elderly
Neuropsychologia 6:43–50. doi:10.1037/0894-4105.6.1.43 from those with Alzheimer disease. J Am Geriatr Soc
7. Sommerville J, Tremont G, Stern RA (2000) The Boston Quali- 40:579–584. doi:10.1111/j.1532-5415.1992.tb02106.x
tative Scoring System as a measure of executive functioning in 24. Appollonio I, Leone M, Isella V et al (2005) The frontal
Rey-Osterrieth Complex Figure Performance. J Clin Exp Neu- assessment battery (FAB): normative values in an Italian popu-
ropsychol 22:613–621 doi:10.1076/1380-3395(200010)22:5;1- lation sample. Neurol Sci 26:108–116. doi:10.1007/s10072-005-
9;FT613 0443-4
8. Beebe DW, Ris MD, Brown TM, Dietrich KN (2004) Executive 25. Monaco M, Costa A, Caltagirone C, Carlesimo GA (2013) For-
functioning and memory for the Rey-Osterreith Complex Fig- ward and backward span for verbal and visuo-spatial data: stan-
ure Task among community adolescents. Appl Neuropsychol dardization and normative data from an Italian adult population.
Neurol Sci 34:749–754. doi:10.1007/s10072-012-1130-x
11:91–98
9. Hanes KR, Andrewes DG, Smith DJ, Pantelis C (1996) A brief 26. Caffarra P, Vezzadini G, Dieci F, Zonato F, Venneri A (2002)
assessment of executive control dysfunction: discriminant valid- Una versione abbreviata del test di Stroop: dati normativi nella
ity and homogeneity of planning, set shift, and fluency measures. popolazione italiana. Rivista di Neurologia 12:111–115
27. Carlesimo GA, Buccione I, Fadda L et al (2002) Standardiz-
Arch Clin Neuropsychol 11:185–191. doi:10.1093/arclin/11.3.
185 zazione di due test di Memoria per uso clinico: Breve Racconto e
10. Caballol N, Marti MJ, Tolosa E (2007) Cognitive dysfunction and Figura di Rey. Nuova Rivista di Neurologia 12:1–13
dementia in Parkinson disease. Mov Disord 22:358–366. doi:10. 28. DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing
1002/mds.21677 the areas under two or more correlated receiver operating char-
11. Boone KB, Lesser IM, Hill-Gutierrez E, Berman NG, D’Elia LF acteristic curves: a nonparametric approach. Biometrics
(1993) Rey-Osterrieth complex figure performance in healthy, 44:837–845
older adults: relationship to age, education, sex, and IQ. Clin 29. Youden WJ (1950) Index for rating diagnostic tests. Cancer
Neuropsychol 7:22–28. doi:10.1080/13854049308401884 3:32–35 doi:10.1002/1097-0142(1950)3:1\32::AID-CNCR28200
12. Canham RO, Smith SL, Tyrrell AM (2000) Automated scoring of 30106[3.0.CO;2-3
a neuropsychological test: the rey osterrieth complex figure. Conf 30. Freeman RQ, Giovannetti T, Lamar M, Cloud BS, Stern RA,
Kaplan E, Libon DJ (2000) Visuoconstructional problems in
Proc EUROMICRO 2:406–413
13. Stern RA, Javorsky DJ, Singer EA et al (1992) BQSS: the Boston dementia: contribution of executive systems functions. Neu-
Qualitative Scoring System for the Rey-Osterrieth Complex ropsychology 14:415–426
Figure. Psychological Assessment Resources, Odessa 31. Ray NJ, Jenkinson N, Brittain J, Holland P, Joint C et al (2009)
14. Elderkin-Thompson V, Boone KB, Kumar A, Mintz J (2004) The role of the subthalamic nucleus in response inhibition: evi-
Validity of the Boston Qualitative Scoring System for the Rey- dence from deep brain stimulation for Parkinson’s disease.
Osterrieth Complex Figure among depressed elderly patients. Neuropsychologia 47:2828–2834. doi:10.1016/j.neuropsycholo
J Clin Exp Neuropsychol 26:598–607. doi:10.1080/138033904 gia.2009.06.011
90504425$16.00 32. Mirabella G, Iaconelli S, Modugno N, Giannini G, Lena F,
15. Kim M, Namgoong Y, Youn T (2008) Effect of organizational Cantore G (2013) Stimulation of subthalamic nuclei restores a
strategy on visual memory in patients with schizophrenia. Eur near normal planning strategy in Parkinson’s patients. PLoS One
8(5):e62793. doi:10.1371/journal.pone.0062793
Arch Psychiatry Clin Neurosci 62:427–434. doi:10.1111/j.1440-
1819.2008.01821.x 33. Wylie SA, Claassen DO, Huizenga HM, Schewel KD, Rid-
16. Jang HG, Kim HS, Ha TH et al (2010) Nonverbal memory and derinkhof KR, Bashore TR, van den Wildenberg WPW (2012)
organizational dysfunctions are related with distinct symptom Dopamine Agonists and the suppression of impulsive motor
actions in Parkinson’s disease. J Cogn Neurosci 24:1709–1724.
dimensions in obsessive-compulsive disorder. Psychiatric Res
180:93–98. doi:10.1016/j.psychres.2010.04.016 doi:10.1162/jocn_a_00241
17. Javorsky DJ, Rosenbaum J, Stern RA (1999) Utility of the Boston 34. Mirabella G, Iaconelli S, Romanelli P, Modugno N, Lena F,
Qualitative Scoring System (BQSS) for the Rey-Osterrieth Manfredi M, Cantore G (2012) Deep brain stimulation of sub-
Complex Figure in the evaluation of traumatic brain injury. Arch thalamic nuclei affects arm response inhibition in Parkinson’s

123
Neurol Sci

patients. Cereb Cortex 22(5):1124–1132. doi:10.1093/cercor/ 36. Alegret M, Vendrell P, Junqu C, Valldeoriola F, Tolosa E (2001)
bhr187 Visuospatial deficits in Parkinsons disease assessed by judgment
35. Owen AM (2004) Cognitive dysfunction in Parkinson’s disease: of line Orientation Test: error analyses and practice effects. J Clin
the role of frontostriatal circuitry. Neuroscientist 10:525–537. Exp Neuropsychol 23:592–598. doi:10.1076/jcen.23.5.592.1248
doi:10.1177/1073858404266776

123