On The Experience
On The Experience
On The Experience
Link:
Link to publication record in Edinburgh Research Explorer
Document Version:
Peer reviewed version
Published In:
Journal of Personnel Psychology
General rights
Copyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)
and / or other copyright owners and it is a condition of accessing these publications that users recognise and
abide by the legal requirements associated with these rights.
Abstract
Systematic Review methodology (SRm) is an increasingly popular choice for literature reviews
in the Social Sciences. Although, compared to traditional narrative reviews SRm appears time-
explore the premise: The advantages of SRm to IWO Psychology researchers outweigh the
disadvantages. We offer observations, insights and potential solutions to challenges faced during
the reviewing process, concluding that SRm is worthwhile for IWO Psychology researchers.
known is crucial, supporting an informed decision about the focus and execution of future
studies. Usually, this involves reviewing the literature on the topic; a range of possible review
strategies (cf. Denyer, 2009; Jesson, Matheson & Lacey, 2011) being available to Industrial,
Work and Organizational (IWO) Psychology researchers, including the Systematic Review
methodology (SRm). SRm is argued to offer advantages over traditional literature reviews,
allowing sense to be made of large bodies of information whilst minimizing bias (Petticrew &
Roberts, 2006), and has already gained acceptance in the Social Sciences (Harlen & Crick,
2004). Yet, IWO Psychology researchers have still to embrace SRm, a search of 13 relevant
Psychology) eliciting no published systematic reviews (SRs). We therefore examine the premise
that the advantages of SRm to IWO Psychology researchers outweigh the disadvantages, offering
a specific methodology that locates existing studies, selects and evaluates contributions,
analyses and synthesizes data, and reports the evidence in such a way that allows
reasonably clear conclusions to be reached about what is and is not known (Denyer &
Originating in the medical sciences, SRm has been used widely, often to evaluate the
effectiveness of specific therapies or treatments (Leucht, Kissling & Davis, 2009). During the
last two decades, the methodology’s importance has been recognized by other disciplines:
Within Social Sciences, the Evidence for Policy and Practice Information and Coordinating
Centre was established in 1995, utilizing SRm to provide more evidence-based guidance for
policy and practice (Harlen & Crick, 2004). More recently, the value of SRm for evidence-based
research has been acknowledged by Management and Organization Sciences (MOS) researchers,
arguing it is a “key methodology for locating, appraising, synthesizing, and reporting ‘best
evidence’” (Briner, Denyer & Rousseau, 2009, p. 24), supporting its potential for IWO
Psychology.
SRm is distinguished from traditional narrative and other forms of literature review in
two interrelated ways (Petticrew & Roberts, 2006). Firstly, SRm adheres closely to a set of
processes to limit researcher bias through attempting to identify, appraise and synthesize all
studies relevant to the research question(s). Secondly, these processes are defined in advance and
However, the key question for IWO Psychology researchers is: What advantages does
this methodology for reviewing the literature offer? This is important given the availability of
alternatives including meta-narrative approaches (e.g. Greenhalgh et al., 2005), critical appraisals
(e.g. Hill & Spittlehouse, 2003) and realist reviews (e.g. Pawson, Greenhalgh, Harvey & Walshe,
2005). With regards to our question, SRm literature highlights distinct advantages over such
Yet literature also acknowledges SRm as laborious and time-consuming. The average SR
requires seven months’ work by a team of reviewers (Allen & Olkin, 1999; cited in Petticrew &
Roberts, 2006), emphasizing the need to answer the question “is it worthwhile?” We undertake
combining literatures from IWO Psychology and MOS. Our motivation for this SR was to
researchers and practitioners alike, since previous reviews were undertaken mostly to the 1990s
(e.g. Arvey & Murphy, 1998) addressing the topic less comprehensively than achieved here.
Whilst we considered the structured and meticulous procedures applied in SRm a particularly
suitable aid in consolidating the ample, heterogeneous literature available; it is useful to explore
Method
The focus and conduct of SRm varies between disciplines. Petticrew and Roberts (2006)
have developed guidelines for the Social Sciences, using SRm to address evidence about a
medical sciences. In MOS research, Denyer and Tranfield (2009) have adapted medical sciences’
transparent, ii) inclusive, iii) explanatory and iv) heuristic. Both sets of guidelines can be
integrated as a process consisting of a scoping study followed by five discrete review stages.
These we now outline and illustrate using the review topic Individual Workplace Performance,
A scoping study typically precedes the actual review to determine the basis of the
performance due to the dispersed spread of evidence. An a-priori search within relevant SR
databases (e.g. Cochrane Database of Systematic Reviews) indicated no previous reviews had
been undertaken on this topic, indicating a possible need. To identify the precise focus for the
review, we undertook an initial, exploratory search for relevant literature. This revealed diverse
Clearly framed, answerable research questions provide the basis for selecting potentially
relevant studies for a SR. As recommended (Denyer & Tranfield, 2009; Petticrew & Roberts,
sample). These comprised Psychology and MOS academics with research foci in workplace
Psychology), chosen on the grounds of having specific expertise on the topi; alongside private
of performance management.
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 6
open questions to ensure that the review questions would remain aligned with our focus, as
elicited from the initial scoping stage, and useful to a wider community.
of responses focusing on main themes to firm up review questions from the experts’ suggestions.
academic and practitioner views; we resolved this issue by ensuring a balance in the review
questions. Feedback from the expert panel indicated that the final research questions were
3. What are the relationships, if any, between overall versus criterion-specific measures of
workplace performance and established predictors (i.e. ability and personality measures)?
review questions. The challenge was to ensure that potential key references were not excluded.
Tailored search strings combined terms relating to the research topic; for example, to find
OR rating OR review OR measure OR manage was used*”, the asterisk enabling searching on
truncated word forms. Pilot database searches proved useful in determining the utility of such
To help ensure maximum saturation, twelve databases and proceedings from four
conferences (e.g. Society for Industrial and Organizational Psychology Conference) were used
following a subject librarian’s advice. Further manual searches (for three journals inaccessible
through the databases, e.g. ‘Assessment & Development Matters’) were undertaken and requests
sent to scholars with relevant research interests. After removal of duplicates, this resulted in
References were screened initially by title for relevance to the three review questions,
reducing their number to 3,010. Subsequent screening by title and abstract reduced their number
to 314. Having obtained full text copies of all, these were read and evaluated using
inclusion/exclusion criteria (Table 3) derived from researchers’ (mostly with experience in SRm)
suggestions (Briner et al., 2009; Cassell, 2010; Denyer, 2009) – this being a time-consuming
process, taking approximately three months. Despite precisely defined criteria, digression was a
potential challenge; being minimized through constant focus on the review questions. The 171
publications that met the inclusion criteria of satisfactory quality and contributing to answering
the review questions were our final pool of references (Petticrew & Roberts, 2006). For each
study key information (e.g. study context, data collection methods, findings in relation to the
SRm literature (e.g. Petticrew & Roberts, 2006; Rousseau, Manning & Denyer, 2008)
indicates a variety of processes for synthesizing evidence recorded on data extraction forms (e.g.
synthesis by explanation or aggregation), depending on the type of review questions asked and
the available data. For the first two review questions, evidence was synthesized qualitatively
through narrative integration, involving comparison and corroboration (Rousseau et al, 2008).
The third review question was addressed quantitatively through aggregation, using statistical
meta-analysis (Hunter & Schmidt, 2004). This combination of analytic methods allowed full
Review findings addressed all three questions accentuating what was known, what was
not known, where future research should focus and how this might inform policy and practice.
Overall, the SR process was laborious, taking nine months and offering a potential challenge
regarding loss of motivation. We avoided this through regular discussions with other systematic
reviewers.
Results
Some 84.8% of the 171 publications used to address the three review questions were
peer-reviewed journal articles, having been published in a wide variety of journals (N=52) (Table
4), over half being published in six journals. Documents included in the SR were published
between 1959 and 2010, with more than 75% published in the last 20 years. Over half (53.2%) of
publications included were considered of high overall quality as defined through the
inclusion/exclusion criteria (Table 3). 19.3% of publications addressed at least two of the review
questions. Questions 1, 2 and 3 were addressed respectively by 27.5%, 36.3% and 49.7% of
publications, indicating that how individual workplace performance was defined and
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 9
conceptualized as a criterion was the least researched aspect, analysis of the data extraction
Discussion
This note has explored the premise that the advantages of SRm to IWO Psychology
researchers outweigh the disadvantages, addressing the question “is it worth it?” Based on the
experiences outlined we consider that, despite SRm being laborious and time-consuming for
establishing best evidence, both process and outcome are worthwhile given the effort required.
Firstly, the rigor and standardization of SRm results in greater transparency, explicitness and
replicability than may be achieved through traditional narrative reviews. It is suited to topics
such as workplace performance, where different disciplines and theoretical and practical
professionals to help determine review questions facilitates theoretical rigor and practical
relevance in the review. Thirdly, using precise inclusion/exclusion criteria alongside continued
Invariably SRm, regardless of the review topic, presents challenges. In our case, a major
challenge, which reviewers with narrower or less commonly researched questions might not face,
was dealing with the large number of references. Further challenges addressed during our SR,
which we believe fellow researchers in IWO Psychology and neighboring disciplines (e.g. HRM)
are advised to ensure stakeholders from constituent disciplines are involved in the scoping study
and determination of specific review questions. SRm is not a universal process to be applied to
all literature reviews. Rather, the purpose is to establish current best evidence regarding specified
research questions, thereby allowing more informed decisions about future studies. It is
therefore necessary to ensure the review focus is appraisal, synthesis and reporting of existing
evidence. While this focus and thus SR may not always be appropriate, this note offers a better
understanding of the utility of this methodology. We have found the experience of conducting a
SR rewarding, owing to the structured approach taken, skills acquired, and increased confidence
from a sound understanding of extant literature. We would therefore encourage IWO Psychology
researchers to embrace and use SRm to their advantage to review best evidence from existing
knowledge.
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 11
References
Arvey, R. D., & Murphy, K. R. (1998). Performance evaluation in work settings. Annual Review
Briner, R. B., Denyer, D., & Rousseau, D. M. (2009). Evidence-based management: Concept
Briner, R. B., & Rousseau, D. M. (2011). Evidence-based I-O psychology: Not there yet.
Industrial and Organizational Psychology: Perspectives on Science and Practice, 4(1),
3-22.
Cassell, C. (2010, January). Criteria for evaluating papers using qualitative research methods.
http://www.bpsjournals.co.uk/journals/joop/qualitative-guidelines.cfm
Denyer, D., & Tranfield, D. (2009). Producing a systematic review. In D. Buchanan & A.
Bryman (Eds.), The Sage handbook of organizational research methods (pp. 671-689).
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005).
Harlen, W., & Crick, R. D. (2004). Opportunities and challenges of using systematic reviews of
18, 54-71.
Hill, A., & Spittlehouse, C. (2010, October). What is critical appraisal? Evidence Based
pdfs/What_is_crit_appr.pdf
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting for error and bias
Jesson, J. K., Matheson, L., & Lacey, F. M. (2011). Doing Your Literature Review: Traditional
Leucht, S., Kissling, W., & Davis, J. M. (2009). How to read and understand and use systematic
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review – a new method of
systematic review designed for complex policy interventions. Journal of Health Services
Petticrew, M, & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide.
Rousseau, D. M., Manning, J., & Denyer, D. (2008). Evidence in management and
Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011). Methodology in conducting a
Table 1
(e.g. Briner & Rousseau, 2011; (Jesson, Matheson & Lacey, (Hunter & Schmidt, 2004)
Rigor, thoroughness and Does not generally have a formal Once appropriate meta-analysis
objectivity through adherence to methodology, thus resulting in method has been chosen (e.g.
clear principles and prescribed lacking rigor, transparency and statistical meta-analysis), it is
allow replicabilty
Consideration and reconciliation Researcher can focus on Researcher can be very selective
of all potentially relevant sources ‘preferred’ literature sources (e.g. as to which studies to include in
existing evidence across relevant selection of materials they bias; not always made clear
studies and integration of believe to be important, thus enough why some studies have
different schools of thought and potentially introducing a one- been included whilst others have
evidence
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 14
Facilitation of reviews of topics There is not usually a review Can facilitate the quantitative
where a vast and heterogeneous protocol or strategy or a defined review of areas within
body of literature is available by method to follow, which can Psychology and other Social
following an a priori developed make it difficult to identify and Sciences in which number of
protocol that clearly states tasks review topics where a vast and available studies is large and
and stages of the reviewing heterogeneous body of literature findings seem contradictory by
criteria)
Combination of analysis and Whilst review may contain a Whilst review may contain
can include both a narrative and a focused on the narrative focused on the meta-analysis
synthesis)
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 15
Table 2
Psybooks database 54
Table 3
Does the study address any/ Only studies that can contribute to answering any or all of the review
all of the review’s questions? questions are useful for the systematic review; all others are excluded
review questions?
Is the study well informed by Outlining of previous findings and existing theory and integration
Are the purpose and aims of Clear specification of the research questions and objectives addressed
Are the methods chosen Clear explanation of what methods of data collection were chosen and
purpose?
Does the study claim a Study creates, extends or advances knowledge in a meaningful way.
Is the study relevant for the Usefulness and applicability of the results for a practitioner, such as an
Are the conclusions well Reference back to initially formulated research questions and aims;
Table 4
Note. An overall quality score ranging from 1 (low overall quality) to 5 (high overall quality) was obtained for each
publication – this was based on the application of previously determined inclusion/exclusion criteria relating to
quality (Table 3).
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 18
Table 5
Potential Challenges and Solutions: SRs in IWO Psychology (#) and other disciplines (*)
and to determine review - Ensure resulting review questions are comprehensive, yet
Analysis and reconciliation Whilst review questions may have a more academic focus, answers
appropriate search - Include any terms in the search strings that are related to the
Stringent - SRs in healthcare (such as Smith et al., 2011) may restrict their
inclusion/exclusion criteria inclusion criteria to studies that are randomized controlled trials;
(Smith, Devane, Begley & such criteria are unlikely to be useful to SRs in IWO Psychology as
Clarke, 2011) * they would delimit the scope of the review too strongly
Across Time management - Expect that most tasks will take longer than anticipated
all - Acquiring new knowledge and skills for SRm will take time, too
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 19
stages Dwindling motivation - Discuss process and progress with other systematic reviewers
Large number of SRs of - SRm being a new approach to reviewing literature in IWO
important topics (Smith et this discipline; in the case of Smith and colleagues (2011), a SR of
Note. Challenges experiences by researchers in disciplines other than IWO Psychology have been marked with an
asterisk (*), those applicable to SRs in IWO Psychology and neighboring disciplines with a hash (#). Challenges that
are unmarked are considered applicable to SRs in any discipline.
Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.
Alternative Proxies: