On The Experience

Download as pdf or txt
Download as pdf or txt
You are on page 1of 20

Edinburgh Research Explorer

On the experience of conducting a Systematic Review in


Industrial, Work, and Organizational Psychology: Yes, it is
worthwhile

Citation for published version:


Rojon, C, McDowall, A & Saunders, MNK 2011, 'On the experience of conducting a Systematic Review in
Industrial, Work, and Organizational Psychology: Yes, it is worthwhile', Journal of Personnel Psychology,
vol. 10, no. 3, pp. 133-138. https://doi.org/10.1027/1866-5888/a000041

Digital Object Identifier (DOI):


10.1027/1866-5888/a000041

Link:
Link to publication record in Edinburgh Research Explorer

Document Version:
Peer reviewed version

Published In:
Journal of Personnel Psychology

Publisher Rights Statement:


© Hogrefe Publishing (2011). This article does not exactly replicate the final version published in the Journal of
Personnel Psychology. It is not a copy of the original published article and is not suitable for citation. DOI:
10.1027/1866-5888/a000041

General rights
Copyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)
and / or other copyright owners and it is a condition of accessing these publications that users recognise and
abide by the legal requirements associated with these rights.

Take down policy


The University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorer
content complies with UK legislation. If you believe that the public display of this file breaches copyright please
contact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately and
investigate your claim.

Download date: 06. Aug. 2023


Running head: SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY

On the Experience of Conducting a Systematic Review in Industrial, Work and Organizational

Psychology: Yes, It Is Worthwhile

Abstract

Systematic Review methodology (SRm) is an increasingly popular choice for literature reviews

in the Social Sciences. Although, compared to traditional narrative reviews SRm appears time-

consuming and laborious, transparency and replicability of the methodology is argued to

facilitate greater clarity of review. Nevertheless, researchers in Industrial, Work and

Organizational (IWO) Psychology have yet to embrace this methodology. Drawing on

experience from conducting a Systematic Review (SR) of individual workplace performance we

explore the premise: The advantages of SRm to IWO Psychology researchers outweigh the

disadvantages. We offer observations, insights and potential solutions to challenges faced during

the reviewing process, concluding that SRm is worthwhile for IWO Psychology researchers.

Keywords: Systematic Review; literature review; evidence based research; performance


SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 2

Prior to embarking on a new research project, an initial exploration of what is already

known is crucial, supporting an informed decision about the focus and execution of future

studies. Usually, this involves reviewing the literature on the topic; a range of possible review

strategies (cf. Denyer, 2009; Jesson, Matheson & Lacey, 2011) being available to Industrial,

Work and Organizational (IWO) Psychology researchers, including the Systematic Review

methodology (SRm). SRm is argued to offer advantages over traditional literature reviews,

allowing sense to be made of large bodies of information whilst minimizing bias (Petticrew &

Roberts, 2006), and has already gained acceptance in the Social Sciences (Harlen & Crick,

2004). Yet, IWO Psychology researchers have still to embrace SRm, a search of 13 relevant

journals (e.g. Journal of Occupational and Organizational Psychology, Journal of Personnel

Psychology) eliciting no published systematic reviews (SRs). We therefore examine the premise

that the advantages of SRm to IWO Psychology researchers outweigh the disadvantages, offering

observations and insights from our experience of conducting a SR of individual workplace

performance; addressing the question “is it worth it?”

What is Systematic Review methodology?

SRm is defined as:

a specific methodology that locates existing studies, selects and evaluates contributions,

analyses and synthesizes data, and reports the evidence in such a way that allows

reasonably clear conclusions to be reached about what is and is not known (Denyer &

Tranfield, 2009, p. 671).


SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 3

Originating in the medical sciences, SRm has been used widely, often to evaluate the

effectiveness of specific therapies or treatments (Leucht, Kissling & Davis, 2009). During the

last two decades, the methodology’s importance has been recognized by other disciplines:

Within Social Sciences, the Evidence for Policy and Practice Information and Coordinating

Centre was established in 1995, utilizing SRm to provide more evidence-based guidance for

policy and practice (Harlen & Crick, 2004). More recently, the value of SRm for evidence-based

research has been acknowledged by Management and Organization Sciences (MOS) researchers,

arguing it is a “key methodology for locating, appraising, synthesizing, and reporting ‘best

evidence’” (Briner, Denyer & Rousseau, 2009, p. 24), supporting its potential for IWO

Psychology.

Reasons for Using Systematic Review methodology

SRm is distinguished from traditional narrative and other forms of literature review in

two interrelated ways (Petticrew & Roberts, 2006). Firstly, SRm adheres closely to a set of

processes to limit researcher bias through attempting to identify, appraise and synthesize all

studies relevant to the research question(s). Secondly, these processes are defined in advance and

reported in sufficient detail to enable replication.

However, the key question for IWO Psychology researchers is: What advantages does

this methodology for reviewing the literature offer? This is important given the availability of

alternatives including meta-narrative approaches (e.g. Greenhalgh et al., 2005), critical appraisals

(e.g. Hill & Spittlehouse, 2003) and realist reviews (e.g. Pawson, Greenhalgh, Harvey & Walshe,

2005). With regards to our question, SRm literature highlights distinct advantages over such

approaches, summarized in Table 1.


SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 4

** Table 1 about here**

Yet literature also acknowledges SRm as laborious and time-consuming. The average SR

requires seven months’ work by a team of reviewers (Allen & Olkin, 1999; cited in Petticrew &

Roberts, 2006), emphasizing the need to answer the question “is it worthwhile?” We undertake

this by drawing on experiences of a transdisicplinary SR of individual workplace performance

combining literatures from IWO Psychology and MOS. Our motivation for this SR was to

provide an up-to-date, exhaustive integration of the available research evidence to inform

researchers and practitioners alike, since previous reviews were undertaken mostly to the 1990s

(e.g. Arvey & Murphy, 1998) addressing the topic less comprehensively than achieved here.

Whilst we considered the structured and meticulous procedures applied in SRm a particularly

suitable aid in consolidating the ample, heterogeneous literature available; it is useful to explore

the extent the advantages of SRm outweigh the disadvantages.

Method

The focus and conduct of SRm varies between disciplines. Petticrew and Roberts (2006)

have developed guidelines for the Social Sciences, using SRm to address evidence about a

variety of questions, rather than focusing solely on cause-effect relationships as emphasized in

medical sciences. In MOS research, Denyer and Tranfield (2009) have adapted medical sciences’

guidance to suit their discipline, offering four (amended) methodological principles: i)

transparent, ii) inclusive, iii) explanatory and iv) heuristic. Both sets of guidelines can be

integrated as a process consisting of a scoping study followed by five discrete review stages.

These we now outline and illustrate using the review topic Individual Workplace Performance,

reflecting on challenges encountered and offering possible solutions.


SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 5

0 Pre-Review Scoping Study

A scoping study typically precedes the actual review to determine the basis of the

literature search, ascertaining if a review is actually needed or if it would be mere replication.

This is of particular importance for cross-disciplinary topics such as individual workplace

performance due to the dispersed spread of evidence. An a-priori search within relevant SR

databases (e.g. Cochrane Database of Systematic Reviews) indicated no previous reviews had

been undertaken on this topic, indicating a possible need. To identify the precise focus for the

review, we undertook an initial, exploratory search for relevant literature. This revealed diverse

understandings of the individual workplace performance construct, in particular its definition,

conceptualization and measurement (for example is the construct uni- or multi-dimensional?).

These issues provided the focus for our subsequent review.

1 Determination of Specific Review Questions

Clearly framed, answerable research questions provide the basis for selecting potentially

relevant studies for a SR. As recommended (Denyer & Tranfield, 2009; Petticrew & Roberts,

2006), we consulted an advisory panel of ten experts (non-probability maximum variation

sample). These comprised Psychology and MOS academics with research foci in workplace

performance (e.g. professors of Human Resource Management (HRM) and Occupational

Psychology), chosen on the grounds of having specific expertise on the topi; alongside private

and public sector HRM practitioners (e.g. an Organisational Development Manager in

government administration) to provide a practitioner perspective, based on real-world experience

of performance management.
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 6

We conducted semi-structured interviews with these heterogeneous stakeholders asking broad

open questions to ensure that the review questions would remain aligned with our focus, as

elicited from the initial scoping stage, and useful to a wider community.

Qualitative content analysis of these interviews involved examination and interpretation

of responses focusing on main themes to firm up review questions from the experts’ suggestions.

Reconciling different stakeholder perspectives proved challenging, especially integrating

academic and practitioner views; we resolved this issue by ensuring a balance in the review

questions. Feedback from the expert panel indicated that the final research questions were

sufficiently focused to allow new meaningful theoretical insights, whilst comprehensive to

inform practical performance contexts, namely:

1. How is individual workplace performance as a criterion defined and conceptualized?

2. How is individual workplace performance measured and why?

3. What are the relationships, if any, between overall versus criterion-specific measures of

workplace performance and established predictors (i.e. ability and personality measures)?

2 Searching the Literature

We undertook a comprehensive search to locate all studies potentially relevant to these

review questions. The challenge was to ensure that potential key references were not excluded.

Tailored search strings combined terms relating to the research topic; for example, to find

references pertaining to ‘measurement’, the string “assess* OR apprais* OR evaluat* OR test

OR rating OR review OR measure OR manage was used*”, the asterisk enabling searching on

truncated word forms. Pilot database searches proved useful in determining the utility of such

strings and specifying a start date.


SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 7

To help ensure maximum saturation, twelve databases and proceedings from four

conferences (e.g. Society for Industrial and Organizational Psychology Conference) were used

following a subject librarian’s advice. Further manual searches (for three journals inaccessible

through the databases, e.g. ‘Assessment & Development Matters’) were undertaken and requests

sent to scholars with relevant research interests. After removal of duplicates, this resulted in

59,465 references (Table 2).

**Table 2 about here**

3 Selecting and Evaluating References

References were screened initially by title for relevance to the three review questions,

reducing their number to 3,010. Subsequent screening by title and abstract reduced their number

to 314. Having obtained full text copies of all, these were read and evaluated using

inclusion/exclusion criteria (Table 3) derived from researchers’ (mostly with experience in SRm)

suggestions (Briner et al., 2009; Cassell, 2010; Denyer, 2009) – this being a time-consuming

process, taking approximately three months. Despite precisely defined criteria, digression was a

potential challenge; being minimized through constant focus on the review questions. The 171

publications that met the inclusion criteria of satisfactory quality and contributing to answering

the review questions were our final pool of references (Petticrew & Roberts, 2006). For each

study key information (e.g. study context, data collection methods, findings in relation to the

review questions) was recorded using a data extraction form.

** Table 3 about here**

4 Analyzing and Synthesizing Findings


SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 8

SRm literature (e.g. Petticrew & Roberts, 2006; Rousseau, Manning & Denyer, 2008)

indicates a variety of processes for synthesizing evidence recorded on data extraction forms (e.g.

synthesis by explanation or aggregation), depending on the type of review questions asked and

the available data. For the first two review questions, evidence was synthesized qualitatively

through narrative integration, involving comparison and corroboration (Rousseau et al, 2008).

The third review question was addressed quantitatively through aggregation, using statistical

meta-analysis (Hunter & Schmidt, 2004). This combination of analytic methods allowed full

integration of evidence considered.

5 Discussion and Utilization of the Findings

Review findings addressed all three questions accentuating what was known, what was

not known, where future research should focus and how this might inform policy and practice.

Overall, the SR process was laborious, taking nine months and offering a potential challenge

regarding loss of motivation. We avoided this through regular discussions with other systematic

reviewers.

Results

Some 84.8% of the 171 publications used to address the three review questions were

peer-reviewed journal articles, having been published in a wide variety of journals (N=52) (Table

4), over half being published in six journals. Documents included in the SR were published

between 1959 and 2010, with more than 75% published in the last 20 years. Over half (53.2%) of

publications included were considered of high overall quality as defined through the

inclusion/exclusion criteria (Table 3). 19.3% of publications addressed at least two of the review

questions. Questions 1, 2 and 3 were addressed respectively by 27.5%, 36.3% and 49.7% of

publications, indicating that how individual workplace performance was defined and
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 9

conceptualized as a criterion was the least researched aspect, analysis of the data extraction

forms enabling identification of knowledge gaps.

**Table 4 about here**

Discussion

This note has explored the premise that the advantages of SRm to IWO Psychology

researchers outweigh the disadvantages, addressing the question “is it worth it?” Based on the

experiences outlined we consider that, despite SRm being laborious and time-consuming for

establishing best evidence, both process and outcome are worthwhile given the effort required.

Firstly, the rigor and standardization of SRm results in greater transparency, explicitness and

replicability than may be achieved through traditional narrative reviews. It is suited to topics

such as workplace performance, where different disciplines and theoretical and practical

orientations need to be integrated. Secondly, using an expert panel of academics and

professionals to help determine review questions facilitates theoretical rigor and practical

relevance in the review. Thirdly, using precise inclusion/exclusion criteria alongside continued

focus on review questions ensures quality whilst minimizing digression.

**Table 5 about here**

Invariably SRm, regardless of the review topic, presents challenges. In our case, a major

challenge, which reviewers with narrower or less commonly researched questions might not face,

was dealing with the large number of references. Further challenges addressed during our SR,

which we believe fellow researchers in IWO Psychology and neighboring disciplines (e.g. HRM)

might encounter, alongside solutions and challenges experienced by reviewers in different


SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 10

disciplines, are summarized in Table 5. Moreover, individuals planning a SR in similar contexts

are advised to ensure stakeholders from constituent disciplines are involved in the scoping study

and determination of specific review questions. SRm is not a universal process to be applied to

all literature reviews. Rather, the purpose is to establish current best evidence regarding specified

research questions, thereby allowing more informed decisions about future studies. It is

therefore necessary to ensure the review focus is appraisal, synthesis and reporting of existing

evidence. While this focus and thus SR may not always be appropriate, this note offers a better

understanding of the utility of this methodology. We have found the experience of conducting a

SR rewarding, owing to the structured approach taken, skills acquired, and increased confidence

from a sound understanding of extant literature. We would therefore encourage IWO Psychology

researchers to embrace and use SRm to their advantage to review best evidence from existing

knowledge.
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 11

References

Arvey, R. D., & Murphy, K. R. (1998). Performance evaluation in work settings. Annual Review

of Psychology, 49, 141-168.

Briner, R. B., Denyer, D., & Rousseau, D. M. (2009). Evidence-based management: Concept

cleanup time? Academy of Management Perspectives, 23(4), 19-32.

Briner, R. B., & Rousseau, D. M. (2011). Evidence-based I-O psychology: Not there yet.
Industrial and Organizational Psychology: Perspectives on Science and Practice, 4(1),
3-22.
Cassell, C. (2010, January). Criteria for evaluating papers using qualitative research methods.

Journal of Occupational and Organizational Psychology. Retrieved from

http://www.bpsjournals.co.uk/journals/joop/qualitative-guidelines.cfm

Denyer, D. (2009, October). Reviewing the literature systematically. Cranfield: Advanced

Institute of Management Research (AIM). Retrieved from http://aimexpertresearcher.org/

Denyer, D., & Tranfield, D. (2009). Producing a systematic review. In D. Buchanan & A.

Bryman (Eds.), The Sage handbook of organizational research methods (pp. 671-689).

London, United Kingdom: Sage.

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O., & Peacock, R. (2005).

Storylines of research in diffusion of innovation: A meta-narrative approach to systematic

review. Social Science & Medicine, 61, 417-430.

Harlen, W., & Crick, R. D. (2004). Opportunities and challenges of using systematic reviews of

research for evidence-based policy in education. Evaluation & Research in Education,

18, 54-71.

Hill, A., & Spittlehouse, C. (2010, October). What is critical appraisal? Evidence Based

Medicine, 3(2), 1-8. Retrieved from http://www.whatisseries.co.uk/whatis/


SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 12

pdfs/What_is_crit_appr.pdf

Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting for error and bias

in research findings (2nd ed.). Thousand Oaks, CA: Sage.

Jesson, J. K., Matheson, L., & Lacey, F. M. (2011). Doing Your Literature Review: Traditional

and Systematic Techniques. London, United Kingdom: Sage.

Leucht, S., Kissling, W., & Davis, J. M. (2009). How to read and understand and use systematic

reviews and meta-analyses. Acta Psychiatrica Scandinavica, 119, 443-450.

Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review – a new method of

systematic review designed for complex policy interventions. Journal of Health Services

Research & Policy, 10(1), 21-34.

Petticrew, M, & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide.

Oxford, United Kingdom: Blackwell Publishing.

Rousseau, D. M., Manning, J., & Denyer, D. (2008). Evidence in management and

organizational science: Assembling the field’s full weight of scientific knowledge

through syntheses. The Academy of Management Annals, 2(1), 475-515.

Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011). Methodology in conducting a

systematic review of systematic reviews of healthcare interventions. BMC Medical

Research Methodology, 11(1), 15-20.


SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 13

Table 1

Advantages of Systematic Review methodology compared to other reviewing approaches

SRm Traditional narrative review Meta-analysis

(e.g. Briner & Rousseau, 2011; (Jesson, Matheson & Lacey, (Hunter & Schmidt, 2004)

Denyer & Tranfield, 2009; 2011)

Petticrew & Roberts, 2006)

Rigor, thoroughness and Does not generally have a formal Once appropriate meta-analysis

objectivity through adherence to methodology, thus resulting in method has been chosen (e.g.

clear principles and prescribed lacking rigor, transparency and statistical meta-analysis), it is

stages of reviewing replicability by others; but: crucial to illustrate clearly

allows more flexibility and process of locating, evaluating,

exploration of researcher’s ideas selecting and coding studies to

allow replicabilty

Consideration and reconciliation Researcher can focus on Researcher can be very selective

of all potentially relevant sources ‘preferred’ literature sources (e.g. as to which studies to include in

of information allows favorite databases) and base their meta-analyses, thus

comprehensive collation of all review on a personal, purposive potentially introducing researcher

existing evidence across relevant selection of materials they bias; not always made clear

studies and integration of believe to be important, thus enough why some studies have

different schools of thought and potentially introducing a one- been included whilst others have

research findings and is sided or even biased argument not

particularly suitable when aware

of main themes concerning the

review topic, but unsure of actual

evidence
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 14

Facilitation of reviews of topics There is not usually a review Can facilitate the quantitative

where a vast and heterogeneous protocol or strategy or a defined review of areas within

body of literature is available by method to follow, which can Psychology and other Social

following an a priori developed make it difficult to identify and Sciences in which number of

protocol that clearly states tasks review topics where a vast and available studies is large and

and stages of the reviewing heterogeneous body of literature findings seem contradictory by

process (e.g. selection and is available adhering to statistical and

evaluation of references by psychometric principles of data

means of inclusion/exclusion analysis in meta-analysis

criteria)

Combination of analysis and Whilst review may contain a Whilst review may contain

synthesis methods possible, i.e. meta-analysis, it is typically narrative element, it is typically

can include both a narrative and a focused on the narrative focused on the meta-analysis

meta-analysis component component (qualitative synthesis) component (quantitative

synthesis)
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 15

Table 2

Number of References Identified in Systematic Review Stages 2 and 3

Stage Source/Process Number of potentially relevant references

2 Searching the literature

Web of Science database 35,173

PsycInfo database 11,381

Business Source Complete database 9,079

Medline database 4,145

Emerald Management eJournals database 2,200

IBSS (International Bibliography of the Social Sciences) database 874

Psychology & Behavioral Sciences Collection database 708

AOM (Academy of Management) Conference Proceedings database 301

Psybooks database 54

CIPD (Chartered Institute of Personnel and Development) Research Summaries database 34

Manual searches (3 journals, 4 conferences, scholars’ literature) 29

British Library e-Theses database 23

I&DeA (Improvement and Development Agency for Local Government) database 5

Total (prior to removal of duplicates) 64,006

Total (after removal of duplicates) 59,465

3 Selecting and evaluating references

Screening by title alone 3,010

Screening by title and abstract 314

Evaluation by full text 171


SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 16

Table 3

Sample Inclusion/Exclusion Criteria for Study Selection

Criterion Explanation of criterion

Does the study address any/ Only studies that can contribute to answering any or all of the review

all of the review’s questions? questions are useful for the systematic review; all others are excluded

Is it relevant to any/all of the outright

review questions?

Is the study well informed by Outlining of previous findings and existing theory and integration

existing theory? within study (cf. Cassell, 2010; Denyer, 2009)

Are the purpose and aims of Clear specification of the research questions and objectives addressed

the study clearly specified? (cf. Denyer, 2009)

Are the methods chosen Clear explanation of what methods of data collection were chosen and

appropriate to the stated why (cf. Cassell, 2010; Denyer, 2009)

purpose?

Does the study claim a Study creates, extends or advances knowledge in a meaningful way.

contribution? Guidance for future research is provided

Is the study relevant for the Usefulness and applicability of the results for a practitioner, such as an

practice? IWO psychologist, a human resources manager etc. Author comments

on how this is the case (cf. Cassell, 2010; Denyer, 2009)

Are the conclusions well Reference back to initially formulated research questions and aims;

linked to the purpose and establishment of clear links (Cassell, 2010).

aims of the research?


SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 17

Table 4

Descriptive Statistics of the Publications Used in the Review (N=171)

Type of %/N Peer-reviewed %/N Year of %/N Quality %/N


document journals publication
Peer- 84.8/145 Journal of 17.0/29 2001-2010 35.1/60 High 53.2/91
reviewed Applied overall
journal Psychology quality
articles
Non-peer- 1.2/2 Personnel 9.4/16 1991-2000 40.9/70 Above 25.1/43
reviewed Psychology average
journal overall
articles quality
Journal 1.2/2 International 7.6/13 1981-1990 10.5/18 Average 21.6/37
articles Journal of overall
(with peer- Selection and quality
review Assessment
status
unclear)
Book 9.4/16 Human 7.0/12 1971-1980 9.4/16
chapters Performance
Doctoral 2.3/4 Journal of 4.7/8 1961-1970 3.5/6
theses Occupational
and
Organizational
Psychology
Conference 1.2/2 Journal of 4.7/8 1951-1960 0.6/1
proceedings Business &
Psychology
Other journals 41.1/59

Note. An overall quality score ranging from 1 (low overall quality) to 5 (high overall quality) was obtained for each
publication – this was based on the application of previously determined inclusion/exclusion criteria relating to
quality (Table 3).
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 18

Table 5
Potential Challenges and Solutions: SRs in IWO Psychology (#) and other disciplines (*)

Stage Potential challenge Potential solution

1 Expert consultation to - Involve a heterogeneous sample of stakeholders

reflect on the review topic - Ask open, broad interview questions

and to determine review - Ensure resulting review questions are comprehensive, yet

question(s) # sufficiently focused

- Justify the role of each stakeholder in the expert panel

Analysis and reconciliation Whilst review questions may have a more academic focus, answers

of different stakeholder need to be formulated in a way meaningful to both

perspectives # academics/researchers and practical contexts

2 Development of - Discuss planned literature searches with librarian

appropriate search - Include any terms in the search strings that are related to the

terms/strings review question(s)

- Conduct pilot searches

3&4 Reference management - Accept only relevant studies of a satisfactory quality

- Establish well defined inclusion/exclusion criteria

- Focus on review question(s) throughout to avoid digression

- Allow sufficient time to acquire and assess references

Stringent - SRs in healthcare (such as Smith et al., 2011) may restrict their

inclusion/exclusion criteria inclusion criteria to studies that are randomized controlled trials;

(Smith, Devane, Begley & such criteria are unlikely to be useful to SRs in IWO Psychology as

Clarke, 2011) * they would delimit the scope of the review too strongly

Across Time management - Expect that most tasks will take longer than anticipated

all - Acquiring new knowledge and skills for SRm will take time, too
SYSTEMATIC REVIEWS IN IWO PSYCHOLOGY 19

stages Dwindling motivation - Discuss process and progress with other systematic reviewers

Large number of SRs of - SRm being a new approach to reviewing literature in IWO

variable quality scope on Psychology, this aspect is unlikely to be a problem to researchers in

important topics (Smith et this discipline; in the case of Smith and colleagues (2011), a SR of

al., 2011) * SRs is suggested to overcome this particular problem

Note. Challenges experiences by researchers in disciplines other than IWO Psychology have been marked with an
asterisk (*), those applicable to SRs in IWO Psychology and neighboring disciplines with a hash (#). Challenges that
are unmarked are considered applicable to SRs in any discipline.

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy