Critical Appraisal of Studies Using Laboratory Animal Models

Annette M. O’Connor and Jan M. Sargeant

Abstract from (source population). Further, it is often inferred that

the results can be applied to populations other than the source
In this manuscript we discuss an approach to critically ap- population (i.e., they can be applied to a target population). If

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

praising papers based on the results of laboratory animal ex- this inference is not made, the results are not useful to end-
periments. The roles of external and internal validity in users who wish to generalize the findings to reinforce con-
critically appraising the results of a paper are introduced. cepts already known, generate new hypotheses, develop
The risk of bias domains used by the Cochrane Handbook new research directions, or make policy based on the results.
of Systematic Reviews of Interventions form the basis for as-
sessing internal validity. The bias domains discussed include What Is Critical Appraisal?
the selection bias, performance bias, outcome assessment
bias, attrition bias, and reporting bias. Further, an approach Critical appraisal is an essential part of the scientific process
to considering the role of chance in research findings is designed to assess the validity of scientific findings. The unit
discussed. of assessment for critical appraisal is a single study, and the
approach to assessing a single study result is present here.
Key Words: animal models; critical appraisal; preclinical Critical appraisal is often conducted informally in a manner
research; bias; validity such that the rationale for a judgment is not clear. Here we
seek to provide a transparent and systematic approach to
Introduction critical appraisal. The approach presented here, in particular
the use of risk domains, is borrowed from the Cochrane
Handbook of Systematic Reviews of Interventions and trans-

P
reclinical assessment of interventions and a great deal
of pathogenic mechanism research is conducted using lated to animal studies in preclinical medicine (Higgins
animal models. Our understanding of the working of bi- et al. 2011).
ological systems, such as the immune and cardiovascular sys- Critical appraisal should be differentiated from an assess-
tems, is often based upon the results of research on animals. ment of comprehensive reporting. Comprehensive reporting
Many pharmaceutical interventions in human health are first simply assesses if the study is reported in a manner that in-
tested in animal studies for efficacy and safety. As such, ani- cludes the important components of a study. There is substan-
mal research is considered critical to the scientific endeavor. tial evidence that reporting of preclinical studies is less than
However, how well animal studies achieve this goal is a sub- comprehensive or has low reproducibility (Kilkenny et al.
ject of debate (van der Worp et al. 2010). 2009; Prinz et al. 2011; Steward et al. 2012). Based on this
Although it frequently seems to researchers that publica- evidence, there have been calls for improved reporting
tion is an endpoint, it is only an intermediary step in the sci- (Begley and Ellis 2012; Landis et al. 2012; van der Worp
entific process (Sargeant and O’Connor 2013). When authors and Macleod 2011). Further, there are several guidelines out-
describe results, they often make the inference that the results lining aspects of study design, analysis, and reporting that
from the study animals (study population) represent the re- should be included in any research report; examples include
sults expected from the population the study animals came the ARRIVE Guidelines (Kilkenny et al. 2010) and the
“Guidance for the Description of Animal Research in Scien-
tific Publications” (National Research Council [US] Institute
Annette M. O’Connor, BVSc, MVSc, DVSc, FANZCVS, is a veterinarian
and Professor of Epidemiology at the Veterinary Medical Research Institute,
for Laboratory Animal Research 2011). The target audience
College of Veterinary Medicine, Iowa State University, Ames, Iowa. Jan for these checklists or guidelines is generally authors, and the
M. Sargeant, DVM, PhD, is a veterinarian and Professor of Epidemiology aim is to provide guidance on to how to present the study.
in the Department of Population Medicine and Director of the Centre for Many, but not all, of the items in these checklists are related
Public Health and Zoonoses at the Ontario Veterinary College, University to enabling critical appraisal by the end-user, and for this rea-
of Guelph, Guelph, Ontario, Canada.
Address correspondence and reprint requests to Annette M. O’Connor,
son sometimes these checklists are used for critical appraisal.
Veterinary Medical Research Institute, Bld 4, College of Veterinary Medi- This is inappropriate. Assessing comprehensive reporting re-
cine, Iowa State University, Ames, IA 50010 or email oconnor@iastate.edu. quires an assessment of presence or absence of an item,

ILAR Journal, Volume 55, Number 3, doi: 10.1093/ilar/ilu038

© The Author 2014. Published by Oxford University Press on behalf of the Institute for Laboratory Animal Research. All rights reserved.
For permissions, please email: journals.permissions@oup.com 405
whereas critical appraisal requires judgment about validity. There is good empirical evidence that reporting of laboratory
For example, a study may report random allocation to treat- animal studies is not comprehensive (Banwell et al. 2009;
ment group and therefore have a “yes” for such an item on Crossley et al. 2008; Sena et al. 2007a, 2007b, 2010; van
a checklist. However, despite random allocation to group, der Worp et al. 2007; Wheble et al. 2008). End-users should
there may be important baseline differences in characteristics also be aware that even when authors do use the ARRIVE
across treatments groups that suggest a high risk of bias. Guidelines (or similar lists), the authors are not obliged to fol-
Checklists for comprehensive reporting are generally made low the checklist item order. For example, there is no strict
for larger areas of science and are based on a study design requirement that the number of study animals be reported in
such as assessments of interventions and diagnostic test the first part of the results. Based on historical, journal, disci-
evaluations (Ell 2003; O’Connor et al. 2010; The Lancet pline, or author’s preference, the methods and materials sec-
2010). These checklists can be used by authors across a vari- tion often contains this information. The reporting of the
ety of disciplines, because the standards for reporting across information is what is important, not its order in the

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

science are fairly standard. Critical appraisal, however, is manuscript.
more narrow and topic specific. For example, most scientists
can determine if the study contains the list of outcomes as-
sessed; however, only content experts can judge if the ap- The Components of Critical Appraisal
proach to measurement is valid and unlikely to introduce
If the results of studies are to be inferred to represent those
bias. This is why there are few “checklists” for critical ap-
from a source and target population, it is important to evaluate
praisal, and those that exist (like this one) recommend tailor-
the external validity and internal validity of the study. Exter-
ing the items to the topic. Although it is widely known what
nal validity refers to the extent to which the results can be in-
information is needed to assess bias, this is not synonymous
ferred to a target population. Internal validity refers to how the
with the presence of bias. The later is topic specific and re-
study results represent the source population. Critical apprais-
quires judgment.
al of a scientific report can evaluate both concepts.
After critical appraisal of individual papers, the next step is
to combine the results of many studies to reach a conclusion
about a body of work. Formal transparent approaches to this External Validity
combination are available but have infrequently been applied
to laboratory animal studies and are not reviewed here (Guyatt The concept of external validity is a difficult one in laboratory
et al. 2008, 2011; Johnson et al. 2014; Koustas et al. 2014; animal studies, because the target population is unclear and
Lam et al. 2014; Woodruff and Sutton 2014). varies by end-user. By the very nature of using animals as
models for basic biological functions, there is the implication
that the results are in some way generalizable to other popu-
Organization of the Remainder of This Paper lations, such as humans. However, often with laboratory an-
imal studies, the next step in the scientific process is to
This remainder of this manuscript is organized as follows. develop a new hypothesis to be tested in other animals, in
First, we discuss the components of validity and introduce which case the target population of the laboratory animal
the terms external validity, internal validity, bias, and bias do- study would be laboratory animal populations. Further, there
mains. We then discuss in greater detail the bias domains that is some empirical evidence in the literature suggesting that re-
affect internal validity that might affect animal experiments. sults from animal models do not generalize to “other” exter-
For each domain of bias, we present a hypothetical example nal populations of interest (Perel et al. 2007; Pound et al.
from an animal experiment publication that demonstrates an 2004; van der Worp et al. 2010). In this paper we assume
approach that would induce the bias. These examples are of- that the critical appraiser has already made the decision that
ten extreme to make them easier to understand. Next, we pro- the study results, if internally valid, will apply to a target pop-
vide a discussion of the rationale for why the risk domain ulation. We do not provide guidelines for assessing external
should be assessed when critically appraising a study. We validity other than to mention that information about external
then discuss how to evaluate the role of random error when validity often relates to the population studied.
conducting a critical appraisal. Finally, we provide comments
on how to reach a judgment about a paper based on the risk
domains and assessment of internal validity. Internal Validity
Throughout the paper, we provide guidance on where in
the manuscript a critical appraiser might expect to find the in- The most common aspect considered in critical appraisal is
formation needed to assess each bias domain. For this com- internal validity. Internal validity relates to the question,
ponent, we make reference to “The ARRIVE Guidelines for “Are the results of the study population representative of
Reporting Animal Research” (Kilkenny et al. 2010). Al- the source population?” Threats to internal validity occur
though we make reference to the ARRIVE Guidelines check- due to bias. For example, consider an experiment that looks
list items, end-users should be aware that not all authors at differences in behavior in mice subjected to 2 treatments (A
report in a manner consistent with the ARRIVE Guidelines. and B). In the experiment, the outcome is a binary variable

406 ILAR Journal

(“yes” or “no”) that classifies each animal as having a charac- clinical trial literature, the terminology used for categorizing
teristic of interest or not. There are 10 mice per treatment systematic errors is selection bias, performance bias, detec-
group. The data demonstrate that in treatment A, 8 of the 10 tion bias, attrition bias, and reporting bias (Higgins et al.
mice score a “yes,” while in treatment B, only 4 of the 10 mice 2011). In trials, selection bias may create a confounding
score a “yes.” The comparison of the treatments can be sum- bias. In this manuscript, we will use the latter terms for con-
marized using the ratio of the proportions with “yes” out- sistency with the risk-of-bias tool developed by the Cochrane
comes in treatment A to treatment B (i.e., 8/10 ÷ 4/10 = 2). Collaboration (Higgins et al. 2011). The rationale for using
From these results, we would infer that treatment A doubles the risk-of-bias domains recommended by the Cochrane Col-
the risk of a “yes” response. It would also be reasonable to laboration is that the biases associated with animal research
summarize the comparison of the treatments by calculating are consistent with those associated with human health and
the difference in risk of the “yes” response (i.e., 80% − 40% animal experiments more closely represent controlled trials
= 40%). If a “yes” response is favorable (e.g., a “yes” out- that observational studies. Further, there are communication

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

come indicates better or improved cognitive function), treat- advantages to using of a common terminology (Higgins
ment A may be considered a candidate therapy to be moved et al. 2011). For example, failure to blind leads to the same
forward to the next stage of testing. If a “yes” response is not concerns about outcome assessment bias in human-based
favorable (e.g., a “yes” outcome indicates lower or dimin- research and animal research.
ished cognitive function), treatment A may be considered a
risk factor or potential cause of the behavior, and further re- Risk-of-Bias Domains
search may evaluate the causal mechanisms. The question the
critical appraiser must ask is, “Did the observed ratio or differ- Selection Bias
ence arise because of the treatments, systematic bias, or impre-
cision?” Alternatively, internal validity can be framed as the Mice in the study were allocated to treatment group by
question, “What is the potential for factors other than the var- sex; treatment A consisted of female mice and treatment
iable of interest to have influenced the results of the study?” B consisted of male mice:
For the remainder of this paper, we will discuss a system- Example 1
atic approach to critically appraising the internal validity of
study findings. We will discuss sources of systematic error Selection bias occurs when there are systematic differences in
(i.e., bias) as well as the impact of imprecision or random baseline characteristics between the treatment groups being
error on the results. compared (Higgins et al. 2011). In laboratory animal studies,
selection bias occurs when nonrandom factors influence the
Bias allocation of animals to treatment groups (Starks et al.
2009). In example 1, it is possible that the sex of the group
Bias refers to a systematic deviation from the true state of na- may account for the observed difference in the group out-
ture (i.e., the association observed in the study population dif- comes, rather than the treatment. Although extreme, this is
fers from the true association in the source population). In an example of selection bias.
comparative research, we aim to measure the effect of an in- When critically appraising a paper for evidence of selection
tervention among the study groups (e.g., how average liver bias, end-users should ask, “Are the groups comparable such
weight differs across 3 treatments). However, the observed that an observed difference is likely attributable to the treatment
difference in the study population may differ from the true re- rather than a confounder?” A confounder is a factor that is re-
sult in the source population for 2 reasons: bias or impreci- lated to the outcome in the source population independent of
sion. Imprecision relates to random error and is discussed at the treatments and related to the treatments in the study popu-
the end of the manuscript. Bias is caused by systematic errors. lation. A common potential confounder in animal studies is
Systematic errors are a function of how the study was con- sex. If the sex of animals is unevenly distributed across the treat-
ducted and, unlike random error, cannot be resolved by in- ment groups and sex has an effect on the outcome, then the ob-
creasing the sample size. For example, if researchers aim to served difference in treatments may be simply a sex effect.
compare the weights of 2 study groups, they may obtain an Ideally, we would like to measure the effect of an interven-
imprecise estimate of the difference in weights, because tion by observing the effects of treatment A and treatment B at
only 5 animals per group were studied. Simply increasing exactly the same time on exactly the same animals; this is re-
the sample size can decrease the extent of imprecision. How- ferred to as the counterfactual. In this situation, the observed
ever, if the researchers use a scale that is systematically incor- difference in the outcome would be attributable to only the dif-
rect (e.g., one that always underestimates weight), increasing ference in treatment. Of course, the counterfactual cannot be
the sample size will not decrease this systematic error. observed. In lieu of being able to observe the counterfactual,
The nomenclature of systematic errors is only moderately researchers try to create intervention groups that are exchange-
consistent across disciplines and may therefore create some able (Greenland and Robins 1986; Lindley and Novak 1981;
confusion. In epidemiology, the categories of systematic error Pearl 2009). For example, imagine a hypothetical exchange
are usually referred to as selection bias, confounding bias, of the 2 treatment groups; if the treated group becomes untreat-
and information bias (Dohoo et al. 2010). However, in the ed, and vice versa, and the effect of the treatment remains the

Volume 55, Number 3, doi: 10.1093/ilar/ilu038 2014 407

same, then the groups are exchangeable. Exchangeability will
arise if there is balance across the groups for all the factors that
may affect the outcome, so that we achieve close to the coun-
terfactual comparison. Any difference in the outcome observed
between the treatment groups can be attributed to the difference
in treatment. Therefore, when critically appraising a study, end-
users should look for the use of design tools that would create
exchangeable groups.
Randomization is the best known of the design tools to bal-
ance the distribution of confounders across the intervention
groups. Randomization of study subjects to intervention
groups means that the differences observed in the makeup

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

of groups are due to chance (“random error”). Randomization
is the only design tool that can address known and unknown
confounders. As such, it is an important tool and should be
used in all studies. Numerous studies have demonstrated
that the results of randomized and nonrandomized studies dif-
fer, suggesting randomization reduces bias (Burns and Figure 1 Sampling distribution of number of head from repeated
O’Connor 2008; Jerndal et al. 2010; Pedder et al. 2014; tosses of 20 fair coins.
Sargeant et al. 2009; Schulz et al. 1995). Thus, randomization
serves as an indicator of a reduced risk of bias. However, ran-
domization does not guarantee the balancing of confounders
and, therefore, does not guarantee the absence of selection
bias as a cause of observed group differences. The reporting
of randomization should not be treated as synonymous with
the absence of selection bias.
The ability of randomization to balance both known and
unknown confounders decreases as the study becomes small-
er. This is an important point. Because of the importance of
sample size in the efficacy of randomization, even in random-
ized studies, end-users need to consider the size of the groups
to determine the potential for randomization to balance
known and unknown confounders. By and large, laboratory
animal studies are very small, so this is relevant to critical ap-
praisal of laboratory animal studies. For example, consider an
experiment with 20 animals—10 males and 10 females. If sex
is associated with the outcome in the source population, the
researcher would want to ensure that there was an even num-
ber of males and females in each group to balance the sex ef-
fect. If the researchers relied solely on simple random
allocation, it is possible that the sex of the animals would Figure 2 Sampling distribution of number of head from repeated
not be evenly distributed across the groups. This concept tosses of 6 fair coins.
can be illustrated with the idea of a coin toss—if a coin is
tossed 20 times, 10 heads would not be guaranteed despite appraisers should look for the use of more complex approach-
the use of a random method. As shown in Figure 1, the es to random allocation. The use of such tools may increase
most common result is to have 10 heads; however, other re- the confidence that the risk of selection bias is low. It is pos-
sults are possible. In Figure 2, it can be seen that the potential sible to employ restricted randomization to deliberately force
for uneven groups increases with decreasing sample size. For the even distribution of known confounders and increase the
example, when 6 coin tosses are made, it is quite common for potential for exchangeability. Block and stratified randomiza-
only 1 head to occur (around 10% of the time), rather than the tion can be used, although the researcher may not use these
expected 3 heads from 6 tosses. As the sample size increases, exact terms, which are more common in clinical trials. In
the probability of extreme unbalance become rarer, and the stratified randomization, the researchers could stratify animals
50% heads result is observed with greater frequency. The by sex, and randomly allocate within each sex, ensuring that
same concept can be applied to small experiments, which sex was evenly distributed across treatment groups. If a con-
laboratory animal studies often are. tinuous variable is a potential confounder, the researchers
Given the potential for study size to affect the efficacy of may use block randomization. For example, 20 piglets might
simple randomization to control selection bias, critical be organized by weight from heaviest to lightest, creating

408 ILAR Journal

5 blocks of 4 animals each. Then all possible allocation se- treated control mice were housed in two group cages in the
quences of 2 treatments (labeled A and B) for a group of 4 older Bl2 facility:
can be created (i.e., AABB, ABBA, ABAB, BABA, Example 2
BBAA, BAAB) (Altman and Bland 1999). The researcher
may randomly allocate a sequence to each block. Such an ap- Performance bias occurs when there are systematic differenc-
proach aims to balance the distribution of weight across the es between groups in the care that is provided, or in exposure
treatment groups and reduce selection bias and confounding. to factors other than the treatment (Higgins et al. 2011). When
Other designs tools also exist to distribute confounders critically appraising papers of laboratory animal studies, end-
across intervention groups. The most commonly used design users should ask, “Was the approach to husbandry the same
tool in laboratory animal studies is restriction of the study for all treatment groups and was caregiving done without
population to exclude a factor that is known to or would likely knowledge of the treatment group?” If the answer is no,
affect the outcome. In our example, restriction would corre- then the potential for performance bias may be high.
In example 2, it is clear that the management of the animals

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

spond to only conducting the experiment in male mice or
only conducting the study in female mice. Restriction is is different between the treatment groups and this could influ-
such a fundamental approach to controlling for confounding ence the performance of the animals. In such a scenario, the
in laboratory animal studies that many researchers may not potential is high for differences in treatment group outcomes
recognize it as a design tool. Laboratory animal studies rou- to be affected by the different approaches to housing. Obvi-
tinely are restricted to a source population of the same age and ously, there are more subtle ways in which animals may be
genetic lines, and by doing so, important known (or suspect- managed that could still contribute to differences in the per-
ed) confounders are removed. It is also important to recognize formance of the groups. In animal studies, performance bias
that the use of restriction as a design tool means that variation may arise in many ways. First, as in the example, the animals
is often lower in laboratory animal studies compared with hu- may be managed differently by design. Location in facilities,
man randomized controlled trials that are often designed to group sizes, and diet are all factors that should be the same
include a diverse group of people. Because of this lower var- across treatment groups. Secondly, the animals may be man-
iation, restriction may in part explain why large sample sizes aged differently by nonblinded caregivers. Blinding of the
are often not needed in laboratory animal studies to observe caregiver is important in all animal studies. Because there is
differences. Unfortunately, restriction also has the impact of a daily interaction between caregivers and animals, it is im-
reducing external validity. Of note, the National Institutes portant that caregivers are not aware of the treatment groups,
of Health recently have been discouraging restriction of stud- so they do not inadvertently manage one group differently.
ies to one sex of animal (Clayton and Collins 2014). Blinding of caregivers to the treatment group should not be
Finally, even in studies that employ all of the tools recom- confused with blinding of outcome assessment, which is re-
mended to increase the exchangeability of groups and reduce lated to the potential for detection bias (see below). Blinding
selection bias, groups may still be uneven. This should be eval- of the caregiver to treatment group can be difficult for studies
uated by looking at the baseline characteristics of the groups, that use only 2 cages (i.e., one for each treatment). Such stud-
frequently reported in the initial portion of the results section. ies theoretically have a high potential for performance bias, if
a caregiver feels they “know” the treatment allocation. This is
an argument for having more than one cage per treatment.
Where Might We Expect Information About
Selection Bias to Be in the Report?
Where Might We Expect Information About Performance
When assessing the potential for selection bias, the critical Bias to Be in the Report?
appraiser needs to look at the eligibility criteria for the study
subjects, the approach to allocation of study subjects to treat- To assess performance bias, the critical appraiser needs to
ment groups, the sample size, and the baseline information know about the management of the animals, the outcomes
for study subjects in each of the treatment groups. In the AR- of interest, and the approach to blinding of caregivers as to
RIVE Guidelines, the information that would enable the end- which treatment groups the animals (or animal cages) belong.
user to judge the balance of known factors between the In the ARRIVE Guidelines, the information that would en-
groups is described in Items 6, 8, 10, and 11. The information able the end-user to judge performance bias is described in
requested in these items will enable the end-user to know the Items 6, 9, and 13.
study population, housing, and demographics and the base-
line data of animals in each treatment group.
Detection Bias
Performance Bias
The cages of the mice in the group treated with the new
fantastic drug were labeled with the term “fantastic
Mice in the infected group were housed in the newer B3 drug” and the cages of the mice that did not received
isolation facility in individual cages. However, as the treatment were labeled “old drug” Each day the staff
there was no biological need and to reduce costs, the un- feeding the mice were asked to described how active the

Volume 55, Number 3, doi: 10.1093/ilar/ilu038 2014 409

 mice where on a scale of 1 to 5, with 1 being not active and Thus, bias can be introduced, even without knowing exactly
5 being very active: what treatment A is.
Example 3

Where Might We Expect Information About Performance

Detection bias can occur when there are systematic differenc- Bias to Be in the Report?
es between treatment groups in the way in which the outcome
is assessed (Higgins et al. 2011). When critically appraising a To assess detection bias, the critical appraiser needs to know
paper, end-users should ask, “Was the approach to assessing about the management of the animals, including any ap-
the outcomes the same in both groups and done without proaches to identification, the measurement of the outcomes
knowledge of the group?” If the answer is no, the potential of interest, and the approach to blinding of the outcome asses-
for detection bias may be high. sor. In the ARRIVE guidelines, the information that would
enable the end-user to determine the likelihood that the out-

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

In example 3, it is clear that the people measuring the out-
come are also aware of the treatment groups of the mice. The come was differentially assessed is described in Items 6, 9,
caregivers may have the expectation that the mice receiving and 13.
the new treatment will perform better and inadvertently over-
estimate the activity levels of these mice. The observed differ-
ences in the outcomes between groups could be due to Attrition Bias
differential measurement of the outcome between the 2
groups rather than treatment. This approach clearly has the “Twenty-four animals were allocated to 2 treatment
potential to introduce detection bias. groups (12 in each group). The average weight (± SD) of
The most common tool used to prevent detection bias is the mice in treatment A was 2.5 g ± 0.3 (n = 7) and the av-
blinding of the individuals assessing the outcome as to which erage weight in treatment B was 2.4 g ± 0.3 (n = 12).”
animals (or cages) belong to which treatment groups. Empir- Example 4
ical evidence shows that failure to blind the outcome assessor
in laboratory animal studies is associated with more favorable Attrition bias refers to systematic differences in withdrawals
outcomes (Jerndal et al. 2010; Minnerup et al. 2010). Ideally, from treatment groups (Higgins et al. 2011). When critically
all studies should blind outcome assessment; however, the appraising a paper, end-users should ask, “Was the loss of an-
risk of bias due to failure to blind may be high or low depend- imals from the groups minimal and unrelated to the treatment
ing upon the outcome. For example, when the outcome is groups?” If the answer is no, the potential for bias due to at-
death, the risk of bias due to lack of blinding may be very trition may be high. The impact of attrition is that the outcome
low, as the outcome is extremely objective. Other outcomes is not observed for all animals. In the example above, it ap-
that may seem objective and quantifiable, such as cells per pears that some animals have not completed the study. If
field, may be prone to bias if the reader is allowed to pick a the outcome of those missing animals was systematically dif-
“suitable slide field.” Risk of bias should be judged for every ferent across the groups, then bias can occur. For example,
outcome reported and the judgment reached may differ imagine a study that measured weight gain over a 50-day pe-
among different outcomes within the same experiment. As- riod as an outcome, as in example 4. The study also had
sessment of the risk of detection bias for a particular outcome weight loss as a withdrawal criterion (e.g., if an animal loses
often requires content expertise. a certain amount of weight they must be withdrawn from the
When blinding is used, it is important not only that the out- study and euthanized). Further, consider the impact of these
come assessor not know what specific treatment an animal criteria, if the effect of treatment A is to create a bimodal dis-
has received, but also what treatment group an animal is in tribution of diseases (i.e., animals within the group are either
(e.g., group labeled “A” or group labeled “B”), even if specif- severely affected or show no clinical signs), and if the effect
ic treatments are not named. This is important, because know- of treatment B is to create a wide spectrum of disease, from
ing the outcome for one animal of a group may influence the severe to mild to no clinical signs. In such a scenario, severely
interpretation of the outcome for the next animal in the same affected animals in treatment A are more likely to be removed
group. Even if a bias does not exist towards a particular out- from the study, leaving only healthy animals. The unobserved
come, knowing the treatment group may inadvertently reduce animals would have had very low weight gain and, if includ-
group-level variation. For example, imagine the scenario ed, would have reduced the treatment A average weight gain.
where the first animal is known to have received treatment However, because only the healthy animals are actually ob-
A and has severe lung consolidation. The pathologist may served to the end of the study, the average weight gain ob-
be more attuned to lung congestion in the next animal in served in treatment A is higher than the true unobserved
group A. Similarly, the pathologist may be more inclined to average, and the observed difference in group averages is
call the pathology severe, if the last animal in group A was due to attrition bias rather than a true treatment effect.
known to have severe pathology. This bias would make Attrition bias is a major issue in human studies where attri-
groups more consistent, reducing heterogeneity, and may de- tion may be voluntary ( participants choose to leave the study)
crease the p value as the true extent of variation is understated. or nonvoluntary (e.g., participants die prior to the end of the

410 ILAR Journal

study). In human clinical trials, participants can leave the et al. 2011). The concept of reporting bias is sometimes dif-
study if they choose. When they leave, the reason for leaving ficult to conceptualize, when critically appraising a single
often is not known. If the reason for leaving is associated with study. It is likely more accurate to say that studies fail to report
the treatment, a bias can occur. This almost certainly cannot some results and that the actual bias occurs in subsequent re-
happen in laboratory animal studies. Therefore the potential views that use studies that fail to report all results. The failure
for voluntary attrition bias in laboratory animal studies is al- to report something makes it impossible to ask the question,
most nonexistent; however, nonvoluntary attrition can occur “Are there other explanations for the results observed?” Re-
if animals die or are withdrawn because they met a priori cri- porting bias is relevant when the aim of the critical appraisal
teria for removal prior to observing the outcome. Because exercise is to combine and summarize a larger body of work.
nonvoluntary attrition can inadvertently introduce bias, au- If there is evidence of reporting bias and some results are
thors are encouraged to report information about attrition missing, the results of the summary will be biased because
for each study group. Authors should also consider including of the absence of some studies. At the individual paper level,

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

outcomes that will be unaffected by attrition; in the hypothet- reporting bias may hide multiplicity issues and, as such, affect
ical example about weight loss, comparing survival time in the ability of the critical appraiser to know the role of chance
each group would not be affected by attrition bias. However, in the findings (see the discussion of multiplicity below).
often outcomes cannot be designed to address attrition bias
and other approaches must be employed. Some design or stat-
istical approaches to minimizing the impact of attrition can be Where Might We Expect Information About Reporting
used, such as using the last observed outcome or imputing the Bias to Be in the Report?
missing data. If these approaches are reported in a manuscript,
the assessment of their validity often requires a statistician be As shown in example 5, evidence of reporting bias is often
consulted. found by comparing the outcomes reported (ARRIVE check-
list item 16) and the statistical methods proposed (ARRIVE
checklist item 13) and the variables assessed (ARRIVE
Where Might We Expect Information About Performance checklist item 12). It is also possible to compare the outcomes
Bias to Be in the Report? studied in the report with those proposed in the original study
proposal or protocol, if available. Finally, some journals now
The ARRIVE Guidelines propose that authors report the suggest the inclusion of a statement of comprehensive report-
number of animals enrolled in each group, the number of an- ing (Altman and Moher 2013), and the inclusion of such a
imals completing the study, and the number included in the statement should suggest that the risk of reporting bias is
analysis. It is also possible that authors will report their ap- low. An example of such a statement modified for animal
proach to dealing with missing data including any imputation studies might be, “The authors affirm that this manuscript is
approaches in the statistical analysis section of the manuscript an honest, accurate, and transparent account of the study be-
(ARRIVE checklist items 13, 14, and 15). ing reported; that no important aspects of the study have been
omitted; and that any discrepancies from the review as
Reporting Bias planned have been documented and explained. The authors
have indicated where results from these study animals are re-
ported in other publications, and have included citations for
“Outcomes measured were weight of the liver, kidney, these publications where relevant.”
brain, spleen and lungs. …
Results: The average weight of the kidneys in treatment A
was 1.5 g (SD = 0.5, n = 12) and the average weight in treat-
ment B was 0.9 g (SD = 0.4, n = 12) (mean difference = 0.6, Random Error
95% CI = 0.2 to 0.98, p value = 0.003867). The conclusion
reached was …” “Mortality in treatment A was 90% (9 of 10) and 10% in
Example 5 treatment B (1 of 10) (Fisher’s exact p-value = 0.001093).”
Example 6
In example 5, we see that although the authors reported as-
sessing 5 outcomes, the results of only 1 are reported. This “Mortality in treatment A was 90% (9 of 10) and 10% in
is an example of incomplete reporting. When critically ap- treatment B (1 of 10) (Fisher’s exact p-value < 0.05).”
praising a paper, end-users should ask, “Were the results of Example 7
all outcome variables assessed reported completely?” If the
answer is no, the potential for reporting bias may be high. Of- Apart from systematic sources of error, random error may be
ten incomplete reporting can only be theoretically assessed. If another explanation for observed differences between group
the authors had omitted reference to the lungs in the methods outcomes. When critically appraising a paper, end-users
and materials, the reader would have no evidence about the should ask, “Is there a low probability that chance played a
inclusivity of the outcomes reported. Reporting bias refers role in the observed difference?” The end-user should use
to the absence of important results from the study (Higgins the p value provided by the original study to assess this

Volume 55, Number 3, doi: 10.1093/ilar/ilu038 2014 411

question and should report the p value threshold used by the dom error. If the 95% confidence interval includes the null val-
end-user as the basis for considering an outcome rare enough ue (1 for ratios and 0 for differences), then the p value is >0.05.
to reject the null hypothesis. A small p value suggests a low For example, if a study reports a risk ratio of 2 and the 95%
probability of chance, a large p value suggests a high proba- confidence interval for the ratio is from 0.386 to 2.44, the p
bility of chance. It is known from sampling distribution the- value must be >0.05. However, sometimes it is more difficult
ory that if we were to obtain multiple random samples from a to determine the exact p value from a confidence interval, be-
subset of individuals from the same population and measure a cause the degrees of freedom are not reported, and in those sit-
population parameter (e.g., mean weight), we would not ob- uations, the exact role of chance might only be inferred rather
tain identical means for each sample, even though each sam- than exactly calculated.
ple came from the same source population. “Random error”
means that the difference observed is simply a function of the
random sampling of 2 groups of individuals from the same Multiple Testing

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

population. The p value of a test statistic under the null hy-
pothesis is the approach used to express the probability that “Statistical Methods. The study had 3 treatment groups,
the observed difference in the outcomes between groups is and we measured wet weights, dry weights, and surface
a function of random error. For the data provided above, after area for the liver, spleen, kidney, heart, lungs, stomach,
performing a Fisher’s exact test to assess treatment group dif- and brain. We compared the average outcomes across
ferences, we obtain a p value of 0.00109. A chi-square test all possible pairwise comparisons of the 3 groups (i.e.,
would not be appropriate for this dataset, because the sample group 1 vs. group 2, group 2 vs. group 3, and group 1
size is too small. This p value suggests that 0.109% of the vs. 3) using t-tests. We used a p value of 0.05 to assess stat-
istical significance.
time with repeated samplings, we would expect to obtain
Results. The wet weights, dry weights, and surface area
the same (or greater) observed difference in proportions in of liver, spleen, kidney, heart, lungs, stomach, and brain
mortality between 2 groups of 10 randomly drawn individuals were not statistically different across any of the pairwise
from the same population. In example 6, we would conclude treatment group comparisons with the exception of a stat-
that the chances are small, because the p value suggests that istically significant difference in the surface area of the
the observed difference of 90% between groups would be ob- spleen between Group 1 and Group 3 ( p < 0.05).”
served only rarely. Example 8
Failure of authors to report the exact p value prevents a crit-
ical appraiser from evaluating the probability that random er- Another factor to evaluate related to random error is multi-
ror is the reason for the observed treatment effect. Often plicity. It is common, by convention, to accept that a compar-
authors simply report that the finding was “statistically signif- ison is “statistically significant” if the p value is < 0.05. The
icant,” meaning that the probability that the observed differ- 0.05 corresponds to a type I error (i.e., the probability that the
ence arose from random error is ≤5% (example 7). The ability observed difference or larger would occur in a population
to assess the role of random error would be substantially im- where the means were the same is 5% or less). However,
proved if one could differentiate between a p value of 0.05 or when multiple comparisons are conducted within the same
0.0001. For example, a p value of 0.05 implies that something experiment, although the probability of a type I error for
occurs 1 in 20 occasions. In reality, this is not a very rare out- each comparison is 5%, the probability that at least one com-
come, when we consider that it is quite common to roll 2 sixes parison was significant due to random error is considerably
with a pair of dice, an event with a probability of 1 in 36. The higher.
magnitude of the p value should not be used to interpret the Authors should consider when multiple outcomes are as-
size of the treatment effect (i.e., a small p value does to equate sessed that differences may have arisen by chance. As well,
to a large difference, just a rare difference). This is a common across the entire study, the more tests conducted, the more op-
misinterpretation. Indeed, in laboratory animal studies, most portunity there is for random error. In human clinical trials,
treatment effects are quite large, as the studies are small and studies are often designed to assess a limited number of out-
only have the statistical power to detect large differences. One comes, reducing the potential for this multiplicity of random
issue to be aware of is that within a test such as an ANOVA it errors. However, it is unclear if limiting the number of out-
is possible to adjust for the p value of pairwise comparison comes in laboratory animal studies is desirable. Laboratory
using statistical methods. Examples of approaches include animal studies are meant to provide preliminary proof of con-
calculation of the least significant difference, a Bonferonni cept or data on treatment safety and thus play a critical role in
adjustment, a Tukey adjustment , Tukey-Kramer adjustment, discovery and hypothesis generation (Henderson et al. 2013).
and numerous possible approaches (Ramsey and Schafer They also often end in euthanasia of the animals. Thus, a
2013). However, such approaches only adjust the p value large number of outcomes may be justified. For discovery
for pairwise comparisons conducted within the ANOVA test studies, we believe that it is preferable that authors report
and do not account for the conduct of multiple ANOVA tests. all outcomes assessed with group-level and summary-level
A 95% confidence interval conveys to the end-user how pre- data (including measures of precision) and authors consider
cisely the estimate of the treatment effect is known. However, not conducting inferential statistics on all outcomes (e.g.,
it can also be used to provide indirect information about ran- consider testing only those outcomes for which the study

412 ILAR Journal

Volume 55, Number 3, doi: 10.1093/ilar/ilu038

Table 1 Sample form that might be used to document the approach to critical appraisal of a laboratory animal study designed to compare outcomes
among groups

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

Design tools associated with ARRIVE guideline
Area to assess Question to ask Possible responses control or reduction of risk items

External validity “If the study was conducted in a No—don’t assess Inclusion criteria for relevant 7, 8, and 9
manner that suggests little internal Yes—continue to assess internal populations, housing, and
bias, will it be useful for the ‘next validity intervention used
step’ because the population is
relevant to ‘the next step?’”
Internal validity (using risk-of-bias domains from Higgins et al. 2011)
Selection bias “Are the groups comparable such Yes—low ROB Blinded allocation to group, 6, 8, 10, and 11
that an observed difference is likely Unclear—unclear ROB restriction, randomization,
attributable to the treatment rather No—high ROB restricted randomization
2014

than a confounder?”
Performance bias “Was the approach to husbandry the Yes—low ROB Blinding of caregivers, use of multiple 6, 9, and 13
same for all treatment groups and Unclear—unclear ROB cages per treatment
was caregiving done without No—high ROB
knowledge of the treatment
group?”
Detection bias “Was the approach to assessing the Yes—low ROB Blinding of outcome assessors, use 6, 9, and 13
outcomes the same in both groups Unclear—unclear ROB of repeatable and objective
and done without knowledge of the No—high ROB outcome measures
group?”
Attrition bias “Was the loss of animals from the Yes—low ROB Minimization of loss to follow-up and 13 to 15
groups minimal and unrelated to Unclear—unclear ROB complete reporting of loss to
the treatment groups?” No—high ROB follow-up for each treatment group
Reporting bias “Were the results of all outcome Yes—low ROB Comprehensive reporting and a 12 to 17
variables assessed reported Unclear—unclear ROB well-designed study protocol
completely?” No—high ROB
Random error
Test-level error “Is there a low probability that chance Yes—low risk of random error The exact p-value and the 95% 10, 13, and 16
played a role in the observed in the test confidence interval
difference?” Unclear—unclear risk of random
error in the test
No—high risk of random error
in the test
Continued
413
 ARRIVE guideline
was specifically designed—i.e., the basis for the power calcu-
lation). For the critical appraiser, if reporting suggests a large

10, 13, and 16

number of outcomes were tested, the role of random chance in

Items above
the study increases. This is the area where reporting bias af-
items fects the critical appraiser. If the authors do not report all out-
comes assessed, the person appraising the study cannot
accurately gauge the role of random error due to multiplic-
ity in the study results. For example, if a researcher tests
and publishes 20 outcomes and only one is statistically
The power of the study, the number

significant, a mindful reader could recognize the likelihood

of false statistical significance and adjust for this. However,
Design tools associated with
control or reduction of risk

if instead the researcher publishes the 1 significant outcome

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

with only 4 nonsignificant outcomes, 20% of outcomes
appear significant, and the potential for the reader to re-
of tests conducted

cognize the random nature of the significant findings is

eliminated.

How to Reach a Conclusion about a Paper

We propose that for individual papers, critical appraisers first
assess external validity (i.e., if the study is judged to be inter-
nally valid, then the result would be useful to apply to the next
Yes—low risk of random error across

No—high risk of random error across

step in the scientific process). Only then assess internal valid-

Unclear—unclear risk of random

ity. To assess internal validity, for each of the risk domains,

we would propose following the approach proposed by the
Cochrane Collaboration (Higgins et al. 2011). For each do-
Unclear internal validity
error across the tests

main, assess whether the risk of bias is high, low, or unclear.

Possible responses

High internal validity

Low internal validity

The unclear response usually arises if there is a lack of com-

prehensive reporting. If reporting is comprehensive, a judg-
ment of high or low risk should be made and the rationale
the tests

the tests

indicated. After assessing each bias domain and the potential

for random error, evaluate the findings together. The final
judgment can be recorded as valid or invalid. We present a
template table (Table 1) that might be used for a critical ap-
praisal exercise. We also present an example of a completed
table (Table 2) to illustrate how the rationale can be included
Based on the overall assessment of
“Did the authors limit the number of

to enhance transparency.
assessment of internal validity?
those the study was designed
hypothesis tests conducted to

Tempting as it may be to create a score or a cut-off, this ap-

the above items what is your

proach should not be used (Higgins et al. 2011). Scores and

cut-points have been discredited in favor of making a judg-
(powered) to assess?”

ment about the validity of study findings (likely valid or likely

invalid) and using risk-of-bias assessments to support that
Question to ask

judgment. This avoids subjective selection of weightings

for different bias domains and also allows the critical apprais-
er to judge the risk of bias for each domain in the specific con-
text of the study or outcome. It is not unreasonable for the
reviewer to decide that on the basis of a single flaw that the
Abbreviation: ROB, risk of bias.

results are likely not internally valid.

We believe that recording the reasoning behind an assess-
ment will be helpful for defending decisions about validity.
Table 1 Continued

Study-level error

Decisions about validity of results have been made for a

Area to assess

very long time and knowledge of risk-of-bias domains has

Conclusion

long been known. The 2 novel aspects to critical appraisal in-

corporated here are the formal partitioning of the risk-of-bias
domains and the transparent documentation of the conclu-
sions reached about these domains.

414 ILAR Journal

Volume 55, Number 3, doi: 10.1093/ilar/ilu038

Table 2 Completed critical appraisal form for a hypothetical study using examples in the text

Area to assess Question to ask Possible responses Rationale

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

External validity “If the study was conducted in a manner that Yes Population, housing, and intervention are relevant
suggests little internal bias, will it be useful for the to next step in the scientific process.
‘next step’ because the population is relevant to
‘the next step?’”
Internal validity
Selection bias “Are the groups comparable such that an observed No—high ROB Example 1: Animals are assigned by sex and sex
difference is likely attributable to the treatment may be a confounder, if related to the outcome.
rather than a confounder?”
Performance bias “Was the approach to husbandry the same for all No—high ROB Example 2: The treatment groups are housed very
treatment groups and was caregiving done without differently and, therefore, the housing rather than
knowledge of the treatment group?” the treatment could be the cause of the observed
2014

differences.
Detection bias “Was the approach to assessing the outcomes the No—high ROB Example 3: The outcome assessors are clearly
same in both groups and done without knowledge aware of the treatment assignment and the
of the group?” approach to measurement is highly subjective, so
these factors could be the cause of the observed
differences between groups rather than the
treatment.
Attrition bias “Was the loss of animals from the groups minimal and No—high ROB Example 4: Some animals are missing from one
unrelated to the treatment groups?” group and no explanation is provided.
Reporting bias “Were the results of all outcome variables assessed Unclear—unclear ROB Example 5: Several measured outcome are not
reported completely?” reported; this suggests incomplete reporting and
may indicate a non-significant finding.
Random error
Test level “Is there a low probability that chance played a role in Yes—low risk of random error Example 6: The p-value is very small suggesting the
the observed difference?” observed difference is rare under the null
hypothesis.
Study level “Did the authors limit the number of hypothesis tests No—high risk of random error Example 8: The authors appear to have conducted
conducted to those the study was designed 9 × 7 = 63 hypothesis tests without adjustment for
( powered) to assess?” multiplicity, and only found one significant
outcome. It is unclear if this is a primary (important)
outcome.
Conclusion Low internal validity

Abbreviation: ROB, risk of bias.

415
Conclusion Kilkenny C, Parsons N, Kadyszewski E, Festing MFW, Cuthill IC, Fry D,
Hutton J, Altman DG. 2009. Survey of the quality of experimental de-
sign, statistical analysis and reporting of research using animals. PLoS
The critical appraisal of studies is a fundamental aspect of One 4:e7824.
the scientific process. Here we propose applying the Koustas E, Lam J, Sutton P, Johnson PI, Atchley DS, Sen S, Robinson KA,
“risk-of-bias” domains used by the Cochrane Collaboration Axelrad DA, Woodruff TJ. 2014. The Navigation Guide—evidence-
to the critical appraisal of laboratory animal studies. Critical based medicine meets environmental health: Systematic review of non-
appraisal requires content expertise and making judgments human evidence for PFOA effects on fetal growth. Environ Health
Perspect 122:1015–1027.
cannot be avoided. However, transparency in the criteria as-
Lam J, Koustas E, Sutton P, Johnson PI, Atchley DS, Sen S, Robinson KA,
sessed and conclusions reached can make the process of crit- Axelrad DA, Woodruff TJ. 2014. The Navigation Guide—evidence-
ical appraisal easier to conduct, more deliberate, and perhaps based medicine meets environmental health: Integration of animal and
more reproducible. human evidence for PFOA effects on fetal growth. Environ Health
Perspect 122:1040–1051.
Landis SC, Amara SG, Asadullah K, Austin CP, Blumenstein R, Bradley EW,

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

Crystal RG, Darnell RB, Ferrante RJ, Fillit H, et al. 2012. A call for trans-
References parent reporting to optimize the predictive value of preclinical research.
Nature 490:187–191.
Altman DG, Bland JM. 1999. How to randomise. BMJ 319:703–704. Lindley DV, Novak MR. 1981. The role of exchangeability in inference. Ann
Altman DG, Moher D. 2013. Declaration of transparency for each research Statist 9:45–58.
article. BMJ 347:f4796. Minnerup J, Wersching H, Diederich K. 2010. Methodological quality of pre-
Banwell V, Sena ES, Macleod MR. 2009. Systematic review and stratified clinical stroke studies is not required for publication in high-impact jour-
meta-analysis of the efficacy of interleukin-1 receptor antagonist in ani- nals. J Cereb Blood Flow Metab 30:1619–1624.
mal models of stroke. J Stroke Cerebrovasc Dis 18:269–276. National Research Council (US) Institute for Laboratory Animal Research.
Begley CG, Ellis LM. 2012. Drug development: Raise standards for preclin- 2011. Guidance for the Description of Animal Research in Scientific
ical cancer research. Nature 483:531–533. Publications. Available online (http://www.ncbi.nlm.nih.gov/books/
Burns MJ, O’Connor AM. 2008. Assessment of methodological quality and NBK84205/), accessed June 25, 2014.
sources of variation in the magnitude of vaccine efficacy: A systematic O’Connor AM, Sargeant JM, Gardner IA, Dickson JS, Torrence ME, Con-
review of studies from 1960 to 2005 reporting immunization with Mor- sensus Meeting P, Dewey CE, Dohoo IR, Evans RB, Gray JT, et al.
axella bovis vaccines in young cattle. Vaccine 26:144–152. 2010. The REFLECT statement: Methods and processes of creating re-
Clayton JA, Collins FS. 2014. Policy: NIH to balance sex in cell and animal porting guidelines for randomized controlled trials for livestock and
studies. Nature 509:282–283. food safety by modifying the CONSORT statement. Zoonoses Public
Crossley NA, Sena E, Goehler J, Horn J, van der Worp B, Bath PM, Health 57:95–104.
Macleod M, Dirnagl U. 2008. Empirical evidence of bias in the design Pearl J. 2009. Causality. Cambridge: Cambridge University Press. p. 285.
of experimental stroke studies: A metaepidemiologic approach. Stroke Pedder H, Vesterinen HM, Macleod MR, Wardlaw JM. 2014. Systematic re-
39:929–934. view and meta-analysis of interventions tested in animal models of lacu-
Dohoo IR, Martin W, Stryhn H. 2010. Veterinary Epidemiologic Research. nar stroke. Stroke 45:563–570.
2nd edition. VER Inc, Prince Edward Island, Canada. Perel P, Roberts I, Sena E, Wheble P, Briscoe C, Sandercock P, Macleod M,
Ell PJ. 2003. STARD and CONSORT: Time for reflection. Eur J Nucl Med Mignini LE, Jayaram P, Khan KS. 2007. Comparison of treatment effects
Mol Imaging 30:803–804. between animal experiments and clinical trials: Systematic review. BMJ
Greenland S, Robins JM. 1986. Identifiability, exchangeability, and epidemi- 334:197.
ological confounding. Int J Epidemiol 15:413–419. Pound P, Ebrahim S, Sandercock P, Bracken MB, Roberts I. 2004. Where is
Guyatt GH, Oxman AD, Kunz R, Falck-Ytter Y, Vist GE, Liberati A, the evidence that animal research benefits humans? BMJ 328:514–517.
Schünemann HJ, Grade Working Group. 2008. Going from evidence Prinz F, Schlange T, Asadullah K. 2011. Believe it or not: How much can we
to recommendations. Br Med J 336:1049–1051. rely on published data on potential drug targets? Nat Rev Drug Discov
Guyatt GH, Oxman AD, Schunemann HJ, Tugwell P, Knottnerus A. 2011. 10:712.
GRADE guidelines: A new series of articles in the Journal of Clinical Ramsey FL, Schafer DW. 2013. The statistical sleuth: A course in methods of
Epidemiology. J Clin Epidemiol 64:380–382. data analysis. Boston: Brooks/Cole, Cengage Learning.
Henderson VC, Kimmelman J, Fergusson D, Grimshaw JM, Hackam DG. Sargeant JM, O’Connor AM. 2013. Issues of reporting in observational stud-
2013. Threats to validity in the design and conduct of preclinical efficacy ies in veterinary medicine. Prev Vet Med 113:323–330.
studies: A systematic review of guidelines for in vivo animal experi- Sargeant JM, Saint-Onge J, Valcour J, Thompson A, Elgie R, Snedeker K,
ments. PLoS Med 10:e1001489. Marcynuk P. 2009. Quality of reporting in clinical trials of preharvest
Higgins J, Altman D, Sterne J. 2011. Chapter 8: Assessing risk of bias in in- food safety interventions and associations with treatment effect.
cluded studies. In: Higgins J, Green S, eds. Cochrane Handbook for Foodborne Pathog Dis 6:989–999.
Systematic Reviews of Interventions. Available online (http://www. Schulz KF, Chalmers I, Hayes RJ, Altman DG. 1995. Empirical evidence of
cochrane-handbook.org), accessed June 25, 2014. bias. Dimensions of methodological quality associated with estimates of
Jerndal M, Forsberg K, Sena ES, Macleod MR, O’Collins VE, Linden T, treatment effects in controlled trials. JAMA 273:408–412.
Nilsson M, Howells DW. 2010. A systematic review and meta-analysis Sena E, van der Worp HB, Howells D, Macleod M. 2007a. How can we im-
of erythropoietin in experimental stroke. J Cereb Blood Flow Metab prove the pre-clinical development of drugs for stroke? Trends Neurosci
30:961–968. 30:433–439.
Johnson PI, Sutton P, Atchley DS, Koustas E, Lam J, Sen S, Robinson KA, Sena E, Wheble P, Sandercock P, Macleod M. 2007b. Systematic review and
Axelrad DA, Woodruff TJ. 2014. The Navigation Guide—evidence- meta-analysis of the efficacy of tirilazad in experimental stroke. Stroke
based medicine meets environmental health: Systematic review of human 38:388–394.
evidence for PFOA effects on fetal growth. Environ Health Perspect Sena ES, Briscoe CL, Howells DW, Donnan GA, Sandercock PA,
122:1028–1039. Macleod MR. 2010. Factors affecting the apparent efficacy and safety
Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG. 2010. Improv- of tissue plasminogen activator in thrombotic occlusion models of stroke:
ing bioscience research reporting: The ARRIVE guidelines for reporting Systematic review and meta-analysis. J Cereb Blood Flow Metab
animal research. PLoS Biol 8:e1000412. 30:1905–1913.

416 ILAR Journal

Starks H, Diehr P, Curtis JR. 2009. The challenge of selection bias and van der Worp HB, Sena ES, Donnan GA, Howells DW, Macleod MR. 2007.
confounding in palliative care research. J Palliat Med 12:181–187. Hypothermia in animal models of acute ischaemic stroke: A systematic
Steward O, Popovich PG, Dietrich WD, Kleitman N. 2012. Replication and review and meta-analysis. Brain 130:3063–3074.
reproducibility in spinal cord injury research. Exp Neurol. 233:597–605. Wheble PC, Sena ES, Macleod MR. 2008. A systematic review and meta-
The Lancet. 2010. CONSORT 2010. The Lancet 375:1136. analysis of the efficacy of piracetam and piracetam-like compounds in
van der Worp HB, Howells DW, Sena ES, Porritt MJ, Rewell S, O’Collins V, experimental stroke. Cerebrovasc Dis 25:5–11.
Macleod MR. 2010. Can animal models of disease reliably inform human Woodruff TJ, Sutton P. 2014. The Navigation Guide systematic review meth-
studies? PLoS Med 7:e1000245. odology: A rigorous and transparent method for translating environmen-
van der Worp HB, Macleod MR. 2011. Preclinical studies of human disease: Time tal health science into better health outcomes. Environ Health Perspect
to take methodological quality seriously. J Mol Cell Cardiol 51:449–450. 122:1007–1014.

Downloaded from https://academic.oup.com/ilarjournal/article/55/3/405/644697 by guest on 11 September 2021

Volume 55, Number 3, doi: 10.1093/ilar/ilu038 2014 417