Tool for assessing Risk Of Bias due to Missing Evidence in a meta-analysis (ROB-ME)

SHORT VERSION (CRIBSHEET)

Version 1 October 2023

ROB-ME Development Group:

Matthew J Page, Jonathan AC Sterne, Isabelle Boutron, Asbjørn Hróbjartsson, Jamie J Kirkham, Tianjing Li, Andreas Lundh, Evan Mayo-Wilson, Joanne E McKenzie, Lesley A
Stewart, Alex J Sutton, Lisa Bero, Adam G Dunn, Kerry Dwan, Roy G Elbers, Raju Kanukula, Joerg J Meerpohl, Erick H Turner, Julian PT Higgins

Correspondence to: Dr. Matthew Page, Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, 553 St Kilda Road,
Melbourne, Victoria, 3004, Australia. Telephone: +61 9903 0248. Email address: matthew.page@monash.edu

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

1
Step 1. Select and define meta-analyses that will be assessed for risk of bias due to missing evidence
Meta- Specify the PICO for all meta-analyses that For each meta-analysis, specify which study designs and results were eligible for inclusion, indicating
analysis will be assessed for risk of bias. For example: whether the meta-analysis was restricted to particular:
ID
Participants: People with shoulder pain • study designs, and;
Intervention: Ibuprofen • outcome definitions (e.g. measures, metrics, time points), and;
• methods of analysis (e.g. analysis populations, crude or adjusted estimates).
Comparator: Placebo
If such information is reported elsewhere in the systematic review, either indicate the relevant section of
Outcome: Pain intensity at short-term (0-12 the review or copy the information here.
weeks)
For example:
Eligible study designs: Randomized trials
Add/delete rows where necessary
Eligible outcome definitions: Pain scores measured using any scale; up to 12 weeks post-randomization
Eligible methods of analysis: Analyses of change from baseline values; intention-to-treat analysis sample;
analyses adjusted for covariates
1 Participants: Eligible study designs:
Intervention: Eligible outcome definitions:
Comparator: Eligible methods of analysis:
Outcome:
2 Participants: Eligible study designs:
Intervention: Eligible outcome definitions:
Comparator: Eligible methods of analysis:
Outcome:
3 Participants: Eligible study designs:
Intervention: Eligible outcome definitions:
Comparator: Eligible methods of analysis:
Outcome:

2
Step 2. Determine which studies meeting the inclusion criteria for the meta-analyses have missing results
For each study meeting the inclusion criteria for one or more of the meta-analyses, assemble available sources of information about the study. This might include the
trials register entry (e.g. at ClinicalTrials.gov), study protocol, statistical analysis plan, reports of results of the study (e.g. journal article, clinical study report), or
information obtained directly from the study authors or sponsor (e.g. data files supplied).
Then compare results available with all available information about what outcomes were measured. If study plans are available (e.g. trials register entry, protocol),
compare results available with details of the pre-specified outcomes, to identify any outcomes with no results reported. It might be helpful to construct a matrix for
each eligible study that lists all outcomes described in the study plans and records whether results were available for each. If no study plans are available, cross-check
the methods and results sections against one another to identify any outcomes with no results reported, or results reported incompletely.
Then complete the Results Matrix below to indicate (using the symbols in the Key below) whether study results are available for inclusion in each meta-analysis to be
assessed for risk of bias, considering the scenarios in Box 1 when making a judgement about reasons for unavailability.
Also specify the total number of participants analysed for an indication of the likely weight of each study in the meta-analysis.

Key for Results Matrix

✓ A study result is available for inclusion in the meta-analysis.
~ No study result is available for inclusion in the meta-analysis, for a reason unrelated to the P value, magnitude or direction of the result.
? Unclear whether an eligible study result was generated.
X No study result is available for inclusion in the meta-analysis, likely because of the P value, magnitude or direction of the result generated.
Optional: Record any information known about the results (if available), such as the direction of effect (e.g. Favours intervention / Favours control), the statistical
significance of the result (e.g. P > 0.05), or narrative descriptions (e.g. “No difference”).

Example of a completed Results Matrix

Study ID Source(s) used Number of Result available Result available Result available Result available
participants for inclusion in for inclusion in for inclusion in for inclusion in
analysed Meta-analysis 1 Meta-analysis 2 Meta-analysis 3 Meta-analysis 4
Smith 2000 PMID: XXXXXXXX 455 ✓ X ✓ ?
Nyqvist 2017 None (not published) 67 X X ~ ~
Stylianos 2019 PMID: XXXXXXXX 87 ? ✓ ✓ X
Hozo 2014 PMID: XXXXXXXX 145 X ✓ X ✓
MacIntyre 2020 NCTXXXXXXXX 280 ~ ✓ X ~

3
Results Matrix (add/delete rows and columns where necessary)
Study ID* Source(s) used Number of Result available Result available Result available
participants for inclusion in for inclusion in for inclusion in
analysed** Meta-analysis 1 Meta-analysis 2 Meta-analysis 3

*List all studies meeting the inclusion criteria for the meta-analyses, regardless of whether a report of the results is available. Ideally, also include in the matrix any
study identified that did not measure the outcome (but met all other inclusion criteria) and any study that was excluded from the systematic review only because it had
no usable outcome data (but met all other inclusion criteria).
**If it is not clear how many participants were analysed, record the total number of participants assigned to the relevant intervention and control groups

Provide any relevant information to support responses

4
Step 3. Consider the potential for missing studies across the systematic review
Answer the following questions to determine whether circumstances indicate potential for some eligible studies not being identified because of the P value,
magnitude or direction of the results generated. Answer these questions once, in relation to the systematic review as a whole.
Question Elaboration Response options
3.1. Were prospectively registered studies or studies Answer ‘Yes’ if the review includes an inception cohort, that is, a set of studies that Y/N
identified for a prospective meta-analysis the only were identified and deemed eligible for inclusion in the meta-analyses before the
type of study eligible for inclusion in the review? results of the studies were generated or became known.
3.2. If N to 3.1: Would you expect every eligible Answer ‘Yes / Probably yes’ if this is a research area for which you expect all studies NA / Y / PY / PN / N
study to be identifiable regardless of its results? to have been prospectively registered, or if there is another reason to expect every
eligible study to be identifiable regardless of its results [and specify the reason in the
box below]
3.3. If Y/PY to 3.2: Were you likely to have found all Answer ‘Yes / Probably yes’ if you searched relevant trials registers and the search NA / Y / PY / PN / N
eligible studies regardless of their results? strategy was designed so that it would retrieve studies regardless of which outcomes
were reported, or if there was another reason why you expect to have found all
eligible studies regardless of their results [and specify the reason in the box below]
Y: ‘Yes’; PY: ‘Probably yes’; PN: ‘Probably no’; N: ‘No’; NA: ‘Not applicable’.

Check the box below if the response to 3.1 was ‘No’ and the response to 3.2 or 3.3 was ‘No / Probably no’
☐ Circumstances indicate potential for some eligible studies not being identified because of the P value, magnitude or direction of the results generated

Provide any relevant information to support responses

5
Step 4. Assess risk of bias due to missing evidence in a meta-analysis (complete for each meta-analysis)
Responses underlined in green are potential markers for low risk of bias, and responses in red are potential markers for a risk of bias.

Details of the meta-analysis being assessed for risk of bias

Specify the meta-analysis For example, “Random-effects meta-analysis of the effect of ibuprofen versus placebo on pain intensity at short-term (0-
12 weeks)”
Specify the meta-analysis result (e.g. For example, “Mean difference -15.00, 95% CI -23.99, -6.01”
summary effect estimate and 95% CI)
Specify the number of included studies and For example, “10 studies (4,934 participants)”
participants
Risk of bias assessment
Signalling questions Elaboration Response options
The following questions relate to the within-study assessment of non-reporting bias (‘known unknowns’)
4.1. Of the studies identified, was there any Note: In software to be developed to implement the tool, responses to this question will Y/N
for which no result was available for be prefilled automatically based on what users enter into the Results Matrix (Step 2).
inclusion in the meta-analysis, likely because
Answer ‘Yes’ if any of the studies in the Results Matrix were marked with an ‘X’ for this
of the P value, magnitude or direction of the
particular meta-analysis.
result generated (refer to Step 2)?
4.2. If Y to 4.1: Is it likely that there would be First, consider whether the amount of missing evidence is large enough that its omission NA / Y / PY / PN / N / NI
a notable change to the summary effect is likely to lead to a notable change in the summary (combined) effect estimate observed
estimate if the omitted results had been (regardless of how large the observed estimate is). Second, consider the direction of
included? effect (e.g. favours experimental intervention) for any studies missing from the meta-
analysis, if such information is known (e.g. when study authors only report that “pain was
lower in the drug group”, without presenting summary statistics or effect estimates). It
may be helpful to append any known studies that are missing from the meta-analysis to a
forest plot, for example using the template presented in Figure 1.
Answer ‘Yes / Probably yes’ if the amount of missing information is non-trivial and, if
known, the direction of effect in omitted studies differs from the direction of effect for
the meta-analysis, and hence the omission is likely to lead to a notable change in the
magnitude of the summary effect estimate. If the meta-analysis was estimated using a
fixed-effect model, consider the total weight of the studies missing from the meta-
analysis. If the weight of missing studies was comparable to or greater than that of the
6
 available studies, there is reason for concern (even more so if the direction of effect in
omitted studies differs from the direction of effect for the meta-analysis). If the meta-
analysis was estimated using a random-effects model, consider the number of studies
missing and extent of statistical heterogeneity in the observed meta-analysis; if there is
minimal heterogeneity, inclusion of just one omitted study could shift the summary
effect estimate because of how the weights are distributed.
Answer ‘No / Probably no’ if the amount of missing information is trivial (i.e. so small
that the study’s omission is likely to have little or no impact on the magnitude of the
summary effect estimate). For example, if a fixed-effect meta-analysis model was used,
and the studies in the meta-analysis each included thousands of participants whereas
studies missing from the meta-analysis each included at most a hundred participants, the
inability to include the missing study results will not notably impact the magnitude of the
summary effect estimate.
Only answer ‘No information’ if the sample size of any of the studies missing from the
meta-analysis was unclear.
4.3. Of the studies identified, was there any Note: In software to be developed to implement the tool, responses to this question will Y/N
for which it was unclear whether an eligible be prefilled automatically based on what users enter into the Results Matrix (Step 2).
result was generated (refer to Step 2)?
Answer ‘Yes’ if any of the studies in the Results Matrix were marked with a ‘?’ for this
particular meta-analysis.
4.4. If Y to 4.3: Is it likely that there would be Consider whether the amount of potentially missing evidence (i.e. from studies for which NA / Y / PY / PN / N / NI
a notable change to the summary effect it was unclear whether an eligible result was generated) is large enough to have a
estimate if the potentially omitted results notable effect on the magnitude of the summary effect estimate observed (regardless of
had been included? how large the observed estimate is).
Answer ‘Yes / Probably yes’ if the amount of potentially missing information is non-
trivial, or ‘No / Probably no’ if the amount of potentially missing information is trivial
(see Elaboration for 4.2 for explanations of ‘trivial’ and ‘non-trivial’). Only answer ‘No
information’ if the sample size of any of the studies potentially missing from the meta-
analysis was unclear.

7
The following questions relate to the across-study assessment of non-reporting bias (‘unknown unknowns’)
4.5 Do circumstances (identified in Step 3) Note: In software to be developed to implement the tool, responses to this question will Y/N
indicate potential for some eligible studies be prefilled automatically based on how users respond to the questions in Step 3.
not being identified because of the P value,
Answer ‘Yes’ if the checkbox in Step 3 was checked.
magnitude or direction of the results
generated?
4.6. If Y to 4.5: Is it likely that studies not Core outcome sets that have been in existence for a long time might be helpful for NA / Y / PY / PN / N
identified had results that were eligible for identifying outcomes that would be expected to be measured in all studies (and hence
inclusion in the meta-analysis? results that are likely to have been generated). Similarly, in reviews of regulated
interventions, consider which outcomes are recommended or required for measurement
by regulators. Also, consider the scope of the review question and definition of the
outcome domain being synthesized. For example, in a review of the effect of any mass
media campaign on any health outcome, if the review outcome domain is broad (e.g.
“health behaviours”), then it is likely that all missing studies will have some measure of
that domain (and hence eligible results generated), while if the review outcome domain
is very specific (e.g. tobacco use), it is less likely that all missing studies will have some
measure of that domain. On the other hand, in a narrower review of the effect of mass
media campaigns designed to help people quit tobacco smoking, it would be reasonable
to suspect all missing studies will have measured tobacco use.
Answer ‘Yes / Probably yes’ if eligible results are likely to have been generated in the
potentially missing studies (e.g. because the outcome of interest is typically measured).
Answer ‘No / Probably no’ if eligible results are unlikely to have been generated in the
potentially missing studies (e.g. because the outcome of interest is rarely measured).
4.7. If Y to 4.1, 4.3 or 4.5: Does the pattern of The pattern of observed study results can be determined using various approaches, NA / Y / PY / PN / N
observed study results suggest that the including (but not limited to): (1) generating and visually inspecting funnel plots; (2)
meta-analysis is likely to be missing results testing for funnel plot asymmetry; (3) comparing the summary effect estimate generated
that were systematically different (in terms under a fixed-effect and random-effects model, or; (4) visually inspecting the P value,
of P value, magnitude or direction) from magnitude or direction of the study results presented in a forest plot or table. These
those observed? approaches may reveal a tendency for the intervention effects estimated in smaller
studies to differ from those estimated in larger studies (‘small-study effects’), for which
non-reporting biases are one of several possible explanations (e.g. smaller studies with
statistically non-significant results were never published). Users should attempt to
distinguish the different possible reasons for small-study effects (e.g. non-reporting
biases, risk of bias in selection of the reported result leading to spuriously inflated effects
8
 in smaller studies, more thorough implementation of the intervention in smaller studies
leading to larger intervention effect estimates). Adding contour lines corresponding to
perceived ‘milestones’ of statistical significance (e.g. P = 0.01, 0.05, 0.1, etc.) to funnel
plots and considering factors such as the risk of bias in included studies and the particular
intervention and the circumstances in which it was implemented in different studies can
help identify the likely cause of small-study effects.
Answer ‘Yes / Probably yes’ if there is a tendency for the intervention effects estimated
in smaller studies to differ from those estimated in larger studies (‘small-study effects’),
and the reason for small-study effects is likely to be non-reporting biases. For example, if
a contour-enhanced funnel plot reveals that studies appear to be missing in areas where
results would be statistically non-significant and unfavourable to the experimental
intervention, then this adds credibility to the possibility that the asymmetry is due to
non-reporting biases.
Answer ‘No / Probably no’ if there is no evidence of small-study effects, or if small-study
effects is likely caused by other factors (e.g. risk of bias arising from the randomization
process in some studies, clinical heterogeneity).
Answer ‘Not applicable’ if results are available for a very small number of studies (and
hence no pattern can be observed).
4.8. If Y/PY/NI to 4.2, 4.4, 4.6 or 4.7: Did Several sensitivity analyses have been developed to assess whether a meta-analysis NA / Y / PY / PN / N
sensitivity analyses suggest that the result is robust. Some are relatively straightforward to implement (e.g. restricting the
summary effect estimate was biased due to meta-analysis to the largest studies) whereas others are more complex, producing
missing results? estimates that depend on strong assumptions about the nature of the likely missing
results (e.g. selection models, regression-based adjustment methods). Furthermore,
existing sensitivity analyses do not distinguish between evidence missing due to
unpublished studies versus non-reported results, making it difficult for the user to
determine whether the methods are flagging bias beyond that already detected by the
within-study assessment above. Careful application of the more complex methods is
required, and we recommend consultation with a statistician with suitable knowledge.
Answer ‘Yes / Probably yes’ if sensitivity analyses suggest that the summary effect
estimate would change notably under a range of plausible assumptions about the nature
of the missing results.

9
 Answer ‘No / Probably no’ if no sensitivity analyses were attempted, or if the summary
effect estimate was relatively stable across the primary and sensitivity analyses
attempted.
Risk of bias judgement See Figure 2 and Table 1. Low / High / Some concerns
Optional: What is the predicted direction of Favours experimental /
bias for this meta-analysis? Favours comparator /
Towards null /Away from
null / Unpredictable
Y: ‘Yes’; PY: ‘Probably yes’; PN: ‘Probably no’; N: ‘No’; NI: ‘No information’; NA: ‘Not applicable’.

10
Box 1. Possible scenarios in which study results are missing (adapted from Kirkham et al. BMJ
2018;362:k3802)

Examples of scenarios where it is reasonable to suspect, given a lack of explanation from the
study investigators, that a result is missing because of the P value, magnitude or direction of the
result itself:
• study authors report in the Methods section, trials register entry, protocol or elsewhere
that they measured (or intended to measure) the outcome of interest, but results are
missing for the outcome;
• all results for an outcome were statistically non-significant and are reported incompletely
(e.g. described only as “results were not significant”, or means were reported with no
measure of precision, or within-group change scores for the experimental group were
reported but no data for the comparison group were presented), whereas results for
other outcomes that were statistically significant are reported completely;
• results are missing for one of two outcomes that tend to be measured together (e.g.
results are available for cause-specific mortality and are favourable to the experimental
intervention, yet results for all-cause mortality, which must have been assessed given
cause-specific mortality was, are missing);
• study authors pre-specified that they would report results separately for different
outcomes (e.g. myocardial infarction, stroke, hypertension) yet instead report results for a
composite outcome (e.g. “cardiovascular events”) which happens to be statistically
significant and favourable to the experimental intervention;
• summary statistics (number of events, or mean scores) are available only globally across
all groups (e.g. study authors state that 10 of 100 participants in the study experienced
adverse events, but do not report the number of events by intervention group); and
• a result is expected to have been generated for an outcome but it is not available and
there is notable concern about conflicts of interest of primary study investigators or
funders involved in the analysis or reporting, which have likely influenced them to
withhold results that are unfavourable to the intervention (an assessment using TACIT –
“Tool for Addressing Conflicts of Interest in Trials” (PMID: 31292209) – for the study
should facilitate this judgement).

Examples of scenarios where it is reasonable to assume that a result is missing for a reason
unrelated to the P value, magnitude or direction of the result include:
• it is clear that the outcome of interest was not measured in the study based on
examination of the study protocol or statistical analysis plan or correspondence with the
authors/sponsors;
• it can be assumed that the outcome of interest was not measured in the study because
the instrument or equipment needed to measure the outcome was not available at the
time or location where the study was conducted;
• the outcome of interest was measured but data were not analysed at all due to a reason
unrelated to the nature of the results (e.g. there was a fault in the measurement
instrument, funding for the research team discontinued, study staff changed jobs);
• study authors state that results for all outcomes measured appear in an appendix, but the
appendix has been removed (or was not uploaded) by accident;
• study authors report results in a format unsuitable for inclusion in a meta-analysis for a
reason unrelated to the P value, magnitude or direction of the result (e.g. study
investigators report median and (interquartile) range for a continuous outcome because
the data were skewed).

11
Figure 1. Example forest plot displaying results missing from a meta-analysis of the effect of preoperative exercise training compared with usual care on postoperative
complications (data from PMID 34429375). The forest plot displays studies with results that were able to be included in the meta-analysis along with studies judged to be
missing from the meta-analysis, likely because of the P value, magnitude or direction of the result generated. If known, the sample size, P-value and direction of effect of such
studies with missing results should be presented on the forest plot. There is no need to include on the forest plot: studies for which the outcome was not measured; studies
for which no result was available for inclusion in the meta-analysis, but for a reason unrelated to the P value, magnitude or direction of the result (e.g. because the
measurement instrument was completed incorrectly by all participants), and; studies for which it was unclear whether an eligible study result was generated.

12
Figure 2. Suggested algorithm for reaching risk-of-bias judgement

13
Table 1. Suggested algorithm for reaching risk-of-bias judgement

Low risk (i) Within-study assessment: No study has missing or potentially missing results OR Some studies have missing or potentially missing results but including
such results is unlikely to lead to a notable change to the summary effect estimate.
AND
(ii) Across-study assessment: (Circumstances do not indicate potential for missing studies OR Circumstances do indicate potential for missing studies but
missing studies are unlikely to have eligible results) AND Pattern of results suggests there are unlikely to be missing results that were systematically different
from those observed.
Some (i.1) Within-study assessment: No study has missing or potentially missing results OR Some studies have missing or potentially missing results but
concerns including such results is unlikely to lead to a notable change to the summary effect estimate.
AND
(i.2) Across-study assessment: (Circumstances indicate potential for missing studies and missing studies are likely to have eligible results OR Pattern of
results suggests there are likely to be missing results that were systematically different from those observed) BUT Sensitivity analyses do not suggest that
the summary effect estimate was biased due to missing results.
OR
(ii.1) Within-study assessment: Some studies have missing results but it is unclear if including such results would lead to a notable change to the
summary effect estimate OR Some studies have potentially missing results and including such results is likely to lead to a notable change to the summary
effect estimate (or this is unclear).
AND
(ii.2) Across-study assessment: Pattern of results suggests there are unlikely to be missing results that were systematically different from those observed
AND Sensitivity analyses do not suggest that the summary effect estimate was biased due to missing results.
High risk (i.1) Within-study assessment: Some studies have missing results and including such results is likely to lead to a notable change to the summary effect
estimate.
AND
(i.2) Across-study assessment: Any response.
OR
(ii.1) Within-study assessment: No study has missing or potentially missing results.
AND
(ii.2) Across-study assessment: Circumstances do indicate potential for missing studies but missing studies are unlikely to have eligible results AND
Pattern of results suggests there are likely to be missing results that were systematically different from those observed AND Sensitivity analyses suggest
that the summary effect estimate was biased due to missing results.
OR
(iii.1) Within-study assessment: Some studies have missing or potentially missing results but including such results is unlikely to lead to a notable change
to the summary effect estimate.
AND

14
 (iii.2) Across-study assessment: Circumstances do not indicate potential for missing studies OR Circumstances do indicate potential for missing studies
but missing studies are unlikely to have eligible results AND Pattern of results suggests there are likely to be missing results that were systematically
different from those observed AND Sensitivity analyses suggest that the summary effect estimate was biased due to missing results.
OR
(iv.1) Within-study assessment: Any response.
AND
(iv.2) Across-study assessment: Circumstances indicate potential for missing studies and missing studies are likely to have eligible results AND (Pattern of
results suggests there are likely to be missing results that were systematically different from those observed OR Sensitivity analyses suggest that the
summary effect estimate was biased due to missing results).

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

15