Secondary Research

Secondary research refers to research conducted using data from other studies. When conducting secondary research, individual patients are not
recruited but eligible studies are ‘recruited’. Two important types of secondary research are 1] Narrative reviews 2] Systematic reviews

Narrative reviews:
These reviews are usually carried out by experts in the field of study and are more or less aligned with the expert’s own opinion on the issue supported
by selected research evidence. These reviews are generally broad without a narrow or specific clinical question and are subjective. They carry a
higher risk of being prone to bias than systematic reviews. Literature is not searched methodically but rather in a random manner to suit one’s flow of
argument. Because of the above, narrative reviews cannot be replicated (unless repeated by the same author!). Nevertheless narrative reviews can
provide a good introduction to a topic, stimulate interest and controversies and can even collate existing data for generating new hypotheses.
Systematic reviews:
Systematic reviews follow rigorous steps in identifying relevant
literature, appraising individual studies and analysing suitable
data to synthesise a conclusion. The characteristics of a
systematic review are
1. Study a focused narrow question
2. Comprehensive & specific data collection (see table)
3. Uniform criteria for study selection (see table)
4. Quantitative synthesis of data (may or may not be
possible)
The 4th feature is optional as not all data are ‘combinable’ to
produce a final common result. Also in some situations good
quality data may not be available to combine and synthesise a
concluding result. If the statistical combination is possible, then
such procedure of quantitative synthesis of individual study
data is called a meta-analysis.

Individual trials’ quality: If the primary studies included have poor quality, then even if sound methodological and statistical procedures are followed
the outcome may be meaningless. This is called GIGO principle of ‘garbage in, garbage out’!!
The quality of individual trials can be assessed by considering
1. The nature of the patient sample,
2. Outcomes studied,
3. Length of follow up and
4. Comparability (e.g. dosage) of treatment and
5. Methodological factors such as
1. Adequacy of sample randomisation, 2.adequate concealment and control of intervention, 3. Analysis with the intention to treat principles, and 4. Having an
objective, blinded, outcome assessment

© SPMM Course
A PROPER LITERATURE SEARCH MUST HAVE FOLLOWING The inclusion criteria for studies in a systematic
CHARACTERISTICS: review should consider

1. Using multiple databases – no just limiting to MEDLINE only or CINAHL (1) Types of study designs to include
only.
(2) Types of subjects to include
2. Cross checking the references list of each individual study retrieved by
direct search (3) Types of publications
3. Hand searching for materials unidentified online. (4) Language restrictions, if any
4. Approaching experts to comment on any missing studies.
(5) Types of interventions
5. Identifying grey literature – i.e. the unpublished literature such as
conference abstracts, presentations and posters. (6) Time frame for included studies

Meta-analysis:

• A meta-analysis is most often used to assess the clinical effectiveness of healthcare interventions.
• Hence RCTs are the most frequently combined studies for a meta-analysis though it is possible to combine other study designs.
• A meta-analysis may either use summary data or individual patient data. For
Quality  of  a  meta-­‐analysis  or  systematic  review  depends  on  the  
example, if one wants to do a meta-analysis of 3 studies each having 50 patients
enrolled, using summary method your N will be 3. In case if one is following following  factors  
individual patient data method this requires contacting the authors of each of the •A comprehensive literature search ( see below)
3 trials and requesting the data of the subjects. Though this is ideal, the inability •Clearly defined eligibility criteria for the studies to be included ( see
of investigators to supply data, and the increased costs associated with the below)
retrieval of individual patient data make researchers carry out summary data •Clearly defined & strictly adhered protocol.
method more often. •Predetermined criteria related to the quality of trials
• The four basic steps for conducting a meta-analysis include •Assessing the quality of a study can be difficult since the information
(1) Literature search, reported is often inadequate for this purpose. In these cases, a thorough
(2) Establishing criteria for including and excluding studies sensitivity analysis must be conducted.
(3) Recording of data from the individual studies, and •Discussion and analysis of statistical and clinical heterogeneity.  
(4) Statistical analysis of the data.

Combining individual trial data:

Ø Methods used for meta-analysis use a weighted average of the results.

Ø Weighting refers to the significance attached to each study based on sample size, precision, external validity ( the extent to which results are
generalisable) and methodological quality. It is assigned independently of the outcome of a study.
Ø Two models of analysis:
• When combining the outcomes from different studies, one may use a fixed and/or random effects model.
© SPMM Course
 • A fixed effects model assumes that all the studies share the same common treatment effect
(homogeneous) while a random effects model assumes that they do not share the same
common treatment effect.
• In fixed effect analysis
o The inference is restricted to included set of studies.
o It assumes that only random error within studies could explain observed differences.
o It ignores between-study variations (hence heterogeneity).
o So this can be applied only if heterogeneity can be safely excluded by testing for it.
• Random effects analysis
o It assumes that each study shows a different effect which are normally distributed
around true mean.
o This assumption gives proportionally greater weight to smaller studies.
o Hence, this model is susceptible to publication bias and results in wider less precise confidence intervals.
• In the absence of significant heterogeneity (often calculated using the Q statistic), both the fixed and random effects models yield
similar confidence intervals.
• Although neither of two models is wrong or inaccurate, a significant difference in the combined effect calculated by the two models will
be seen only if studies are very heterogeneous.
• In the presence of significant heterogeneity, the random effects model will yield wider confidence intervals which may be more
representative.
Clinical heterogeneity

Heterogeneity can occur for clinical reasons in trials. For example, even if the same outcome / end result is considered (e.g. increase in insight), the
interventions may be diverse, such as providing psychoeducation sessions, leaflets, videotapes, audiocassettes or talking to the family. Though the
results can be combined theoretically (all odds risk differences having weighted mean) this would not be meaningful. The interventions are so
different that combining them does not make clinical sense. This is an example of clinical heterogeneity. Other circumstances that may give rise to
clinical heterogeneity include differences in selection of patients, the severity of disease and dose or duration of treatment. Clinical heterogeneity is
describable but not measurable. (Retrieved from http://www.bmj.com/content/334/7584/94)

Methodological Heterogeneity:

This refers to heterogeneity resulting from the differential use of study methodology. These may lead to different conclusions in different studies, even
though the clinical characteristics are the same. Methodological heterogeneity is describable but does not need quantification.

Statistical heterogeneity

In a systematic review, individual studies may purport to measure the same outcome but often report results that are not consistent with each other.
Some trials may show a benefit while others show a lack of response, or the trials may be inconsistent in the size of the observed difference between the
placebo and active treatment arms. This variation is called statistical heterogeneity. A homogenous sample refers to a set of individual studies that have
comparable outcomes without much variation (i.e they do not disagree greatly among each other). Heterogeneity refers to the presence of significant
variation among the individual studies in a sample.

Test for statistical heterogeneity across studies:

© SPMM Course
 § Heterogeneity may be judged graphically or be measured statistically.
§ Forest Plot & L'Abbé plot can be used to explore the inconsistency of studies visually. Forest plot is described elsewhere (see blobbogram).
§ L’Abbe plot is nothing but a modified scatter plot wherein CER is plotted aganst EER from individual trials included in the meta-analysis.
Trials that report experimental treatment to be superior to the control arm (EER > CER) will be in the upper left of the plot, between the y-axis
and the diagonal line of equality. If experimental is no better than control then the point will fall on the line of equality (EER = CER), and if
control is better than experimental then the point will be in the lower right of the plot, between the x-axis and the line of equality (EER < CER).
This provides a visually clear idea of how heterogeneous the results are.

àL’Abbe plot:

Note that proportions improved in both

arms of individual studies are plotted
across the X and Y axis.(From
Bandolier website)

§ The Galbraith plot is an alternative to a

forest plot as where on the horizontal axis one
plots 1/standard error (1/SE = a measure of
precision) of the study effect estimate while on
the vertical axis one plots the study effect
estimate divided by its standard error (e.g. log
odds ratio/ SE = called standard normal deviate).
§ Statistical procedures can test whether the results of a study reflect a single underlying effect
(homogenous) or a distribution of effects (heterogeneous). A major limitation of this approach is that the statistical tests lack the power to detect
heterogeneity in most meta-analyses.
§ Chi-square tests of heterogeneity can be either Q test or I2 test.
§ The chi-square tests provide a test of significance for heterogeneity, but do not measure it.
§ As a rule of thumb (Greenhalgh, 2001) a χ2 value on average is equal to its degree of freedom, which is n-1 (number of studies – 1). So for a
meta-analysis of 8 trials, n-1 is 7, and χ2 value of less than or = 7 means no significant heterogeneity is present. But because of the low power of
the test, one cannot say that homogeneity is present either!
§ A well-known approach to quantify heterogeneity is Cochran’s Q, calculated as the weighted sum of squared differences between individual
study effects and the pooled effect across studies.
§ The I² statistic describes the percentage of variation across studies that are due to heterogeneity rather than chance.
§ In these chi-square tests, high P suggests that the heterogeneity is insignificant and that one can perform a meta-analysis.
§ It is important that heterogeneity should not be considered as a mere problem to carry out meta-analysis; it must be investigated, and reasons
must be dissected (similar to transference analysis in psychodynamics!).
§ In case there is a significant heterogeneity, one can either study the reasons such heterogeneity, which can be
o 1. Clinical differences,
o 2. Methodological issues such as problems with randomization,
o 3. Early termination of trials,
o 4. Use of absolute rather than relative measures of risk, etc. and
© SPMM Course
 o 5. Publication bias.
Or, one can resort to random effects analysis. Procedures such as meta-regression analyses and subgroup analysis can throw light on the
causes of heterogeneity (see below).

Publication bias:

• This refers to the tendency for authors to submit and/or editors to publish only those studies that yield statistically significant results (i.e. ‘positive’
and rejecting the null hypothesis).
• There are several methods available to diagnose the presence of publication bias in the literature. Most of these methods operate on the basis that
small studies are more susceptible to publication bias and may, consequently, show larger treatment effects.
• Application of tests for publication bias in the presence of heterogeneity (described below) is not valid, and may lead to false-positive claims for
publication bias (Ioannidis, 2007).
• When all available studies are equally large (i.e., have similar precision), the tests are not meaningful.

Funnel plot: The most common method to detect publication bias is to investigate the presence of asymmetry in (inverted) funnel plots. A funnel plot
shows the relation between the effect size and precision of the individual studies in a meta-anaysis. Small studies are more likely to remain unpublished
if their results are nonsignificant or unfavourable, and larger studies get published regardless prducing an asymmetrical funnel-plot. Although visual
inspection of funnel plots is often unreliable, several statistical tests can be used to quantify
the asymmetry.
Failsafe N: Meta-analytic results could be wrong, due to non-significant studies that remained
in researchers’ ‘file drawers’ and not published. One can speculate whether the inclusion of
these studies would nullify the statistical significance of an observed mean effect in a meta-
analysis. Rosenthal & Cooper developed a formula to calculate the number of zero-effect
studies (no effect) that would be required to nullify the mean effect seen in a meta-analysis.
This number was termed the failsafe N or file-drawer number.
Other tests for publication bias:  Either linear regression method or correlation method (Tau)
can be used to quantify the publication bias. Details of such methods are beyond the scope of
MRCPsych exams.
Cumulative meta-analysis: This can be used to assess the potential impact of publication
bias in tilting or nullifying an effect. A cumulative meta-analysis is done in the following
manner:
o The studies are sorted from largest to smallest (or from most to least precise).
o A meta-analysis is initially run with one study, then repeated with a second study
added, then a third, and so on;
o A forest plot/blobbogram (see elsewhere in the notes) is plotted as one goes along the process of accumulation. Here the first row will show
the effect based on one study, the second row shows the cumulative effect based on two àFunnel plot:
studies and so on.
o If the point estimate stabilizes based on the initially added larger studies, then there is no Note that the funnel is well formed in this example. X axis of
evidence that the smaller studies are producing a biased overall effect. funnel plot carries a measure of effect size (OR, log OR,
o If, however, the point estimate shifts when the smaller studies are added, this can be Cohen’s d etc). Y axis carries a measure of precision such as
considered as an evidence of bias and one can also see the direction of the biased effect. sample size, inverse of standard error (1/SE) etc. (From
Cochrane database training section)
© SPMM Course
The trim-and-fill procedure: This was developed by Duval and Tweedie (2000). This helps to assess whether the effect would change if the bias were to
be removed. Trim and fill procedures is a kind of sensitivity analysis where missing studies are imputed and added to the analysis, and then effect size
is recomputed.

(Other biases that can occur include location bias – studies not being located due to citation habits, the database used, key terms used or multiple
replications of same data.

Inclusion bias refers to the reviewers’ tendency to include studies they agree with.)

Sensitivity analysis: This refers to the analysis of the extent to which an outcome derived from research will change when ‘hypothetical data’ are
introduced. One always wonders if an established result could be different if certain aspects of a research process were different. Such an exploration
of ‘what ifs’ is called sensitivity analysis (Greenhalgh, 2001).This applies to mainly to economic analysis and meta-analysis. In a meta-analysis, there are
various nodal points where such hypothetical data can be introduced. For example
1. What if there was significant heterogeneity (or absence of it) – how will this affect the outcome? One can do analysis in both random and
fixed effects models.
2. Factors determining methodological quality e.g. numbers allocated in either arm, type of outcome measure used, case definition used, etc.
can be manipulated for sensitivity analyses.
3. Publication bias can be examined using sensitivity analysis methods e.g. failsafe N.

Blobbograms:

o A blobbogram or forest plot presents the effect (point estimate) from each individual study as a blob or square (the measured effect), with a
horizontal line (usually the 95% confidence interval, indicating the precision) across the blob.
o The size of the blob or square Forest plot: Things to note in a forest plot
reflects the amount of information in
that individual study (often the 1. The combined /pooled effect size (given by various summary statistics including OR, RR, standardised mean
sample size); differences, Cohen’s d etc)
o The length of the horizontal line 2. Confidence interval of individual point (line width) estimates and combined estimate (lozenge width)
represents the degee of uncertainty 3. Weighting assigned to studies (rectangle size)
of the estimated treatment effect for 4. Heterogeneity of results (absence of heterogeneity indicated by vertical linearity of rectangles or non significant chi
that study. square test for heterogeneity)
American Journal of Psychiatry 2003; 160:1919–1928
o A mid-vertical line is often
constructed to represent null effect for the differences in outcome; on either side lies the area favouring intervention A or B. When the blob or
square lies on one side of this vertical line, this means that the outcome of that specific favoured the intervention marked on that side. If an
individual study result is statisticsally significant then the horizontal lines representing uncertaininty will lie entirely on the same side of the
blob, not touching or cross the midvertical line.
o A final synthesis of individual studies is represented in the form of a diamond lozenge. The position of the lozenge depends upon whether the
intervention rates favourably or not at the end of the meta-analysis.
o The following figure is a forest plot

© SPMM Course
Effect size:
§ Effect size measures for dichotomous variables: In meta-analyses, relative risk and odds ratio have the advantage in comparison with the absolute
risk that they depend less on the baseline risk. OR or RR are appropriate as an effect size measure when both variables are binary (cured vs. not
cured). Conventionally, the odds ratio is commonly used in reporting treatment effects. OR is more or less equal to relative risk and so an odds
ratio of 3 can be interpreted as ‘the outcome of interest happens
about thrice as often in the intervention group as in the control
group’. But this is not accurate (simply equating OR and RR) as this
will tend to overestimate the effect of treatment. But the level of this
overstatement is almost always quite small, especially when the
experimental and control event rates are less than 20%. However,
for frequent events (approximately over 20%), the odds ratio turns
out significantly greater than the relative risk, and falsely equating
relative risk and odds ratio, then leads to an overestimation of the
risk. For this reason, the relative risk is generally used in Cochrane
reviews.
§ Effect sizes for continuous variables: There are basically two
effect size measures for continuous variables: Simple difference
between the mean values (DM) keeps the original units and simply
is the mean value of group X minus the mean value of group Y.
Standardized difference between the mean values (SMD): This is
carried out by dividing the DM by the pooled standard deviation of
the two groups in the basic formula: SMD = (mean value of group X )
mean value of group Y) ! pooled standard deviation of the two
groups. Various formulas for the calculation of SMDs exist such as
e.g. Cohen’s d or Hedges' g, the latter avoiding a bias in Cohen’s d,
seen with small number of subjects.
Statistical methods for fixed vs. random effects:

Model Statistical procedures Measured effects

Mantel-Haenszel ( see SPMM notes on advanced statistics) OR, Rate ratios, Risk ratios (RR)
Fixed effects
Peto Approximates OR

Random effects DerSimonian-Laird Ratios and rate differences

None of the both above mentioned fixed effects methods allow for confounding to be assessed – hence they are not useful if confounding is suspected due to methodological issues. They are
chi-square like tests, which need 2X2 tables for each individual study.

© SPMM Course
A meta-analysis may not always give results that equate with large mega-trials that may be conducted later. For example google ‘Lessons from an
effective, safe, simple intervention that was not’ (use of intravenous magnesium after heart attacks).

More about meta-analyses:

¤ Consensus statement on standard format for meta-analysis:

QUORUM statement
¤ Consensus statement on standard format for RCTs: CONSORT
statement
¤ Meta-regression analyses: Refers to a technique of regression
wherein regression model is applied to meta-analysis to analyse
which characteristics of the studies actually contributed to overall
effect size.
References:

The QUOROM guidelines: Moher D et al. Improving the quality of reports of meta-analyses of
randomized controlled trials: the QUOROM statement. Lancet. 1999;354:1896-1900

Hunter JE, Schmidt FL. Fixed effects vs. random effects meta-analysis models: implications for cumulative
research knowledge. International Journal of Selection and Assessment 8:275-292, 2000

Fletcher, J. What is heterogeneity and is it important? BMJ 2007; 334: 94-96

Egger, M et al. Meta-analysis: principles and procedures BMJ 1997;315:1533-1537

Ioaniidis, JPA & Trikalinos TA. The appropriateness of asymmetry tests for publication bias in meta-
analyses: a large survey. CMAJ 2007; 176 (8). doi:10.1503/cmaj.060410
 

DISCLAIMER: This material is developed from various revision notes assembled while preparing for MRCPsych exams. The content is periodically updated with excerpts from various
published sources including peer-reviewed journals, websites, patient information leaflets and books. These sources are cited and acknowledged wherever possible; due to the
structure of this material, acknowledgements have not been possible for every passage/fact that is common knowledge in psychiatry. We do not check the accuracy of drug-
related information using external sources; no part of these notes should be used as prescribing information.

© SPMM Course