Skip to content

Critical Appraisal of QUANTitative research

Tips from Cardiff University RSSDP course

by  Fiona Morgan on 14th February 2012


  • Critical appraisal = assess the quality of the papers in terms of stats, relevance
  • It’s arguably a science of trashing paper
  • The are vast of literature out there. Hence it’s very time-consuming to find all of what you want. Then how to judge those loads.
  • Moreover quality of peer-reviewed journal papers are questionable.
  • Some studies have contradict results or different effect size, also a reason for using Meta Analysis.
  • Many of abstracts and conclusions are not substantiated in the paper.

Assessing validity

  • “if you’re going to trash a paer, you should do it before you even look at the results” (Trish Greenhalgh)
  • Judged by the research methods used in the paper.

General questions

  • What is the paper about?
  • Is it relevant? – Using set of inclusion and exclusion criteria beforehand
  • Do I trust it?
  • What are the results? – the last question!

Systematic review

  • Procedure should be clear in order to be able to reproduced the result with the same or new (updated) data.
  • More trustworthy when well conducted.
  • Present results from all the relevant, reliable research
  • Reliability of results dependent on the quality of included studies
  • Quality of the included studies should be counted in the Meta Analysis

Key questions

  • Is there a focused research question?
    – PICO / PECO
    P = Population/Problem
    I/E = Intervention, Exposure
    C = Comparison / Control
    O = Outcome
  • Will the search strategy find the evidence?
    – Range of database
    –  Comprehensive search terms? (Indexed terms and keywords)
    – Sufficient range of years?
    – Includes a flow diagram?
  • Was the study selection and data extraction objective?
  • Will the search find the evidence?
    – Hand-searching
    – Reference lists
    – Contracting experts and practitioners
    – Grey literature e.g., conference abstracts or website
    – Unpublished research – different results, publication bias
  • Was study quality considered?
    – Critical appraisal of included studies? – reliability dependent on quality of included studies
    – Well conducted reviews – two reviewers appraise studies independently
  • Were the included studies similar?
    – Do you have enough information to judge>
    – Inclusion criteria should identify comparable studies
    – Heterogeneity – difference between studies

Meta-analysis or Narrative Synthesis?

  • How different are the studies ?
  • Homogeneity  -> Fixed effect model,
  • Moderate heterogeneity -> Random effect model
  • Significant heterogeneity -> narrative synthesis

Meta Analysis

  • Results od studies combined to produce a weighted average
  • More weight given to larger studies with precise effect estimates
  • Wider the confidence interval, the less precise the result
  • If the confidence interval crosses the line of no effect, the result is NOT significant

Are the  results due to chance?

  • Result may simply have occurred by chance
  • Role of chance dealt with by statistical techniques, p-values, confidence interval

Is it relevant?

  • Is the population sufficiently similar to yours?
  • Were the important outcomes measured?
  • Were the results theoretically significant? sound?

Also think about

  • Sponsorship and conflict of interest
  • Date of review
  • Author-identified review limitations
  • Are the conclusions the same in the abstract and the full text

Initial Screening 

  • Is this the right study type?
    – Ab intervention looking at treatment outcomes
  • Can you identify a PICO? Population, Intervention, Control group, Outcomes (objective? validated? measurable? primary or surrogate?)

Randomisation / Allocation Concealment

  • Randomisation process appropriate?
  • Allocation hidden? -> source of bias of selection/allocation


  • Are the results clearly presented?
    – P-values with confidence intervals
  • Are they meaningful?
    – Clear statement of outcomes?
    – Statistically significant?
  • Information provided?
    – Missing data

Key considerations

  • Do the authors provide a power calculation?
  • Relevance? similar population, theoretically sinificant?
%d bloggers like this: