Critical Appraisal of QUANTitative research

Tips from Cardiff University RSSDP course

by  Fiona Morgan on 14th February 2012

Why?

• Critical appraisal = assess the quality of the papers in terms of stats, relevance
• It’s arguably a science of trashing paper
• The are vast of literature out there. Hence it’s very time-consuming to find all of what you want. Then how to judge those loads.
• Moreover quality of peer-reviewed journal papers are questionable.
• Some studies have contradict results or different effect size, also a reason for using Meta Analysis.
• Many of abstracts and conclusions are not substantiated in the paper.

Assessing validity

• “if you’re going to trash a paer, you should do it before you even look at the results” (Trish Greenhalgh)
• Judged by the research methods used in the paper.

General questions

• What is the paper about?
• Is it relevant? – Using set of inclusion and exclusion criteria beforehand
• Do I trust it?
• What are the results? – the last question!

Systematic review

• Procedure should be clear in order to be able to reproduced the result with the same or new (updated) data.
• More trustworthy when well conducted.
• Present results from all the relevant, reliable research
• Reliability of results dependent on the quality of included studies
• Quality of the included studies should be counted in the Meta Analysis

Key questions

• Is there a focused research question?
  – PICO / PECO
  P = Population/Problem
  I/E = Intervention, Exposure
  C = Comparison / Control
  O = Outcome
• Will the search strategy find the evidence?
  – Range of database
  –  Comprehensive search terms? (Indexed terms and keywords)
  – Sufficient range of years?
  – Includes a flow diagram?
• Was the study selection and data extraction objective?
• Will the search find the evidence?
  Snowballing
  – Hand-searching
  – Reference lists
  – Contracting experts and practitioners
  – Grey literature e.g., conference abstracts or website
  – Unpublished research – different results, publication bias
• Was study quality considered?
  – Critical appraisal of included studies? – reliability dependent on quality of included studies
  – Well conducted reviews – two reviewers appraise studies independently
• Were the included studies similar?
  – Do you have enough information to judge>
  – Inclusion criteria should identify comparable studies
  – Heterogeneity – difference between studies

Meta-analysis or Narrative Synthesis?

• How different are the studies ?
• Homogeneity  -> Fixed effect model,
• Moderate heterogeneity -> Random effect model
• Significant heterogeneity -> narrative synthesis

Meta Analysis

• Results od studies combined to produce a weighted average
• More weight given to larger studies with precise effect estimates
• Wider the confidence interval, the less precise the result
• If the confidence interval crosses the line of no effect, the result is NOT significant

Are the  results due to chance?

• Result may simply have occurred by chance
• Role of chance dealt with by statistical techniques, p-values, confidence interval

Is it relevant?

• Is the population sufficiently similar to yours?
• Were the important outcomes measured?
• Were the results theoretically significant? sound?

Also think about

• Sponsorship and conflict of interest
• Date of review
• Author-identified review limitations
• Are the conclusions the same in the abstract and the full text

Initial Screening 

• Is this the right study type?
  – Ab intervention looking at treatment outcomes
• Can you identify a PICO? Population, Intervention, Control group, Outcomes (objective? validated? measurable? primary or surrogate?)

Randomisation / Allocation Concealment

• Randomisation process appropriate?
• Allocation hidden? -> source of bias of selection/allocation

Results?

• Are the results clearly presented?
  – P-values with confidence intervals
• Are they meaningful?
  – Clear statement of outcomes?
  – Statistically significant?
• Information provided?
  – Missing data

Key considerations

• Do the authors provide a power calculation?
• Relevance? similar population, theoretically sinificant?