Critical Appraisal of QUANTitative research
Tips from Cardiff University RSSDP course
by Fiona Morgan on 14th February 2012
- Critical appraisal = assess the quality of the papers in terms of stats, relevance
- It’s arguably a science of trashing paper
- The are vast of literature out there. Hence it’s very time-consuming to find all of what you want. Then how to judge those loads.
- Moreover quality of peer-reviewed journal papers are questionable.
- Some studies have contradict results or different effect size, also a reason for using Meta Analysis.
- Many of abstracts and conclusions are not substantiated in the paper.
- “if you’re going to trash a paer, you should do it before you even look at the results” (Trish Greenhalgh)
- Judged by the research methods used in the paper.
- What is the paper about?
- Is it relevant? – Using set of inclusion and exclusion criteria beforehand
- Do I trust it?
- What are the results? – the last question!
- Procedure should be clear in order to be able to reproduced the result with the same or new (updated) data.
- More trustworthy when well conducted.
- Present results from all the relevant, reliable research
- Reliability of results dependent on the quality of included studies
- Quality of the included studies should be counted in the Meta Analysis
- Is there a focused research question?
– PICO / PECO
P = Population/Problem
I/E = Intervention, Exposure
C = Comparison / Control
O = Outcome
- Will the search strategy find the evidence?
– Range of database
– Comprehensive search terms? (Indexed terms and keywords)
– Sufficient range of years?
– Includes a flow diagram?
- Was the study selection and data extraction objective?
- Will the search find the evidence?
– Reference lists
– Contracting experts and practitioners
– Grey literature e.g., conference abstracts or website
– Unpublished research – different results, publication bias
- Was study quality considered?
– Critical appraisal of included studies? – reliability dependent on quality of included studies
– Well conducted reviews – two reviewers appraise studies independently
- Were the included studies similar?
– Do you have enough information to judge>
– Inclusion criteria should identify comparable studies
– Heterogeneity – difference between studies
Meta-analysis or Narrative Synthesis?
- How different are the studies ?
- Homogeneity -> Fixed effect model,
- Moderate heterogeneity -> Random effect model
- Significant heterogeneity -> narrative synthesis
- Results od studies combined to produce a weighted average
- More weight given to larger studies with precise effect estimates
- Wider the confidence interval, the less precise the result
- If the confidence interval crosses the line of no effect, the result is NOT significant
Are the results due to chance?
- Result may simply have occurred by chance
- Role of chance dealt with by statistical techniques, p-values, confidence interval
Is it relevant?
- Is the population sufficiently similar to yours?
- Were the important outcomes measured?
- Were the results theoretically significant? sound?
Also think about
- Sponsorship and conflict of interest
- Date of review
- Author-identified review limitations
- Are the conclusions the same in the abstract and the full text
- Is this the right study type?
– Ab intervention looking at treatment outcomes
- Can you identify a PICO? Population, Intervention, Control group, Outcomes (objective? validated? measurable? primary or surrogate?)
Randomisation / Allocation Concealment
- Randomisation process appropriate?
- Allocation hidden? -> source of bias of selection/allocation
- Are the results clearly presented?
– P-values with confidence intervals
- Are they meaningful?
– Clear statement of outcomes?
– Statistically significant?
- Information provided?
– Missing data
- Do the authors provide a power calculation?
- Relevance? similar population, theoretically sinificant?