Tips from Cardiff University RSSDP course
by Fiona Morgan on 14th February 2012
- Critical appraisal = assess the quality of the papers in terms of stats, relevance
- It’s arguably a science of trashing paper
- The are vast of literature out there. Hence it’s very time-consuming to find all of what you want. Then how to judge those loads.
- Moreover quality of peer-reviewed journal papers are questionable.
- Some studies have contradict results or different effect size, also a reason for using Meta Analysis.
- Many of abstracts and conclusions are not substantiated in the paper.
- “if you’re going to trash a paer, you should do it before you even look at the results” (Trish Greenhalgh)
- Judged by the research methods used in the paper.
- What is the paper about?
- Is it relevant? – Using set of inclusion and exclusion criteria beforehand
- Do I trust it?
- What are the results? – the last question!
- Procedure should be clear in order to be able to reproduced the result with the same or new (updated) data.
- More trustworthy when well conducted.
- Present results from all the relevant, reliable research
- Reliability of results dependent on the quality of included studies
- Quality of the included studies should be counted in the Meta Analysis
- Is there a focused research question?
- PICO / PECO
P = Population/Problem
I/E = Intervention, Exposure
C = Comparison / Control
O = Outcome
- Will the search strategy find the evidence?
- Range of database
- Comprehensive search terms? (Indexed terms and keywords)
- Sufficient range of years?
- Includes a flow diagram?
- Was the study selection and data extraction objective?
- Will the search find the evidence?
- Reference lists
- Contracting experts and practitioners
- Grey literature e.g., conference abstracts or website
- Unpublished research – different results, publication bias
- Was study quality considered?
- Critical appraisal of included studies? – reliability dependent on quality of included studies
- Well conducted reviews – two reviewers appraise studies independently
- Were the included studies similar?
- Do you have enough information to judge>
- Inclusion criteria should identify comparable studies
- Heterogeneity – difference between studies
Meta-analysis or Narrative Synthesis?
- How different are the studies ?
- Homogeneity -> Fixed effect model,
- Moderate heterogeneity -> Random effect model
- Significant heterogeneity -> narrative synthesis
- Results od studies combined to produce a weighted average
- More weight given to larger studies with precise effect estimates
- Wider the confidence interval, the less precise the result
- If the confidence interval crosses the line of no effect, the result is NOT significant
Are the results due to chance?
- Result may simply have occurred by chance
- Role of chance dealt with by statistical techniques, p-values, confidence interval
Is it relevant?
- Is the population sufficiently similar to yours?
- Were the important outcomes measured?
- Were the results theoretically significant? sound?
Also think about
- Sponsorship and conflict of interest
- Date of review
- Author-identified review limitations
- Are the conclusions the same in the abstract and the full text
- Is this the right study type?
- Ab intervention looking at treatment outcomes
- Can you identify a PICO? Population, Intervention, Control group, Outcomes (objective? validated? measurable? primary or surrogate?)
Randomisation / Allocation Concealment
- Randomisation process appropriate?
- Allocation hidden? -> source of bias of selection/allocation
- Are the results clearly presented?
- P-values with confidence intervals
- Are they meaningful?
- Clear statement of outcomes?
- Statistically significant?
- Information provided?
- Missing data
- Do the authors provide a power calculation?
- Relevance? similar population, theoretically sinificant?
Tips from Cardiff University RSSDP course
by Fiona Morgan on 13th February 2012
- Vast literature
(1.4 million academics papers published in 32,000 peer-reviewed scientific journals worldwide)
- But limited time to read
- This is actually how to trash the paper
- Methodology should be presented clearly to the readers in order to replicate, assess the quality of your work.
Asking general questions
- What is this paper about?
- Is it relevant to what you are looking for
- Do I trust it? – Methodology
- What are the results? (The last question!)
- Interpretive / subjective e.g., Meaning, experiences, feelings or insights
- Variety of methods
- Focuses in understanding how people think, behave in a particular way.
- Language used in qualitative research mostly are user-unfriendly and difficult to read
Assessing qualitative research
- Do we have confidence in the results?
- Can they be applied in similar settings?
- appropriate design, methodology and process?
- Clear statement of aims?
- What is the purpose
- What outcomes are expected?
- The above should be in the introduction
- Research question – SPICE
- Is a qualitative methodology appropriate?
- Was the recruitment strategy appropriate to the research aims?
- From where?
- Who did it?
- What sample size?
- Why people did/did not take part? - Selection bias
- Were the data collected in a way that considered the research issue?
- Is it clear?
- Where was the study set?
- how were data collected?
- How ere they recorded?
- Were method modified?
- Have ethical issues been taken into consideration?
- Ethical approval
- How was the research explained to the participants e.g., expectation, timescale.
- Was informed consent obtained?
- Reflexivity (research bias)
- Is the research role examined?
- Prepare your presentation for your audience – what do they know before hand?
This should determine language you will use i.e., use of jargons, terminology.
Also this will reflect their interest, what do they want to know from your presentation.
- Speaking is inefficient method of communicating relatively to writing. Hence you need to cut out the unnecessary from the presentation.
- Normal speed of speaking in presentation ~ 120 -150 words per minute = 25% slower than normal speaking.
- Therefore in 10 minutes, maximum number of words you can cover = 15o x 10 = 1,500 words
- Give preparation time before presentation i.e., checking presenting equipment facilities
- Number of slides
Rules depend on the disciplines
History may be just 5 slides for 30 minutes
I n scientific one would be around 1 slide per minute (+ intro and Q&A).
- Visual aids can be used for the following function either “Drivers” or :Supplementary” or “Explanatory”.
Style of Presentation
- Name (unless the chair has already introduced you)
- Thanks – Chair, audience
- Title of Research
- Acknowledgement (or at the end of presentation)
- Broad scope of presentation including what you won’t cover but may be expected the the audiences
- Short hook
- Asking question
- Refer to the current affair (especially in social science)
- Statistics related to the subject
- Aims and Objectives
- 1 – 1.5 minutes for Introduction + Opening
End (summary and then conclusion)
- Summary = reiterate what have been presented already, not the new things.
in bullet points
- Conclusion: Includes comments and suggestion of the future works.
- short punchy phrase
Gesture and Body language
- Gaze – eye contact
- You can look down to your note but don’t speck during that time, just pause. Speck after you look up again.
- Use your eye contact like a lighthouse
- Don’t look at other things, focus on your audiences.
- Don’t pat too much attention on a particular person e.g., experts or persons who nodding their heads all the time.
- Whole body orientation
- Gesture - hands/arms
- Show your hands to people
- get your hands in fornt of you
- Don’t over gesture. bring your hands back to the focal point (e.g., stand or note or other object)
- If you use cards, put numbers on them and stick them together so that you will not mess them when you drop them.
- Use open-palm gesture as it is the normal gesture that doesn’t offense anyone.
- use of space
Tips for questions
- Offer to deal with the details afterwards
- Be prepared to concede
- Useful expression “It’s a case that we may have to agree to disagree”.
- Keep calm and don’t take it personally
- Do not comment on anything outside your area of expertise
- Use the experts in the audience
- If no question, you may ask yourself. “This is the question that I have been asked previously”.
PhD study is mainly about how to do a research. It normally takes more than three years to complete the course. Many PhD candidates have withdrew or failed in the end. I found that on of the root causes is that ‘Students are fail to select the right place to study’.
Based-on my experience I decided to study PhD with Professor Stephen Disney at Logistics Systems Dynamics Group, Cardiff Business School because of the following factors.
This is the most important factor. Choose your supervisor wisely since switching one is difficult and costly in terms of time, effort and moral.
– You should be able to work with your supervisor effectively and efficiently. Think that you can work with him/her for at least 3 years
– Your supervisor should know either the subject or the method of your PhD proposal or both. If you are lucky, you may find two supervisors who are experts in subject and method in the supervisory panel.My primary supervisor is a quantitative professor in operations management and my secondary supervisor got a nationwide award on her qualitative PhD thesis. Hence, I am very lucky!
- Research Group / Academic Section
Doing PhD in a well-supporting research group or academic section will help you financially and academically. Choosing a research group which is one of the leading group in the subject will give you an opportunity to learn from other group members and fellow PhD students as well.My research group, Logistics Systems Dynamics Group, is a leading logistics and supply chain in the UK and Europe.
- Schools / Faculty
Similar to research group, choosing a right school or faculty, you will study in the right environment. This is quite significant for PhD in Logistics and Supply Chain Management. You may go for business school or engineering one or even economics. The decision perhaps depends on your research methods. Please note that you may find many engineering researchers in some business schools and vice versa.Cardiff Business School is ranked 4th in the UK in term of research quality!
Studying in a great university will offer you a cross-disciplinary knowledge and experience. At Cardiff University, I have had a great chance to learn such a rigor social science research methods from School of Social Science and another view of logistics and transports from the School of City and Regional Planning as both are the best in the UK. Moreover, Cardiff University is one of 20 Doctoral Training Centre in the UK.
- City / Country
Not only study but you also have to live happily whilst you are study. PhD is a long journey. Living in a nice place will improve your quality of life, reduce stress from study. Moreover, choosing a place where you can learn outside the university is also important.Cardiff was the biggest marine port in the world. Now, the city is very well developed. Welsh people are very nice too. The living cost is also relatively lower than other big cities in the UK.
Albeit based on my experience, these five factors are from my own experience, one may find them useful when considering where to do a PhD. Think well since you have to be there for three years or more!