Skip to content

Posts from the ‘Research Methods’ Category

Critical Appraisal of QUANTitative research


Tips from Cardiff University RSSDP course

by  Fiona Morgan on 14th February 2012

Why?

  • Critical appraisal = assess the quality of the papers in terms of stats, relevance
  • It’s arguably a science of trashing paper
  • The are vast of literature out there. Hence it’s very time-consuming to find all of what you want. Then how to judge those loads.
  • Moreover quality of peer-reviewed journal papers are questionable.
  • Some studies have contradict results or different effect size, also a reason for using Meta Analysis.
  • Many of abstracts and conclusions are not substantiated in the paper.

Assessing validity

  • “if you’re going to trash a paer, you should do it before you even look at the results” (Trish Greenhalgh)
  • Judged by the research methods used in the paper.

General questions

  • What is the paper about?
  • Is it relevant? – Using set of inclusion and exclusion criteria beforehand
  • Do I trust it?
  • What are the results? – the last question!

Systematic review

  • Procedure should be clear in order to be able to reproduced the result with the same or new (updated) data.
  • More trustworthy when well conducted.
  • Present results from all the relevant, reliable research
  • Reliability of results dependent on the quality of included studies
  • Quality of the included studies should be counted in the Meta Analysis

Key questions

  • Is there a focused research question?
    – PICO / PECO
    P = Population/Problem
    I/E = Intervention, Exposure
    C = Comparison / Control
    O = Outcome
  • Will the search strategy find the evidence?
    – Range of database
    –  Comprehensive search terms? (Indexed terms and keywords)
    – Sufficient range of years?
    – Includes a flow diagram?
  • Was the study selection and data extraction objective?
  • Will the search find the evidence?
    Snowballing
    – Hand-searching
    – Reference lists
    – Contracting experts and practitioners
    – Grey literature e.g., conference abstracts or website
    – Unpublished research – different results, publication bias
  • Was study quality considered?
    – Critical appraisal of included studies? – reliability dependent on quality of included studies
    – Well conducted reviews – two reviewers appraise studies independently
  • Were the included studies similar?
    – Do you have enough information to judge>
    – Inclusion criteria should identify comparable studies
    – Heterogeneity – difference between studies

Meta-analysis or Narrative Synthesis?

  • How different are the studies ?
  • Homogeneity  -> Fixed effect model,
  • Moderate heterogeneity -> Random effect model
  • Significant heterogeneity -> narrative synthesis

Meta Analysis

  • Results od studies combined to produce a weighted average
  • More weight given to larger studies with precise effect estimates
  • Wider the confidence interval, the less precise the result
  • If the confidence interval crosses the line of no effect, the result is NOT significant

Are the  results due to chance?

  • Result may simply have occurred by chance
  • Role of chance dealt with by statistical techniques, p-values, confidence interval

Is it relevant?

  • Is the population sufficiently similar to yours?
  • Were the important outcomes measured?
  • Were the results theoretically significant? sound?

Also think about

  • Sponsorship and conflict of interest
  • Date of review
  • Author-identified review limitations
  • Are the conclusions the same in the abstract and the full text

Initial Screening 

  • Is this the right study type?
    – Ab intervention looking at treatment outcomes
  • Can you identify a PICO? Population, Intervention, Control group, Outcomes (objective? validated? measurable? primary or surrogate?)

Randomisation / Allocation Concealment

  • Randomisation process appropriate?
  • Allocation hidden? -> source of bias of selection/allocation

Results?

  • Are the results clearly presented?
    – P-values with confidence intervals
  • Are they meaningful?
    – Clear statement of outcomes?
    – Statistically significant?
  • Information provided?
    – Missing data

Key considerations

  • Do the authors provide a power calculation?
  • Relevance? similar population, theoretically sinificant?

Critical Appraisal QUALitative Research


Tips from Cardiff University RSSDP course

by  Fiona Morgan on 13th February 2012

Why?

  • Vast literature
    (1.4 million academics papers published in 32,000 peer-reviewed scientific journals worldwide)
  • But limited time to read
  • This is actually how to trash the paper

Reporting

  • Methodology should be presented clearly to the readers in order to replicate, assess the quality of your work.

Asking general questions 

  • What is this paper about?
  • Is it relevant to what you are looking for
  • Do I trust it? – Methodology
  • What are the results? (The last question!)

Qualitative research

  • Interpretive / subjective e.g., Meaning, experiences, feelings or insights
  • Variety of methods
  • Focuses in understanding how people think, behave in a particular way.
  • Language used in qualitative research mostly are user-unfriendly and difficult to read

Assessing qualitative research

  • Credible
    – Do we have confidence in the results?
  • Transferable
    – Can they be applied in similar settings?
  • Dependable
    – appropriate design, methodology and process?

Check list

  • Clear statement of aims?
    – What is the purpose
    – What outcomes are expected?
    – The above should be in the introduction
  • Research question – SPICE
    Setting
    Perspective
    Intervention/Phenomena
    Comparison
    Evaluation/Exploration
  • Is a qualitative methodology appropriate?
  • Was the recruitment strategy appropriate to the research aims?
    – From where?
    – Who did it?
    – How?
    – What sample size?
    – Why people did/did not take part?  – Selection bias
  • Were the data collected in a way that considered the research issue?
    – Is it clear?
    – Where was the study set?
    – how were data collected?
    – How ere they recorded?
    – Were method modified?
  • Have ethical issues been taken into consideration?
    – Ethical approval
    – How was the research explained to the participants e.g., expectation, timescale.
    – Was informed consent obtained?
  • Reflexivity (research bias)
    – Is the research role examined?

Presenting Research Orally


Tips

  • Prepare your presentation for your audience – what do they know before hand?
    This should determine language you will  use i.e., use of jargons, terminology.
    Also this will reflect their interest, what do they want to know from your presentation.
  • Speaking  is inefficient method of communicating relatively to writing. Hence you need to cut out the unnecessary from the presentation.
  • Normal speed of speaking in presentation ~ 120 -150 words per minute = 25% slower than normal speaking.
  • Therefore in 10 minutes, maximum number of words you can cover = 15o x 10 = 1,500 words
  • Give preparation time before presentation i.e., checking presenting equipment facilities

Visual aids

  • Number of slides
    Rules depend on the disciplines
    History may be just 5 slides for 30 minutes
    I n scientific one would be around 1 slide per minute  (+ intro and Q&A).
  •   Visual aids can be used for the following function either “Drivers” or :Supplementary” or “Explanatory”.

Style of Presentation

Introduction

  • Name (unless the chair has already introduced you)
  • Thanks – Chair, audience
  • Title of Research
  • Acknowledgement (or at the end of presentation)
  • Broad scope of presentation including what you won’t cover but may be expected the the audiences

Opening

  • Short hook
    – Asking question
    – Quotation
    – Refer to the current affair (especially in social science)
    – Statistics related to the subject
  • Aims and Objectives
  • 1 – 1.5 minutes for Introduction + Opening

End (summary and then conclusion)

  • Summary = reiterate what have been presented already, not the new things.
    in bullet points
  • Conclusion: Includes comments and suggestion of the future works.
    – short punchy phrase

Gesture and Body language

  • Gaze – eye contact
    – You can look down to your note but don’t speck during that time, just pause. Speck after you look up  again.
    – Use your eye contact like a lighthouse
    – Don’t look at other things, focus on your audiences.
    – Don’t pat too much attention on a particular person e.g., experts or persons who nodding their heads all the time.
  • Whole body orientation
  • Posture
  • Gesture  – hands/arms
    – Show your hands to people
    – get your hands in fornt of you
    – Don’t over gesture. bring your hands back to the focal point (e.g., stand or note or other object)
    – If you use cards, put numbers on them and stick them together so that you will not mess them when you drop them.
    – Use open-palm gesture as it is the normal gesture that doesn’t offense anyone.
  • use of space

Tips for questions

  • Offer to deal with the details afterwards
  • Be prepared to concede
  • Useful expression “It’s a case that we may have to agree to disagree”.
  • Keep calm and don’t take it personally
  • Do not comment on anything outside your area of expertise
  • Use the experts in the audience
  • If no question, you may ask yourself. “This is the question that I have been asked previously”.

Time Management Tips


Tips

  • Create a to-do list
  • Having small tasks in the list rather than a big one. Therefore you can tick those taks off easier
  • Prioritise the tasks based on their “Importance” and “Emergency”.
  • Here is the suggested priority (ranked by Urgent then Important
    1. Urgent and  important
    2. Urgent but not important
    3. Not urgent but important
    4. Neither urgent nor important
  • Be flexible -> Plan for unplanned things. Don’t fill all the slot in the whole day, give some room for emergency.
  • A bit under-estimate yourself on how long it would take to finish the task. If you think the task would take 2 hours, you may put 2.5 or 3 hours on the list/diary.
  • Don’t forget the small things that relates to the task e.g., travelling time.
  • Pomodoro technique (Pamodoro = Italian for tomato) -> Working in blocks of 25 minutes.
  • More info in the Website PomodoroTechnique.com
  • and the book as below (click the picture to see on Amazon)
  • Pomodoro technique suggests you to just focus on only one task in the slot of 25 minutes (or the amount of time that fit you).
  • Vitae Booklet “Balanced Researcher” (PDF)
  • Priority is moving around i.e., things getting more urgent or more important. Thus long term plan needed!
  • Know when you are the most effective and not the active. Allocate tasks accordingly.
    e.g., If you are a bit lazy during 3-4pm of the day, assign yourself small taks during that time.
  • Your head/brain is like a hard drive. Don’t put too much in there unless its processing will get slower and slower. Organise and Re-organise your data, information, tasks.
  • Dare to put unnecessary stuffs into the trash.
  • Create waiting list, next action with specific required time and deadline.
  • Using Calendar to organise … sync with your mobile device if you have or consider Cloud based application.
  • Email tip … if you can finish it within 2 minutes, then do it now and delete it. If it requires > 2 minutes, put it in the waiting list. Subject to how urgent it is.
  • Time wasters! -> You need to have your own time for things you love … hobby, sports, Internet or Social Network Site. They are not time wasters as they make your life more lively unless you overspend your time on them. Hence balance them with your duty, prioritising and stick with it.
  • Learn to say “No” to things that are not necessary or will make your life difficult or will mess up your schedule/priorities.
  • Don’t check your Email/Social Network first, do your planned task on the day first.
  • If you need concentration, create such environment … turn off distractions e.g., email notifications, TV if they are so.
  • Remember that it’s not always gonna be a perfect day, prepare for it!

Where to get a PhD?


 

PhD study is mainly about how to do a research. It normally takes more than three years to complete the course. Many PhD candidates have withdrew or failed in the end. I found that on of the root causes is that ‘Students are fail to select the right place to study’.

Based-on my experience I decided to study PhD with Professor Stephen Disney at Logistics Systems Dynamics Group, Cardiff Business School because of the following factors.

  1. Supervisor(s)
    This is the most important factor. Choose your supervisor wisely since switching one is difficult and costly in terms of time, effort and moral.
    – You should be able to work with your supervisor effectively and efficiently. Think that you can work with him/her for at least 3 years
    – Your supervisor should know either the subject or the method of your PhD proposal or both. If you are lucky, you may find two supervisors who are experts in subject and method in the supervisory panel.My primary supervisor is a quantitative professor in operations management and my secondary supervisor got a nationwide award on her qualitative PhD thesis. Hence, I am very lucky!
  2. Research Group / Academic Section
    Doing PhD in a well-supporting research group or academic section will help you financially and academically. Choosing a research group which is one of the leading group in the subject will give you an opportunity to learn from other group members and fellow PhD students as well.My research group, Logistics Systems Dynamics Group, is a leading logistics and supply chain in the UK and Europe.
  3. Schools / Faculty
    Similar to research group, choosing a right school or faculty, you will study in the right environment. This is quite significant for PhD in Logistics and Supply Chain Management. You may go for business school or engineering one or even economics. The decision perhaps depends on your research methods. Please note that you may find many engineering researchers in some business schools and  vice versa.Cardiff Business School is ranked 4th in the UK in term of research quality!
  4. University
    Studying in a great university will offer you a cross-disciplinary knowledge and experience. At Cardiff University, I have had a great chance to learn such a rigor social science research methods from School of Social Science and another view of logistics and transports from the School of City and Regional Planning as both are the best in the UK. Moreover, Cardiff University is one of 20 Doctoral Training Centre in the UK.
  5. City / Country
    Not only study but you also have to live happily whilst you are study. PhD is a long journey. Living in a nice place will improve your quality of life, reduce stress from study. Moreover, choosing a place where you can learn outside the university is also important.Cardiff was the biggest marine port in the world. Now, the city is very well developed. Welsh people are very nice too. The living cost is also relatively lower than other big cities in the UK.
Albeit based on my experience, these five factors are from my own experience, one may find them useful when considering where to do a PhD. Think well since you have to be there for three years or more!