Skip to Main Content

Health Sciences - Systematic Reviews & Meta-Analyses: Step 6: Critical appraisal of studies

LibGuide on Systematic Reviews and Meta-Analyses

Critical Appraisal Guidance - Example Checklists, Rating Sheets

Cochrane RoB 2.0 (Risk of Bias) Tool

This is the newly revised Cochrane Collaboration Risk of Bias tool, released October 2018.  The Risk of Bias tool developed by the Cochrane Collaboration evaluates studies in several domains, including selection bias, performance bias, attrition bias, reporting bias, etc.  It ONLY can be used for randomized control trials (RCTs).  More information on this tool can be located here: 

Critical Appraisal Skills Programme Tools

  • CASP offers a variety of critical appraisal tools.

NIH Study Quality Assessment Tools

  • The NIH has a variety of study quality assessment tools grouped by study design. 

Joanna Briggs Institute Critical Appraisal Tools

  • Several critical appraisal tools grouped by study design. 

GRADE

  • Grading Recommendation Assessment, Development, and Evaluation

AMSTAR 2

  • A MeaSurement Tool to Assess systematic Reviews

MECIR

  • Methodological Expectations of Cochrane Intervention Reviews

Center for Evidence-Based Medicine Critical Appraisal Tools

  • Consolidated Standards of Reporting Trials (CONSORT) Critical Appraisal Checklist

The STROBE Statement for Observational Studies -- Critical Appraisal Checklists

RE-AIM Checklist

  • A checklist that looks at the external validity of intervention programs across the domains of reach, effectiveness, adoption, implementation and maintenance.  
     

Mixed Methods Appraisal Tools

Non-Randomized Studies

The Newcastle-Ottawa Scale for Assessing the Quality of Nonrandomized Studies

  • A critical appraisal tool for nonrandomized studies. 
     

Subject/Discipline-based tools

Meta Quality Appraisal Tool (MetaQAT)

  • Public Health studies

SYRCLE's checklist for Animal Studies

  • A checklist (and article) about critical appraisal for animal studies

Click here to see what kind of biases to look out for

This website contains a plethora of biases that you may want to consider for your systematic review. Remember, you don't have to use an established tool if it doesn't work for you. Try creating your own!

Systematic reviews and meta-analyses: a step-by-step guide

Step 6

Critical appraisal of studies (quality assessment)

There is no consensus on the best way to assess study quality, but most methods encompass issues such as:

- Appropriateness of study design to the research objective

- Risk of bias

- Other issues related to study quality:

         - Choice of outcome measure

         - Statistical issues

         - Quality of reporting

         - Quality of the intervention

         - Generalisability

The consensus reporting guidelines for different study designs proposed by EQUATOR (http://www.equator-network.org/ ) are a useful starting point, but note these are guidelines for reporting of original studies, NOT for assessment of study quality. 

STROBE (http://www.strobe-statement.org)  also provides useful guidelines for good reporting of observational research, including checklists of items that should be included in this type of research.

Useful resources for assessing quality of different study designs can be found here, and some specific examples are QUADAS for studies of diagnostic accuracy.

This article by Sanderson, Tatt and Higgins (2007) provides a review of the wide range of tools used to assess study quality. It does not recommend any specific tool for general use, but lists the domains which should be included [1) appropriate selection of participants 2) appropriate measurement of variables and 3) appropriate control of confounding, as well as considering design specific biases]. You may need to develop your own quality assessment tool, but do seek advice on the best method of quality assessment for your review.

This articly by The Cochrane Collaboration describes a tool they developed for assessing risk of bias in random trials http://www.bmj.com/content/343/bmj.d5928.full.pdf

Move to Step 5         Move to Step 7                    Main Menu

Overview and Planning

The critical appraisal process evaluates studies based on quality measures specific to your research question components, the discipline/s related to your research topic, and those specific to the study design, outcome measures, methodology, data analysis, and reporting methods that relate to your research question and identified outcomes.

During the critical appraisal process, you will be assessing each study's risk of bias. See many of the links below to tools often used to critically appraise research studies.  

Critical Appraisal Planning:

  • Identify or Develop one or more checklists or forms for evaluating studies
  • Pilot test one or more checklists or forms for evaluating studies
  • Ensure that you are evaluating all aspects you would like to / need to address
  • Test and ensure inter-rater reliability in evaluating studies

Which tool is right for you:

Here's a paper that evaluates 18 critical appraisal tools that are available for researchers to use.  

Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review

Abstract

Background: Several scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools.

Methods: We searched for potentially relevant articles from inception to February 2017. One author screened all titles, abstracts and full text articles, and collected data on tool characteristics.

Results: We identified 18 tools that include an assessment of the risk of reporting bias. Tools varied in regard to the type of reporting bias assessed (eg, bias due to selective publication, bias due to selective non-reporting), and the level of assessment (eg, for the study as a whole, a particular result within a study or a particular synthesis of studies). Various criteria are used across tools to designate a synthesis as being at ‘high’ risk of bias due to selective publication (eg, evidence of funnel plot asymmetry, use of non-comprehensive searches). However, the relative weight assigned to each criterion in the overall judgement is unclear for most of these tools. Tools for assessing risk of bias due to selective non-reporting guide users to assess a study, or an outcome within a study, as ‘high’ risk of bias if no results are reported for an outcome. However, assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is outside the scope of most of these tools. Inter-rater agreement estimates were available for five tools.

Conclusion: There are several limitations of existing tools for assessing risk of reporting biases, in terms of their scope, guidance for reaching risk of bias judgments and measurement properties. Development and evaluation of a new, comprehensive tool could help overcome present limitations.