About the AQASR

A clinician in a white coat holding a clipboard observes a smiling young man as he looks down at his cell phone.

Assessing the Quality and Applicability of Systematic Reviews (AQASR)

Why AQASR was developed

The world’s clinical and scientific literature is growing so fast that it has become impossible even for someone who subspecializes in a particular topic to stay current with new developments. As a result, more professionals must rely on reviews to stay on top of the research and to get recommendations about what they should be doing (or should stop doing) in treating their patients/clients. This reliance on reviews creates its own problems, however. Some reviews are good, some are poor, and the worst ones are poor and biased. Systematic reviews are the best class of reviews for answering specific clinical questions (on diagnosis, prognosis, treatment, costs, etc.). The systematic review, which has become more common since 2000, is “a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies” (Institute of Medicine, 2011b). Systematic reviews approach the examination of a body of literature as though it were a research project, which involves a protocol designed to reduce errors in findings and extracting and synthesizing information to optimize the level of objectivity of the results and recommendations.

Many clinicians and researchers learned little about systematic reviews during their schooling or are not confident that they can evaluate the quality of such a review, even if they did study the topic during their training. It is one thing to know what a systematic review is; it is quite another to be able to detect possible weaknesses or biases in a review that recommends a particular course of action, and to evaluate to what extent it can be trusted. The basic purpose of this document and the checklist it presents (Assessing the Quality and Applicability of Systematic Reviews [AQASR]) is to help busy clinicians, administrators, and researchers ask critical questions to reveal the strengths and weaknesses of a review, in general, and as relevant to their particular clinical question or other practical concern(s). Its primary audience is clinicians, as most systematic reviews are optimized to answer the clinical questions they have. Systematic reviews that address the questions of researchers and policymakers also may address focused questions and follow similar procedures. However, the illustrations and justifications we present here will be based on issues of concern to clinicians.

It should be noted that this document addresses systematic reviews (and meta-analyses, a subgenre) only. Often, systematic reviews are the basis for the creation of clinical practice guidelines or similar documents that assist practitioners in making decisions on assessment, diagnosis, prevention, and/or treatment. However, a number of other factors are considered when developing a clinical practice guideline, such as the weighing of risks and benefits of alternative treatments, the costs of treatment, and the values and preferences of patients and clinicians (Dijkers, 2013). Furthermore, a systematic review typically focuses on answering a single clinical question, or a few questions at most, and clinical practice guidelines use findings from a number of reviews (if available) and clinical expertise to address all issues surrounding a particular clinical entity, e.g., diagnosis, comprehensive management and treatment, and prognosis for a specific disorder. This document and the AQASR checklist do not address how such issues are or are not addressed and combined in developing recommendations. Instruments such as the Appraisal of Guidelines for Research and Evaluation (AGREE II) provide guidance on the evaluation of clinical practice guidelines (Brouwers et al., 2010).

Who created AQASR?

The AQASR checklist and related materials were created by the Task Force on Systematic Review and Guidelines of the National Center for the Dissemination of Rehabilitation Research (NCDRR, funded from 1995–2011 by the National Institute on Disability and Rehabilitation Research [NIDRR]). The NCDDR project was replaced in 2012 by the Center on Knowledge Translation for Disability and Rehabilitation Research (KTDRR Center, funded by the National Institute on Disability, Independent Living, and Rehabilitation Research [NIDILRR]). The Task Force comprised a group of disability and rehabilitation clinicians and researchers with experience in creating and/or using systematic reviews. They began by “mining” the existing literature on the quality of systematic reviews for items/questions that have been suggested by various scholars to evaluate the quality of systematic reviews. These items were sorted into the categories currently used in AQASR and then discussed from a number of viewpoints:

  • Does the item/question address the quality of a review?
  • Can the answer be found by reading the review at hand (or must a potential user read all the individual primary studies too, and/or other existing systematic reviews on the topic)?
  • Is it important to ask the question?
  • Does asking the question help users of the systematic review better understand the strengths and limitations of the review, and does it assist them in making better decisions on whether or not to use it?

The items presented are those that the Task Force members regarded as essential. They also did some combining and splitting of issues found while reviewing the literature, or that emerged in their discussions, to enhance the utility of the end result for the checklist user. For the 2023 update, a subset of the original authors and other systematic review experts reviewed the items with a focus on clarifying items that could elicit more than one answer or that could benefit from additional description. In addition, the updated AQASR is a more user-friendly, web-based resource.

Credits

Prepared by the Task Force on Systematic Review and Guidelines. Convened by the National Center for the Dissemination of Disability Research

Task Force Members:
Marcel Dijkers, PhD
Michael Boninger, MD
Tamara Bushnik, PhD
Peter Esselman, MD
Allen Heinemann, PhD
Tamar Heller, PhD
Alex Libin, PhD
Chad Nye, PhD
Joann Starks, MEd
Mark Sherer, PhD
Dave Vandergoot, PhD
Michael Wehmeyer, PhD

2023 Revision Team Members:
Tamara Bushnik, PhD
Devin Dedrick, MEd
Allen Heinemann, PhD
Chad Nye, PhD
Joann Starks, MEd
Oliver Wendt, PhD

September 2011 (Revised August, December 2013)
Revised August 2023

Direct any comments or suggestions to the KTDRR Center by emailing ktdrr@air.org.

Suggested citation

  • Center on Knowledge Translation for Disability and Rehabilitation Research. (2023). Assessing the quality and applicability of systematic reviews (AQASR). American Institutes for Research. https://www.ktdrr.org/aqasr

© AIR, 2011, 2013, 2023

The contents of this document were developed under a grant from the National Institute on Disability and Rehabilitation Research (NIDRR grant number H133A060028) and grants from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR grant numbers90DP0027 and 90DPKT0001). NIDILRR is a Center within the Administration for Community Living (ACL), Department of Health and Human Services (HHS). The contents of this document do not necessarily represent the policy of NIDILRR, ACL, or HHS, and you should not assume endorsement by the Federal Government.