An Introduction to the Process of Creating Systematic Reviews

Systematic reviews are an indispensable part of evidence-based practice (EBP): They help clinicians decide on the advisability of a particular course of action, for example, what instrument to utilize to assess the seriousness of a problem, what procedure to use for treating a problem, and what information to give patients/clients when they ask questions about prognosis. The systematic review is not the only part of EBP, and clinicians should not forget that the values and preferences of their patients/clients should play a role in decision making, as well as the clinician’s own expertise and level of training in advanced assessment and treatment techniques. However well designed, implemented, and reported, a systematic review is never the only part of the puzzle.

All systematic reviews start with a focused clinical question (See Figure 1, Box 2), and are designed to answer that question using only the findings of relevant, quality-assessed research that has been completed but not necessarily published. It is the responsibility of the clinician or other user of the systematic review to determine whether there is a match between that question and their own question(s) and need for information, including the fit with the patients’/clients’ characteristics, needs, and values (Box 1). A protocol (Box 3) is then written that specifies the research process that will be followed to find the answer to the focused question(s). The protocol typically indicates how the data (the results of existing research) will be identified, evaluated, extracted, synthesized, and used to answer the focused question that started the process, and what criteria will be used to ensure the quality of the synthesis and the dependability of the recommendations, if any. The protocol should specify what methods (Boxes 11–22) and standards or instruments (Boxes 4–10) will be used in all later steps. To minimize bias, the protocol should be developed without knowledge of the findings of primary studies. Ideally, at this stage the authors are still blind to what the review might conclude. Sometimes a group separate from the protocol authors reviews the protocol to make sure that the researchers have indeed proposed feasible and optimal ways of completing all the steps in the review process, at least within the scope of the available resources (Box 22).

Figure 1. Schematic overview of systematic review production and linking the results to the reader’s interests

The protocol specifies which bibliographic and other databases will be used and identifies the inclusion/exclusion criteria as well as key free-text words, controlled vocabulary terms, and thesaurus terms (Box 4) to be used in searching for relevant research (Box 11). Most databases will produce reference information, including an abstract of the paper that was published. However, other databases, such as clinical trial registries, may indicate only that a study was planned, and that follow-up with the investigator or sponsor is necessary to determine whether any findings were published or at least are available. These abstracts are used to screen studies/papers (Box 12), using specific criteria (Box 5) regarding what can be eliminated and what must go on to the next stage: the scanning of complete documents. The best abstract scanning uses two or more individuals who review abstracts independently; their agreement should be reported as an indication of the quality of the screening process.

In the next stage, full papers are scanned (Box 13) to determine whether they indeed are applicable to the clinical question and whether they satisfy the criteria (for age group, treatment type, comorbidities, etc. [Box 6]) set forth in the protocol. Additionally, full texts of published papers are commonly used for ancestor search (Box 19), which is checking the list of references for prior relevant publications that for some reason (a very old paper, a journal that is not indexed, an error by an indexer, etc.) did not make it into the batch of abstracts retrieved from the bibliographic databases consulted. Another method often used to identify research, especially studies that may not have been published or that were published only in reports or other documents often not included in the bibliographic databases, involves contacting experts in a particular area (Box 18). “Hand searches” of the most relevant journals (Box 20) are sometimes used. Systematic reviewers may avoid the latter step, either because of the costs involved or because they trust that other databases (e.g., the Cochrane Central Register of Controlled Trials) have been created on the basis of such hand searches. Even with a “small, simple” clinical question, the number of full papers that are thought to be relevant on the basis of a reading of only the abstract can be large, and scanning of the full papers is recommended to determine which papers need to go on to the next step. Again, scanning by multiple readers (Box 13) is used to make sure no paper is accidentally set aside as not relevant.

Many systematic reviews assess the methodological quality of the primary studies they have identified (Box 14), using a quality checklist or even a formal quality rating scale (Box 7). The resulting information may be used to exclude papers or studies altogether, or to weight individual studies in the synthesis phase of the review, and/or in a sensitivity analysis to determine whether research quality makes a difference in the nature of the findings. Because many research reports leave out some information on methods or findings crucial to systematic reviewers, or describe their methods in ambiguous terms, researchers doing a systematic review may want to communicate with the authors of the primary studies to retrieve as much missing information as possible (Box 21). With or without the supplemental information, those completing the quality rating scale or checklist may easily make errors of omission or commission. Therefore, having two or more well-trained individuals (Box 14) do this independently for each paper is recommended.

The next step in the sequence is to extract the data from the papers (studies) that survived the previous stages (Box 15). Using customized forms (or data entry screens linked to a database) and instructions (Box 8), members of the review team identify the information needed in the sources and enter the data in the appropriate fields. Depending on the purpose of the systematic review, this may be bibliographic information (e.g., source journal and year of publication), study characteristics (e.g., number of subjects, use of randomization), and outcomes reported (for instance, specific outcome measures, effect sizes) or aspects of the conclusions drawn by the study’s authors. Also in this stage (Box 15), use of multiple independent extractors is recommended, and the authors of the studies being reviewed may be contacted to get details missing from the published report (Box 21). Steps 13, 14, and 15 can be combined, and they often are combined, in that the same individuals scan in a single step the full papers for eligibility, extract or rate information relevant to the methodological quality of the primary studies, and extract substantive outcome information.

In the data synthesis step, the various primary studies—or at least the elements extracted in Step 15—are combined (Box 16). If the question “Are these studies or findings combinable?” has been answered with “yes,” the common theme (message, finding, etc.) of the primary studies is determined, especially regarding how they answer the clinical questions: “How many studies give answer A, what is the methodological quality of these investigations, and how strong is their support for this (for instance, what are the relevant effect sizes)?; How many give answer B or another answer?” Further analysis in the synthesis phase may address systematic differences between the studies that resulted in answer A versus those that found answer B; authors may also assess whether the trend is different for subgroups of patients/clients, for weaker and stronger studies, etc. In meta-analyses, answering the question of combinability and synthesizing the data are both quantitative.

The existence of explicit synthesis rules and standards that have been defined beforehand (Box 9) is the strong suit of systematic reviews. Decisions regarding how information is combined across studies are guided by the clear rules specified in the protocol rather than someone’s preferences or biases. But the reader should keep in mind that biases may have led to the specification of the rules in the first place, and that sometimes rules are not obeyed. The fact that the protocol mentions rules and standards does not guarantee that the results of the systematic review are dependable. The current document and the AQASR checklist were written to help readers of systematic reviews become critical readers.

While the data synthesis step is akin to statistical analysis in a traditional primary study, the next step—drawing conclusions and making recommendations (Box 17)—is very similar to what is done in primary research. One major difference, however, is that systematic reviewers rely on preset criteria for the strength (quality, quantity, variety) of the evidence when drawing conclusions and making recommendations. These evidence grading schemes (Box 10) may, for instance, state that an intervention can be recommended strongly only if there are at least two large, well-executed randomized controlled trials (RCTs) supporting it. If, however, there are only observational studies, regardless of how many and how well done, the intervention might only be suggested as one out of many options.

Some systematic reviews, especially those sponsored by professional groups or performed with government funds, are different from other types of research in that the protocol calls for a round of external peer review before the findings and recommendations are distributed. This group of experts (which may include methodologists, clinicians, and consumers and which may be the same or different from those who reviewed the protocol before undertaking the study) reviews the draft report, assesses whether the investigators followed their protocol, and determines whether there are, in spite of adherence to a well-written protocol, any major errors (omission of studies, misinterpretation of primary studies, flaws in synthesis, etc.) that resulted in erroneous findings, conclusions, and recommendations. The peer review (Box 22) may be the basis for redoing part of the work, possibly from the step of writing the protocol forward.