This issue of KT Update presents another in a series from Dr. Marcel Dijkers. This article describes the origin of PROSPERO (an international database of prospectively registered systematic reviews in health and social care) and PRISMA-P (a listing of preferred reporting items for a protocol for a systematic review or meta-analysis).
PROSPERO and PRISMA-P
Marcel Dijkers, PhD, FACRM
Icahn School of Medicine at Mount Sinai
Department of Rehabilitation Medicine
[ Download PDF version 329kb ]
It must have been two decades ago that, during the discussion period for a paper that I was presenting at a conference, a colleague remarked something like, “Dr. Dijkers tortures the data until it confesses.” He meant that as a compliment, and that’s how I took it. In hindsight, I would say that he approved that I habitually engaged in data dredging, selective reporting, and various other unsavory tactics to make it possible to report at least one significant p-value—the holy grail of simplistic research. We (or at least I) have since learned about the deleterious effects of publication bias, switching the primary outcome of studies for a secondary one and vice versa, selective reporting of outcomes, (unplanned) subgroup analyses, and so on.
Already back in the 1970s a proposal was made to register trials as a (partial) antidote to these shady tactics, which aim to make investigators, their studies, and (sometimes) their sponsors look their best. The reasoning went as follows: If researchers register their study protocols before they enroll the first subject, that permanent record makes it possible to determine what studies have not been published, or which ones in their reporting make unjustified changes to a protocol or statistical analysis plan. Pressure from biomedical journal editors, researchers themselves (especially those engaged in creating systematic reviews [SRs]), and governmental and nongovernmental sponsors of research, as well as the Food and Drug Administration and other regulatory agencies, resulted in the creation of registries such as ClinicalTrials.gov, which began operation in 2000.
These trial registries work in a very simple fashion. A researcher about to start a study creates (voluntarily or forced by her sponsor or the journal in which he wants to publish) an entry for the study, completing a number of fields with administrative and design information. This minimal protocol can be changed if realities on the ground make that necessary, but the registry keeps all versions on display; as a result, for anyone curious to know how the published data relate to the original research plan, there is a paper trail to follow back. ClinicalTrials.gov and other national and international registries have largely lived up to expectations and now there is an ongoing discussion whether researchers should also be required to submit to the website all or at least summary data from their studies. This would satisfy a second purpose of registries: the public availability of information on research sponsored with public or private funds and involving the volunteer time and effort of hundreds—if not thousands—of subjects.
Clinical trials and other primary studies are not the only research that may undergo selective reporting. There is now a modest body of research demonstrating that SRs also may be reported selectively, for instance, Kirkham, Altman, & Williamson, 2010; Silagy, Middleton, & Hopewell, 2002; and Siontis, Hernandez-Boussard, & Ioannidis, 2013. Because SRs increasingly are considered the most comprehensive and least biased evidence for use in Evidence Based Practice and Evidence Based Medicine (EBP & EBM) (at the apex of the “evidence pyramid”), that is of no small concern. And a parallel solution has been suggested: the prospective registration of SRs. Although there are multiple registries for primary research, for SRs there is only one: PROSPERO—International prospective register of systematic reviews. While the name suggests “PROSPEctive Register,” I have found no mention of the origin of the appellation; maybe it was borrowed from Shakespeare, whose Prospero (in The Tempest) was a sorcerer. PROSPERO is financed by the United Kingdom’s National Institute for Health Research (NIHR) and run by the University of York Centre for Reviews and Dissemination.
After an international Delphi project to determine what elements should be contained in the registry (Booth et al., 2011), PROSPERO opened for business in 2011 and quickly started cataloguing more than 90 SRs per month (Booth et al., 2013; Booth, 2013); now (February 2017) it contains more than 20,000 records, of which more than 1,300 concern rehabilitation and disability. The concept of registering SRs, or the PROSPERO site specifically, has been endorsed by a number of entities, including the Canadian Institutes of Health Research, which has gone as far as requiring applicants seeking grant funding for a trial to conduct an SR to demonstrate the need for the trial (Graham, 2012). Others include Guidelines International Network (Van der Wees et al., 2012); NIHR (Davies, 2012); and the Public Library of Science journals (Booth et al., 2012), as well as journals such as Open Medicine (Palepu, Kendall, & Moher, 2011), British Journal of Obstetrics and Gynaecology (Chien, Khan, & Siassakos, 2012), and Systematic Reviews (Booth et al., 2012). Also on board are organizations such as the International Network of Agencies for Health Technology Assessment and the international Campbell and Cochrane Collaborations (Booth et al., 2012).
The PROSPERO registry has a number of major purposes:
- Function as an archive of SR proposals, which journal editors, peer reviewers, readers, and others can use to determine the discrepancies between the protocol and the published report that authors do not declare and justify in their journal report.
- Serve transparency by making publicly available information on which SRs are being done, using what methods, and by whom (Davies, 2012).
- Improve the quality of SRs by inviting researchers to create a protocol before searching the literature (planless SRs apparently are still a common occurrence, per Moher, Tetzlaff, Tricco, Sampson, & Altman, 2007) to reduce confusion and “ensure that all investigators are working from the same work plan” as Chang and Slutsky (2012, p. 2) formulate it.
- Fill in gaps in knowledge by offering an opportunity to see which areas still lack a systematic review or meta-analysis (Graham, 2012).
- Offer an opportunity for researchers to avoid unplanned duplication of SRs, and to find ongoing work by scientists with similar interests whose efforts they might join.
Duplication is not an imaginary problem. The number of SRs being published each year is mushrooming (even in the area of rehabilitation; see Figure 1), and overlapping SRs are not an uncommon phenomenon. Replication is something to be desired, but, as Moher (2013) pointedly asks, how much replication is too much?
Figure 1. Number of PubMed entries of “rehabilitation” articles with “systematic review” in the title, and number of all other “rehabilitation” articles, by time period *
[ Larger View | Long Text Description ]
Numbers for "other rehabilitation" articles have been divided by 100 to make trends clear.
Source: Author's compilation of data obtained from PubMed search.
PROSPERO works in a way similar to the clinical trial registries: A researcher who wants to register an SR submits administrative and scientific information to the website. These SRs may focus on “health and social care, welfare, public health, education, crime, justice, and international development, as long as there is a health related outcome” (PROSPERO, 2016, p. 3), but scoping reviews and literature reviews are not eligible. The information submitted gets a quick administrative review (Right type of study? Sufficiently clear and informative entries in all required and optional fields?), and when approved (typically, in a few business days) becomes publicly available. Registering an SR is free, as is searching the registry for information. Registration should happen before the systematic reviewers start extracting information from the primary studies, but there is some flexibility. Cochrane protocols are automatically added to PROSPERO (Moher, 2013). Table 1 contains an overview of the items on initial registration; there are 22 required and 18 optional fields, and it is estimated that registration takes about 60 minutes—even less for researchers who already have a written protocol (Booth et al., 2012).
Table 1. Required and optional fields for a PROSPERO registration
* indicates a required field
- Review title*
- Original language title
- Anticipated or actual start date*
- Anticipated completion date*
- Stage of the review at the time of submission*
- Name of contact person*
- Email address of contact person*
- Address of contact person
- Telephone number of contact person*
- Organization with which the review is affiliated*
- Members of the review team and the organization(s) with which they are affiliated
- Funding sources/sponsors*
- Conflicts of interest*
- Review question(s)*
- Literature searches (bibliographic databases; language or publication date restrictions)*
- URL for search strategy
- Condition or domain being studied*
- Population(s) being studied*
- Interventions or exposures*
- Comparators or controls*
- Types of study designs to be included initially*
- Context for the SR; background for inclusion/exclusion criteria*
- Primary outcome(s)*
- Secondary outcomes*
- Data extraction (selection and coding)
- Risk of bias (quality) assessment*
- Strategy for data synthesis*
- Analysis of subgroups or subsets*
- Type (method) of review (menu)
- Service delivery
Area of review (menu)
- Alcohol/substance misuse/abuse
- Crime and justice
- Health inequalities
- Mental health and behavioral conditions
- Physical therapy
- Public health
- Social care
- Language(s) of SR (menu)
- Country (menu)
- Other registration details
- Reference and/or URL for published protocol
- Dissemination plans
- Details of any existing reviews on the same topic by the same authors
- Review status (menu)
- Completed but not yet published
- Completed and published
- Completed, published, and being updated
- Additional information
- Link to publication of the final report
NOTE: Adapted from PROSPERO, 2016, pp. 7–21. Slightly modified from the document Guidance notes for registering a systematic review protocol with PROSPERO from the website PROSPERO – International prospective register of systematic reviews. https://www.crd.york.ac.uk/prospero/aboutreg.php
Any changes in the protocol (SRs often undergo changes because of information found or not found in the literature) are to be submitted, and the paper trail is kept for public inspection. PROSPERO sends investigators e-mails to remind them to update their registration as stages in the review are completed (Booth & Stewart, 2013). It also encourages users to submit their published paper or other report to the website once the study is complete.
PRISMA and PRISMA-P
Which brings us to PRISMA-P. PRISMA stands for “Preferred Reporting Items for Systematic Reviews and Meta-Analyses,” and it was first published in 2009 to offer authors guidance on how to report their systematic reviews (Liberati et al., 2009; Moher, Liberati, Tetzlaff, Altman, & PRISMA Group, 2009). Under Methods, PRISMA suggests reporting an SR’s “Protocol and registration: item 5: Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number” (Moher et al., 2009, p. 339). The authors of PRISMA suggested that this information be provided because they recognized how important it was for reviewers and readers of an SR to have the opportunity to compare the plan with the actuality, and to be able to determine changes in review methodology as well as selective reporting. They have now gone a step further and have published specific guidance for the contents of such a protocol: the final P of PRISMA-P stands for protocol (Moher et al., 2015; Shamseer et al., 2015). As do all the other members of the PRISMA family, PRISMA-P comes in the form of a checklist, which authors can use to determine whether they have included all relevant items in their protocol (see Table 2). It consists of 17 items—26 if subitems are counted as separate entities (see Table 2). Completing a protocol is expected to benefit SR authors, funders of secondary research, clinical practice guidelines developers, policymakers, journal editors, and peer reviewers, as well as educators and students (Moher et al., 2015; Palepu et al., 2011; Stewart, Moher, & Shekelle, 2012).
Table 2. PRISMA-P: preferred items for reporting the protocols for systematic reviews and meta-analyses
||Identify the report as a protocol of a systematic review
||If the protocol is for an update of a previous systematic review, identify as such
||If registered, provide the name of the registry (e.g., PROSPERO) and registration number
||Provide name, institutional affiliation, and e-mail address of all protocol authors; provide physical mailing address of corresponding author
||Describe contributions of protocol authors and identify the guarantor of the review
||If the protocol represents an amendment of a previously completed or published protocol, identify as such and list changes; otherwise, state plan for documenting important protocol amendments
||Indicate sources of financial or other support for the review
||Provide name for the review funder and/or sponsor
|Role of sponsor/funder
||Describe roles of funder(s), sponsor(s), and/or institution(s), if any, in developing the protocol
||Describe the rationale for the review in the context of what is already known
||Provide an explicit statement of the question(s) the review will address with reference to participants, interventions, comparators, and outcomes (PICO)
||Specify the study characteristics (e.g., PICO, study design, setting, time frame) and report characteristics (e.g., years considered, language, publication status) to be used as criteria for eligibility for the review
||Describe all intended information sources (e.g., electronic databases, contact with study authors, trial registers, or other grey literature sources) with planned dates of coverage
||Present draft of search strategy to be used for at least one electronic database, including planned limits, such that it could be repeated
||Describe the mechanism(s) that will be used to manage records and data throughout the review
||State the process that will be used for selecting studies (e.g., two independent reviewers) through each phase of the review (i.e., screening, eligibility, and inclusion in meta-analysis)
|Data collection process
||Describe planned method of extracting data from reports (e.g., piloting forms, done independently, in duplicate), any processes for obtaining and confirming data from investigators
||List and define all variables for which data will be sought (e.g., PICO items, funding sources), any pre-planned data assumptions and simplifications
|Outcomes and prioritization
||List and define all outcomes for which data will be sought, including prioritization of main and additional outcomes, with rationale
|Risk of bias in individual studies
||Describe anticipated methods for assessing risk of bias of individual studies, including whether this will be done at the outcome or study level, or both; state how this information will be used in data synthesis
||Describe criteria under which study data will be quantitatively synthesized
||If data are appropriate for quantitative synthesis, describe planned summary measures, methods of handling data, and methods of combining data from studies, including any planned exploration of consistency (e.g., I 2, Kendall's tau)
||Describe any proposed additional analyses (e.g., sensitivity or subgroup analyses, meta-regression)
||If quantitative synthesis is not appropriate, describe the type of summary planned
||Specify any planned assessment of meta-bias(es) (e.g., publication bias across studies, selective reporting within studies)
|Confidence in cumulative evidence
||Describe how the strength of the body of evidence will be assessed (e.g., GRADE)
Source: Moher et al., 2015.
http://prisma-statement.org/documents/PRISMA-P Statement - Moher Sys Rev Jan 2015.pdf
With manuscript page numbers entered, authors can submit the PRISMA-P checklist along with their protocol manuscript to a journal such as Systematic Reviews to make review of their manuscript easier for the editors and peer reviewers. Such publication of protocols is increasingly common; in January 2017, more than 150 were published (as shown in a PubMed search), of which 189 concerned rehabilitation and disability topics. The PRISMA-P authors structured the checklist in such a way that a protocol can easily be turned into a report on the results of the SR or meta-analysis: the items are harmonized with the PRISMA items (Moher et al., 2015). PRISMA-P is intended primarily for the preparation of protocols for reviews of aggregate data reported in the literature. It is not optimal for meta-analysis of individual patient/participant data, the topic of a previous KT Update article (Dijkers, 2016), but it would not be surprising if PRISMA-IPDMA were to follow in the near future.
If the objective is to go on public record to declare the aim and methods of an SR, registration on PROSPERO and a published manuscript following the guidance of PRISMA-P seem duplicative but cannot hurt. If sunshine (the opportunity for or threat of public scrutiny) helps to keep researchers, including SR authors, on the straight and narrow, then more sunshine should be welcome. In the end, it is better for all patients, participants, clients, and subjects for whom we write our SRs.
Booth, A. (2013). PROSPERO's progress and activities 2012/13. Systematic Reviews, 2, 111-4053-2-111. doi:10.1186/2046-4053-2-111
Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M.,... Stewart, L. (2012). The nuts and bolts of PROSPERO: An international prospective register of systematic reviews. Systematic Reviews, 1, 2-4053-1-2. doi:10.1186/2046-4053-1-2
Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2013). PROSPERO at one year: An evaluation of its utility. Systematic Reviews, 2, 4-4053-2-4. doi:10.1186/2046-4053-2-4
Booth, A., Clarke, M., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2011). Establishing a minimum dataset for prospective registration of systematic reviews: An international consultation. PLOS ONE, 6(11), e27319. doi:10.1371/journal.pone.0027319
Booth, A., & Stewart, L. (2013). Trusting researchers to use open trial registers such as PROSPERO responsibly. The BMJ (Clinical Research Ed.), 347, f5870. doi:10.1136/bmj.f5870
Chang, S. M., & Slutsky, J. (2012). Debunking myths of protocol registration. Systematic Reviews, 1, 4-4053-1-4. doi:10.1186/2046-4053-1-4
Chien, P. F., Khan, K. S., & Siassakos, D. (2012). Registration of systematic reviews: PROSPERO. BJOG: An International Journal of Obstetrics and Gynaecology, 119(8), 903-905. doi:10.1111/j.1471-0528.2011.03242.x
ClinicalTrials.gov. Retrieved from https://clinicaltrials.gov
Davies, S. (2012). The importance of PROSPERO to the National Institute for Health Research. Systematic Reviews, 1, 5-4053-1-5. doi:10.1186/2046-4053-1-5
Dijkers, M. (2016). IPDMA: Individual Patient/Participant Data Meta-Analysis. KT Update, 4(7). Retrieved from http://ktdrr.org/products/update/v4n7/dijkers_ktupdate_v4n7_508.pdf
Graham, I. D. (2012). Knowledge synthesis and the Canadian Institutes of Health Research. Systematic Reviews, 1, 6-4053-1-6. doi:10.1186/2046-4053-1-6
Kirkham, J. J., Altman, D. G., & Williamson, P. R. (2010). Bias due to changes in specified outcomes during the systematic review process. PLOS ONE, 5(3), e9810. doi:10.1371/journal.pone.0009810
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P.,... Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Journal of Clinical Epidemiology, 62(10), e1–34. doi:10.1016/j.jclinepi.2009.06.006
Moher, D. (2013). The problem of duplicate systematic reviews. The BMJ (Clinical Research Ed.), 347, f5040. doi:10.1136/bmj.f5040
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. The BMJ (Clinical Research Ed.), 339, b2535. doi:10.1136/bmj.b2535
Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M.,... PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4, 1-4053-4-1. doi:10.1186/2046-4053-4-1
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007). Epidemiology and reporting characteristics of systematic reviews. PLOS Medicine, 4(3), e78. doi:06-PLME-RA-0453R3 [pii]; 10.1371/journal.pmed.0040078
Palepu, A., Kendall, C., & Moher, D. (2011). Open Medicine endorses PROSPERO. Open Medicine, 5(1), e65-6.
PROSPERO—International prospective register of systematic reviews. Retrieved from http://www.crd.york.ac.uk/PROSPERO/
PROSPERO. (2016). Guidance notes for registering a systematic review protocol with PROSPERO. York, UK: NIHR Centre for Reviews and Dissemination, University of York. Retrieved from
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M.,... PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. The BMJ (Clinical Research Ed.), 349, g7647. doi:10.1136/bmj.g7647
Silagy, C. A., Middleton, P., & Hopewell, S. (2002). Publishing protocols of systematic reviews: Comparing what was done to what was planned. JAMA, 287(21), 2831–2834. doi:joc11902
Siontis, K. C., Hernandez-Boussard, T., & Ioannidis, J. P. (2013). Overlapping meta-analyses on the same topic: Survey of published studies. The BMJ (Clinical Research Ed.), 347, f4501. doi:10.1136/bmj.f4501
Stewart, L., Moher, D., & Shekelle, P. (2012). Why prospective registration of systematic reviews makes sense. Systematic Reviews, 1, 7-4053-1-7. doi:10.1186/2046-4053-1-7
Van der Wees, P., Qaseem, A., Kaila, M., Ollenschlaeger, G., Rosenfeld, R., & Board of Trustees of the Guidelines International Network (G-I-N). (2012). Prospective systematic review registration: Perspective from the Guidelines International Network (G-I-N). Systematic Reviews, 1, 3-4053-1-3. doi:10.1186/2046-4053-1-3