This issue of KT Update presents another in a series of brief articles by Dr. Marcel Dijkers. Here he examines issues around updating systematic reviews.
[ Download PDF version (193KB) ]
Living Systematic Reviews
Marcel Dijkers, PhD, FACRM
Icahn School of Medicine at Mount Sinai, Dept. of Rehabilitation Medicine
'Living systematic reviews' was part of the title of a paper that was published this year in PLOS Medicine, and it intrigued me (Elliott et al., 2014). Does it refer to a clinician who is so intent on observing evidence-based practice (EBP) that she 'lives systematic reviews'? You know the type—won't treat a new patient without searching for guidelines and systematic reviews (SRs) that tell her what is the best approach to treating problem X.
Well, it turned out that what the authors were writing about was more prosaic, or less so, depending on one's point of view. They were referring to the fact that we need 'systematic reviews that are alive,' that is, continuously updated, so that we address the problem that by the time a SR is published it is a year, and possibly a few years, out of date (Sampson et al., 2008). The time from the last search for literature to submission of the SR to a journal is typically six months, and peer review and preparing a manuscript for publication takes another six. During this time new studies relevant to the review topic are being published, which may impact the conclusions (Shojania et al., 2007). Even in The Cochrane Collaboration (The Cochrane Collaboration - Cochrane Reviews, 2010), which has standing review groups and a policy recommendation that reviews are to be updated every two years (Garritty, Tsertsvadze, Tricco, Sampson, & Moher, 2010; Jaidee, Moher, & Laopaiboon, 2010), there is a delay with incorporating new evidence, and the biennial updates do not always take place on schedule.
How bad is the problem? EBP circles always emphasize regular updating when it comes to fast-developing areas of clinical science, where there are, on a regular basis, new studies relevant to a practical question. Is rehabilitation and disability research such a field, and are practitioners in this area hindered by the fact that there are no SRs, or that they are not updated fast enough—or not at all?
To get an idea I turned to a set of reviews that are well known and well respected—the three SRs produced by Keith Cicerone and his colleagues on the evidence for cognitive rehabilitation (Cicerone et al., 2000; Cicerone et al., 2005; Cicerone et al., 2011). In this series, a group of clinician-scientists drawn from the American Congress of Rehabilitation Medicine's Special Interest Group on Brain Injury scoured the literature to determine what evidence there is for cognitive rehabilitation to remedy deficits (resulting from stroke or traumatic brain injury) in attention, visuospatial/constructional functioning, language/communication, memory, and executive functioning/problem solving. The reviews are extremely well-cited; per Web of Science, at the end of February 2014 'Cicerone 2000' had been cited 363 times, 'Cicerone 2005,' 342 times, and 'Cicerone 2011,' 89 times. I wish my research had such impact!
Figure 1 gives some idea of the speed of production of evidence in the area of cognitive rehabilitation, and when it was incorporated in a review. From 1985 through 1999 about 10 papers were published each year, but then the research community kicked things into higher gear, with about 16 papers per year seeing the light from 2000 through 2009. This certainly is an area of fairly intensive publication, although the majority of the material is designated 'Class III' (clinical series without concurrent controls) by Cicerone et al., and there are limited numbers of what they call 'Class I' evidence (well-designed prospective randomized controlled trials). In Cicerone 2000, the evidence had been published, on average, 11.1 years previously (standard deviation [SD] 5.7); for Cicerone 2005, the lag time on average was 4.9 years (SD 1.5); and for Cicerone 2011 it was 5.6 years (SD 1.5). It appears that, at least in this area of disability and rehabilitation science, there is a high level of publication activity and the idea of 'living systematic reviews' might be of benefit.
Figure 1. Number of studies accepted as evidence, by year of publication and year of review
So what do the authors of the PLOS Medicine article propose be done to reduce the gap between knowledge production and knowledge use, the evidence-practice or know-do gap, as it has been called? They see or call for various innovations in the production of SRs to make it simpler and faster to incorporate new knowledge into an existing SR, pretty much as that knowledge becomes available, maybe even as it is accepted for publication and published on the journal's website as an author's copy or published-ahead-of-print copy.
- New, integrated platforms and tools to replace the hodgepodge of software (spreadsheets, email, statistical software, reference managers, etc.) now used by the typically far-flung members of the team that produces a SR.
- Use of artificial intelligence-based automation, especially for searching the literature being published. PubMed now every day adds more than 85 papers on disability and rehabilitation topics, and that is just part of the deluge of publications. Depending on the topic area, PubMed may cover 50% to 80% of health care papers being published worldwide, in all languages. The issue is one of creating software for scouring the bibliographic databases that hits the sweet spot between sensitivity (finding close to 100% of all new relevant publications) and specificity (not including large amounts of non-relevant papers, that then need to be manually eliminated by human reviewers). It seems that great progress has already been made in this area (Cohen, Ambert, & McDonagh, 2012; Wallace et al., 2012).
- Registration of SRs to avoid unnecessary duplication of effort, and storing of the 'raw materials' and descriptions of the process of SR production in standardized formats, so that these can be 'harvested' by other teams preparing similar or overlapping SRs. A plan for such a 'repository' of SR data has been published (Ip et al., 2012). PROSPERO (International prospective register of systematic reviews, University of York, Centre for Reviews and Dissemination) is a site where SRs can be registered to avoid duplication [www.crd.york.ac.uk/PROSPERO]. (It also offers readers of SR reports an opportunity to compare the report with the registered protocol, but that is not directly relevant here).
- Crowdsourcing, e.g. of citation screening, up to the point of using 'citizen scientists.'
Elliott and colleagues (2014) also have suggestions for more efficient peer and editorial review: editorial review only of updates that report 'new studies, no changes in recommendations, no changes in effect size estimates' and expedited peer review of updated SRs that have changes in recommendations or effect size estimates. Journal editors may want to accept this suggestion, because it speeds up publishing of one of their most popular products, thus improving the journal's impact factor.
The issue of systematic reviews going out of date once additional evidence is published to inform clinicians and other decision-makers, has been recognized since at least 1993 (Chalmers, Enkin, & Keirse, 1993). The evidence shows, over and over, that regular updating just does not happen, even with SRs created by or at the behest of institutes or organizations (Garritty et al., 2010). Various semi-automated methods of determining when an update is in order have been proposed and compared (Pattanittum, Laopaiboon, Moher, Lumbiganon, & Ngamjarus, 2012), rather than calling for automatic updating every two years. The suggestions made by Elliott et al. for 'living systematic reviews' may help achieve the goal of regular updates. The steps proposed would reduce the burden on the team creating the SR and would reduce the time required for updates to be prepared and published.
Who knew that the systematic review enterprise was in need of rehabilitation?
Chalmers, I., Enkin, M., & Keirse, M. J. (1993). Preparing and updating systematic reviews of randomized controlled trials of health care. The Milbank Quarterly, 71(3), 411-437.
Cicerone, K. D., Dahlberg, C., Kalmar, K., Langenbahn, D. M., Malec, J. F., Bergquist, T. F.,... Morse, P. A. (2000). Evidence-based cognitive rehabilitation: recommendations for clinical practice. Archives of Physical Medicine and Rehabilitation, 81(12), 1596-1615. doi:S0003999300105787 [pii]
Cicerone, K. D., Dahlberg, C., Malec, J. F., Langenbahn, D. M., Felicetti, T., Kneipp, S.,... Catanese, J. (2005). Evidence-based cognitive rehabilitation: updated review of the literature from 1998 through 2002. Archives of Physical Medicine and Rehabilitation, 86(8), 1681-1692. doi:S0003-9993(05)00330-8 [pii]; 10.1016/j.apmr.2005.03.024 [doi]
Cicerone, K. D., Langenbahn, D. M., Braden, C., Malec, J. F., Kalmar, K., Fraas, M.,... Ashman, T. (2011). Evidence-based cognitive rehabilitation: updated review of the literature from 2003 through 2008. Archives of Physical Medicine and Rehabilitation, 92(4), 519-530. doi:10.1016/j.apmr.2010.11.015
The Cochrane Collaboration - Cochrane Reviews. (2010). Retrieved May 11, 2010 from http://www.cochrane.org/cochrane-reviews
Cohen, A. M., Ambert, K., & McDonagh, M. (2012). Studying the potential impact of automated document classification on scheduling a systematic review update. BMC Medical Informatics and Decision Making, 12, 33-6947-12-33. doi:10.1186/1472-6947-12-33; 10.1186/1472-6947-12-33
Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P., Mavergames, C., & Gruen, R. L. (2014). Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Medicine, 11(2), e1001603. doi:10.1371/journal.pmed.1001603; 10.1371/journal.pmed.1001603
Garritty, C., Tsertsvadze, A., Tricco, A. C., Sampson, M., & Moher, D. (2010). Updating systematic reviews: an international survey. PloS One, 5(4), e9914. doi:10.1371/journal.pone.0009914; 10.1371/journal.pone.0009914
Ip, S., Hadar, N., Keefe, S., Parkin, C., Iovin, R., Balk, E. M., & Lau, J. (2012). A Web-based archive of systematic review data. Systematic Reviews, 1, 15-4053-1-15. doi:10.1186/2046-4053-1-15; 10.1186/2046-4053-1-15
Jaidee, W., Moher, D., & Laopaiboon, M. (2010). Time to update and quantitative changes in the results of cochrane pregnancy and childbirth reviews. PloS One, 5(7), e11553. doi:10.1371/journal.pone.0011553; 10.1371/journal.pone.0011553
Pattanittum, P., Laopaiboon, M., Moher, D., Lumbiganon, P., & Ngamjarus, C. (2012). A comparison of statistical methods for identifying out-of-date systematic reviews. PloS One, 7(11), e48894. doi:10.1371/journal.pone.0048894; 10.1371/journal.pone.0048894
PROSPERO. International prospective register of systematic reviews. University of York, Centre for Reviews and Dissemination. Available: http://www.crd.york.ac.uk/PROSPERO/
Sampson, M., Shojania, K. G., Garritty, C., Horsley, T., Ocampo, M., & Moher, D. (2008). Systematic reviews can be produced and published faster. Journal of Clinical Epidemiology, 61(6), 531-536. doi:10.1016/j.jclinepi.2008.02.004; 10.1016/j.jclinepi.2008.02.004
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007). How quickly do systematic reviews go out of date? A survival analysis. Annals of Internal Medicine, 147(4), 224-233.
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., Schmid, C. H., Bertram, L., Lill, C. M., Cohen, J. T., & Trikalinos, T. A. (2012). Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining. Genetics in Medicine: Official Journal of the American College of Medical Genetics, 14(7), 663-669. doi:10.1038/gim.2012.7