KT Update

TABLE OF CONTENTS

KTDRR's Expert Review Panel

Our Expert Review Panel (CRP) convened on April 15 to give valuable feedback on the usability of a tool that KTDRR and partners at the University of Washington are creating to help make plain language summaries of systematic reviews. These insights helped us reshape some of the content and accessibility features of the tool. We are grateful for the panel members' time and discernment.

We are also accepting applications for membership on the Expert Review Panel. If you have a disability and would like to share your vision on how the work of KTDRR could be more responsive to the needs of people with disabilities, or if you know someone who might, please contact Ann Outlaw for membership criteria.

KT Casebook

Staff at KTDRR are preparing the KT Casebook to highlight innovative examples of KT and related outcomes from the NIDILRR community. The examples documented in the KT Casebook showcase how NIDILRR-funded projects are transferring knowledge into practice to benefit people with disabilities. The purposes of the casebook are to help other NIDILRR grantees build upon these KT successes, along with identifying barriers to KT in our community. If you would like more information about how to participate in this year's KT Casebook, or future years, please contact us at ktdrr@air.org.

2014 KT Conference

Save the Date! KTDRR's annual online KT Conference will be held the week of October 27-31, 2014. This year our conference will focus on how NIDILRR grantees can use a variety of outreach strategies to get their message out to the wider public in order to increase the use of the knowledge generated though NIDILRR-funded research. Stay tuned for more information in the upcoming months.

This issue of KT Update presents another in a series of brief articles by Dr. Marcel Dijkers. Here he examines issues around updating systematic reviews.

[ Download PDF version (193KB) ]

Living Systematic Reviews

Marcel Dijkers, PhD, FACRM
Icahn School of Medicine at Mount Sinai, Dept. of Rehabilitation Medicine

'Living systematic reviews' was part of the title of a paper that was published this year in PLOS Medicine, and it intrigued me (Elliott et al., 2014). Does it refer to a clinician who is so intent on observing evidence-based practice (EBP) that she 'lives systematic reviews'? You know the type—won't treat a new patient without searching for guidelines and systematic reviews (SRs) that tell her what is the best approach to treating problem X.

Well, it turned out that what the authors were writing about was more prosaic, or less so, depending on one's point of view. They were referring to the fact that we need 'systematic reviews that are alive,' that is, continuously updated, so that we address the problem that by the time a SR is published it is a year, and possibly a few years, out of date (Sampson et al., 2008). The time from the last search for literature to submission of the SR to a journal is typically six months, and peer review and preparing a manuscript for publication takes another six. During this time new studies relevant to the review topic are being published, which may impact the conclusions (Shojania et al., 2007). Even in The Cochrane Collaboration (The Cochrane Collaboration - Cochrane Reviews, 2010), which has standing review groups and a policy recommendation that reviews are to be updated every two years (Garritty, Tsertsvadze, Tricco, Sampson, & Moher, 2010; Jaidee, Moher, & Laopaiboon, 2010), there is a delay with incorporating new evidence, and the biennial updates do not always take place on schedule.

How bad is the problem? EBP circles always emphasize regular updating when it comes to fast-developing areas of clinical science, where there are, on a regular basis, new studies relevant to a practical question. Is rehabilitation and disability research such a field, and are practitioners in this area hindered by the fact that there are no SRs, or that they are not updated fast enough—or not at all?

To get an idea I turned to a set of reviews that are well known and well respected—the three SRs produced by Keith Cicerone and his colleagues on the evidence for cognitive rehabilitation (Cicerone et al., 2000; Cicerone et al., 2005; Cicerone et al., 2011). In this series, a group of clinician-scientists drawn from the American Congress of Rehabilitation Medicine's Special Interest Group on Brain Injury scoured the literature to determine what evidence there is for cognitive rehabilitation to remedy deficits (resulting from stroke or traumatic brain injury) in attention, visuospatial/constructional functioning, language/communication, memory, and executive functioning/problem solving. The reviews are extremely well-cited; per Web of Science, at the end of February 2014 'Cicerone 2000' had been cited 363 times, 'Cicerone 2005,' 342 times, and 'Cicerone 2011,' 89 times. I wish my research had such impact!

Figure 1 gives some idea of the speed of production of evidence in the area of cognitive rehabilitation, and when it was incorporated in a review. From 1985 through 1999 about 10 papers were published each year, but then the research community kicked things into higher gear, with about 16 papers per year seeing the light from 2000 through 2009. This certainly is an area of fairly intensive publication, although the majority of the material is designated 'Class III' (clinical series without concurrent controls) by Cicerone et al., and there are limited numbers of what they call 'Class I' evidence (well-designed prospective randomized controlled trials). In Cicerone 2000, the evidence had been published, on average, 11.1 years previously (standard deviation [SD] 5.7); for Cicerone 2005, the lag time on average was 4.9 years (SD 1.5); and for Cicerone 2011 it was 5.6 years (SD 1.5). It appears that, at least in this area of disability and rehabilitation science, there is a high level of publication activity and the idea of 'living systematic reviews' might be of benefit.

Figure 1. Number of studies accepted as evidence, by year of publication and year of review

So what do the authors of the PLOS Medicine article propose be done to reduce the gap between knowledge production and knowledge use, the evidence-practice or know-do gap, as it has been called? They see or call for various innovations in the production of SRs to make it simpler and faster to incorporate new knowledge into an existing SR, pretty much as that knowledge becomes available, maybe even as it is accepted for publication and published on the journal's website as an author's copy or published-ahead-of-print copy.

• New, integrated platforms and tools to replace the hodgepodge of software (spreadsheets, email, statistical software, reference managers, etc.) now used by the typically far-flung members of the team that produces a SR.
• Use of artificial intelligence-based automation, especially for searching the literature being published. PubMed now every day adds more than 85 papers on disability and rehabilitation topics, and that is just part of the deluge of publications. Depending on the topic area, PubMed may cover 50% to 80% of health care papers being published worldwide, in all languages. The issue is one of creating software for scouring the bibliographic databases that hits the sweet spot between sensitivity (finding close to 100% of all new relevant publications) and specificity (not including large amounts of non-relevant papers, that then need to be manually eliminated by human reviewers). It seems that great progress has already been made in this area (Cohen, Ambert, & McDonagh, 2012; Wallace et al., 2012).
• Registration of SRs to avoid unnecessary duplication of effort, and storing of the 'raw materials' and descriptions of the process of SR production in standardized formats, so that these can be 'harvested' by other teams preparing similar or overlapping SRs. A plan for such a 'repository' of SR data has been published (Ip et al., 2012). PROSPERO (International prospective register of systematic reviews, University of York, Centre for Reviews and Dissemination) is a site where SRs can be registered to avoid duplication [www.crd.york.ac.uk/PROSPERO]. (It also offers readers of SR reports an opportunity to compare the report with the registered protocol, but that is not directly relevant here).
• Crowdsourcing, e.g. of citation screening, up to the point of using 'citizen scientists.'

Elliott and colleagues (2014) also have suggestions for more efficient peer and editorial review: editorial review only of updates that report 'new studies, no changes in recommendations, no changes in effect size estimates' and expedited peer review of updated SRs that have changes in recommendations or effect size estimates. Journal editors may want to accept this suggestion, because it speeds up publishing of one of their most popular products, thus improving the journal's impact factor.

The issue of systematic reviews going out of date once additional evidence is published to inform clinicians and other decision-makers, has been recognized since at least 1993 (Chalmers, Enkin, & Keirse, 1993). The evidence shows, over and over, that regular updating just does not happen, even with SRs created by or at the behest of institutes or organizations (Garritty et al., 2010). Various semi-automated methods of determining when an update is in order have been proposed and compared (Pattanittum, Laopaiboon, Moher, Lumbiganon, & Ngamjarus, 2012), rather than calling for automatic updating every two years. The suggestions made by Elliott et al. for 'living systematic reviews' may help achieve the goal of regular updates. The steps proposed would reduce the burden on the team creating the SR and would reduce the time required for updates to be prepared and published.

Who knew that the systematic review enterprise was in need of rehabilitation?

SEDL operates the Center on KTDRR. SEDL is an Equal Employment Opportunity/Affirmative Action Employer and is committed to affording equal access to education and employment opportunities for all individuals. SEDL does not discriminate on the basis of age, sex, race, color, creed, religion, national origin, sexual orientation, marital or veteran status, or the presence of a disability.