KT Update

• Upcoming Webcast Series
• Community of Practice
• Pre-NARRTC Training
• 2015 KT Casebook
• What is a Scoping Review?

---

Stay Connected:
www.ktdrr.org

     

Send email to:
ktdrr@air.org

---

The contents of this newsletter were developed under grant number 90DP0027 from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR). NIDILRR is a Center within the Administration for Community Living (ACL), Department of Health and Human Services (HHS). The contents of this newsletter do not necessarily represent the policy of NIDILRR, ACL, HHS, and you should not assume endorsement by the Federal Government.

Copyright © 2015 by SEDL, an affiliate of American Institutes for Research

Upcoming KTDRR Webcast Series

Community of Practice on KT and Evidence for Disability & Rehabilitation Research

SAVE THE DATE: May 3-4, 2016
Pre-NARRTC Training on Systematic Reviews

2015 Knowledge Translation Casebook

Share your NIDILRR-funded knowledge translation (KT) experience with the Center on KTDRR for inclusion in the 2015 KT Casebook! All NIDILRR grantees share a goal "to promote the generation and effective use of knowledge in areas of importance to individuals with disabilities and their families" (NIDILRR Long-Range Plan for 2013-2017). Grantee contributions to the KT Casebook will help advance our knowledge about KT activities working within the parameters of a NIDILRR grant activity. To highlight your project's work in the 2015 KT Casebook, please contact Ann Outlaw by December 14, 2015! The 2014 KT Casebook highlighting grantees' KT efforts in our community is now available.

In scanning abstracts resulting from a CINAHL search or in leafing through the table of contents of a hardcopy journal you may have come across the term "scoping review", and wondered what it is and how it differs from a systematic review. Alternative terms being used are scoping study, scoping project, scoping exercise, scoping report, scoping method, scoping exercise method, as well as literature mapping, mapping of research, evidence mapping, systematic mapping, literature review, and rapid review (Colquhoun et al., 2014; Pham et al., 2014). We would need to do a careful review of the papers in question to make sure all these various attempts at knowledge synthesis are scoping reviews—if we could come to consensus as to what a scoping review is. It certainly is a new enough animal that in their "scoping review of scoping reviews" Pham et al. (2014) reported that 63% of authors of 344 papers found it necessary to describe what they meant by a "scoping review" (or whatever other "scoping" term they used). Our first indication that a scoping review still is a "rare bird," or worse, a chameleon, might be that one could hardly imagine systematic review authors feeling obliged to define the term "systematic review".

The scoping review as it exists today may have first been named by Mays, Roberts and Popay (2001). They wrote that scoping reviews "aim to map rapidlythe key concepts underpinning a research area and the main sources and types of evidence available, and can be undertaken as stand-alone projects in their own right, especially where an area is complex or has not been reviewed comprehensively before" [p. 194, emphasis in original.] However, most authors of scoping types of reviews claim that directly or indirectly their methodology is derived from a paper by Arksey and O’Malley (2005). They wrote in this key paper:

So what might we consider to be the main differences between a systematic review and a scoping study? First, a systematic review might typically focus on a well‐defined question where appropriate study designs can be identified in advance, whilst a scoping study tends to address broader topics where many different study designs might be applicable. Second, the systematic review aims to provide answers to questions from a relatively narrow range of quality assessed studies, whilst a scoping study is less likely to seek to address very specific research questions nor, consequently, to assess the quality of included studies (p. 20).

That seems to settle the matter, but other authors have described a similar enterprise as a literature mapping, and of the canonical five steps in doing a scoping review described (prescribed?) by Arksey and O’Malley and outlined below most authors omit the last one, which Arksey and O’Malley indeed call an optional stage. Plus: some authors apply the term ‘scoping review’ to a systematic exploration of which concepts and terms are used in what manner in a particular domain, rather than to the creation of an inventory of research findings. There are even those who define what most researchers might call primary research, e.g. collecting the opinions of a sample of experts, as a scoping review. There is plenty of confusion.

Let's start with what Arksey and O’Malley (2005) stated as being the steps in doing a scoping review:

1. Identify the research questions: what domain needs to be explored?
2. Find the relevant studies, through the usual means: electronic databases, reference lists (ancestor searching), websites of organizations, conference proceedings, etc.
3. Select the studies that are relevant to the question(s)
4. Chart the data, i.e. the information on and from the relevant studies
5. Collate, summarize and report the results
6. (Optional) consult stakeholders (clinicians, patients and families, policy makers, or whatever is the appropriate group) to get more references, provide insights on what the literature fails to highlight, etc.

Arksey and O’Malley emphasize that this is not a linear process (as typically dictated by the protocol for a systematic review) but a back-and-forth between early finds and new insights and changes in the search terms and even the questions. They do emphasize that because a scoping review goes wide (although maybe not deep), Mays et al.'s ‘rapid’ may not be applicable.

Other methodologists have distinguished various ways of doing steps 4 and 5, with Schmucker, Motschall, Antes and Meerpohl (2013, p. 1391) claiming that evidence mapping limits itself to tabular displays of the evidence found, while scoping reviews also summarize the search results "descriptively narratively." If we follow their lead, we may have the rough progression depicted in Table 1 below.

Table 1. A possible ordering of knowledge-synthesis methodologies

	Term	Steps Involved
1	Narrative Review	Review of literature that is unsystematically searched and minimally extracted to answer a broad question that may be vaguely stated. 1
2	Evidence Mapping	Adds: explicit questions, systematic search for evidence, and tabular summaries of the nature and findings of the studies.
3	Scoping Review	Adds: a narrative integration of the relevant evidence.
4	Systematic Review	Uses a narrowquestion, adds: evaluation of the quality of the evidence, and recommendations based on a qualitative synthesis of all the evidence or only the high-quality evidence.
5	Meta-Analysis	Adds: a quantitative synthesis of the evidence based on statistical pooling of the findings of studies selected.

¹Narrative reviews do not address a broad question by definition, but it would seem many, if not most, do.

However, it must be admitted that while the term evidence mapping exists (with as prominent protagonists Bragge et al. (2011), who have focused on evidence relevant to treatment and rehabilitation of spinal cord injury and traumatic brain injury, most authors who have written about the methodology of scoping reviews do not make a distinction based on how the literature found is presented. On the other hand, Arksey and O’Malley (2005, p. 21) give "four common reasons why a scoping review might be undertaken," of which the first one is "to examine the extent, range and nature of research activity; this type of rapid review might not describe research findings in any detail but is a useful way of mapping fields of study where it is difficult to visualize the range of material that might be available." The second common reason is determining whether a systematic review is feasible and of value. The third and fourth common reasons seem to describe scoping reviews that are not exploratory or preparatory, but done in their own right: "summarizing and disseminating research findings to policy makers, practitioners and consumers" (#3) and identifying gaps in the existing evidence base followed by conclusions as to the overall state of the research activity in a particular area (#4).

In their review of "scoping studies" that had been commissioned by the United Kingdom's Service Delivery and Organisation Research Programme, an office located within the National Institute of Health Research, Anderson, Allen, Peckham and Goodwin (2008, p. 3) stress that the "scoping exercise" "may or may not entail conceptual mapping, the mapping of policy documents and consultations with stakeholders, as well as mapping the literature." With conceptual mapping they mean exploring the terminology being used relevant to a particular topic: key terms, supplemental terms, etc. Policy mapping is collecting all the relevant policy documents from government bodies and non-governmental entities such as professional organizations. These two may be typical of how scoping reviews are seen in the United Kingdom, or by governmental organizations with responsibility for a research agenda or a research implementation agenda.

Arksey and O’Malley (2005) emphasize that a scoping review, however much it may resemble a systematic review, does not involve an assessment of the quality of the primary studies, with which most authors who have written about the methodology of scoping reviews (Davis, Drey & Gould, 2009; Levac, Colquhoun & O’Brien, 2010) agree, although Whittemore, Chao, Jang, Minges and Park (2014) state that there are "no well-established criteria" for a scoping review to have or not have quality appraisal of the primary studies. Daudt, van Mossel, & Scott (2013) do suggest that scoping reviews should use some form of quality assessment. In the most recent attempt to create some clarity as to what a scoping review is, Colquhoun et al. (2014) suggest that a scoping review or study is a "form of knowledge synthesis that addresses an exploratory research question aimed at mapping key concepts, types of evidence, and gaps in research related to a defined area or field by systematically searching, selecting and synthesizing existing knowledge." They propose to make an enhanced Arksey and O’Malley methodology the standard for the method (i.e.,no quality assessment of the evidence)—potentially disappointing the 22% of authors who, according to Pham et al. (2014), did assess the methodology of the studies being scoped.

There is a potentially crucial issue here: if a scoping study is undertaken to map the research in a particular area, specifically to identify the gaps in research, how useful is such an exercise if the quality of that research is not addressed? Using a hypothetical example, it may be comforting to know that there are as many as 20 studies of aspect X of this area of interest, suggesting that our knowledge is sufficiently broad. But if all those 20 studies were done poorly, do we really have knowledge, whether the results are displayed graphically, tabularly, or narratively? Or can we only be certain of our knowledge, and possibly of the recommendations being made to policy makers, practitioners or patients, if we have done some form of quality assessment first—an assessment which may result in a sizeable reduction in the number of studies in the evidence base? Pham et al. (2014) seem to think so, as do Grant and Booth (2009). Including a careful assessment of the studies that produced the evidence may bring a scoping review closer to a systematic review than Colquhoun et al. (2014) would like to see. It behooves disability and rehabilitation researchers who plan to use a scoping review, whether it is an inventory of measures of participation (Seekins et al., 2012) or of strategies used in stakeholder engagement (Camden et al., 2014), to review carefully the methodology that was used to map the literature in the area in question. Until such time that we indeed have a standard definition of and methodology for scoping studies, the chameleon needs to be studied carefully every time we find him traipsing through our literature.

References

Anderson, S., Allen, P., Peckham, S., & Goodwin, N. (2008). Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems, 6(7). doi:10.1186/1478-4505-6-7 http://www.health-policy-systems.com/content/6/1/7

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology: Theory & Practice, 8(1), 19-32.
http://www.tandfonline.com/doi/abs/10.1080/1364557032000119616

Bragge, P., Clavisi, O., Turner, T., Tavender, E., Collie, A., & Gruen, R. L. (2011). The Global Evidence Mapping Initiative: scoping research in broad topic areas. BMC Medical Research Methodology, 11, 92-2288-11-92. doi:10.1186/1471-2288-11-92

Camden, C., Shikako-Thomas, K., Nguyen, T., Graham, E., Thomas, A., Sprung, J., . . . Russell, D. J. (2014). Engaging stakeholders in rehabilitation research: a scoping review of strategies used in partnerships and evaluation of impacts. Disability and Rehabilitation, 37(15), 1-11. doi:10.3109/09638288.2014.963705

Clavisi, O., Bragge, P., Tavender, E., Turner, T., & Gruen, R. L. (2013). Effective stakeholder participation in setting research priorities using a Global Evidence Mapping approach. Journal of Clinical Epidemiology, 66(5), 496-502.e2. doi:10.1016/j.jclinepi.2012.04.002

Colquhoun, H. L., Levac, D., O’Brien, K. K., Straus, S., Tricco, A. C., Perrier, L., Kastner, M. & Moher, D. (2014). Scoping reviews: time for clarity in definition, methods, and reporting. Journal of Clinical Epidemiology, 67(12), 1291-1294. doi:10.1016/j.jclinepi.2014.03.013

Daudt, H. M., van Mossel, C., & Scott, S. J. (2013). Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley's framework. BMC Medical Research Methodology, 13, 48-2288-13-48. doi:10.1186/1471-2288-13-48

Davis, K., Drey, N., & Gould, D. (2009). What are scoping studies? A review of the nursing literature. International Journal of Nursing Studies, 46(10), 1386-1400. doi:10.1016/j.ijnurstu.2009.02.010

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26(2), 91-108. doi:10.1111/j.1471-1842.2009.00848.x

Levac, D., Colquhoun, H., & O’Brien, K. K. (2010). Scoping studies: advancing the methodology. Implementation Science, 5(69) doi:10.1186/1748-5908-5-69
http://www.implementationscience.com/content/5/1/69

Mays, N., Roberts, E., & Popay, J. (2001). Synthesising research evidence. In N. Fulop, P. Allen, A. Clarke, & N. Black (Eds.), Studying the organisation and delivery of health services: Researchmethods(pp. 188-219). London: Routledge.

Pham, M. T., Rajic, A., Greig, J. D., Sargeant, J. M., Papadopoulos, A., & McEwen, S. A. (2014). A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Research Synthesis Methods, 5, 371–385. doi:10.1002/jrsm.1123

Schmucker, C., Motschall, E., Antes, G., & Meerpohl, J. J. (2013). Methods of evidence mapping. A systematic review. [Methoden des Evidence Mappings. Eine systematische Ubersichtsarbeit] Bundesgesundheitsblatt, Gesundheitsforschung, Gesundheitsschutz, 56(10), 1390-1397. doi:10.1007/s00103-013-1818-y

Seekins, T., Shunkamolah, W., Bertsche, M., Cowart, C., Summers, J. A., Reichard, A., & White, G. (2012). A systematic scoping review of measures of participation in disability and rehabilitation research: a preliminary report of findings. Disability and Health Journal, 5(4), 224-232. doi:10.1016/j.dhjo.2012.05.002

Whittemore, R., Chao, A., Jang, M., Minges, K. E., & Park, C. (2014). Methods for knowledge synthesis: an overview. Heart & Lung: The Journal of Critical Care, 43(5), 453-461. doi:10.1016/j.hrtlng.2014.05.014