Articles on Measurement of KT Outcomes

Successful measurement of KT implementation includes procedures and instruments for the collection and analysis of practitioners' use of evidence–based knowledge, “gap closure”, or consumer–related outcomes (i.e., student, patient, and economic).

Following are some articles from the KT Library that address this topic. KTDRR staff reviewed a number of articles, developed a brief abstract, and assigned ratings based on strength of evidence, readability, and consumer orientation. For more information on these ratings, see KT Library Descriptor Scales.

Coalition for Evidence-Based Policy. (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide. NCEE EB2003. Washington, DC: US Department of Education. Institute of Education Sciences, National Center for Educational Evaluation and Regional Assistance.

Abstract: The Coalition for Evidence-based Policy document provides educators with guidelines in evaluating the random assignment process, outcome data, and results report to determine whether or not the research is "evidence-based." The document includes a rationale for the guidelines as well as a checklist for use in the evaluation.

Descriptor Scales

Demner-Fushman, D., Few, B., Hauser, S. E., &Thoma, G. (2006). Automatically identifying health outcome information in MEDLINE records. Journal of the American Medical Informatics Association, 13, 52-60.     

Abstract: Demner-Fushman et al. target health care professionals with limited time to review research. The authors describe an automated evidence-based medicine model approach to identifying relevant information in medical research quickly without needing to analyze the entire document. The approach was ranked against PubMed Clinical Queries and the authors found that the outcome-based ranking provided significantly more accurate information.

Descriptor Scales

Dromerick, A. W. (2003). Evidence-based rehabilitation: The case for and against constraint-induced movement therapy. Journal of Rehabilitation Research and Development, 40(1), viii - ix.     

Abstract: Dromerick's editorial notes the lack of empirical research to support practices in rehabilitation. The author suggests that the widespread use of constraint-induced movement therapy (CIMT) as a rehabilitation treatment is not justified based on the lack of quality multicenter randomized control trials comparing CIMT to other active motor treatments. Dromerick does cite Vander Lee (2001) and Dromerick, Edwards & Hahn, (2006) as quality research in this field suggesting this work may be a preliminary research basis for CIMT.

Descriptor Scales

Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003). Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. American Journal of Public Health, 93(8), 1261–1267.

Abstract: Glasgow, Lichtenstein, and Marcus note the large discrepancy between the number of efficacy and effectiveness studies. The authors suggest that the gap is due to inherent differences in how the studies are planned and implemented. The article includes recommendations for placing greater emphasis on external validation. The authors suggest how to enhance Consolidated Standards of Reporting Trials (CONSORT) criteria for reporting randomized clinical trials that reflect consideration of external validity. A companion article is available in this collection entitled, "Evaluating the relevance, generalization, and applicability of research" (Green, L. W., & Glasgow, R. E., 2006).

Descriptor Scales

Gold, M., & Taylor, E. F. (2007). Moving research into practice: Lessons from the US Agency for Healthcare Research and Quality's IDSRN program. Implementation Science, 2(9), 1-28.

Abstract: Gold and Taylor describe their methodology and results in evaluating the Integrated Delivery Systems Research network (IDSRN) program of the U.S. Agency for Healthcare Research and Quality (AHRQ). The authors conducted interviews and reviewed program documents as well as analyzed projects and case studies. The results supported the concept of linking researchers with users of the research in a team-based approach. Further, IDSRN provided a mechanism to address issues in an expedited manner. The results noted weaknesses in the administrative structure of IDSRN, which resulted in its reorganization into ACTION (Accelerating Change and Transformation of Organizations and Networks).

Descriptor Scales

Green, L. W., & Glasgow, R. E. (2006). Evaluating the relevance, generalization, and applicability of research: Issues in external validation and translation methodology. Evaluation & the Health Professions, 29(1), 126-153. source.

Abstract: Green and Glasgow suggest that current research does not include sufficient emphasis on external validation or generalizability. The authors propose criteria to evaluate the external validity of research, such as the inclusion of members of the target population in the study; use of intended settings; reporting the expertise and training of people providing implementation, as well as any adaptations made for different settings; effects beyond primary outcomes including quality of life issues; and reporting costs. The article recommends that external validity should be included in the planning process, thus making the research relevant to the people who will use the outcomes for setting policy or for decision making on an individual level. A companion article is available in this collection entitled, "Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness translation" (Glasgow, 2003).

Descriptor Scales

Grimshaw, J., Eccles, M. Thomas, R., MacLennan, G., Ramsay, C., Fraser, C., & Vale, L. (2006). Toward evidence-based quality improvement: Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966-1998. Journal of General Internal Medicine, 21, S14-S20. source.

Abstract: The Grimshaw et al. systematic review addresses the effectiveness and costs of guideline dissemination and implementation strategies. The authors included studies from 1966 through 1998 and found them to be weak in reporting of methodology. The authors noted the advantages of using paper or electronic reminders to improve care rather than a multifaceted approach of educational outreach. However, the article cautions that after 30 years of research on guideline dissemination and implementation, there continues to be a lack of quality studies to inform the field about quality improvement strategies.

Descriptor Scales

Harmsworth, S., Turpin, S., & the TQEF National Co-ordination Team. (2000). Creating an effective dissemination strategy: An expanded interactive workbook for educational development projects

Abstract: The Harmsworth and Turpin document is a step-by-step workbook for research teams to use in developing an effective plan for dissemination of their findings. The authors focus attention on what is to be disseminated, identification of the target populations, establishment of reasonable timeframes, and venue of dissemination. The authors note that effective dissemination involves the recipient to participate in the awareness, understanding or action dictated by the new knowledge.

Descriptor Scales

Haynes, R. B., Cotoi, C., Holland, J., Walters, L, Wilczynski, N., Jedraszewski, D.,McKinlay, J., Parrish, R., & McKibbon, K. A. (2006). Second-order peer review of the medical literature for clinical practitioners. JAMA, 295(15), 1801-1808.

Abstract: Haynes et al. describe The McMaster Online Rating of Evidence (MORE) system that utilizes practicing physicians to rate peer-reviewed journal articles in their discipline as the basis for inclusion in the McMaster Premium Literature Service (PLUS) Internet access program. Following a review by staff, volunteer physicians rate articles by whether the article is important to the field (relevance) and whether it is new information (newsworthy). The ratings provide a screen for articles to be included in an Internet service that notifies physicians of recent research. The project demonstrated the value of a peer review of published journal articles by discipline.

Descriptor Scales

Lauer, P. A. (2004). A policymakers primer on education research: How to understand, evaluate, and use it. Aurora, CO: Mid-continent Research for Education and Learning.

Abstract: The Lauer document to provides decision-makers and other interested persons with a comprehensive explanation of the various aspects of educational research. Its intended audience is people who have limited knowledge regarding educational research. A joint effort of the Education Commission of the States (ECS) and Mid-continent Research for Education and Learning (McREL), the Primer is "part of a larger project that seeks to improve the connection between research and policy" (p. iii). It offers sufficient information to support evidence-based decisions based on empirical research.

Descriptor Scales

NCDDR. (2005). FOCUS Technical Brief (10). What is knowledge translation?

Abstract: This issue of FOCUS discusses knowledge translation, a relatively new term that is used to describe a relatively old problem-the underutilization of evidence-based research in systems of care. This article describes relevant KT concepts, KT planning models, and suggests a working definition for KT that is designed to reflect NIDILRR's research and development priorities.

Descriptor Scales

NCDDR. (2005). FOCUS Technical Brief (11). Communities of practice: A strategy for sharing and building knowledge.

Abstract: This issue of FOCUS discusses the use of Communities of Practice (CoPs) as a knowledge transfer (KT) strategy. CoPs are "groups of people who share a concern, a set of problems, a passion about a topic, and who deepen their knowledge and expertise in this area by interacting on an ongoing basis" (Wenger, McDermott, & Snyder, 2002). By building on its members' shared knowledge, a CoP can be useful in developing new ideas and new strategies. The NCDDR's efforts to support a CoP for NIDILRR grantees are also described.

Descriptor Scales

NCDDR. (2005). FOCUS Technical Brief (12). What Consumers and Researchers Say About Research.

Abstract: The NCDDR and the Research Utilization Support and Help (RUSH) project at the Southwest Educational Development Laboratory conducted two studies in 2005 with different audiences in order to learn more about their perceptions of research and how best to get information to diverse groups of end users. This issue of FOCUS shares the findings from the two studies and suggests potential implications.

Descriptor Scales

Nelson, T. D., & Steele, R. G. (2007). Predictors of practitioner self-reported use of evidence-based practices: Practitioner training, clinical setting, and attitudes toward research. Administration and Policy in Mental Health and Mental Health Services Research, 34(4), 319 - 330.

Abstract: Nelson and Steele present the results of an online survey of 214 mental health practitioners in 15 states regarding the implementation of evidence-based practice (EBP). The survey instrument consists of 97 items requiring the practitioner to make a judgment on a 4-point scale. The authors examine the relationships between self-reported EBP use and several factors: perceived openness of the clinical setting toward EBP; theoretical orientation (i.e. cognitive-behavioral, behavioral, family systems, psychodynamic, humanistic); the practitioner’s attitudes toward treatment research (i.e. positive or negative); the practitioner’s receipt of training in EBP; practitioner’s academic degree; and practitioner’s years of experience.

Descriptor Scales

Patton, M. Q. (2002). Utilization-focused evaluation (U-FE) checklist.

Abstract: The Patton checklist provides step-by-step procedures for implementing a utilization-focused evaluation (U-FE). The U-FE is a process in which the evaluator and intended user collaborate on the design and implementation of the evaluation as well as the projected use of the outcomes. The process may be applied to any evaluation design, purpose, topic or type of data. The checklist also includes challenges for the evaluation facilitation at each step in the process.

Descriptor Scales

Renger, R., & Hurley, C. (2006). From theory to practice: Lessons learned in the application of the ATM approach to developing logic models. Evaluation and Program Planning, 29(2),106–119.

Abstract: Renger and Hurley give an overview of the three-step systematic Antecedent, Target, Measurements (ATM) approach to developing logic models. The authors offer the advantages of the ATM approach in comparison to others that have been developed. Further, the article includes practical lessons for implementing the model in a variety of settings.

Descriptor Scales

Schlosser, R.W. (2007). FOCUS Technical Brief (17). Appraising the Quality of Systematic Reviews.

Abstract: This issue of FOCUS written by Ralf W. Schlosser, PhD, is part two of a three part series on systematic reviews. This issue describes critical considerations for appraising the quality of a systematic review including the protocol, question, sources, scope, selection principles, and data extraction. The author also describes tools for appraising systematic reviews.

Descriptor Scales

Scott, A. (2006). Peer review and the relevance of science. SPRU Electronic Working Paper Series 145.     

Abstract: Scott analyzes of the use of peer reviews in funding decisions, academic publishing, and promotions in research institutions and universities. The author suggests that peer reviews tend to downplay social relevance and innovation in preference or judging potential research in comparison to existing knowledge. The article notes that social problems require a peer review process that includes a wider range of competencies and interests, such as external validity, relevance, interdisciplinary effort and level of risk. Further, the author recommends that peer reviews be conducted in a structured manner to make the process more transparent.

Descriptor Scales

Shadish, W., & Myers, D. (2006). SPRU Electronic Working Paper Series 145.

Abstract: Scott analyzes of the use of peer reviews in funding decisions, academic publishing, and promotions in research institutions and universities. The author suggests that peer reviews tend to downplay social relevance and innovation in preference or judging potential research in comparison to existing knowledge. The article notes that social problems require a peer review process that includes a wider range of competencies and interests, such as external validity, relevance, interdisciplinary effort and level of risk. Further, the author recommends that peer reviews be conducted in a structured manner to make the process more transparent.

Descriptor Scales

Southwest Educational Development Laboratory (SEDL) (2005), Utilizing measurement: Focusing on the "U" in "D&U". Research Utilization Support and Help (RUSH), Authority Section 202, 29 U.S. C. 761a; Federal Register, 2/6/97. pp. 5711-5721.

Abstract: This special report focuses on the importance of utilization as a part of the dissemination process. Information is presented on strategies that can be helpful in designing ways to measure the effectiveness of your dissemination activities through their use by intended audiences. The report also highlights the need for utilization evaluation to be a helpful tool for project staff by suggesting ways to improve dissemination and resulting utilization outcomes. Steps for planning and implementing an effective utilization evaluation effort are described.

Descriptor Scales

Teutsch, S. M., & Berger, M. L. (2005). Evidence synthesis and evidence-based decision making: Related but distinct processes. Medical Decision Making, 25, 487-489.

Abstract: Teutsch and Berger note the increasing use of evidence syntheses to assist a variety of leaders and policymakers in evidence-based decision making. The authors reports that evidence-based reviews and syntheses have very specific guidelines, including an appeals process, in order to make the process transparent to decision makers. However, the authors add that although evidence-based decision making should be transparent to stakeholders, there are no similar standards guiding the use of the information in evidence-based decision making.

Descriptor Scales

Woloshin, S., & Schwartz, L. M. (2006). What’s the rush? The dissemination and adoption of preliminary research results. Journal of the National Cancer Institute, 98(6), 372-373.

Abstract: Woloshin and Schwartz raise the issue of adopting the results of interim research into clinical practice. The authors recommended that meeting organizers and medical journal editors coordinate the presentation of results with the publication process, thus allowing for more complete information to be presented and establishing the expectation that research would be published near the same time as the meeting. The article includes a checklist for practitioners to consult in determining whether to use preliminary research findings. Finally, the authors recommend that any preliminary finding should include a caution that the information may change before the study is completed.

Descriptor Scales