Articles on Research Quality

Quality research most commonly refers to the scientific process encompassing all aspects of study design; in particular, it pertains to the judgment regarding the match between the methods and questions, selection of subjects, measurement of outcomes, and protection against systematic bias, nonsystematic bias, and inferential error. Quality research is necessary to obtain reliable and valid knowledge, particularly knowledge relevant to interventions, environmental adaptations, and products or devices.

Following are some articles from the KT Library that address these topics. KTDRR staff reviewed a number of articles, developed a brief abstract, and assigned ratings based on strength of evidence, readability, and consumer orientation. For more information on these ratings, see KT Library Descriptor Scales.

American Dietetic Association. (2007). ADA evidence analysis manual (5th ed.)     

Abstract: The American Dietetic Association manual provides a step-by-step process for evidence analysis including specific actions to be taken at each step. Numerous charts, checklists and worksheets to guide the user’s process, as well as a glossary of terms related to research design, are found in the appendices.

Descriptor Scales

American Educational Research Association Task Force on Reporting of Research Methods. (2006). Standards for reporting on empirical social science research in AERA publications.

Abstract: The American Educational Research Association (AERA) adopted standards on reporting empirical research in AERA publications in June, 2006. The standards were developed to assist researchers, editors, reviewers and readers of AERA journals. They are based on two major principles: that empirical research should be warranted, and that the research reporting process is transparent at every step. The standards are organized into the following areas: problem formulation, design and logic of the study, sources of evidence, measurement and classification, analysis and interpretation, extrapolation, ethics in reporting and, title.

Descriptor Scales

Coalition for Evidence-Based Policy. (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide. NCEE EB2003. Washington, DC: US Department of Education. Institute of Education Sciences, National Center for Educational Evaluation and Regional Assistance.

Abstract: The Coalition for Evidence-based Policy document provides educators with guidelines in evaluating the random assignment process, outcome data, and results report to determine whether or not the research is "evidence-based." The document includes a rationale for the guidelines as well as a checklist for use in the evaluation.

Descriptor Scales

DeJong, G., & Horn, S. (2005). Randomized controlled trials in rehabilitation research. New Zealand Journal of Disability Studies, 11, 120-124. 

Abstract: DeJong and Horn present aspects of rehabilitation that make randomized controlled trials (RCT) difficult to accomplish. Acknowledging that rehabilitation involves the implementation of multiple interventions, the authors raise issues with RCT methods of isolating one intervention, creating a clinical environment controlling variables, restricting selection—thus affecting generalizability, statistical challenges of using very small numbers of participants, and the problem of not being able to provide an intervention that is "blind" to the participant and researcher. The authors suggest that researchers in rehabilitation look for alternative research designs that address the unique characteristics of rehabilitation practice and RCTs be restricted to less complicated interventions.

Descriptor Scales

Demner-Fushman, D., Few, B., Hauser, S. E., &Thoma, G. (2006). Automatically identifying health outcome information in MEDLINE records. Journal of the American Medical Informatics Association, 13, 52-60.     

Abstract: Demner-Fushman et al. target health care professionals with limited time to review research. The authors describe an automated evidence-based medicine model approach to identifying relevant information in medical research quickly without needing to analyze the entire document. The approach was ranked against PubMed Clinical Queries and the authors found that the outcome-based ranking provided significantly more accurate information.

Descriptor Scales

Dijkers, M. P. J. M. for the NCDDR Task Force on Systematic Review and Guidelines. (2009). When the best is the enemy of the good: The nature of research evidence used in systematic reviews and guidelines. Austin, TX: SEDL.

Abstract: When the Best is the Enemy of the Good: The Nature of Research Evidence Used in Systematic Reviews and Guidelines was developed in August, 2009, by the Task Force on Systematic Review and Guidelines. This task force paper explores critical issues related to the "gold standard" for research designs, the emergence of systematic reviews, and implications for evidence-based rehabilitation and clinical practice.

Descriptor Scales

Dromerick, A. W. (2003). Evidence-based rehabilitation: The case for and against constraint-induced movement therapy. Journal of Rehabilitation Research and Development, 40(1), viii - ix.     

Abstract: Dromerick's editorial notes the lack of empirical research to support practices in rehabilitation. The author suggests that the widespread use of constraint-induced movement therapy (CIMT) as a rehabilitation treatment is not justified based on the lack of quality multicenter randomized control trials comparing CIMT to other active motor treatments. Dromerick does cite Vander Lee (2001) and Dromerick, Edwards & Hahn, (2006) as quality research in this field suggesting this work may be a preliminary research basis for CIMT.

Descriptor Scales

Ebell, M. H., Siwek, J., Weiss, B. D., Woolf, S. H., Susman, J. Ewigman, B., & Bowman, M. (2004). Strength of recommendation taxonomy (SORT): A patient-centered approach to grading evidence in the medical literature. American Family Physician, 69, 548-556.     

Abstract: Ebell et al. suggest using the Strength of Recommendation Taxonomy (SORT) scale to determine the quality of medical reviews, . The SORT scale was developed by representatives of family medicine and primary care journals as well as the Family Practice Inquiries Network. SORT is based on evaluation of quality of study design, quantity of studies included in the review and consistency with outcomes reported. In addition, the scale includes a determination of whether the outcomes are patient-oriented or disease-related. Further, the authors link SORT to other compatible taxonomies. The authors suggest that use of a single scale across studies and journals help to analyze outcomes for translation into practice.

Descriptor Scales

Glanville, J. M., Lefebvre, C., Miles, J. N. V., & Camosso-Stefinovic, J. (2006). How to identify randomized controlled trials in MEDLINE: Ten years on. Journal of the Medical Library Association, 94(2), 130-136.

Abstract: Glanville et al. examines whether the 1994 Cochrane Highly Sensitive Search Strategy to search for randomized controlled trials in MEDLINE could be improved after ten years of use. They found that "clinical trial" was the best discriminating term. In years in which Cochrane had assessed MEDLINE records, few additional records were found. However, for records not assessed by Cochrane, the term "randomized controlled trial" was very accurate at identifying non-indexed trials, almost equaling the precision of the Cochrane Highly Sensitive Search Strategy.

Descriptor Scales

Green, L. W., & Glasgow, R. E. (2006). Evaluating the relevance, generalization, and applicability of research: Issues in external validation and translation methodology. Evaluation & the Health Professions, 29(1), 126-153. source.

Abstract: Green and Glasgow suggest that current research does not include sufficient emphasis on external validation or generalizability. The authors propose criteria to evaluate the external validity of research, such as the inclusion of members of the target population in the study; use of intended settings; reporting the expertise and training of people providing implementation, as well as any adaptations made for different settings; effects beyond primary outcomes including quality of life issues; and reporting costs. The article recommends that external validity should be included in the planning process, thus making the research relevant to the people who will use the outcomes for setting policy or for decision making on an individual level. A companion article is available in this collection entitled, "Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness translation" (Glasgow, 2003).

Descriptor Scales

Grimshaw, J., Eccles, M. Thomas, R., MacLennan, G., Ramsay, C., Fraser, C., & Vale, L. (2006). Toward evidence-based quality improvement: Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966-1998. Journal of General Internal Medicine, 21, S14-S20. source.

Abstract: The Grimshaw et al. systematic review addresses the effectiveness and costs of guideline dissemination and implementation strategies. The authors included studies from 1966 through 1998 and found them to be weak in reporting of methodology. The authors noted the advantages of using paper or electronic reminders to improve care rather than a multifaceted approach of educational outreach. However, the article cautions that after 30 years of research on guideline dissemination and implementation, there continues to be a lack of quality studies to inform the field about quality improvement strategies.

Descriptor Scales

Haynes, R. B., Cotoi, C., Holland, J., Walters, L, Wilczynski, N., Jedraszewski, D.,McKinlay, J., Parrish, R., & McKibbon, K. A. (2006). Second-order peer review of the medical literature for clinical practitioners. JAMA, 295(15), 1801-1808.

Abstract: Haynes et al. describe The McMaster Online Rating of Evidence (MORE) system that utilizes practicing physicians to rate peer-reviewed journal articles in their discipline as the basis for inclusion in the McMaster Premium Literature Service (PLUS) Internet access program. Following a review by staff, volunteer physicians rate articles by whether the article is important to the field (relevance) and whether it is new information (newsworthy). The ratings provide a screen for articles to be included in an Internet service that notifies physicians of recent research. The project demonstrated the value of a peer review of published journal articles by discipline.

Descriptor Scales

Johnston, M. V., Sherer, M., & Whyte, J. (2006). Applying evidence standards to rehabilitation research. American Journal of Physical Medicine and Rehabilitation, 85, 292-309. full-text source.

Abstract: Johnston et al. explain evidenced based practice standards used in systematic reviews. In addition, the authors apply the evidence based methods to analyze the quality of research in spinal cord injury, traumatic brain injury, and burn rehabilitation. The article concludes that although the rehabilitation field has experienced a dramatic increase in systematic reviews published each year, the number of studies that met the highest level of criteria was very small in all three areas of research.

Descriptor Scales

Lauer, P. A. (2004). A policymakers primer on education research: How to understand, evaluate, and use it. Aurora, CO: Mid-continent Research for Education and Learning.

Abstract: The Lauer document to provides decision-makers and other interested persons with a comprehensive explanation of the various aspects of educational research. Its intended audience is people who have limited knowledge regarding educational research. A joint effort of the Education Commission of the States (ECS) and Mid-continent Research for Education and Learning (McREL), the Primer is "part of a larger project that seeks to improve the connection between research and policy" (p. iii). It offers sufficient information to support evidence-based decisions based on empirical research.

Descriptor Scales

NCDDR. (2005). FOCUS Technical Brief (12). What Consumers and Researchers Say About Research.

Abstract: The NCDDR and the Research Utilization Support and Help (RUSH) project at the Southwest Educational Development Laboratory conducted two studies in 2005 with different audiences in order to learn more about their perceptions of research and how best to get information to diverse groups of end users. This issue of FOCUS shares the findings from the two studies and suggests potential implications.

Descriptor Scales

NCDDR. (2005). FOCUS Technical Brief (9). What are the standards for quality research?

Abstract: This issue of FOCUS discusses principles and standards for quality research, the basis for these standards, and strategies for reporting quality research. In the fields of disability and rehabilitation research, there is a healthy debate regarding the specific criteria for quality research, and the specific checklists to be used to standardize reporting. As the debate ensues, there are many ideas emerging in the public domain related to quality research and quality evidence that can be used to help guide the discussion.

Descriptor Scales

Schlosser, R. W. (2006). FOCUS Technical Brief (15). The Role of Systematic Reviews in Evidence-Based Practice, Research, and Development.

Abstract: This issue of FOCUS written by Ralf W. Schlosser, PhD, is part one of a three part series on the topic of evidence-based technology. This issue provides an overview of systematic reviews in research and development. Systematic reviews aim to synthesize the results of multiple original studies by using strategies that delimit bias. Systematic reviews can be used to inform evidence-based practice, which is increasingly shaping the disability and rehabilitation research field.

Descriptor Scales

Schlosser, R.W. (2007). FOCUS Technical Brief (17). Appraising the Quality of Systematic Reviews.

Abstract: This issue of FOCUS written by Ralf W. Schlosser, PhD, is part two of a three part series on systematic reviews. This issue describes critical considerations for appraising the quality of a systematic review including the protocol, question, sources, scope, selection principles, and data extraction. The author also describes tools for appraising systematic reviews.

Descriptor Scales

Scott, A. (2006). Peer review and the relevance of science. SPRU Electronic Working Paper Series 145.

Abstract: Scott analyzes of the use of peer reviews in funding decisions, academic publishing, and promotions in research institutions and universities. The author suggests that peer reviews tend to downplay social relevance and innovation in preference or judging potential research in comparison to existing knowledge. The article notes that social problems require a peer review process that includes a wider range of competencies and interests, such as external validity, relevance, interdisciplinary effort and level of risk. Further, the author recommends that peer reviews be conducted in a structured manner to make the process more transparent.

Descriptor Scales

Shadish, W., & Myers, D. (2004). Campbell Collaboration research design policy brief.

Abstract: The Research Design Policy Brief provides a rationale and proposed policies regarding The Campbell Collaboration’s systematic reviews on the effectiveness of an intervention. The policies propose development of two databases for randomized and nonrandomized studies, standard design codes to be used in reviews, and designated searchable fields to identify research in the databases.

Descriptor Scales

Shadish, W. R., & Rindskopf, D. M. (2007). Methods for evidence-based practice: Quantitative synthesis of single-subject designs. New Directions for Evaluation, 113, 95-109.

Abstract: Shadish and Rindskopf describe the use of single-subject designs in meta-analyses. The article reviews methods for analyzing multiple single-subject designs, suggests methods for conducting a meta-analysis using single-subject designs, and includes a list of current meta-analyses.

Descriptor Scales

Task Force on Systematic Review and Guidelines. (2013). Assessing the quality and applicability of systematic reviews (AQASR). Austin, TX: SEDL, Center on Knowledge Translation for Disability and Rehabilitation Research.

Abstract: The basic purpose of the AQASR document and checklist is to help busy clinicians, administrators, and researchers to ask critical questions that help to reveal the strengths and weaknesses of a systematic review, in general, and as relevant to their particular clinical question or other practical concerns. Its primary audience is clinicians, as most systematic reviews are optimized to answer the clinical questions they have.

Descriptor Scales

Valentine, J. C., Cooper, H.,,Patall, E. A., Tyson, D., & Robinson, J. C. (2010). A method for evaluating research syntheses: The quality, conclusions, and consensus of 12 syntheses of the effects of after-school programs. Research Synthesis Methods, 1(1), 20-38. doi: 10.1002/jrsm.3

Abstract: Valentine et al. examine the methods used in 12 research syntheses of after-school programs (ASPs). Their goals are to determine if "best practices" are demonstrated in these syntheses, to compare the conclusions from the syntheses with the data observed by this review team, and to look for consistency regarding conclusions about the effects of ASPs. Evaluation tools and a coding guide are created and implemented to compare the various syntheses. A wide variety of methods are identified across the syntheses and few consistencies in the findings are identified. The tools and procedures used by the authors can help researchers and policymakers to review more than one synthesis for decision making.

Descriptor Scales

Additional Articles

Dijkers, M. P. J. M., Brown, M., & Gordon, W. A. (2008). FOCUS Technical Brief (19). Getting Published and Having an Impact: Turning Rehabilitation Research Results Into Gold.

Abstract: The FOCUS authored by Drs. Marcel Dijkers, Margaret Brown, and Wayne Gordon from the Mount Sinai School of Medicine, Department of Rehabilitation Medicine, New York, suggests strategies that rehabilitation researchers can use to maximize their work-turning "research results into gold." In the disability and rehabilitation research community, it is important for researchers to be cognizant of how published results of research studies can facilitate or limit their use in answering important evidence-based questions.

Descriptor Scales