Utilization Measurement: Focusing on the "U" in "D&U"

Special Report

Introduction

The importance of addressing utilization. Many disability researchers, when they begin planning for dissemination and utilization (D&U), think primarily about the "D." Dissemination is the important item on most people's agendas: how to get research results to intended audiences in the most effective, cost-efficient manner. If someone pauses to consider utilization, it's generally to wonder why three words are employed when, it seems, one would suffice. The phrase dissemination and utilization seems too big a mouthful, literally and conceptually.

But utilization is a critical element in increasing the effective reach of disability research outcomes. Focusing only on the "D" in D&U is like dialing nine numbers of a ten-digit telephone number: You may be 90 percent finished, but unless you dial that last digit, you'll never make the correct intended connection.

What do dissemination and utilization address? Where does one kind of activity end and the other begin? There is no single, clear line of demarcation. In fact, if you're familiar with the literature (see NCDDR, 1996; NCDDR, 1999), you probably know that some definitions of dissemination go so far as to encompass use. Generally, though, dissemination has been unable to break free of its roots — its Latin roots, that is, its literal reference to scattering seed. People associate dissemination with spreading the word; how ideas and information get used seems another issue altogether.

So it's helpful to pull the two ideas apart, to assure that each can be addressed explicitly. We've found the following to be a useful way of thinking about D&U:

• Dissemination speaks primarily to the process of knowledge transfer — the who, what, when and how of moving ideas and information from the source(s) to intended recipient(s).
• Utilization speaks primarily to purpose and to impact — why you want people to get the research outcomes you're putting forth, what use you want people to make of the ideas, information, or products, and how people are actually using them.

Evaluation as key to improving utilization. Central to the utilization portion of D&U is evaluation. To understand how effectively you are reaching your audiences — not merely in terms of whether audience members hear a Public Service Announcement about employment of people with disabilities, for example, but in terms of how listeners do or do not apply the information they hear — you must have some means of evaluating use.

This paper is intended as an introduction to the role of evaluation in the utilization process. Its purpose is to help disability researchers grasp the importance of incorporating a focus on assessing use into plans for disseminating research outcomes. The paper examines basic evaluation principles as they relate to dissemination and utilization, and describes the major steps involved in evaluation planning. Future publications in this series will examine specific evaluation strategies that may be particularly useful to disability researchers.

Some Basics about Evaluation

Evaluation is too rarely thought of as a tool that is substantively important to a project's implementation. If you're like many people involved in research and development (R&D), you probably have little to do with the evaluation of your programmatic activities, either at the planning or the implementation stages. When preparing a proposal or plan of work, many people either bring in an evaluation specialist to draft the evaluation plan, or recycle language and procedures used in previous projects. Proposal writers are often careful to include both "formative" and "summative" activities, and to identify indicators if required, then it's on to the "real" work of the project. Implementing the evaluation plan becomes the job of another department, or a staff member or consultant hired specifically for the task. Because the R&D program staff have had little or nothing to do with designing the evaluation, their attention to the results is based mostly on a desire for the project to get a "clean bill of health."

That is not the approach to evaluation we're discussing here. The following are some basic ideas that may help bring evaluation into better focus as a practical tool that can be of real service in your work.

Evaluation is more than a compliance activity. Everything, including evaluation, comes back to the question of utility. According to Michael Quinn Patton, on whose book, Utilization-Focused Evaluation (Edition 3, 1997), this paper is based, the first thing to consider about evaluation is its use. The purpose of evaluation is not merely to comply with funding guidelines or to lend an air of legitimacy to a research and development effort. Rather, the goal is improvement. If your goal is to do the best possible job of developing and/or disseminating disability research outcomes that people actually use, then your evaluation activities should yield information that you and your staff can use. But it's up to you to make evaluation useful, by getting involved in both its design and its implementation.

Evaluation is not research. Often people resist thinking about evaluation because they conceive of it as a necessarily elaborate, highly technical research activity. Although there are significant correspondences between the two, evaluation as we are discussing it does not necessarily require a research methodology in order to yield useful results. As Patton (p. 24) notes,

Program evaluation differs fundamentally from research in the purpose of data collection and standards for judging quality… Research aims to produce knowledge and truth. Useful evaluation supports action.

One important commonality that evaluation does share with research, however, is that, like research, evaluation must be grounded in reliable data. In some cases, you may use research methods to generate that data; or you may use "management information system data, program monitoring statistics, or other forms of systematic information that are not research-oriented" (Patton, p. 24).

Effective evaluation must meet basic criteria. If evaluation activities are to be of help in improving D&U, they should meet four basic criteria, or standards of quality. These criteria were developed in 1980 by a national Joint Committee on Standards, representing a number of professional organizations, and were restated in 1994. The four basic criteria are:

• Utility. Utility involves taking steps to ensure that "an evaluation will serve the practical information needs of intended users." As committee chair Daniel Stufflebeam observed, "an evaluation should not be done at all if there is no prospect for its being useful to some audience."
• Feasibility — To be judged as feasible, an evaluation must be "realistic, prudent, diplomatic, and frugal." Stufflebeam suggests an evaluation should not be conducted "if it is not feasible to conduct it in political terms, or practicality terms, or cost-effectiveness terms."
• Propriety — Propriety involves ensuring that "an evaluation will be conducted legally, ethically, and with due regard for the welfare" of both the individuals participating in the evaluation and those who are affected by its results.
• Accuracy — To be accurate, an evaluation must "reveal and convey technically adequate information about the features that determine worth or merit" of what's being evaluated. This criterion addresses the methodological appropriateness of both the evaluation plan and its implementation (Patton, pp. 16-17).

There is no single, best way to "do" an evaluation. There are many approaches to evaluation that can meet the four basic criteria listed above. "The notion that there is one right way to do things dies hard." Patton concludes:

Every evaluation situation is unique. A successful evaluation (one that is useful, practical, ethical, and accurate) emerges from the special characteristics and conditions of a particular situation — a mixture of people, politics, history, context, resources, constraints, values, needs, interests, and chance. (p. 126)

According to evaluation experts including Patton, designing the "right" evaluation for any specific program "requires interaction, negotiation, and situational analysis." It also involves keeping a clear eye on the purpose of the evaluation activity, which is to yield useful results. Utility should never have to play second fiddle to methodology.

There are multiple dimensions to measuring use. The role of evaluation in relation to the D&U process is to help disability researchers understand how much and how effectively consumers are using the research outcomes that have been disseminated. Utilization of research outcomes, then, can be considered in terms of two basic dimensions.

When researchers think about evaluating consumers' use, they generally think of only one of these dimensions: the level or extent of use. Patton describes this dimension as the "use —; non-use continuum." Evaluation along this continuum involves measuring the degree or magnitude of use among intended audiences.

There is a second, equally important, dimension, however: the use —; misuse continuum. In most instances, it's important to assess the manner as well as the degree of use of disability research outcomes, asking and obtaining answers to the question: How — and how effectively — are consumers using research outcomes?

One important caution about evaluation along the use - misuse continuum: The common tendency is to focus on evaluating use in terms of the researchers' intentions, that is, assessing the extent to which consumers actually use research outcomes in the way that disability researchers intended. But intended use and effective use are not always synonymous. Sometimes consumers make adaptations or find effective uses never considered by researchers; sometimes cultural differences between researchers and consumers shift the assumptions as to what constitutes effective use.

You may want to know whether consumers are using research outcomes as you intended; certainly that's a legitimate and important function of evaluation. It's also important, though, to include in your data collection plan ways of looking beyond your specific intentions, and to be careful in the assumptions you make about use versus misuse.

Steps in the Evaluation Planning Process

As is true for dissemination, evaluation planning needs to begin in the earliest stages of a project. Planning evaluation activities that will yield useful results, however, is likely to be quite different from the pro forma or compliance-oriented planning typical of many research and development (R&D) projects. The following steps are key to planning an effective, utilization-focused evaluation.

1. Determine who the intended users of the evaluation are. The first step is to identify the people who will use the evaluation results, so that evaluation questions and results can be tailored to their needs. Patton observes, "Clearly and explicitly identifying people who can benefit from an evaluation is so important that evaluators have adopted a special term for potential evaluation users: stakeholders" (p. 41).

In evaluating the dissemination and utilization of disability research outcomes stakeholders may include many groups, including the grant-sponsored staff and other organization staff, managers, advisory boards, boards of directors, funding sources, consumers and their families, and direct service providers. It is difficult, and often impossible, to devise evaluation activities that address the diverse interests of all these stakeholders.

That means that some process is necessary for narrowing the range of possible questions to focus the evaluation. In utilization-focused evaluation, this process begins by narrowing the list of potential stakeholders to a much shorter, more specific group of primary intended users. Their information needs, that is, their intended uses, focus the evaluation (Patton, p. 42).

The disability researchers involved in developing and disseminating research outcomes are, of course, an important stakeholder group whose information needs should be addressed in any evaluation activity. But inclusion is not enough. As studies conducted by Patton and others show, it is people's interest in, and commitment to, using evaluation results that determine whether those results actually support program improvement. Patton describes this as "the personal factor," and offers the following definition:

The personal factor is the presence of an identifiable individual or group of people who personally care about the evaluation and the findings it generates. Where such a person or group was present, evaluations were used; where the personal factor was absent, there was a correspondingly marked absence of evaluation impact (p. 44).

It is essential, then, for disability researchers and related program staff to recognize the ways in which evaluation results can strengthen their work, and to think of evaluation as an integral part of their programmatic work.

2. Identify the purpose of the evaluation. Three primary purposes for evaluation include: "rendering judgments, facilitating improvements, and/or generating knowledge" (Patton, p. 65). For disability researchers, the first two purposes will generally be of greatest concern. Most evaluation efforts will be concerned with generating data that demonstrate the project's value to funding sources and institutional leaders and that help the project and its successors work more effectively.

While it is possible for a single evaluation to address more than one, even all three, of these basic purposes, there are hazards in doing so.

One [purpose] is likely to become the dominant motif and prevail as the primary purpose informing design decisions and priority uses… I also find that confusion among these quite different purposes, or failure to prioritize them, is often the source of problems and misunderstandings along the way and can become disastrous at the end when it turns out that different intended users had different expectations and priorities (Patton, p. 67).

The fundamental differences between evaluation results that are used to render judgments about program merits and those used for program improvement may be characterized as:

Judgment-oriented evaluation requires preordinate, explicit criteria and values that form the basis for judgment. Improvement-oriented approaches tend to be more open ended, gathering varieties of data about strengths and weaknesses with the expectation that both will be found and each can be used to inform an ongoing cycle of reflection and innovation (Patton, p. 68).

Given that, in most cases, any evaluation of your efforts will need to address both of these purposes, it is important to design an evaluation that gets at both kinds of information. And just as questions of utility should not play second fiddle to a rigid methodological perspective (as we noted earlier), they shouldn't be subordinated to questions of judgment, either.

3. Specify the questions that the evaluation needs to answer. This step is really a further elaboration of the evaluation's purpose. Most people are well acquainted with judgment-related questions, which shape most compliance-oriented evaluations. In addition, Patton (p. 68) identifies a number of questions that are relevant to improvement-oriented evaluation. In the list below, some questions have been adapted to more specifically address disability research:

• What are the research project's strengths and weaknesses?
• To what extent are intended audiences making use of disability research outcomes?
• Which types of consumers or stakeholders are using your disability research outcomes and which types aren't?
• Which types of dissemination media have been most effective in reaching targeted consumers? Which have been least effective?
• How are consumers using disability research outcomes? How closely do those uses correspond with project staff's identified uses?
• What kinds of implementation problems have emerged and how are they being addressed?
• What's happening that wasn't expected?
• How are research project staff and consumers interacting?
• What are staff and consumer perceptions of the program? What do they like? dislike? want to change?
• What are perceptions of the research project's culture and climate?
• How are funds being used compared to initial budgetary expectations?
• How is the research project's external environment affecting internal operations?
• Where can increased or improved efficiencies be realized?
• What new ideas are emerging that can be tried out and tested?

4. Specify evaluation criteria. As noted earlier, the importance of delineating evaluation criteria varies according to the evaluation's purpose. Patton emphasizes that, "in judgment-oriented evaluations, specifying the criteria for judgment is central and critical" (p. 66). Different stakeholders may have far different perceptions as to what is important in determining a research project's success. For example, a legislator who helped to authorize a funding initiative may be most concerned about the number of consumers reached by a project, while project staff may be much more concerned about the extent of behavior change among a much smaller, targeted beneficiary group. It's important, then, to clarify expectations at the planning stage, so that evaluation methods will yield the kinds of information you need.

Evaluation criteria are often derived from the goals and objectives outlined in a grant proposal or other plan of work. As Patton points out, however, there are often big problems with this approach. Such goals tend to be stated in terms of research project activities or services rather than in terms of consumer outcomes. Patton itemizes a number of examples of these problematic goals/objectives, such as:

• To develop needed services for chronically chemically dependent clients.
• Develop a supportive, family-centered, empowering, capacity-building intervention system for families and children (p. 155).

What's needed instead is "a clear statement of the targeted change in circumstances, status, level of functioning, behavior, attitude, knowledge, or skills" among intended users (Patton, p. 159).

Because of the difficulty in collecting data that fully capture the kinds of changes listed in the preceding paragraph, it's usually necessary to come up with some indicators that reflect the desired changes you want to assess. Patton emphasizes the need to remember that outcomes and indicators are not synonymous.

Some kind of indicator is necessary, however, to measure degree of outcome attainment. The key is to make sure that the indicator is a reasonable, useful, and meaningful measure of the intended client outcome (Patton, pp. 160-161).

As stated earlier, improvement-oriented evaluation is much more open-ended than is judgment-oriented evaluation. Patton notes that there are alternatives to "goals-based evaluation" (p. 177). One alternative, labeled "goal-free evaluation" by evaluation expert Michael Scriven (cited in Patton, p. 181), involves "gathering data on a broad array of actual effects and evaluating the importance of these effects in meeting [the] demonstrated needs" of intended audiences. Another alternative that Patton recommends, and one that addresses the improvement-related information needs of disability researchers, is to focus on the kinds of questions that the users of evaluation results want answered, questions such as those listed in the preceding section. Rather than specific criteria for success, the questions serve to shape the evaluation methodology and data analysis.

Even if you focus your improvement-oriented evaluation activities on such questions, however, it will likely be necessary to generate some explicit statements related to your programmatic intentions. For example, in order to assess the extent to which consumers are using research outcomes as researchers intended, it will be necessary to explicitly describe those intended uses and what constitutes "use" on the part of the consumer.

5. Identify evaluation methods. One of the hazards in trying to shift to utilization-focused evaluation is that, in the planning process, project staff may work their way through the preceding four steps and then, at step five, turn the planning over to an evaluation expert. These evaluation specialists, in turn, often simply revert back to old, familiar but non-utilitarian evaluation methods. In contrast, Patton notes:

In utilization-focused evaluation, methods decisions, like decisions about focus and priority issues, are guided and informed by our evaluation goal: intended use [of the evaluation results] by intended users (p. 241).

The best way to achieve this goal is to involve those intended users — meaning, among others, you and your research project staff — in making decisions about evaluation methods. What's needed is an evaluator who can help guide you through this decision making process.

The utilization-focused evaluator advises intended users about options; points out the consequences of various choices; offers creative possibilities; engages with users actively, reactively, and adaptively to consider alternatives; and facilitates their methods decisions. At the stage of choosing methods, the evaluator remains a technical adviser, consultant, and teacher. The primary intended users remain decision makers about the evaluation (Patton, p. 243).

There are, of course, a number of important decisions to be made regarding the evaluation methodology; these may include the use of:

• Mail questionnaires, telephone interviews, or personal face-to-face interviews?
• Individual interviews or focus groups?
• Even-numbered or odd-numbered scales on survey items?
• Opinion, knowledge, and/or behavioral questions?
• All closed questions or some open-ended? If some open-ended, how many?
• Norm-referenced or criterion-referenced tests?
• Development of our own instruments or adoption of measures already available?
• Experimental design, quasi-experimental design, or case studies?
• Participant observation or spectator observation?
• A few in-depth observations or many shorter observations?
• Standardized or individualized protocols?
• Fixed or emergent design?
• Follow up after two weeks, three months, six months, or a year?
• Follow up everyone or a sample?
• What kind of sample: simple random, stratified, and/or purposeful?
• What size sample?
• Will interviewers have the same characteristics as program participants: gender? age? [disability?] race?
• What comparisons to make: past performance? intended goals? hoped-for goals? other research project outcomes? (Patton, p. 247)

The list of decisions can seem overwhelming, but with good technical assistance, the processing is not as daunting as it may seem. One helpful bit of information, is that, contrary to the beliefs of most non-researchers, "there are no universal and absolute standards for judging" the appropriateness of evaluation methods. Rather, the criteria for quality "are necessarily situational and context bound" (Patton, p. 249).

Conclusions

Adapting utilization-focused evaluation represents a significant departure from the more typical current practice in the field. The challenge to change is by no means trivial; it requires more work of project staff and — perhaps most difficult — a reorientation in thinking on the part of both evaluators and project staff. There are many factors that may tend to divert you, your staff, and the evaluators engaged in your project from a focus on utilization of evaluation results. Patton suggests eight such diversions, and calls them seven "use-deadly" sins (plus one). These include:

1. Making evaluators the primary decision makers and, therefore, the primary users.
2. Identifying vague, passive audiences as users [of evaluation results] instead of real people.
3. Targeting organizations as users (e.g. 'the feds') instead of specific persons.
4. Focusing on decisions instead of decision makers.
5. Assuming the evaluation's "funding entity" is automatically the primary stakeholder.
6. Waiting until the findings are in to identify intended users and intended uses.
7. Taking a stance of standing above the fray of people and politics.
8. Identifying primary intended users at the outset of the study, then ignoring them until the final report is ready (Patton, pp. 58, 60).

With a genuine commitment to project/practice improvement, however, it is more than possible to shift your evaluation plans to produce results you can actually use. Given the imperative to increase the use of disability research outcomes, we must all do no less.

Source Materials

Michael Quinn Patton (1997). Utilization-focused evaluation: The new century text. Edition 3. Thousand Oaks, California: SAGE Publications.

NCDDR (1996). A review of the literature on dissemination and knowledge utilization. Austin, TX: Southwest Educational Development Laboratory. Available: https://ktdrr.org/ktlibrary/articles_pubs/ncddrwork/NCDDR_lit_review_on_dissem.pdf

NCDDR (1999). Disability, diversity, and dissemination: A review of the literature on topics related to increasing the utilization of rehabilitation research outcomes among diverse consumer groups. Austin, TX: Southwest Educational Development Laboratory.