Modeling Technology Innovation: Combining Science, Engineering, and Industry Methods to Achieve Beneficial Socioeconomic Impacts Systematically and Deliberately

This FOCUS Technical Brief summarizes a paper recently published in the open-access journal Implementation Science¹ (Stone & Lane, 2012). The full paper presents a conceptual framework that integrates all three knowledge-generating methods—scientific research (R), engineering development (D), and industry production (P)—into a logic model format, which is useful for planning, obtaining, and measuring the intended beneficial impacts by implementing knowledge in practice (Lane & Flagg, 2010).

Background

Government-sponsored science, technology, and innovation (STI) programs support the socioeconomic aspects of public policies in addition to expanding the knowledge base. For example, beneficial healthcare services and devices are expected to result from investments in research and development (R&D) programs, an expectation that assumes a causal link to commercial innovation. Such programs are increasingly held accountable for evidence of impact—that is, innovative goods and services resulting from R&D activity.

However, the absence of comprehensive models and metrics skews evidence gathering toward bibliometrics about research outputs (published discoveries), with less focus on transfer metrics about development outputs (patented prototypes) and almost no discernible econometrics related to production outputs (commercial innovations). This disparity is particularly problematic for the expressed intent of such programs because the last category, production outputs, leads to most of the measurable socioeconomic benefits, such as improved health and function from product and service use, revenues generated by corporations, and tax payments into government coffers resulting from the new net wealth generated in the commercial marketplace.

Methods

This issue of FOCUS summarizes a paper recently published in the open-access journal Implementation Science¹ (Stone & Lane, 2012). The full paper presents a conceptual framework that integrates all three knowledge-generating methods—scientific research (R), engineering development (D), and industry production (P)—into a logic model format, which is useful for planning, obtaining, and measuring the intended beneficial impacts by implementing knowledge in practice (Lane & Flagg, 2010).

The framework also integrates the Context-Input-Process-Product (CIPP) model of evaluation (Stufflebeam, 2004). The CIPP model takes a systemic approach by referring to project goals, inputs, processes and outputs. Needs analysis, which is central to context evaluation, lends direction to the project by orienting it more toward the target audience’s needs, while bringing relevance (worth) to the planned output. Input evaluation ensures that the project is feasible. Process evaluation for research, development, or production methods promotes efficiency and effectiveness. Output evaluation ensures and assesses the quality (merit) of the output and continues to follow up. Thus, the CIPP approach to evaluation builds relevance into STI policies and programs while sustaining appropriate focus on the issue of rigor (Worthen, Sanders, & Fitzpatrick, 1997).

Results

The resulting logic model framework explicitly traces the progress of knowledge from inputs as it moves through the three knowledge-generating processes (R, D, P) and their respective knowledge outputs (discovery, invention, innovation) to the intended sociobeneficial impacts. The framework is a hybrid model for creating technology-based innovations by merging best practices in new product development with a widely accepted knowledge translation (KT) approach.

Given the emphasis on evidence-based practice in the medical and health fields and "bench to bedside" expectations for knowledge transfer, sponsors and grantees alike should find the model useful for planning, implementing, and evaluating innovation processes.

Figure 1 presents the logic model framework, which shows the results of integrating the CIPP model with the three related methods of scientific research, engineering development, and industrial production. This hybrid model provides technology-based innovation program sponsors and project managers with an operational role for evaluation, from incorporating relevance along with rigor to tracking outcomes and impacts as well as activity outputs.

Figure 1 combines all the relevant components in a comprehensive diagram to show how the role of evaluation spans the entire innovation process and how KT serves to bridge the components. The diagram illustrates the links between R, D, and P methods and how they combine to create and deliver a technology-based innovation to the marketplace. It also depicts the mechanisms involved in generating the socioeconomic benefits expressed in public policies and supported through government programs.

The time and effort required to progress through this sequence is partly dependent on the path taken. One can use the figure to trace paths of differing length from the output of any method (R, D, or P) to the outcomes and impact. The time frame for research outputs to achieve impacts—particularly for technology-based projects—is longer because of the need to pass through the two downstream methods of development and production. For research projects, achieving impacts is often beyond the scope of funding and the project's time frame. This issue is an important point for project and program accountability, which typically tracks results only up to the termination of the funding time frame. It is unlikely that research projects can demonstrate downstream impacts during the award period. At best, researchers can demonstrate the downstream plan through which they or other stakeholders will complete the development and/or production activities and thereby transform outputs into impacts.

The integrated logic model in Figure 1 emphasizes the importance of performing two context evaluations prior to initiating any efforts intended to generate technology-based innovations. This initial planning phase of a project is the opportune time to apply the two forms of context evaluation: (1) program context—the analysis of the broader situational context around a project's identified problem, which informs funding priorities and requests for proposals; and (2) project context—the needs and opportunities analysis specific to a project's immediate context, which provides information necessary for defining project objectives. Using this approach helps ensure that evidence-based program priorities lead to the funding of evidence-based project objectives.

The logic model highlights three additional elements:

1. The CIPP evaluation activities are juxtaposed above, to the left, and below the project activities column and are connected respectively to its objectives, structure, and process.
2. The project activities column shows R, D, and P activities and how they collectively advance new knowledge toward commercial innovations. Note the KT bridges within the project process and how they link methodological outputs vertically and horizontally. Note also the initial gate (G) before the R phase, where one can avoid the time and expense involved in sponsoring new research if the necessary knowledge already exists in the literature base inside or outside the field of application. Similarly, the transitions from research to development and then from development to production should be considered decision gates where one can opt to proceed or to stop work. This option prevents the mindset of proceeding regardless of the likely results simply because resources have been allocated for that purpose.
3. The KT bridges go outward from outputs to outcomes; note the forked KT symbols going to short-term outcomes. Here, KT happens in two ways: a general KT (the white boxes in the Short-Term Outcomes column) and a more focused KT (the shaded boxes in the same column). The first case involves delivering outputs to all stakeholders with potential interest. The second case involves perhaps limiting KT to a specific group, organization, or individual, such as a manufacturer, that is positioned to treat the knowledge as an input to the next method.

Figure 1 also extends the logic model over time. For each KT case—general KT and focused KT—the diagram shows a sequence of outcomes that should result from the planned and coordinated outputs of R, D, and P activities. Programs and projects are expected to obtain a sequence of outcomes, from changing stakeholders' awareness of specific knowledge, to building their interest in the knowledge, to their eventual implementation of the knowledge. Implementation should result in changes to practice (e.g., the use of evidence-based applications, prototype construction and testing, and commercial device and service manufacturing) or to policy (e.g., regulation and reimbursement of devices and services).

The initial context evaluation mentioned earlier helps ensure that programs and projects achieve this sequence of outcomes leading to the expected impact. At the program level, evidence-based information about socioeconomic needs amenable to technology-based innovations helps funding agencies assess grant proposals for relevance, define indicators of impact, and determine how to monitor and evaluate funded projects. The context evaluation ensures that needs remain central to funding priorities and project deliverables.

At the project level, the evidence-based needs analysis aligns the project deliverables with the sponsor's mission while ensuring the relevance of project outputs to the intended knowledge users prior to initiating activities. Figure 1 shows how this prior-to-grant perspective flows and suggests why this approach is preferable to end-of-grant or integrated approaches to KT, particularly for those programs and projects explicitly intended to result in beneficial socioeconomic impacts from technology-based innovations. Given the option, why would program teams pursue any other path?

Figure 1 also integrates key concepts and creates connections to guide the construction of integrated logic models for utilization-focused R&D. As a static graphic, the figure is of necessity simple and linear in form. Yet it can serve as a basis for constructing nonlinear and complex models, as needed, to incorporate and explicate elements that have a bearing on the causal sequence represented by this simple model. In addition, the figure can be readily expanded for individual programs and projects to reflect their unique characteristics and contexts.

The main point of the integration and connection among concepts in Figure 1 is to champion the cause of relevance, alongside rigor, through a continuous KT effort that starts at the beginning of an R&D program or project. Future additions to this work will address how to better integrate government and academic R&D programs with privately funded industry efforts to implement the outputs from R&D in technology-based innovations.

The outcomes in the commercial marketplace are necessary as incentives for companies to generate the desired socioeconomic benefits. These incentives include revenues from sales paid to corporations to cover their costs (e.g., salaries, materials, and facilities) and profits to owners and shareholders. Of course, a portion of these revenues are paid to the government as taxes (profits to companies, income to employees, taxes on sales), which cycle back through the public coffers to be allocated as public funds used to sponsor R&D. Moreover, profitable companies benefit their home nations, so the balance of trade translates directly into national R&D capacity.

Summary

Efforts to improve society while competing economically necessarily include programs that support technology-based innovations. Quality-of-life issues are paramount in the fields of medicine and healthcare. The process through which scientific knowledge is translated and technological knowledge is transferred should be accurately modeled for planning, implementation, and evaluation purposes. Describing the mechanisms underlying technology-based innovations and tracking the indicators of progress are necessary for establishing coherent milestones and accomplishing systematic results. If successful, sponsoring organizations will shift their perspective from the solution-driven "bench to bedside" approach to the need-driven "bedside to bench and back" approach.

The approach described here integrates elements from the CIPP model of evaluation into the basic, linear logic model format that currently guides program planning and evaluation practice. What links the two models in this framework is the provision of a context evaluation prior to initiating any activity. This prior-to-grant perspective elevates the quality of relevance to a level equal to the quality of rigor—an orientation encompassing the stakeholders who determine success or failure of the entire effort. As a result, funding agencies can focus program goals to ground project objectives in the context of validated needs. The framework also clarifies the roles of process and product evaluations, which strengthen the merit and worth of project outputs. The role of outcome evaluations beyond the traditional measures of outputs (i.e., publications, patents) is to assess the actual socioeconomic impacts and deliver those evidence-based results to funders and stakeholders alike.

CENTER ON KT4TT

The 5-year Center on Knowledge Translation for Technology Transfer (KT4TT) project (http://publichealth.buffalo.edu/cat/kt4tt.html) was awarded to the University at Buffalo (SUNY), Center for Assistive Technology (CAT) on October 1, 2008. SEDL and Western New York Independent Living, Inc., are partners in the project. SEDL's role focuses on utilization-oriented methods of dissemination, training, and technical assistance to effectively communicate with knowledge producers and knowledge users. This FOCUS Technical Brief is a product of the SEDL-KT4TT partnership.

The project focuses on three key outcomes:

• Improved understanding of the barriers preventing successful knowledge translation for technology transfer and ways to overcome these barriers
• Advanced knowledge of best models, methods, and measures of knowledge translation and technology transfer for achieving outcomes
• Increased utilization of these validated best practices by NIDILRR's technology-oriented grantees

Center on Knowledge Translation for Technology Transfer

View FOCUS Technical Brief Archived Issues