recombination with the knowledge generated by heterogeneous partners;

(Howells, 2006; Lee et al., 2010; Muller & Zenker, 2001; Spithoven et al., 2010):

collaborative partners (scanning and information processing);

the academic labs. However, the knowledge transfer process between these two systems is characterised by high transaction costs and cognitive distance that hamper the direct contribution that universities can offer toward the commercialization of viable technologies (Gilsing et al., 2011; Kodama, 2008; Polt et al., 2001; Yusuf, 2008) (Figure 1). First, the establishment of university-industry linkages implies high search costs, since firms need to invest more time and resources in seeking and assessing academic partners than they do with those belonging to their supply chain. This can be ascribed, on one hand, to information asymmetry that makes it difficult for firms to stay up-to-date on the state-ofthe-art projects and the related results carried out at the university and, on the other hand, to the uncertainty about the future output deriving from the application of general and theoretical knowledge. Moreover, the information asymmetry and the uncertainty which characterize the TT process also generate high bargaining costs in terms of negotiation and coordination of the parties. Indeed, "firms and universities are exposed to different incentive schemes that shape their interest in the transfer process" (Gilsing et al., 2011: 4). Scientists aim to contribute to the generation of public knowledge through dissemination; whereas firms want to appropriate the advantages deriving from the rapid commercialization of products/services that embody the new knowledge. This leads to the identification of complex solutions to overcome motivation problems and to increase the reliability among partners. Finally, the differences between universities and firms in terms of systems of perception, interpretation and evaluation, as well as of ''shared meanings'' linked to the organizational culture generate barriers to the combining of knowledge.

Fig. 1. Incentive and barriers on the relationship between firms and Public Research Source: Polt et al., 2001

the academic labs. However, the knowledge transfer process between these two systems is characterised by high transaction costs and cognitive distance that hamper the direct contribution that universities can offer toward the commercialization of viable technologies (Gilsing et al., 2011; Kodama, 2008; Polt et al., 2001; Yusuf, 2008) (Figure 1). First, the establishment of university-industry linkages implies high search costs, since firms need to invest more time and resources in seeking and assessing academic partners than they do with those belonging to their supply chain. This can be ascribed, on one hand, to information asymmetry that makes it difficult for firms to stay up-to-date on the state-ofthe-art projects and the related results carried out at the university and, on the other hand, to the uncertainty about the future output deriving from the application of general and theoretical knowledge. Moreover, the information asymmetry and the uncertainty which characterize the TT process also generate high bargaining costs in terms of negotiation and coordination of the parties. Indeed, "firms and universities are exposed to different incentive schemes that shape their interest in the transfer process" (Gilsing et al., 2011: 4). Scientists aim to contribute to the generation of public knowledge through dissemination; whereas firms want to appropriate the advantages deriving from the rapid commercialization of products/services that embody the new knowledge. This leads to the identification of complex solutions to overcome motivation problems and to increase the reliability among partners. Finally, the differences between universities and firms in terms of systems of perception, interpretation and evaluation, as well as of ''shared meanings'' linked to the organizational culture generate barriers to the combining of knowledge.

Fig. 1. Incentive and barriers on the relationship between firms and Public Research

Source: Polt et al., 2001

In order to increase the closeness between the different actors as well as to reduce the search and bargaining costs that this KT process implies, different types of organizations have been sponsored by public institutions over the last few decades. KTOs are knowledge-intensive providers which aim to facilitate the interaction between these heterogeneous actors and to support the exploitation of the research results of universities by industrial firms. For instance, science parks have been established in order to nurture innovative environments in which firms and academic institutions can interact via informal channels favouring knowledge spillover. Examples of well-known science parks include Stanford Research Park and Research Triangle Park of North Carolina in the US (Link & Link, 2003), Cambridge in the U.K. and Sophia-Antipolis in France (Longhi, 1999; Keeble et al., 1999). Another type of KTO, the academic incubator, has the mission to foster the creation of start-up firms based on university-owned or licensed technologies (Phan et al., 2005). Moreover, technology transfer offices or industrial liaison offices have been established in order to reduce the information asymmetry typically encountered in the scientific knowledge market through the management of intellectual property rights generated by scientists (Macho-Stadler et al., 2007).

As discussed in the previous sections, the complexity of applying scientific knowledge, the technological convergence trend which calls for multidisciplinary research, and the fierce competition on prices and time-to-market, spur firms to focus their in-house R&D and complement it with external scientific knowledge (Tether & Tajar, 2008). In this regard, in order to meet the firms' requirements, recent research highlights that KTOs are expanding rapidly and further typologies are emerging, such as business innovation centres, innovation agencies, R&D labs, technology consultants, technical testing and analysis labs (Comacchio et al., 2011; Consoli & Elche-Hortelano, 2010; Laranja, 2009). Moreover, studies show that KTOs are progressively evolving in terms of wider TT services portfolios and additional roles performed. Considering the three main phases of the innovation value chain, the offer of KTOs nowadays encompasses a broad range of services concerning (Howells, 2006; Lee et al., 2010; Muller & Zenker, 2001; Spithoven et al., 2010):

More recently, KTOs have started to fulfil a further activity throughout the innovation value chain of firms, namely the intermediation between firms-receivers and their scientific and technological partners. A recent research study describes this activity in terms of roles performed by KTOs, which can act as (Johnson, 2008): *mediators or arbitrators* in cases of

Performance Evaluation for Knowledge Transfer Organizations:

**TYPOLOGIES MISSION STATEMENT**

Table 1. Typologies of KTOs and mission statements

that can be helpful in facilitating R&D collaboration.

KTOs may be beneficial from different points of view.

innovation policies and allocate resources effectively.

highlighting their strengths and weaknesses.

Laboratory of applied research and/or testing and

Source: Comacchio et al. 2011

analysis

Best European Practices and a Conceptual Framework 135

dispute, thus facilitating contract negotiation for the accomplishment of a project; *sponsors and distributors* of funding for innovation efforts; *filters and legitimators* of projects that are worthy of support; *technology brokers* acting as a repository of information about technology experts and new technology opportunities and as a bridge between seekers and solvers of innovative solutions; *resource/management providers* in terms of project management practices

The variety of KTO types (European Commission, 2004) and their differentiated offer (see Table 1) together with their relevance in promoting innovation and entrepreneurship at the firm level as well as at regional level raise the issue of identifying the appropriate criteria to assess the effectiveness of the knowledge transfer process performed by these organizations.

The importance of evaluating the performance of KTOs has become paramount due to the increasing need for companies to rely on providers and intermediaries that are able to support them during the different phases of the innovation value chain. In order to nurture a market for what KTOs offer, there is a quest for more information about their characteristics in terms of services/products provided as well as for the performance they are able to achieve. The availability and the diffusion of this information may contribute to reduce the asymmetry, and thus increase the transparency of the transactions which occur in the market for knowledge. Promoting the disclosure of the capabilities and performance of

First, KTOs identifying their value proposition and assessing their performance are better able to define their product/service portfolio and craft their competitive strategy at regional, national and international level. Second, from the demand side, firms-receivers can use a univocal method to compare and evaluate the offer of the different KTOs, thus they may benefit from the reduction of transactional costs related to the search for and the selection of appropriate partners, as well as the monitoring of the results attained. Finally, local and regional stakeholders, such as universities, institutions and policy makers, may define their

However, the variety of KTOs that have been set up over the last few decades together with the progressive broadening of their mission and the heterogeneity of stakeholders with different perspectives increase the complexity of the evaluation process. In the following section, we will present different approaches and attempts to KTO performance assessment,

**2.4 Summary: Why it is important to evaluate the KTOs' performance**

analysis and testing to client firms.

Supplying qualified services of research and development,

universities and companies. Technological transfer office Supporting the academic staff to identify and manage the

development.

Incubator Accelerating the growth and success of entrepreneurial

Business Innovation Centre Offering a range of integrated guidance and support

Chamber of commerce.

Topic centre Promoting a specific industry or a specific technological

Multi-sector centre Supplying of diversified services to firms operating in several sectors. Public research organization Performing research in its own Institutes, promoting

area inside a geographical context.

the qualification of human resources.

development analysis, testing and experimentation of products, processes and new technologies, technical and scientific dissemination of knowledge and documentation

competitiveness of regions and cities by creating new business opportunities and adding value to mature companies; fostering entrepreneurship and incubating new innovative companies; generating knowledge-based

organization's intellectual assets, including protecting intellectual property and transferring or licensing rights to

companies through an array of business support resources and services that could include physical space, capital, coaching, common services, and networking connections.

technological innovation through the offer of services that meet the requirements of the firms associated to the

Gathering and coordinating scientific, organizational and financial resources in the region in order to transfer acquired information on new production processes and

innovation and competitiveness of the national industrial system, promoting the internationalization of the national research system, providing technologies and solutions to emerging public and private needs, offering advice to the Government and other public bodies, and contributing to

services for projects carried out by innovative SMEs, thereby contributing to regional and local development.

Furthering the development and expansion of

research results to the entrepreneurial context.

jobs; building attractive spaces for the emerging knowledge workers; enhancing the synergy between

other parties to enhance prospects for further

in specific sectors at the national level.

Promoting the economic development and

Experimental station Carrying out pre-competitive industrial research and

**TYPOLOGIES MISSION STATEMENT**

Scientific park and technological hub

Chamber of commerce special

agency and laboratory

Territorial development

enterprise

Table 1. Typologies of KTOs and mission statements Source: Comacchio et al. 2011

dispute, thus facilitating contract negotiation for the accomplishment of a project; *sponsors and distributors* of funding for innovation efforts; *filters and legitimators* of projects that are worthy of support; *technology brokers* acting as a repository of information about technology experts and new technology opportunities and as a bridge between seekers and solvers of innovative solutions; *resource/management providers* in terms of project management practices that can be helpful in facilitating R&D collaboration.

The variety of KTO types (European Commission, 2004) and their differentiated offer (see Table 1) together with their relevance in promoting innovation and entrepreneurship at the firm level as well as at regional level raise the issue of identifying the appropriate criteria to assess the effectiveness of the knowledge transfer process performed by these organizations.

#### **2.4 Summary: Why it is important to evaluate the KTOs' performance**

The importance of evaluating the performance of KTOs has become paramount due to the increasing need for companies to rely on providers and intermediaries that are able to support them during the different phases of the innovation value chain. In order to nurture a market for what KTOs offer, there is a quest for more information about their characteristics in terms of services/products provided as well as for the performance they are able to achieve. The availability and the diffusion of this information may contribute to reduce the asymmetry, and thus increase the transparency of the transactions which occur in the market for knowledge. Promoting the disclosure of the capabilities and performance of KTOs may be beneficial from different points of view.

First, KTOs identifying their value proposition and assessing their performance are better able to define their product/service portfolio and craft their competitive strategy at regional, national and international level. Second, from the demand side, firms-receivers can use a univocal method to compare and evaluate the offer of the different KTOs, thus they may benefit from the reduction of transactional costs related to the search for and the selection of appropriate partners, as well as the monitoring of the results attained. Finally, local and regional stakeholders, such as universities, institutions and policy makers, may define their innovation policies and allocate resources effectively.

However, the variety of KTOs that have been set up over the last few decades together with the progressive broadening of their mission and the heterogeneity of stakeholders with different perspectives increase the complexity of the evaluation process. In the following section, we will present different approaches and attempts to KTO performance assessment, highlighting their strengths and weaknesses.

Performance Evaluation for Knowledge Transfer Organizations:

case of the spin-off processes and research programs.

which enable them to provide KT services).

enterprise survival rate following graduation (final output).

**Key Performance Indicators**

**3.2 Output evaluation**

three key areas.

the KT goals.

**Area of evaluation**

Best European Practices and a Conceptual Framework 137

costs are incurred when the service is provided (Gardner et al., 2010; Hogan, 1996), as in the

The aforementioned causes of complexity in achieving a uniform methodology for the evaluation of KTOs explain the wide diversity of approaches implemented by these organizations so far. This section provides a review of the extant methods used in assessing the performance of KTOs drawing on initiatives and best practices identified in the European context. The review aims to identify the set of core areas of evaluation and the related Key Performance Indicators (KPIs) which are likely to be common to all KTOs, in order to find convergence in appraisal methods. Drawing on institutional reports elaborated by European KTOs associations (such as the Association of European Science and Technology Transfer Professionals ASTP, the European BIC Network or the Pan-European Network of Knowledge Transfer Offices ProTon) as well as on appraisal tools elaborated by some European KTOs, the following sub-sections will present and discuss two main sets of KPIs related to two macro-areas of evaluation, which are the *output* of the KT process (in terms of research results, economic-financial returns and impact on firms-receivers and the local innovation system) and *input* (the structural, human and social capital of the KTOs

The evaluation approach which focuses on the results attained by the KTO may embrace

The first area aims to capture the level of effectiveness of the KTO in achieving its KT goals, such as establishing high-tech firms in the region or promoting research collaborations between academia and firms. In this regard, metrics should assess both intermediate and final outputs of the knowledge transfer process. For instance, the effectiveness a KTO whose goal is to nurture the creation of new firms through incubation services can be assessed by monitoring the number of tenants hosted by the KTO (intermediate output) but also the

Table 2 shows in the first column the main aims included in the mission statement of KTOs: i) to establish channels of communication and collaboration between firms and research institutions; ii) to promote the commercialization of intellectual property rights generated by scientists; iii) to nurture entrepreneurship in the local innovation system; iv) to support the creation of new companies. For each area of evaluation the second column reports the Key Performance Indicators that are usually used to monitor the achievement of

Research activities Number of collaborative research agreements stipulated during the

implementation and share the project outputs

year with other trans-regional or international KTOs

year where both firms, universities, institutions and other KTOs participate in the design of the research project, contribute to its

Number of collaborative research agreements stipulated during the

#### **3. Knowledge transfer evaluation and a comparative analysis of European best practice**

#### **3.1 Introduction: The challenge of evaluating KTOs**

During the last decade, European and local institutions have sponsored initiatives to identify methodologies in supporting KTOs to define their KT objectives, to assess the way these objectives are being fulfilled, and to set up permanent monitoring systems that can guarantee the evaluation of performance over time (European Commission, 2009; Guy, 1996). However, the attempts towards the definition of an evaluation system of KTOs have encountered several difficulties that can be ascribed to the characteristics of KTOs and the specific features of the knowledge transfer process.

First, KTOs present a high level of heterogeneity in terms of mission statements and business models adopted. Indeed, the institutional statements are often too broad, encompassing a wide range of aims that make it difficult to identify the KT objectives to set, monitor and evaluate. Moreover, each type of KTO has its own particular aims to pursue, increasing the difficulties of making comparisons across categories of KTOs (Hogan, 1996). Furthermore, the strategy and the subsequent business models implemented are determined by multiple stakeholders who show different expectations regarding the KTO performance measurement system. On the one hand, local and regional institutions aim to obtain an objective approach to measure allocated resources and to plan future investments consistently with the regional policy guidelines; on the other, firms want to obtain a performance appraisal tool that enables them to compare KTOs and select the effective partner to involve in their innovation process (Bigliardi et al., 2006). This makes designing a common performance measurement system, that conciliates these multiple interests, highly complex.

Second, the broad services portfolio provided nowadays by KTO increases the complexity of performance measurement due to the fact that each type of service should be assessed by a set of specific indicators. Furthermore, the knowledge contents transferred by KTO are characterized by different levels of tacitness and uncertainty that generate complexity in identifying their value. Indeed, a KTO may provide explicit and codified knowledge more easily valuable in a market for technological knowledge (such as the licensing of IP rights or the testing and prototyping of activities), but also knowledge characterized by a high degree of uncertainty and tacitness that makes it difficult to define appraisal metrics (for instance the promotion and coordination of research projects for new product development).

Third, the measurement of the impact of the KT process encompasses both the macro and the micro level of analysis, since the KTOs' activities have an effect on the single firmreceiver as well as on the local economic environment (region, industrial cluster) in which the KTO operates. This generates problems in terms of data availability and accessibility concerning the benefits attained by the local system and the receivers, but also in terms of isolating the results that can be ascribed to the specific intervention of KTOs (Gardner et al., 2010; Guy, 1996).

Finally, KTOs do not produce instant results. Indeed, the impact of the KT process on the firms-receivers and on the local environment takes time to emerge, whereas many of the costs are incurred when the service is provided (Gardner et al., 2010; Hogan, 1996), as in the case of the spin-off processes and research programs.

The aforementioned causes of complexity in achieving a uniform methodology for the evaluation of KTOs explain the wide diversity of approaches implemented by these organizations so far. This section provides a review of the extant methods used in assessing the performance of KTOs drawing on initiatives and best practices identified in the European context. The review aims to identify the set of core areas of evaluation and the related Key Performance Indicators (KPIs) which are likely to be common to all KTOs, in order to find convergence in appraisal methods. Drawing on institutional reports elaborated by European KTOs associations (such as the Association of European Science and Technology Transfer Professionals ASTP, the European BIC Network or the Pan-European Network of Knowledge Transfer Offices ProTon) as well as on appraisal tools elaborated by some European KTOs, the following sub-sections will present and discuss two main sets of KPIs related to two macro-areas of evaluation, which are the *output* of the KT process (in terms of research results, economic-financial returns and impact on firms-receivers and the local innovation system) and *input* (the structural, human and social capital of the KTOs which enable them to provide KT services).

#### **3.2 Output evaluation**

136 Management of Technological Innovation in Developing and Developed Countries

During the last decade, European and local institutions have sponsored initiatives to identify methodologies in supporting KTOs to define their KT objectives, to assess the way these objectives are being fulfilled, and to set up permanent monitoring systems that can guarantee the evaluation of performance over time (European Commission, 2009; Guy, 1996). However, the attempts towards the definition of an evaluation system of KTOs have encountered several difficulties that can be ascribed to the characteristics of KTOs and the

First, KTOs present a high level of heterogeneity in terms of mission statements and business models adopted. Indeed, the institutional statements are often too broad, encompassing a wide range of aims that make it difficult to identify the KT objectives to set, monitor and evaluate. Moreover, each type of KTO has its own particular aims to pursue, increasing the difficulties of making comparisons across categories of KTOs (Hogan, 1996). Furthermore, the strategy and the subsequent business models implemented are determined by multiple stakeholders who show different expectations regarding the KTO performance measurement system. On the one hand, local and regional institutions aim to obtain an objective approach to measure allocated resources and to plan future investments consistently with the regional policy guidelines; on the other, firms want to obtain a performance appraisal tool that enables them to compare KTOs and select the effective partner to involve in their innovation process (Bigliardi et al., 2006). This makes designing a common performance measurement system, that conciliates these multiple interests, highly

Second, the broad services portfolio provided nowadays by KTO increases the complexity of performance measurement due to the fact that each type of service should be assessed by a set of specific indicators. Furthermore, the knowledge contents transferred by KTO are characterized by different levels of tacitness and uncertainty that generate complexity in identifying their value. Indeed, a KTO may provide explicit and codified knowledge more easily valuable in a market for technological knowledge (such as the licensing of IP rights or the testing and prototyping of activities), but also knowledge characterized by a high degree of uncertainty and tacitness that makes it difficult to define appraisal metrics (for instance

the promotion and coordination of research projects for new product development).

Third, the measurement of the impact of the KT process encompasses both the macro and the micro level of analysis, since the KTOs' activities have an effect on the single firmreceiver as well as on the local economic environment (region, industrial cluster) in which the KTO operates. This generates problems in terms of data availability and accessibility concerning the benefits attained by the local system and the receivers, but also in terms of isolating the results that can be ascribed to the specific intervention of KTOs (Gardner et al.,

Finally, KTOs do not produce instant results. Indeed, the impact of the KT process on the firms-receivers and on the local environment takes time to emerge, whereas many of the

**3. Knowledge transfer evaluation and a comparative analysis of European**

**3.1 Introduction: The challenge of evaluating KTOs**

specific features of the knowledge transfer process.

**best practice**

complex.

2010; Guy, 1996).

The evaluation approach which focuses on the results attained by the KTO may embrace three key areas.

The first area aims to capture the level of effectiveness of the KTO in achieving its KT goals, such as establishing high-tech firms in the region or promoting research collaborations between academia and firms. In this regard, metrics should assess both intermediate and final outputs of the knowledge transfer process. For instance, the effectiveness a KTO whose goal is to nurture the creation of new firms through incubation services can be assessed by monitoring the number of tenants hosted by the KTO (intermediate output) but also the enterprise survival rate following graduation (final output).

Table 2 shows in the first column the main aims included in the mission statement of KTOs: i) to establish channels of communication and collaboration between firms and research institutions; ii) to promote the commercialization of intellectual property rights generated by scientists; iii) to nurture entrepreneurship in the local innovation system; iv) to support the creation of new companies. For each area of evaluation the second column reports the Key Performance Indicators that are usually used to monitor the achievement of the KT goals.

Performance Evaluation for Knowledge Transfer Organizations:

intervention

**Key Performance Indicators**

collaboration with KTO

Sources: Bigliardi et al., 2006; EBN, 2011; European Commission, 2009.

 Total turnover for services and its trend Total turnover for area location and its trend

Total license revenue and its trend

foreign countries

foreign countries

**Key Performance Indicators**

 ROA and its trend ROE and its trend

expenditure

European Commission, 2009; Piccaluga et al., 2011.

staff

Table 3. Key Performance Indicators of KTOs' impact

**Area of evaluation**

**Area of evaluation**

Financial returns

Economic performance

Impact on the firms-receivers and the local innovation system

Best European Practices and a Conceptual Framework 139

Growth in turnover in companies attributed to KTO intervention

 Number of collaborative relationships and joint ventures among local, extra-regional and international firms favored by KTO Number and type of environmental improvement carried out in

Investment flows installed by KTO from other regions or from

Personnel flows installed by KTO from other regions or from

 Total financial value of all research agreements and its trend Revenues generated during the year from profits and/or sales of

 Average number of start-ups per FTE member of KTO staff Average number of jobs created per FTE member of KTO staff Average number of business plans created per FTE member of KTO

 Average number of jobs created per 100K€ of KTO income Average number of business plans created per 100K€ of KTO

Sources: Area SciencePark, 2011; Arundel & Bordoy, 2010; Bigliardi et al., 2006; EBN, 2011;

Average number of companies assisted per 100K€ of KTO income

Average number of companies assisted per FTE member of KTO staff

equity in spin-offs in which the KTO holds equity Financial autonomy (own current revenues / endowment)

Total revenues per FTE member of KTO staff

 Cost per job created with the support of a KTO Public financial contribution per job created Average number of start-ups per 100K€ of KTO

Table 4. Key performance indicators of KTOs' financial and economic returns

Laboratories of extra-regional or foreign firms installed by the KTO

Number of jobs created in companies attributed to KTO

Table 2. Key Performance Indicators of KTOs' knowledge transfer output Sources: Area SciencePark, 2011; Arundel & Bordoy, 2010; Bigliardi et al., 2006; EBN, 2011; European Commission, 2009; Piccaluga et al., 2011.

where all research is performed by the KTO

during the year by established labs

research

Entrepreneurship promotion

Enterprise creation

products/processes

their relative impact factor

IP exploitation Total number of licenses executed during the year

agreement with the KTO

agreement with the KTO

entrepreneurship

entrepreneurship

during the year

incubated firms

Number of contract research agreements stipulated during the year

 Number of consultancy agreements stipulated during the year where the KTO provides expert advice without performing new

Number and type of pre-competitive research programs initiated

 Number and type of conventions stipulated during the year with established labs aiming at the concurrent development of

Number of scientific publications published during the year, with

Number of patent researches carried out during the year

Number of active spin-off companies with a formal license

Number of active spin-off companies with a formal equity

Number of events organized during the year to promote

Number of training events/number of training hours organized

Number and type of new product prototypes launched by

 Enterprise Survival Rate (within the incubation period) Enterprise Survival Rate (3 years following graduation) Total numbers of contacts for enterprise creation Number of feasibility studies created during the year Number of enterprise creation projects during the year Number of business plans produced during the year

Number of people that attended events to promote

Number of people that attended training events

Average incubator space occupancy rate (%)

 Number of start-ups created during the year Number of jobs created in start-ups in the year

Sources: Area SciencePark, 2011; Arundel & Bordoy, 2010; Bigliardi et al., 2006; EBN, 2011;

 Average incubation time (years) Number of tenants in incubators Total employment by tenants

Table 2. Key Performance Indicators of KTOs' knowledge transfer output

European Commission, 2009; Piccaluga et al., 2011.

 Number of spin-offs established during the year Active spin-off companies' average during the year Enterprise Survival Rate of spin-off companies

Table 3. Key Performance Indicators of KTOs' impact Sources: Bigliardi et al., 2006; EBN, 2011; European Commission, 2009.

Table 4. Key performance indicators of KTOs' financial and economic returns Sources: Area SciencePark, 2011; Arundel & Bordoy, 2010; Bigliardi et al., 2006; EBN, 2011; European Commission, 2009; Piccaluga et al., 2011.

Performance Evaluation for Knowledge Transfer Organizations:

*IP assets*

**Key Performance Indicators**

Capital *Size, breakdown and level of education of staff*

transfer activities Number of permanent staff Number of temporary staff Number of contract staff

Number of graduates

*Competences development*  Number of training days

*Retention*

Staff turnover

*Receiver satisfaction*

initiatives

European Commission, 2009; Piccaluga et al., 2011.

**Area of evaluation**

Human

Relational

Capital *Partners*

Best European Practices and a Conceptual Framework 141

Number of invention disclosures received during the year

Number of priority patent applications submitted in the year

 Total number of KTO staff in full-time equivalents (FTEs) – including all professional, administrative and support staff for knowledge

Total number of priority patent applications

Number of patents granted in the year

Number of professional staff (FTEs)

Number of post-graduate degrees and doctorates

Employee satisfaction (work environment), evaluation scale 1-5

 Number of new KTO firms-receivers acquired during the year Customer satisfaction: positive comments from participants in KT

Sources: Area SciencePark, 2011; Arundel & Bordoy, 2010; Bigliardi et al., 2006; EBN, 2011;

Number of new partnership established during the year

Number of participants in training courses

 Total staff expenses for training Total staff expenses - HR development

Average seniority of staff

 Number of foreign partners Number of R&D partners Number of institutional partners

Number of KTO firms-receivers

Table 5. Key performance indicators of KTOs' intellectual capital

Number of complaints from receivers

Number of partners

A second area of result concerns the evaluation of the success of the KT process, therefore its impact on the firms-receivers' performance and on the economic growth of the region (table 3). It is worth bearing in mind that these effects require a long period of time to emerge, and they are mediated by the innovation activities carried out by each receiver and by the different actors operating in the local innovation system.

The last area of result encompasses the financial and the economic performance attained by the KTO (Table 4). The related KPIs capture the ability of the KTO to generate financial returns from the supply of its services and to attain efficiency.

#### **3.3 Evaluation of inputs**

The measurement approach based on input relies on the Intellectual Capital model (Edvinsoon, 1997), often used by KTOs in their annual disclosure activities. Table 5 shows the Intellectual Capital components and the related KPIs tailored for the KTOs:

A second area of result concerns the evaluation of the success of the KT process, therefore its impact on the firms-receivers' performance and on the economic growth of the region (table 3). It is worth bearing in mind that these effects require a long period of time to emerge, and they are mediated by the innovation activities carried out by each receiver and

The last area of result encompasses the financial and the economic performance attained by the KTO (Table 4). The related KPIs capture the ability of the KTO to generate financial

The measurement approach based on input relies on the Intellectual Capital model (Edvinsoon, 1997), often used by KTOs in their annual disclosure activities. Table 5 shows

> Number of research laboratories hosted by the KTOs Number of high-tech firms hosted by the KTOs

Average square meters available for research activities

Value of fixed assets (plant, machinery, etc.)

management costs, contract costs, etc.)

Total expenditures on infrastructure

Average square meters available for incubation activities of owned

Total expenditures on technology transfer activities (patent portfolio

Total expenditures on basic and applied research (staff dedicated to

research, capital expenditures on new equipment, etc.)

the Intellectual Capital components and the related KPIs tailored for the KTOs:

which represents the potential commercialization of public science.

by the different actors operating in the local innovation system.

returns from the supply of its services and to attain efficiency.

firms and universities (Comacchio et al., 2011);

policy makers) and with the firms-receivers.

incubators

*R&D assets*

**Key Performance Indicators**

Number of years since establishment

Total square meters available

**3.3 Evaluation of inputs**

**Area of evaluation**

Structural Capital

Table 5. Key performance indicators of KTOs' intellectual capital

Sources: Area SciencePark, 2011; Arundel & Bordoy, 2010; Bigliardi et al., 2006; EBN, 2011; European Commission, 2009; Piccaluga et al., 2011.

Performance Evaluation for Knowledge Transfer Organizations:

comparisons among different types of KTOs.

coherent model different indicators of KTO performance.

**4.1 The balanced scorecard approach**

Fig. 2. The Balanced Scorecard Framework

**approach**

firm's strategy.

Best European Practices and a Conceptual Framework 143

In the following section we will discuss how the KTO performance management system could be improved by providing a preliminary comprehensive frame that draws on the Balanced Scorecard (BSC) approach, in order to bring within an overall integrated and

**4. An analytical model for the measurement of KTOs: A balanced scorecard**

Among the seminal performance measurement models, the Kaplan-Norton Balanced Scorecard continues to be the most referenced framework in the literature (Kaplan & Norton, 1992; 2004; 2007; Kaplan et al., 2010). This is due to its ability to support firms' strategic learning, namely "gathering feedback, testing the hypotheses on which strategy was based, and making the necessary adjustments" (Kaplan & Norton, 2007: 159). Indeed, the BSC helps firms to articulate their vision and to define in clear and operational terms the results that they aim to achieve. Moreover, the BSC is a comprehensive frame that enables companies to get a feedback system thanks to the specification of the causal relationships between performance drivers and objectives: "Companies build their strategy maps from the top down, starting with their long-term financial goals and then determining the value proposition that will deliver the revenue growth specified in those goals, identifying the processes most critical to creating and delivering that value proposition, and, finally, determining the human, information, and organization capital the processes require." (Kaplan & Norton, 2004: 55). Figure 2 provides a representation of the four perspectives (financial, customer, internal process, learning and growth) and the relationships with a

The BSC framework has been widely applied in many manufacturing and service industries, but only recently has been extended to the research setting (Bremser & Barsky, 2004; Eilat et

performance, but could also help to find an overall accurate framework for worthy

KTOs which have a structured, multidisciplinary, and international network of partners are able to obtain critical information more rapidly, to involve complementary partners in joint research programs and to more easily access the circuit of financed innovation projects (Adler & Kwon, 2002; Nahapiet & Ghoshal, 1998). Moreover, the activation of a network of external qualified relations increases the identity and the reputation of the KTO within the innovation system.

The relational capital also captures the customer relationships developed by the KTOs. In this regard, metrics rely on follow-up procedures or satisfaction surveys.

#### **3.4 Summary**

This section provided a review of the recent debate on the difficulties and opportunities of measuring KTOs' performance. Furthermore, the in-depth analysis carried out has shown the state-of-the-art of metrics and indicators used by KTOs and institutions to evaluate knowledge transfer benefits. This analysis has shed light on the fragmentation of practices implemented by KTOs, in terms of aims, areas of results and metrics.

According to a recent article, which provides a broad analysis of methods for quantifying and qualifying the benefits of KT around the world, four key issues have to be taken into account: inputs vs outputs, quality vs quantity, subjectivity vs objectivity and time series vs cross-sectional analysis (Gardner et al., 2010)*.* We would contribute to this discussion, providing new implications for each issue, on the base of our in-depth analysis of evaluation practices and related indicators in Europe.

performance, but could also help to find an overall accurate framework for worthy comparisons among different types of KTOs.

In the following section we will discuss how the KTO performance management system could be improved by providing a preliminary comprehensive frame that draws on the Balanced Scorecard (BSC) approach, in order to bring within an overall integrated and coherent model different indicators of KTO performance.

#### **4. An analytical model for the measurement of KTOs: A balanced scorecard approach**

#### **4.1 The balanced scorecard approach**

142 Management of Technological Innovation in Developing and Developed Countries

KTOs which have a structured, multidisciplinary, and international network of partners are able to obtain critical information more rapidly, to involve complementary partners in joint research programs and to more easily access the circuit of financed innovation projects (Adler & Kwon, 2002; Nahapiet & Ghoshal, 1998). Moreover, the activation of a network of external qualified relations increases the identity and the reputation of the KTO within the

The relational capital also captures the customer relationships developed by the KTOs. In

This section provided a review of the recent debate on the difficulties and opportunities of measuring KTOs' performance. Furthermore, the in-depth analysis carried out has shown the state-of-the-art of metrics and indicators used by KTOs and institutions to evaluate knowledge transfer benefits. This analysis has shed light on the fragmentation of practices

According to a recent article, which provides a broad analysis of methods for quantifying and qualifying the benefits of KT around the world, four key issues have to be taken into account: inputs vs outputs, quality vs quantity, subjectivity vs objectivity and time series vs cross-sectional analysis (Gardner et al., 2010)*.* We would contribute to this discussion, providing new implications for each issue, on the base of our in-depth analysis of evaluation

this regard, metrics rely on follow-up procedures or satisfaction surveys.

implemented by KTOs, in terms of aims, areas of results and metrics.

practices and related indicators in Europe.

within the same measurement system.

overwhelm qualitative ones.

provider.

innovation system.

**3.4 Summary**

Among the seminal performance measurement models, the Kaplan-Norton Balanced Scorecard continues to be the most referenced framework in the literature (Kaplan & Norton, 1992; 2004; 2007; Kaplan et al., 2010). This is due to its ability to support firms' strategic learning, namely "gathering feedback, testing the hypotheses on which strategy was based, and making the necessary adjustments" (Kaplan & Norton, 2007: 159). Indeed, the BSC helps firms to articulate their vision and to define in clear and operational terms the results that they aim to achieve. Moreover, the BSC is a comprehensive frame that enables companies to get a feedback system thanks to the specification of the causal relationships between performance drivers and objectives: "Companies build their strategy maps from the top down, starting with their long-term financial goals and then determining the value proposition that will deliver the revenue growth specified in those goals, identifying the processes most critical to creating and delivering that value proposition, and, finally, determining the human, information, and organization capital the processes require." (Kaplan & Norton, 2004: 55). Figure 2 provides a representation of the four perspectives (financial, customer, internal process, learning and growth) and the relationships with a firm's strategy.

Fig. 2. The Balanced Scorecard Framework

The BSC framework has been widely applied in many manufacturing and service industries, but only recently has been extended to the research setting (Bremser & Barsky, 2004; Eilat et

Performance Evaluation for Knowledge Transfer Organizations:

KTOs in the region.

network namely degree centrality.

Best European Practices and a Conceptual Framework 145

The *Financial Perspective* describes the tangible outcomes of the strategy in financial terms, such as profitability, revenue growth and cost efficiency. It can be conceived as the final result of the cause-effect relationship among the other three dimensions of the scorecard (learning and growth performances affects internal business processes, which impact on customer outcomes, which finally drive financial results). This perspective applied to KTOs' performance measurement encompasses metrics related to the capability of the organization to gather external resources through its knowledge transfer services. It regards both the short-term profitability generated by commercialized services, the productivity and the efficiency of business processes as well as long-term financial sustainability of research and intermediation activities. Examples of financial performance indicators are the following: total turn-over from services, amount of public funded projects, ROA, costs. Moreover, within this perspective the impact of KTOs on a local innovation system is considered and measured by economic indicators, for instance personnel and investment flows attracted by

The *Customer Perspective* looks at customers' satisfaction with the product or services delivered by a firm and their loyalty. We suggest that as far as a KTO is concerned, customer perspective could be applied to acquisition of clients such as firms and to their retention. However, considering the brokerage role that KTOs play across firms and research institutes, this perspective could be extended to other clients such as Universities or private research bodies. Moreover, the relationships between a KTO and receivers could be characterised by different degrees of interdependences and duration. This dimension could be re-defined as *Receivers Perspective* in order to measure the performance of the overall receivers network. Beside the well known indicators of customer satisfaction and retention, other indicators are those that measure the network size, namely number of clients or agreements and network dynamics, such as partners development rate. Finally, indicators of social network analysis could be applied to assess the structural features of a KTO's ego

The *Internal Perspective* regards the key internal processes that make up the value proposition to customers, thus those processes by which a firm is able to meet customers' needs. From a knowledge transfer point of view this dimension should consider client's requirements that are tightly related to the phases of the innovation value chain. Consequently, we suggest to reconceptualise this dimension as *Innovation Value Chain perspective*. The measures it includes assess whether a KTO is able to act as an external competent partner along the three main phases: knowledge generation, transformation and exploitation. Indicators for the knowledge generation phase are related to three main areas of results: i) basic research (number of scientific publications per year, number of patent researches carried out during the year); ii) technology scouting (number of technological audit provided); iii) brokerage (number of events organised to promote networking among innovators). Indicators of results in the second phase of knowledge transformation are the following: type of conventions stipulated during the year aiming at the concurrent development of products/processes and number of application for patent; number of research and development projects; number of training events organized; number of prototyping and testing services provided; etc. Indicators of results in the third phase of knowledge exploitation are: number of application for patent; number of business plans;

al., 2008; Sartorius et al., 2010; Wang et al., 2010). Drawing on the KT and BSC literature, we provide a preliminary contribution to this debate re-conceptualising the Balanced Scorecard coherently to the KTO context.

#### **4.2 The balanced scorecard of KT**

#### **4.2.1 Introduction**

The BSC strategy map (Fig.3) provides a framework for linking intangible assets to shareholder value creation through four interrelated perspectives: financial, customer, internal process, learning and growth. We suggest that this scheme is a coherent architecture by which the fragmented evaluation system of KTOs could be unified. Moreover, this framework provides a strategic way of thinking about performance assessment of KT services, useful for providers, receivers and policy-makers. Indeed a well-developed and comprehensive performance measurement systems could reduce the barriers that hamper the relationship between demand and supply of services (Polt et al., 2001).

Fig. 3. The Balanced Scorecard Framework for KTO

#### **4.2.2 The managerial perspectives of BSC applied to KTOs**

The four managerial perspectives of the BSC provide KTOs' management with a comprehensive tool to measure performances along different but integrated dimensions. We propose that each dimension of the BSC could be adapted to the specific activities pursued by KT bodies. Accordingly in the following discussion each dimension of the original BSC frame is presented and is adjusted in order to be applied to the context of the market of knowledge transfer services. We finalize the discussion of each dimension by giving some examples of key indicators that can be used, coherently with metrics of KTO performance management, presented in Section 3.

al., 2008; Sartorius et al., 2010; Wang et al., 2010). Drawing on the KT and BSC literature, we provide a preliminary contribution to this debate re-conceptualising the Balanced Scorecard

The BSC strategy map (Fig.3) provides a framework for linking intangible assets to shareholder value creation through four interrelated perspectives: financial, customer, internal process, learning and growth. We suggest that this scheme is a coherent architecture by which the fragmented evaluation system of KTOs could be unified. Moreover, this framework provides a strategic way of thinking about performance assessment of KT services, useful for providers, receivers and policy-makers. Indeed a well-developed and comprehensive performance measurement systems could reduce the barriers that hamper

the relationship between demand and supply of services (Polt et al., 2001).

Fig. 3. The Balanced Scorecard Framework for KTO

management, presented in Section 3.

**4.2.2 The managerial perspectives of BSC applied to KTOs**

The four managerial perspectives of the BSC provide KTOs' management with a comprehensive tool to measure performances along different but integrated dimensions. We propose that each dimension of the BSC could be adapted to the specific activities pursued by KT bodies. Accordingly in the following discussion each dimension of the original BSC frame is presented and is adjusted in order to be applied to the context of the market of knowledge transfer services. We finalize the discussion of each dimension by giving some examples of key indicators that can be used, coherently with metrics of KTO performance

coherently to the KTO context.

**4.2.1 Introduction**

**4.2 The balanced scorecard of KT**

The *Financial Perspective* describes the tangible outcomes of the strategy in financial terms, such as profitability, revenue growth and cost efficiency. It can be conceived as the final result of the cause-effect relationship among the other three dimensions of the scorecard (learning and growth performances affects internal business processes, which impact on customer outcomes, which finally drive financial results). This perspective applied to KTOs' performance measurement encompasses metrics related to the capability of the organization to gather external resources through its knowledge transfer services. It regards both the short-term profitability generated by commercialized services, the productivity and the efficiency of business processes as well as long-term financial sustainability of research and intermediation activities. Examples of financial performance indicators are the following: total turn-over from services, amount of public funded projects, ROA, costs. Moreover, within this perspective the impact of KTOs on a local innovation system is considered and measured by economic indicators, for instance personnel and investment flows attracted by KTOs in the region.

The *Customer Perspective* looks at customers' satisfaction with the product or services delivered by a firm and their loyalty. We suggest that as far as a KTO is concerned, customer perspective could be applied to acquisition of clients such as firms and to their retention. However, considering the brokerage role that KTOs play across firms and research institutes, this perspective could be extended to other clients such as Universities or private research bodies. Moreover, the relationships between a KTO and receivers could be characterised by different degrees of interdependences and duration. This dimension could be re-defined as *Receivers Perspective* in order to measure the performance of the overall receivers network. Beside the well known indicators of customer satisfaction and retention, other indicators are those that measure the network size, namely number of clients or agreements and network dynamics, such as partners development rate. Finally, indicators of social network analysis could be applied to assess the structural features of a KTO's ego network namely degree centrality.

The *Internal Perspective* regards the key internal processes that make up the value proposition to customers, thus those processes by which a firm is able to meet customers' needs. From a knowledge transfer point of view this dimension should consider client's requirements that are tightly related to the phases of the innovation value chain. Consequently, we suggest to reconceptualise this dimension as *Innovation Value Chain perspective*. The measures it includes assess whether a KTO is able to act as an external competent partner along the three main phases: knowledge generation, transformation and exploitation. Indicators for the knowledge generation phase are related to three main areas of results: i) basic research (number of scientific publications per year, number of patent researches carried out during the year); ii) technology scouting (number of technological audit provided); iii) brokerage (number of events organised to promote networking among innovators). Indicators of results in the second phase of knowledge transformation are the following: type of conventions stipulated during the year aiming at the concurrent development of products/processes and number of application for patent; number of research and development projects; number of training events organized; number of prototyping and testing services provided; etc. Indicators of results in the third phase of knowledge exploitation are: number of application for patent; number of business plans;

Performance Evaluation for Knowledge Transfer Organizations:

**Area of results**

Human capital

Structural capital

Network size and quality

Network dynamics

Table 6. The Balanced Scorecard of KTOs

the year; number of tenants in incubators, etc.

**BSC**

**perspectives**

Human and Structural Capital Perspective

Partner and Network Perspective

Best European Practices and a Conceptual Framework 147

**Sample metrics**

the year

Exploitation Active spin-off companies' average during

equivalents (FTEs)

*Competences development*  Number of training days

satisfaction

activities

research

during the year

Number of partners

during the year

over time

number of spin-offs established during the year; active spin-off companies' average during

The *Learning and Growth Perspective* is the dimension that measures the intangible assets of an organization. As argued by Kaplan and Norton (2004: 54) "the objectives in this perspective identify which jobs (the human capital), which systems (the information capital), and what kind of climate (the organization capital) are required to support the value-creating internal processes". The same dimensions could be applied to KTO in which

*Retention*

*R&D assets*

*IP assets*

degrees and doctorates

Number of tenants in incubators

*Size, breakdown and level of education of staff*  Total number of KTO staff in full-time

Number of graduates and post-graduate

Number of participants in training courses

Staff turnover, average seniority of staff or

 Total expenditures on infrastructure Total expenditures on technology transfer

Total expenditures on basic and applied

Number of invention disclosures received

Number of patents granted in the year

Number of new partnership established

Skill and capabilities provided by partners.

Composition of partners and relationships

Number of external consultants;

**Sample metrics**

its trend

 Number of clients Number of agreements

KTO degree centrality

 Customer retention rate New clients development rate

Customer satisfaction

Number of complaints from customers

publications per year, number of patent researches carried out during the year)

 Brokerage (number of events organised to promote networking among innovators)

 Number and type of conventions stipulated during the year aiming at the concurrent development of products/processes Number of research and development project

Number of spin-offs established during the

 Number of training events organized Number of prototyping and testing services

 Number of application for patent Number of business plans

provided

year

Basic research (number of scientific

 Technology scouting (number of technological audit provided)

 Total turnover for services and its trend; Total license revenue and its trend

Financial autonomy; amount of public

Total revenues per KTO employee; ROA and

 Personnel flows installed by KTO from other regions or from foreign countries

founded research projects

**Area of results**

Profitability of KT services

Financial sustainability

Productivity

Network size

Network structure

Network dynamics

Knowledge Generation

Knowledge Transformation

Knowledge

on local innovation system

Economic impact

**BSC**

**perspectives**

Financial Perspective

Receivers Perspective

Innovation Value Chain Perspective

Table 6. The Balanced Scorecard of KTOs

number of spin-offs established during the year; active spin-off companies' average during the year; number of tenants in incubators, etc.

The *Learning and Growth Perspective* is the dimension that measures the intangible assets of an organization. As argued by Kaplan and Norton (2004: 54) "the objectives in this perspective identify which jobs (the human capital), which systems (the information capital), and what kind of climate (the organization capital) are required to support the value-creating internal processes". The same dimensions could be applied to KTO in which

Performance Evaluation for Knowledge Transfer Organizations:

much this framework is implemented by KTOs.

stakeholders expectations.

*2008*, UNU-MERIT.

489-505.

**6. References**

manage resource allocation and support regional development.

*Management Review*, Vol.27, No.1, pp. 17–40.,

Area SciencePark (2011), *Intellectual capital report 2009*. Trieste.

*Industrial & Corporate Change*, Vol.19, No.3, pp. 775-803.

*Technology Management*, Vol.10, No.7/8, pp. 643-664.

technology transfer, *Research Policy*, Vol.24, pp.97–114.

*Management Review*, Vol.50, N0.4, pp. 69-76.

*Science*, Vol.15, No.5, pp.555–568.

Best European Practices and a Conceptual Framework 149

Third, this comprehensive scheme could become a monitoring tool for policy makers willing to make cross-sectional comparisons of providers based on a univocal method, in order to

In terms of future lines of research, we suggest that from a theoretical point of view the model could benefit from further conceptual development in two directions: by a more indepth analysis of each perspective and related metrics and by an analysis of the causal relationships among perspectives within a KT organization. From an empirical point of view a field work by case studies could be a first test of the theoretical model to understand how

We also draw some preliminary managerial implications for KTO, considering how this framework could be applied by a KTO, for instance what information is easily available to implement the framework and how much detailed the frame should be to meet the different

Adler, P.S. & Kwon, S.W. (2002). Social capital: Prospects for a new concept, *Academy of*

Amesse, F. & Cohendet, P. (2001). Technology transfer revisited from the perspective of the

Arora, A. & Gambardella, A. (2010). Ideas for rent: an overview of markets for technology,

Arora, A., Fosfuri, A. & Gambardella, A. (2001). *Markets for technology: The economics of*

Arundel, A & Bordoy, C. (2010), Summary *Respondent Report: ASTP Survey for Fiscal Year*

Autio, E. & Laamanen, T. (1995). Measurement and evaluation of knowledge transfer:

Balconi, M. & Laboranti, M. (2006). University–industry interactions in applied research: The

Bessant, J. & Rush, H. (1995). Building bridges for innovation: The role of consultants in

Bigliardi, B. Dormio, A.I., Nosella, A. & Petroni, G. (2006). Assessing science parks'

Boudreau, K.J. & Lakhani, K.R. (2009). How to manage outside innovation, MITSloan

Bremser, W.G. & Barsky N.P. (2004). Utilising the balanced scorecard for R&D performance

Breschi, S., Lissoni, F. & Montobbio, F. (2005). The geography of knowledge spillovers:

Review of knowledge transfer mechanism and indicators, *International Journal of*

performances: Direction from selected italian case studies, *Technovation,* Vol.26, pp.

Conceptual issues and measurement problems. In S. Breschi & F. Malerba (Eds.), *Clusters, networks and innovation* (pp. 343–378). Oxford: Oxford University Press. Carlile, P.R. (2004). Organization science transferring, translating, and transforming: An

integrative framework for managing knowledge across boundaries, *Organization*

knowledge-based economy, *Research Policy*, Vol.30, pp.1459–1478.

*innovation and corporate strategy*, The MIT Press, Cambridge MA.

case of microelectronics, *Research Policy*, Vol.35, pp.1616–1630.

measurement, *R&D Management*, Vol. 34, No.3, pp. 229-238.

research activity relies both on human competencies and prior R&D experience, thus we suggest to name this aspect as *Human and Structural Capital Perspective*. Coherently, indicators are those traditionally used to measure KTO intangible input and already presented in Section 3, a sample of which is: indicators of *size, breakdown and level of education of staff (*total number of KTO staff in full-time equivalents (FTEs), number of graduates and post-graduate degrees and doctorates)*,* indicators of *competences development (*number of training days or number of participants in training courses) and finally indicators of *retention and satisfaction (*staff turnover, average seniority of staff or satisfaction). Concerning *structural capital* some indicators are those of *R&D assets (*total expenditures on infrastructure*/*on technology transfer activities /on basic and applied research) and of *IP assets (*number of invention disclosures received during the year*,* number of patents granted in the year).

In addition to human and structural capital a further enabler of the competitive advantage of KTOs is the network of external contractors and organizational partners by which a KTO provides its services, such as consultants, researchers. Thus, accordingly with the most recent contributions on BCS in R&D settings (Lazzarotti et al., 2011) we enrich the original BCS map with an additional dimensions of KTO performance measurement: *Partner and Network Perspective.* Indicators are the following: number of partners, number of new partnership established during the year and number of external consultants; skills and capabilities provided by partners (network size and quality) and composition of partners and relationships over time (network dynamics).

Table 6 shows the BSC framework tailored for KTOs setting, illustrating the five complementary perspectives and the related area of results and indicators. The fragmented measurement systems presented in the Section 3 are therefore composed in a comprehensive framework.

#### **5. Conclusion**

In this chapter, after having illustrated the main drivers of the evolution of the market of knowledge transfer services, we maintained the relevance of the evaluation of KTO performances, notwithstanding its intrinsic difficulties. Grounding our discussion on an indepth analysis of key indicators used in Europe to measure knowledge transfer performances, we highlighted the limits of the extant evaluation systems, specifically we argued that they are fragmented and disentangled from the strategy of KTOs. In line with these critiques we maintained that a performance measurement system based on an integrated frame such as the BSC has several advantages and positive effects.

First, as an assessment tool, it helps to overcome the main barriers between the demand and supply of KT services. From the point of view of firms-receivers, a BSC might increase transparency, lower information asymmetries reducing transaction costs (Polt et al., 2001).

Second, as a managerial tool, the BSC helps KTO to craft their strategy in terms of key services offered, targeted receivers served, core competences to mobilize internally and externally in order to achieve financial and economic returns (Bremser & Barsky, 2004). Moreover the description of the multiple strategic goals a KTO wants to accomplish through the five BSC perspectives might simplify the selection of the most appropriate measures (Kaplan, 2010).

Third, this comprehensive scheme could become a monitoring tool for policy makers willing to make cross-sectional comparisons of providers based on a univocal method, in order to manage resource allocation and support regional development.

In terms of future lines of research, we suggest that from a theoretical point of view the model could benefit from further conceptual development in two directions: by a more indepth analysis of each perspective and related metrics and by an analysis of the causal relationships among perspectives within a KT organization. From an empirical point of view a field work by case studies could be a first test of the theoretical model to understand how much this framework is implemented by KTOs.

We also draw some preliminary managerial implications for KTO, considering how this framework could be applied by a KTO, for instance what information is easily available to implement the framework and how much detailed the frame should be to meet the different stakeholders expectations.

#### **6. References**

148 Management of Technological Innovation in Developing and Developed Countries

research activity relies both on human competencies and prior R&D experience, thus we suggest to name this aspect as *Human and Structural Capital Perspective*. Coherently, indicators are those traditionally used to measure KTO intangible input and already presented in Section 3, a sample of which is: indicators of *size, breakdown and level of education of staff (*total number of KTO staff in full-time equivalents (FTEs), number of graduates and post-graduate degrees and doctorates)*,* indicators of *competences development (*number of training days or number of participants in training courses) and finally indicators of *retention and satisfaction (*staff turnover, average seniority of staff or satisfaction). Concerning *structural capital* some indicators are those of *R&D assets (*total expenditures on infrastructure*/*on technology transfer activities /on basic and applied research) and of *IP assets (*number of invention disclosures received during the year*,* number of patents granted

In addition to human and structural capital a further enabler of the competitive advantage of KTOs is the network of external contractors and organizational partners by which a KTO provides its services, such as consultants, researchers. Thus, accordingly with the most recent contributions on BCS in R&D settings (Lazzarotti et al., 2011) we enrich the original BCS map with an additional dimensions of KTO performance measurement: *Partner and Network Perspective.* Indicators are the following: number of partners, number of new partnership established during the year and number of external consultants; skills and capabilities provided by partners (network size and quality) and composition of partners

Table 6 shows the BSC framework tailored for KTOs setting, illustrating the five complementary perspectives and the related area of results and indicators. The fragmented measurement systems presented in the Section 3 are therefore composed in a comprehensive

In this chapter, after having illustrated the main drivers of the evolution of the market of knowledge transfer services, we maintained the relevance of the evaluation of KTO performances, notwithstanding its intrinsic difficulties. Grounding our discussion on an indepth analysis of key indicators used in Europe to measure knowledge transfer performances, we highlighted the limits of the extant evaluation systems, specifically we argued that they are fragmented and disentangled from the strategy of KTOs. In line with these critiques we maintained that a performance measurement system based on an

First, as an assessment tool, it helps to overcome the main barriers between the demand and supply of KT services. From the point of view of firms-receivers, a BSC might increase transparency, lower information asymmetries reducing transaction costs (Polt et al., 2001). Second, as a managerial tool, the BSC helps KTO to craft their strategy in terms of key services offered, targeted receivers served, core competences to mobilize internally and externally in order to achieve financial and economic returns (Bremser & Barsky, 2004). Moreover the description of the multiple strategic goals a KTO wants to accomplish through the five BSC perspectives might simplify the selection of the most appropriate measures

integrated frame such as the BSC has several advantages and positive effects.

in the year).

framework.

**5. Conclusion**

(Kaplan, 2010).

and relationships over time (network dynamics).

Performance Evaluation for Knowledge Transfer Organizations:

Vol.85, No.6, pp. 121-130.

*Policy*, Vol.35, pp. 715–728.

*School Working Paper*, 10-074

Vol.33, No.4, pp. 319-332.

*Policy*, Vol.37, pp. 1224–1240.

*transfer*, Vol.28, pp. 81–85.

*Organization*, Vol.25, pp. 483–510.

No.4, pp. 333-342.

pp. 1501–1516.

pp. 495–505.

Best European Practices and a Conceptual Framework 151

Hansen, M. & Birkinshaw, J. (2007). The innovation value chain, *Harvard Business Review*,

Hogan, B. (1996). Evaluation of science and technology parks*,* In: *The Science Park Evaluation Handbook*, European Innovation Monitoring System (EIMS) publication n. 61. Howells, J. (2006). Intermediation and the role of intermediaries in innovation, *Research*

Johnson, W.H.A. (2008). Roles, resources and benefits of intermediate organizations

Kaplan, R.S. (2010). Conceptual foundations of the balanced scorecard, *Harvard Business*

Kaplan, R.S. & Norton, D.P. (1992). The balanced scorecard-measures that drive performance, *Harvard Business Review*, Vol.70, No.1, January-February, pp. 71-79. Kaplan, R.S. & Norton, D.P. (2004). Measuring the strategic readiness of intangible assets,

Kaplan, R.S. & Norton, D.P. (2007). Using the balanced scorecard as a strategic management system, *Harvard Business Review*, July-August, Vol.85, No.7/8, pp. 150-161. Kaplan, R.S., Norton, D.P. & Rugelsjoen, B. (2010). Managing alliances with the Balanced Scorecard, *Harvard Business Review*, Vol.88, No.1/2, January-February, pp. 114-120. Keeble, D., Lawson, C., Moore B. & Wilkinson F. (1999). Collective learning processes,

Kodama, T. (2008). The role of intermediation and absorptive capacity in facilitating

Laranja, M. (2009). The development of technology infrastructure in Portugal and the need to

Lazzarotti, V., Manzini, R. & Mari L. (2011). A model for R&D performance measurement,

Lee, S., Park, G., Yoon, B. & Park, J (2010). Open innovation in SMEs—An intermediated

Lichtenthaler, U. & Ernst, H. (2007). Developing reputation to overcome the imperfections in

Link, A.N. & Link, K.R. (2003). On the growth of U.S. science parks, *Journal of Knowledge*

Longhi, C. (1999). Networks, collective learning and technology development in innovative

Macho-Stadler, I., Pérez-Castrillo, D. & Veugelers, R. (2007). Licensing of university

McEvily, B. & Zaheer, A. (1999). Bridging the ties: A source of firms heterogeneity in competitive capabilities, *Strategic Management Journal*, Vol.20, pp.1133–1156. Mendonça, S., (2009). Brave old world: Accounting for 'high-tech' knowledge in 'low-tech'

Muller, E. & Zenker, A. (2001). Business services as actors of knowledge transformation: the

high technology regions: The case of Sophia-Antipolis, *Regional Studies*, Vol.33,

inventions: The role of a technology transfer office, *International Journal of Industrial*

role of KIBS in regional and national innovation systems, *Research Policy*, Vol.30,

*International journal of production economics*, Vol.134, pp. 212-223

the markets for knowledge, *Research Policy*, Vol.36, pp. 37–55.

network model, *Research Policy*, Vol.39, pp. 290–300.

industries., *Research Policy*, Vol.38, No.3, pp. 470-482.

*Harvard Business Review*, January-February, Vol.82, No.2pp. 52-63.

supporting triple helix collaborative R&D: The case of Precarn, *Technovation*, Vol.28,

networking and 'institutional thickness' in the Cambridge Region, *Regional Studies*,

university-industry linkages - An empirical study of TAMA in Japan, *Research*

pull innovation using proactive intermediation policies, *Technovation*, Vol.29, pp. 23–34.

Chesbrough, H.W. (2003). O*pen innovation: The new imperative for creating and profiting from*

Comacchio, A., Bonesso, S. & Pizzi C. (2011). Boundary spanning between industry and

Consoli, D. & Elche-Hortelano D. (2010). Variety in the knowledge base of Knowledge

Corley, P., Boardman, C. & Bozeman, B. (2006). Design and the management of multi-

Daim, T.U., Kocaoglu, D.F. & Anderson, T.R. (2009). Knowledge driven planning tools for

Decter, M., Bennett, D. & Leseure, M. (2007). University to business knowledge transfer—

Dosi G., Llerena, P. & Sylos Labini, M. (2006). The relationships between science, technologies

Edvinsson, L. (1997). Developing intellectual capital at Skandia, *Long Range Planning*, Vol.30,

Eilat, H., Golany, B. & Shtub, A. (2008). R&D project evaluation: An integrated DEA and

Etzkowitz, H. & Leydesdorff, L. (2000). The dynamics of innovation: from National Systems

European Commission (2004). *Knowledge transfer institutions in Europe: An overview*, Brussels. European Commission (2009). Metrics for knowledge transfer from public research

Fleming, L. & Waguespack, D. M. (2007). Brokerage, boundary spanning, and leadership in open innovation communities, *Organization Science*, Vol.18, No.2, pp.165–180. Gardner, P.L., Fong, A.Y. & Huang, R.L., (2010). Measuring the impact of knowledge transfer

Gassmann, O., Daiber, M. & Enkel, E. (2011). The role of intermediaries in cross-industry

Geuna, A. & Muscio, A. (2009). The governance of university knowledge transfer: A critical

Gilsing, V., Bekkers, R., Bodas Freitas, I.M. & van der Stehen, M. (2011). Differences in

Transfer mechanisms and barriers, *Technovation*, Vol.31, No.12, pp.638-647. Guy, K. (1996). Designing a science park evaluation, In: *The Science Park Evaluation Handbook*.

Hagardon, A. & Sutton, R. (1997). Technology brokering and innovation in a product development firm, *Administrative Science Quarterly*, Vol.42, pp.716–749.

innovation processes, *R&D Management*, Vol.41, No.5, pp. 457-469.

European Innovation Monitoring System (EIMS) publication n. 61.

review of the literature, *Minerva*, Vol.47, pp.93–114.

the so-called 'European Paradox'*, Research Policy*, Vol.35, pp. 1450–1464.

Intensive Business Services, *Research Policy*, Vol.39, pp. 1303–1310.

UK and USA comparisons, *Technovation*, Vol.27, pp. 145–155.

EBN (2011), *BIC Observatory 2011 (Data 2010)*, European BIC Network.

balanced scorecard approach, *Omega*, Vol.36, pp.895-912.

university: the role of Technology Transfer Centres, *Journal of Technology Transfer*,

institutional research collaborations: Theoretical implications from two case

emerging and converging technologies, *Technological Forecasting & Social Change*,

and their industrial exploitation: An illustration through the myths and realities of

and ''Mode 2'' to a Triple Helix of university–industry–government relations,

organizations in Europe, In: *Report from the European Commission's expert group on knowledge transfer metrics,* European Commission Directorate-General for Research

from public research organisations: a comparison of metrics used around the world, *International Journal of Learning and Intellectual Capital*, Vol.7, No.3/4, pp. 318-327. Gassmann, O. (2006). Editorial. Opening up the innovation process: towards an agenda. In

knowledge transfer between science-based and development-based industries:

*technology*, Harvard Business School Press, Boston.

studies, *Research Policy*, Vol.35, pp.975–993.

DOI 10.1007/s10961-011-9227-6.

Vol.76, pp.1.

No.3, pp. 266-373.

*Research Policy*, Vol.29, pp. 109-123.

Communication Unit, Brussel.

*R&D Management*, Vol.36, pp.223-228.

**8**

*United Kingdom*

**Understanding Innovation Deployment**

*Brunel University, Department of Information Systems and Computing*

This chapter encompasses two objectives. The first objective is to understand different conceptualisation of technology, context and the user according to various theoretical perspectives within information systems (IS) literature and in the field of healthcare/medical informatics; and in addition, outline perspectives that have expounded the nature of relationship between these three entities. The need to explore such notions stems from authors' interest to elicit how these theoretical perspectives and concepts have been applied in the field of healthcare to understand technological innovation deployment and evaluation processes by taking the tacit, ephemeral, and complicated nature of

The second objective involves drawing parallels between these two bodies of literature and conceptualising a framework that can be employed to explain how and why technological innovation deployment and evaluation processes in the field of healthcare cause many

The chapter starts by drawing on well known theories in the field of IS to study deployment processes. These include the social construction of technology (SCOT), actor network theory (ANT), diffusion of innovation, contextual approach and the work of sociologist, Anthony Giddens in structuration theory and high modernity, and its derivatives in the field of IS by authors such as Barley and Orlikowski. By drawing on these perspectives, the conceptual framework is presented in its skeletal form outlining the salience of considering technological innovation, context, and user as three entities interrelated to each other

In the following section of this chapter, theories which provide understanding on deployment and evaluation processes in the field of healthcare are presented. These include theories such as normalisation process theory (NPT), routinisation theory, the structure, process and outcome framework and, concepts such as virtual and invisible work. These theories and concepts provide an insight into the micro-dynamics that occur between the

**1. Introduction**

challenges to arise.

**1.1 Chapter outline**

through recursive relationship.

healthcare work practices into account.

**and Evaluation in Healthcare:**

Urvashi Sharma, Julie Barnett and Malcolm Clarke

**The Triality Framework**

## **Understanding Innovation Deployment and Evaluation in Healthcare: The Triality Framework**

Urvashi Sharma, Julie Barnett and Malcolm Clarke *Brunel University, Department of Information Systems and Computing United Kingdom*

#### **1. Introduction**

152 Management of Technological Innovation in Developing and Developed Countries

Muscio, A. (2010). What drives the university use of knowledge transfer offices? Evidence

Nahapiet, J. & Ghoshal, S. (1998). Social capital, intellectual capital, and the organizational advantage, *Academy of Management Review*, Vol.23, No.2, pp. 242–266. Nambisan, S. & Sawhney, M. (2007). A buyer's guide to the innovation bazaar, *Harvard*

Parvan, S.V. (2007). *Community Innovation Statistics. Weak link between innovative enterprises*

Phan, P.H., Siegel, D.S. & Wright M. (2005). Science parks and incubators: observations, synthesis and future research, *Journal of Business Venturing*, Vol.20, pp. 165–182. Piccaluga, A., Balderi, C., Patrono, A. (2011). *The ProTon Europe. Seventh annual survey report*

Polt, W., Rammer, C., Schartinger, D., Gassler, H. & Schibany, A. (2001). Benchmarking

Reisman, A. (2005). Transfer of technologies: a cross-disciplinary taxonomy, *Omega*, Vol.33,

Roper, S., Dub, J. & Love, J.H. (2008). Modelling the innovation value chain, *Research Policy*,

Samtani, L.A., Mohannak, K. & Hughes, S.W. (2008). Knowledge transfer evaluation in the

Sartorius, K, Trollip, N. & Eitzen, C. (2010). Performance measurement frameworks in a

Tether, B.S. & Tajar, A. (2008). Beyond industry–university links: Sourcing knowledge for

Wang, J., Lin, W., Huang, Y.H. (2010). A performance-oriented risk management framework

Yusuf, S. (2008). Intermediating knowledge exchange between universities and businesses,

for innovative R&D projects, *Technovation*, Vol.30, pp.601-611.

*and public research institutes/universities*, Statistics in focus, Science and Technology,

*(fiscal year 2009)*. Institute of Management Scuola Superiore Sant'Anna (Pisa, Italy),

industry–science relations: the role of framework conditions, *Science and Public*

high technology industry: An interdisciplinary perspective*,* In Goa, J., Jay, L., Jun, N., Lin, M. & Joseph, M., Eds. Proceedings 3rd World Congress on Engineering Asset Management and Intelligent Maintenance Systems Conference (WCEAM-IMS 2008): *Engineering Asset Management – A Foundation for Sustainable Development*,

state controlled research organization: Can the Balanced Scorecard (BSC) be modified? *South African Journal of Business Management*, Vol.41, No.(2), pp.51-63. Spithoven, A., Clarysse, B. & Knockaert, M. (2010). Building absorptive capacity to organise

inbound open innovation in traditional industries, *Technovation*, Vol.30, No.2, pp.

innovation from consultants, private research organisations and the public science-

from Italy, *Journal of Knowledge transfer*, Vol.35, No.2, pp. 181-202.

*Business Review*, Vol.86, No.6, pp.109-118.

81/2007.

ProTon Europe.

pp. 189–202.

130–141.

Vol.37, pp. 961-977.

*Policy*, Vol.28, No.4, pp.247–258.

pp. 1357-1365, Beijing, China.

base, *Research Policy*, Vol.37, pp. 1079–1095.

*Research Policy*, Vol.37, pp. 1167–1174.

OECD (2010). *Measuring innovation. A new perspective*. OECD publishing.

This chapter encompasses two objectives. The first objective is to understand different conceptualisation of technology, context and the user according to various theoretical perspectives within information systems (IS) literature and in the field of healthcare/medical informatics; and in addition, outline perspectives that have expounded the nature of relationship between these three entities. The need to explore such notions stems from authors' interest to elicit how these theoretical perspectives and concepts have been applied in the field of healthcare to understand technological innovation deployment and evaluation processes by taking the tacit, ephemeral, and complicated nature of healthcare work practices into account.

The second objective involves drawing parallels between these two bodies of literature and conceptualising a framework that can be employed to explain how and why technological innovation deployment and evaluation processes in the field of healthcare cause many challenges to arise.

#### **1.1 Chapter outline**

The chapter starts by drawing on well known theories in the field of IS to study deployment processes. These include the social construction of technology (SCOT), actor network theory (ANT), diffusion of innovation, contextual approach and the work of sociologist, Anthony Giddens in structuration theory and high modernity, and its derivatives in the field of IS by authors such as Barley and Orlikowski. By drawing on these perspectives, the conceptual framework is presented in its skeletal form outlining the salience of considering technological innovation, context, and user as three entities interrelated to each other through recursive relationship.

In the following section of this chapter, theories which provide understanding on deployment and evaluation processes in the field of healthcare are presented. These include theories such as normalisation process theory (NPT), routinisation theory, the structure, process and outcome framework and, concepts such as virtual and invisible work. These theories and concepts provide an insight into the micro-dynamics that occur between the

Understanding Innovation Deployment and Evaluation in Healthcare: The Triality Framework 155

address, and closure by redefinition of a problem that sees the use of technology as the solution to an entirely different problem than originally proposed. For example, when the prelimatic tyre was originally introduced to solve the vibration problems in cycles, it was rejected; however, when used instead in racing cycles, it was accepted. This illustrates the

Context according to SCOT includes the social and political dimensions. It is argued to shape the values and norms of the groups. It does not however provide detailed insight into

The user in SCOT, is conceptualised as a member belonging to a particular social group, who despite sharing a common interpretation of technological artefact as a member of the group also has an individual interpretation which can be subjected to the group's scrutiny over time. This notion is constituted within the concept of technology frames, and has been regarded as its contribution. Another contribution of SCOT includes its attention to the

However, despite the vital contributions, SCOT has been argued to have drawbacks that include the lack of acknowledging that the design of an artefact might be embedded with a particular group and their intention, and therefore may be unsuitable for other contexts and groups. In addition it is further argued that SCOT does not consider user opinion and

In the field of IS, SCOT is furthered by Orlikowski and Gash (1994), who introduced the concept of technology frames of reference (TFR) by drawing on Bijker's notion of the technology frame. According to TFR, the perceptions, expectations and experiences of an individual pertaining to a provided technology, constitute their view and attitude about its nature, value and use. Individuals are argued to have separate and different TFRs and these are often created when a change is introduced that causes disturbance to established

In the case where the TFRs of individuals match, they are called to be in congruence such that "*alignment of frames on key elements or categories*" exists, but when there are differences, they lead to "*incongruence of technology frames*" (Orlikowski & Gash, 1994). Significant incongruence would result in technology being abandoned, while congruent technology

By drawing from SCOT and TFR, it can be argued that users of technology constitute perceptions as frames that are influenced by their interaction with their surrounding environment and group members (Davidson, 2006; Davidson, 2002; Davidson, 1997). These frames govern their reaction to any change introduced and if congruent then positive impacts such as use of provided technology can be achieved. On the other hand, if incongruent frames dominate, the use of technology will cease and efforts to deploy or evaluate would be wasted. The notion of frames discussed here and the way in which SCOT views the three entities of context, technology and the user, facilitates to draw holistic definition of each of the three entities in Triality Framework. The framework is presented

The focus now moves to ANT which differs greatly from SCOT in terms of its analytical

closure by redefinition of the problem.

this area and has been criticised for this weakness.

influence of the designer on design during the design process.

extrapolates findings by only following trends (Faulkner, 2009).

routines and work practices (Ciborra & Lanzara, 1994).

later in the chapter.

approach.

frames would achieve use and become embedded in routines.

context, technology and the user, which in this research, are considered to be the attributes of recursive relationships between the three entities. This leads to the final iteration of the framework, called the Triality framework, according to which, the technology is conceptualised as technological innovation, the context is conceptualised as healthcare social system and the user is conceptualised as human agent. It further highlights that each recursive relationship has set of attributes associated with it and there are nine attributes in total.

By drawing on this framework, this chapter aims at eliciting the complexities associated with healthcare context and the dilemmas surrounding healthcare professionals when technological innovations such as telehealth and electronic records are deployed and evaluated.

#### **2. Theoretical perspectives from the field of IS: Understanding conceptualisation of technological content, context and user, and the nature of interrelationship between them**

Exploring theoretical perspectives from IS provides insight into how technological content, context and the user have been conceptualised, and what is the nature of relationship between the three entities. Thus, theoretical perspectives that are of essence are discussed next.

#### **2.1 Social Construction of Technology (SCOT)**

SCOT stems from science and technology studies, which concern themselves with the impact of technology on society and its dynamics. These studies argue that the technology design and its acceptance trajectory are affected by social dynamics within a given society (William & Edge, 1996). SCOT therefore, strives to explain why certain technological artefacts or "*variants*" rise in society at given point in time whereas the others "*die*" (Pinch & Bijker, 1984). This view was proposed in response to technology determinism that establishes an opposite perspective that technology impacts the surroundings and its users (William & Edge, 1996). SCOT presents specific conceptualisation of technology, context and the user. These are discussed below.

Technology according to SCOT is conceptualised as an artefact of which the design and use is the subject of various interpretations resulting due to enabling and constraining properties of the technology as experienced by the groups under a given social context (Avgerou, 2002; Bijker, 1995; Klein & Klienman, 2002; Pinch & Bijker, 1984; Jackson et al., 2002). This constitutes the concept of interpretive flexibility, which also ascertains that despite different interpretations of an artefact within a society, these interpretations can coexist together as they are dependent on desired outcomes.

In addition, the degree to which a technological artefact is accepted within a society is defined by the concept of closure and stabilisation. Closure is achieved when "*all groups' problems have been addressed and groups achieve consensus on a particular design*" (Klein & Klienman, 2002). Stabilization on the other hand, occurs when an artefact no longer requires high specifications. Closure can be attained in two ways, through rhetorical closure where the relevant social group sees the technology as solving the problem that it was intended to

context, technology and the user, which in this research, are considered to be the attributes of recursive relationships between the three entities. This leads to the final iteration of the framework, called the Triality framework, according to which, the technology is conceptualised as technological innovation, the context is conceptualised as healthcare social system and the user is conceptualised as human agent. It further highlights that each recursive relationship has set of attributes associated with it and there are nine attributes in

By drawing on this framework, this chapter aims at eliciting the complexities associated with healthcare context and the dilemmas surrounding healthcare professionals when technological innovations such as telehealth and electronic records are deployed and

Exploring theoretical perspectives from IS provides insight into how technological content, context and the user have been conceptualised, and what is the nature of relationship between the three entities. Thus, theoretical perspectives that are of essence are discussed

SCOT stems from science and technology studies, which concern themselves with the impact of technology on society and its dynamics. These studies argue that the technology design and its acceptance trajectory are affected by social dynamics within a given society (William & Edge, 1996). SCOT therefore, strives to explain why certain technological artefacts or "*variants*" rise in society at given point in time whereas the others "*die*" (Pinch & Bijker, 1984). This view was proposed in response to technology determinism that establishes an opposite perspective that technology impacts the surroundings and its users (William & Edge, 1996). SCOT presents specific conceptualisation of technology, context and

Technology according to SCOT is conceptualised as an artefact of which the design and use is the subject of various interpretations resulting due to enabling and constraining properties of the technology as experienced by the groups under a given social context (Avgerou, 2002; Bijker, 1995; Klein & Klienman, 2002; Pinch & Bijker, 1984; Jackson et al., 2002). This constitutes the concept of interpretive flexibility, which also ascertains that despite different interpretations of an artefact within a society, these interpretations can co-

In addition, the degree to which a technological artefact is accepted within a society is defined by the concept of closure and stabilisation. Closure is achieved when "*all groups' problems have been addressed and groups achieve consensus on a particular design*" (Klein & Klienman, 2002). Stabilization on the other hand, occurs when an artefact no longer requires high specifications. Closure can be attained in two ways, through rhetorical closure where the relevant social group sees the technology as solving the problem that it was intended to

**2. Theoretical perspectives from the field of IS: Understanding conceptualisation of technological content, context and user,**

**and the nature of interrelationship between them**

exist together as they are dependent on desired outcomes.

**2.1 Social Construction of Technology (SCOT)**

the user. These are discussed below.

total.

next.

evaluated.

address, and closure by redefinition of a problem that sees the use of technology as the solution to an entirely different problem than originally proposed. For example, when the prelimatic tyre was originally introduced to solve the vibration problems in cycles, it was rejected; however, when used instead in racing cycles, it was accepted. This illustrates the closure by redefinition of the problem.

Context according to SCOT includes the social and political dimensions. It is argued to shape the values and norms of the groups. It does not however provide detailed insight into this area and has been criticised for this weakness.

The user in SCOT, is conceptualised as a member belonging to a particular social group, who despite sharing a common interpretation of technological artefact as a member of the group also has an individual interpretation which can be subjected to the group's scrutiny over time. This notion is constituted within the concept of technology frames, and has been regarded as its contribution. Another contribution of SCOT includes its attention to the influence of the designer on design during the design process.

However, despite the vital contributions, SCOT has been argued to have drawbacks that include the lack of acknowledging that the design of an artefact might be embedded with a particular group and their intention, and therefore may be unsuitable for other contexts and groups. In addition it is further argued that SCOT does not consider user opinion and extrapolates findings by only following trends (Faulkner, 2009).

In the field of IS, SCOT is furthered by Orlikowski and Gash (1994), who introduced the concept of technology frames of reference (TFR) by drawing on Bijker's notion of the technology frame. According to TFR, the perceptions, expectations and experiences of an individual pertaining to a provided technology, constitute their view and attitude about its nature, value and use. Individuals are argued to have separate and different TFRs and these are often created when a change is introduced that causes disturbance to established routines and work practices (Ciborra & Lanzara, 1994).

In the case where the TFRs of individuals match, they are called to be in congruence such that "*alignment of frames on key elements or categories*" exists, but when there are differences, they lead to "*incongruence of technology frames*" (Orlikowski & Gash, 1994). Significant incongruence would result in technology being abandoned, while congruent technology frames would achieve use and become embedded in routines.

By drawing from SCOT and TFR, it can be argued that users of technology constitute perceptions as frames that are influenced by their interaction with their surrounding environment and group members (Davidson, 2006; Davidson, 2002; Davidson, 1997). These frames govern their reaction to any change introduced and if congruent then positive impacts such as use of provided technology can be achieved. On the other hand, if incongruent frames dominate, the use of technology will cease and efforts to deploy or evaluate would be wasted. The notion of frames discussed here and the way in which SCOT views the three entities of context, technology and the user, facilitates to draw holistic definition of each of the three entities in Triality Framework. The framework is presented later in the chapter.

The focus now moves to ANT which differs greatly from SCOT in terms of its analytical approach.

Understanding Innovation Deployment and Evaluation in Healthcare: The Triality Framework 157

However, the very criticism of ANT is also seen as its main strength as some argue that by using such an analytical concept, ANT accounts for negotiating, redefining and appropriation of interest of human actors and those "*inscribed within a technical artefact*" (Orlikowski et al., 1996). One example that used the notion of negotiation in healthcare is study where the implementation of telehealth is studied, and it is argued that the interaction resulting between context, information communication technology (ICT), and the user due to ICT implementation is a social process. It requires negotiating between power, politics and meaning encompassed within ICT and its intended use (this concept uses the term ICT

Another important concept that ANT provides is that of normalisation. This concept facilitates understanding about technology and its assimilation within networks. More importantly this concept highlights the longitudinal nature of technology assimilation by

In addition, ANT's concept of 'Unintended consequences' outlines that the outcomes of introducing technology in a network might not always be desired and expected, and that unintended ways of technology use increases when users and designers are spatially absent or distanced (Nicolini, 2007). One study using this concept in healthcare (Harrison et al., 2007) suggested that unintended consequences can be attributed to the change introduced by the healthcare information technology (HIT) implementation, and that the role of such unintended consequence can be assessed by evaluating five types of sociotechnical interactions, shown in the form of an interactive sociotechnical analysis (ISTA) model. The model allows examination of the actual use of HIT, the impact of the technical and physical settings of work on HIT, users' interpretation of HIT use, and the recursive dependence of

The enhanced understanding that ANT provides on conceptualisation of technology, context and the user is drawn upon while defining these three entities within Triality

So far, we looked at the variants of Science and Technology Studies (STS), the focus now moves to another theoretical perspective, Diffusion Of Innovation (DOI). This perspective provides a valuable understanding on how technology diffuses within a society by

In this perspective, technology is conceptualised as innovation where innovation is encompassed in a broad definition as "*an idea, practice, or object that is perceived as new by an individual or other unit of adoption*" (Rogers, 2003). An innovation is argued to have five attributes of: relative advantage, compatibility, complexity, trialability, and observability.

Innovation as technology is also suggested to be a composite of hardware and software, and

The context according to this perspective is a social system consisting of social structures and norms. The structures are a type of information that regulate the individual's behaviour and are defined as "*patterned arrangements of units in a system*". Norms on the other hand are

perceiving technology, context and the user from different perspective.

The definition of each attribute is presented in table1.

an enabler of accomplishing actions to achieve desired goals.

instead of IS innovation) (Constantinides & Barrett, 2006a, 2006b).

users in their routines (Avgerou, 2002).

these factors on each other.

**2.3 Diffusion of innovation**

Framework.

#### **2.2 Actor Network Theory (ANT)**

In actor network theory (ANT), there is no analytical difference between a technology artefact and a human user (Latour, 1999; Greenhalgh & Stones, 2010). Both are described as actants that pose as an agency, and together aim at achieving a goal within a heterogeneous network, while drawing on artefacts, text, conventions such as money, people or a hybrid of these intermediaries.

An agency that is described as the technical artefact and human user is argued to be semiotically equivalent; which requires thinking symmetrically about human and nonhuman agents. Both Pickering (1993) and Orlikoswki (2007, 2008) favour the use of this notion and have extended it. For example Pickering introduced the notion of the 'Mangle of practice', according to which, he argues that the human agency and the machine agency are temporally emergent in practice, with the goals of the human agency governing the outcome. In other words, human agency uses technology as its "*temporal extension*" and by which it can succeed in its endeavours. He further suggests that it is the interaction between the human agency and the machine agency which leads to resistance and accommodation.

More recently, using the underlying philosophy of ANT, Orlikowski (2007) introduced the concept of 'Cognitive entanglement', which "*presumes that there are no independently existing entities with inherent characteristics*". The entities in this statement pertain to the context, technology and the user. The notion suggests that there is no recursive relationship between the entities, and according to this approach the "*focus is on agencies that have so thoroughly saturated each other that previously taken for granted boundaries are dissolved*" (Orlikowski, 2008).

An approach encompassing such a perspective towards technology and human as a user is favoured because its argued to solves the problem of representation (Pickering, 1993; Orlikowski, 2007, 2008); which because of the subject object dichotomy employed within other perspectives, fails to acknowledge the issues arising due to power imbalance and shift in knowledge that occurs when a technological artefact is introduced within heterogeneous networks (Latour, 1999).

A heterogeneous network in ANT is a reference to the context in which human-machine interactions are situated (Law & Callon, 1992). The context is governed by sociology of translation. Translation here refers to the processes and actions that are involved in achieving a result due to the interaction between the actants, and it depends upon irreversibility and alignment of the interests of the actors with the overall network aims and goals. In addition, it is suggested that networks are successfully translated when they conform to the regulation, norms (local) and are constituted by legitimate actors. Such translations allow networks to converge and therefore, be effective and durable (Avgerou, 2001, 2002). This notion of sociology of translation is applied in the field of healthcare by Nicolini (2010) in his work to understand the adoption and assimilation of telemedicine innovation.

Among the main drawbacks of ANT is its analytical dimension in which human and machine agencies are treated as the same (Pickering, 1993; Avgerou, 2002). Labelled as the post-humanist approach, it is argued that the use of this analytical lens crumbles when the complexity and number of networks increases. In addition, it is debated that the context in which both human and nonhuman actors interact is only acknowledged to be politically influenced and the social and cultural avenues are ignored.

However, the very criticism of ANT is also seen as its main strength as some argue that by using such an analytical concept, ANT accounts for negotiating, redefining and appropriation of interest of human actors and those "*inscribed within a technical artefact*" (Orlikowski et al., 1996). One example that used the notion of negotiation in healthcare is study where the implementation of telehealth is studied, and it is argued that the interaction resulting between context, information communication technology (ICT), and the user due to ICT implementation is a social process. It requires negotiating between power, politics and meaning encompassed within ICT and its intended use (this concept uses the term ICT instead of IS innovation) (Constantinides & Barrett, 2006a, 2006b).

Another important concept that ANT provides is that of normalisation. This concept facilitates understanding about technology and its assimilation within networks. More importantly this concept highlights the longitudinal nature of technology assimilation by users in their routines (Avgerou, 2002).

In addition, ANT's concept of 'Unintended consequences' outlines that the outcomes of introducing technology in a network might not always be desired and expected, and that unintended ways of technology use increases when users and designers are spatially absent or distanced (Nicolini, 2007). One study using this concept in healthcare (Harrison et al., 2007) suggested that unintended consequences can be attributed to the change introduced by the healthcare information technology (HIT) implementation, and that the role of such unintended consequence can be assessed by evaluating five types of sociotechnical interactions, shown in the form of an interactive sociotechnical analysis (ISTA) model. The model allows examination of the actual use of HIT, the impact of the technical and physical settings of work on HIT, users' interpretation of HIT use, and the recursive dependence of these factors on each other.

The enhanced understanding that ANT provides on conceptualisation of technology, context and the user is drawn upon while defining these three entities within Triality Framework.

So far, we looked at the variants of Science and Technology Studies (STS), the focus now moves to another theoretical perspective, Diffusion Of Innovation (DOI). This perspective provides a valuable understanding on how technology diffuses within a society by perceiving technology, context and the user from different perspective.

#### **2.3 Diffusion of innovation**

156 Management of Technological Innovation in Developing and Developed Countries

In actor network theory (ANT), there is no analytical difference between a technology artefact and a human user (Latour, 1999; Greenhalgh & Stones, 2010). Both are described as actants that pose as an agency, and together aim at achieving a goal within a heterogeneous network, while drawing on artefacts, text, conventions such as money, people or a hybrid of

An agency that is described as the technical artefact and human user is argued to be semiotically equivalent; which requires thinking symmetrically about human and nonhuman agents. Both Pickering (1993) and Orlikoswki (2007, 2008) favour the use of this notion and have extended it. For example Pickering introduced the notion of the 'Mangle of practice', according to which, he argues that the human agency and the machine agency are temporally emergent in practice, with the goals of the human agency governing the outcome. In other words, human agency uses technology as its "*temporal extension*" and by which it can succeed in its endeavours. He further suggests that it is the interaction between the human agency and the machine agency which leads to resistance and accommodation. More recently, using the underlying philosophy of ANT, Orlikowski (2007) introduced the concept of 'Cognitive entanglement', which "*presumes that there are no independently existing entities with inherent characteristics*". The entities in this statement pertain to the context, technology and the user. The notion suggests that there is no recursive relationship between the entities, and according to this approach the "*focus is on agencies that have so thoroughly saturated each other that previously taken for granted boundaries are dissolved*" (Orlikowski, 2008). An approach encompassing such a perspective towards technology and human as a user is favoured because its argued to solves the problem of representation (Pickering, 1993; Orlikowski, 2007, 2008); which because of the subject object dichotomy employed within other perspectives, fails to acknowledge the issues arising due to power imbalance and shift in knowledge that occurs when a technological artefact is introduced within heterogeneous

A heterogeneous network in ANT is a reference to the context in which human-machine interactions are situated (Law & Callon, 1992). The context is governed by sociology of translation. Translation here refers to the processes and actions that are involved in achieving a result due to the interaction between the actants, and it depends upon irreversibility and alignment of the interests of the actors with the overall network aims and goals. In addition, it is suggested that networks are successfully translated when they conform to the regulation, norms (local) and are constituted by legitimate actors. Such translations allow networks to converge and therefore, be effective and durable (Avgerou, 2001, 2002). This notion of sociology of translation is applied in the field of healthcare by Nicolini (2010) in his work to understand the adoption and assimilation of telemedicine

Among the main drawbacks of ANT is its analytical dimension in which human and machine agencies are treated as the same (Pickering, 1993; Avgerou, 2002). Labelled as the post-humanist approach, it is argued that the use of this analytical lens crumbles when the complexity and number of networks increases. In addition, it is debated that the context in which both human and nonhuman actors interact is only acknowledged to be politically

influenced and the social and cultural avenues are ignored.

**2.2 Actor Network Theory (ANT)**

these intermediaries.

networks (Latour, 1999).

innovation.

In this perspective, technology is conceptualised as innovation where innovation is encompassed in a broad definition as "*an idea, practice, or object that is perceived as new by an individual or other unit of adoption*" (Rogers, 2003). An innovation is argued to have five attributes of: relative advantage, compatibility, complexity, trialability, and observability. The definition of each attribute is presented in table1.

Innovation as technology is also suggested to be a composite of hardware and software, and an enabler of accomplishing actions to achieve desired goals.

The context according to this perspective is a social system consisting of social structures and norms. The structures are a type of information that regulate the individual's behaviour and are defined as "*patterned arrangements of units in a system*". Norms on the other hand are

Understanding Innovation Deployment and Evaluation in Healthcare: The Triality Framework 159

Having discussed theories that acknowledge the three entities of technology, context and the user as important yet do not discuss the nature of relationship between these three entities, we move our attention to the theories that provide such conceptualisation of relationship between the entities. These theories and concepts include the contextualist approach, the structuration theory and the technology-in-practice. We start with contextualist approach that elaborates on the recursive relationship between the

The contextualist approach elucidates the importance of considering context when studying change and the processual nature of such a change. According to this approach, context is conceptualised to consist of outer and inner contexts. The outer context refers to interorganisational conditions such as political, social, economical and competitive environment (Pettigrew, 1985, 1987, 1997). On the other hand, inner context accounts for intraorganisational aspects such as structure, organisational culture and political circumstances. In IS, the contribution of the contextualist approach is considered important as defining the context can be often problematic (Avgerou & Madon, 2004); and by extrapolating this notion to IS, context can be defined as a set of variables that affect information systems and in turn are influenced by it. This is termed "*the environment*", and informs that context should be

Conceptualisation of technology according to this perspective is that of a content that can be understood as "*the particular areas of transformation under investigation*" (Pettigrew, 1987). Apart from introducing new technology, when viewed from the perspective of organisational context, these areas of transformation can also include geographical positioning, change in organisational culture and work force. It may be further argued that content and context are bound in a mutual relationship, where one is continually shaped by and shapes the other. Outlining the recursive relationship between content and the context

The relationship between context and content is enabled through process, where process encapsulates the actions, reactions and interactions of the various parties that are involved in changing of an organisation from one state to another (Markus & Daniel, 1988; Pettigrew , 1987). Thus, it represents continuity and interdependent sequence of actions and events related to a given phenomena and allows understanding of its (phenomena's) origin, continuance and resultant outcome. At an actor level, process is represented and described by verb forms such as interacting, acting, reacting, responding and adapting. Whereas, at a system level, emerging, elaborating, mobilizing, continuing, changing, dissolving and

The strength of the contextualist approach lies in the provision of "*guiding assumptions*" to carry out the research appropriately (Pettigrew, 1997). Among these assumptions is the inclusion of vertical and horizontal analysis. It is argued that having multiple levels of analysis enables the emergent and situational nature of process under the given context to be captured. Analysis at the vertical dimension includes studying group dynamics and organisational issues, and the interrelationship between those levels. Horizontal analysis involves interconnecting longitudinally the phenomena of interest studied at the vertical

is one of the contributions of this approach (Pettigrew, 1985, 1987, 1997).

(technological) content and the context.

considered as emergent (Avgerou, 2001).

transforming describe process.

**2.4 Contextualist approach**

defined as regularised behaviour patterns that are acceptable. The notion of a social system is inclusive of an organisation or a network of organisations. When an innovation is proposed, an organisation is argued to go through a process of innovation which, involves defining organisational need, matching the need with an innovation, going through the process of redefining and restructuring the innovation and organisational routines to achieve fit, clarify if any gaps remain, and finally achieve routinization where innovation is accepted with activities such that it "*loses its identity*" over time (Rogers, 2003; Lee 2004).

The user, according to this perspective is understood to be an individual who is a part of the innovation-decision process that involves mapping the journey of an individual through gathering knowledge on the innovation, developing a perception towards the innovation, evaluating and taking the decision to either accept or reject the innovation, use the innovation, and finally confirm the decision (to use it or not). The user is also conceptualised to communicate his/her understanding about the technological innovation to the other individuals within the social system, thus, impacting its diffusion.

One of the main contributions of this perspective is the concept of consonance and dissonance, which argues that an individual's behaviour is attributed to their constant effort towards eliminating or reducing the uncomforting feeling due to change. In IS, this notion is used to understand how use of IT changes work practices at a micro level, and how practices change due to perceived dissonance between the context, technology and the action entailed within practices. Overtime, individuals enact practice as routines to experience consonance (Vaast & Walsham, 2005).

This perspective also proposes that the change in social structures and norms due to the introduction of innovation leads to resistance (Greenhalgh et al., 2008). Using this perspective to understand the diffusion of electronic records in the NHS. U.K. resulted in a conceptual model that considers "*the determinants of diffusion, dissemination, and implementation of innovations in health service delivery and organization*" (Greenhalgh et al., 2004; Greenhalgh et al., 2008).

This perspective and how it had been used by other authors to understand the processes that impact diffusion of innovation provides a valuable contribution to the framework in this work to conceptualise the context, technology and user.

Having discussed theories that acknowledge the three entities of technology, context and the user as important yet do not discuss the nature of relationship between these three entities, we move our attention to the theories that provide such conceptualisation of relationship between the entities. These theories and concepts include the contextualist approach, the structuration theory and the technology-in-practice. We start with contextualist approach that elaborates on the recursive relationship between the (technological) content and the context.

#### **2.4 Contextualist approach**

158 Management of Technological Innovation in Developing and Developed Countries

defined as regularised behaviour patterns that are acceptable. The notion of a social system is inclusive of an organisation or a network of organisations. When an innovation is proposed, an organisation is argued to go through a process of innovation which, involves defining organisational need, matching the need with an innovation, going through the process of redefining and restructuring the innovation and organisational routines to achieve fit, clarify if any gaps remain, and finally achieve routinization where innovation is accepted with activities such that it "*loses its identity*" over time (Rogers, 2003; Lee 2004).

*"is the degree to which an innovation is perceived as being consistent with existing values, past*

The user, according to this perspective is understood to be an individual who is a part of the innovation-decision process that involves mapping the journey of an individual through gathering knowledge on the innovation, developing a perception towards the innovation, evaluating and taking the decision to either accept or reject the innovation, use the innovation, and finally confirm the decision (to use it or not). The user is also conceptualised to communicate his/her understanding about the technological innovation to the other

One of the main contributions of this perspective is the concept of consonance and dissonance, which argues that an individual's behaviour is attributed to their constant effort towards eliminating or reducing the uncomforting feeling due to change. In IS, this notion is used to understand how use of IT changes work practices at a micro level, and how practices change due to perceived dissonance between the context, technology and the action entailed within practices. Overtime, individuals enact practice as routines to

This perspective also proposes that the change in social structures and norms due to the introduction of innovation leads to resistance (Greenhalgh et al., 2008). Using this perspective to understand the diffusion of electronic records in the NHS. U.K. resulted in a conceptual model that considers "*the determinants of diffusion, dissemination, and implementation of innovations in health service delivery and organization*" (Greenhalgh et al.,

This perspective and how it had been used by other authors to understand the processes that impact diffusion of innovation provides a valuable contribution to the framework in

*"is the degree to which an innovation is perceived as better than the idea it supersedes"*

*"is the degree to which an innovation is perceived as difficult to understand and use"*

*"is the degree to which an innovation may be experimented with on a limited basis"*

 *"is the degree to which the results of an innovation are visible to others"*

Table 1. Five attributes of innovation (taken from Rogers, 2003)

individuals within the social system, thus, impacting its diffusion.

experience consonance (Vaast & Walsham, 2005).

this work to conceptualise the context, technology and user.

2004; Greenhalgh et al., 2008).

*Relative advantage*

*experiences, and needs of the potential adopters"*

*Compatibility*

*Complexity*

*Trialability*

*Observalibility*

The contextualist approach elucidates the importance of considering context when studying change and the processual nature of such a change. According to this approach, context is conceptualised to consist of outer and inner contexts. The outer context refers to interorganisational conditions such as political, social, economical and competitive environment (Pettigrew, 1985, 1987, 1997). On the other hand, inner context accounts for intraorganisational aspects such as structure, organisational culture and political circumstances.

In IS, the contribution of the contextualist approach is considered important as defining the context can be often problematic (Avgerou & Madon, 2004); and by extrapolating this notion to IS, context can be defined as a set of variables that affect information systems and in turn are influenced by it. This is termed "*the environment*", and informs that context should be considered as emergent (Avgerou, 2001).

Conceptualisation of technology according to this perspective is that of a content that can be understood as "*the particular areas of transformation under investigation*" (Pettigrew, 1987). Apart from introducing new technology, when viewed from the perspective of organisational context, these areas of transformation can also include geographical positioning, change in organisational culture and work force. It may be further argued that content and context are bound in a mutual relationship, where one is continually shaped by and shapes the other. Outlining the recursive relationship between content and the context is one of the contributions of this approach (Pettigrew, 1985, 1987, 1997).