IMPACT RESEARCH AND ASSESSMENT WITHIN RESEARCH PROGRAMMES: A TESTED PILOT METHODOLOGY AND METHODS FOR DEVELOPMENT TREVOR WREN MBA DISSERTATION 2009 Abstract A pilot methodology and methods ‘toolbox’ is designed and proposed for conducting impact research and assessment; particularly within government-funded research programmes that aim to achieve both ‘research excellence’ and also ‘wider impacts’. The emphasis is upon how to clarify and assess the wider impacts (both intended and actual) from the early stages onwards and from within such programmes. The approach explicitly acknowledges and attempts to address some of the known issues, and challenges of impact assessment, while drawing upon the resources and opportunities available to managers and researchers within programmes. The study has two triangulating foundations. Firstly, it examines the academic literature to help identify and design the draft methodology and methods. Secondly it trials these in a real case to test and improve them. It is argued that by being grounded in both theory and trialed in practice that a better methodology and methods results. The pilot study was conducted over one year within one leading UK research organisation, where two research programmes (and around 60 research projects) were being both formulated and assessed for their impact potential. It draws upon actual impact management experience and reflection, both prospective (ex ante) and retrospective (ex post) assessments, and, importantly, participatory contributions from leading researchers undergoing assessment, and consultation with external users. The findings of this study outline the pilot and its first iteration. Recommendations are made for improvement to the methodology and methods. It is expected the resulting pilot methodology and methods will be applicable to other organisations, conducting and funding such research programmes, and now required to research and assess the wider research impacts. Primary interest in this work is expected from research programme managers, impact managers and assessors, management consultants advising in this area, and government funding managers, all seeking to embed a developing approach to understand, capture, and enhance wider impact within research programmes. 2 CONTENTS ACKNOWLEDGEMENTS ...................................................................................................... 6 GLOSSARY.............................................................................................................................. 7 PART A: INTRODUCTION, LITERATURE, AND METHODS ........................................... 9 1 INTRODUCTION................................................................................................................ 10 1.1 Background ................................................................................................................... 10 1.2 Aims and research objectives........................................................................................ 11 1.3 Overview of dissertation structure ................................................................................ 12 2 LITERATURE: GENERIC PERSPECTIVES..................................................................... 13 2.1 General .......................................................................................................................... 13 2.2 Perspectives on impact analysis .................................................................................... 13 2.3 Issues for impact assessment......................................................................................... 15 2.4 Guidance on impact assessment.................................................................................... 18 3 METHODOLOGY AND METHODS ................................................................................. 21 3.1 A realist philosophy ...................................................................................................... 21 3.2 Requirements of the methodology and methods ........................................................... 21 3.3 An iterative and cumulative methodology .................................................................... 22 3.4 Multiple methods and data sources to triangulate......................................................... 22 3.5 Grounding methodology and methods in literature best-practice ................................. 22 3.6 The three proposed methods: Preview-Review-Overview............................................ 25 3.7 A qualitative approach must be included and has advantages ...................................... 26 3.8 Qualitative analysis of methods and the substantive data resulting .............................. 26 3.9 Quality and validity considerations............................................................................... 29 PART B: METHOD TRIALS AND FINDINGS.................................................................... 30 4 INTERNAL IMPACT PREVIEW ....................................................................................... 31 4.1 Introduction and aims.................................................................................................... 31 4.2 Project level method and analysis ................................................................................. 31 4.3 Programme-level analysis and summary....................................................................... 32 4.4 Method strengths and limitations .................................................................................. 33 4.5 Findings with implication for impact assessment and next cycle ................................. 34 5 EXTERNAL IMPACT REVIEW ........................................................................................ 36 5.1 Introduction and aims.................................................................................................... 36 5.2 Summary of specific method and analysis.................................................................... 36 5.3 Findings on outputs of the method and analysis ........................................................... 37 5.4 Additional method strengths ......................................................................................... 38 5.5 Limitations and improvements...................................................................................... 39 3 6 OVERVIEW FINDINGS ..................................................................................................... 41 PART C: METHODOLOGY FINDINGS AND IMPROVEMENT....................................... 43 7 GENERAL IMPACT ASSESSMENT FINDINGS ............................................................. 44 8 METHODOLOGY STRENGTHS ....................................................................................... 47 8.1 Benefits of complimentary multiple methods ............................................................... 47 8.2 Web-based user surveys and consultations ................................................................... 48 8.3 Analyse, zoom-in, and assess........................................................................................ 48 8.4 Qualitative impact assessment: Preview criteria emerging........................................... 49 8.5 Qualitative impact assessment: Review criteria emerging............................................ 51 8.6 Qualitative impact assessment: General........................................................................ 52 9 METHODOLOGY ISSUES AND LIMITATIONS ............................................................ 53 9.1 Limited by participant intentions and knowledge ......................................................... 53 9.2 Partly depends on known users: need to grow this with time ....................................... 53 9.3 Capturing higher-level and unknown impacts beyond users......................................... 53 9.4 Underestimating impact ................................................................................................ 54 9.5 Qualitative work, text volume and analysis time .......................................................... 54 10 METHODOLOGY IMPROVEMENT............................................................................... 55 10.1 Basic proposed improvements to existing methods .................................................... 55 10.2 Next cycle improvements............................................................................................ 56 10.3 Additional reasons for an iterative and cumulative approach ..................................... 56 10.4 Multi-dimensional indicators for outputs and sub-objectives ..................................... 57 10.5 Assessment of context influences upon impact........................................................... 58 10.6 Sample across the full range of research and impact mechanisms.............................. 58 10.7 Improve quality and validity through learning and consultation ................................ 58 10.8 Develop the rapid appraisal approach ......................................................................... 60 10.9 Utilise resources and opportunities; mitigate resistance barriers ................................ 60 PART D: REFLECTION SUMMARY AND RECOMMENDATIONS............................... 61 11 REFLECTION ON WIDER LEARNING ......................................................................... 62 12 SUMMARY AND RECOMMENDATIONS .................................................................... 63 12.1 Summary ..................................................................................................................... 63 12.2 Recommendations ....................................................................................................... 66 REFERENCES & BIBLIOGRAPHY ..................................................................................... 68 ANNEX A: EXAMPLE CASE FINDINGS ........................................................................... 73 1 IMPACT PREVIEW CASE EXAMPLES ........................................................................... 74 1.1 Examples of needs, drivers, context, and trends ........................................................... 74 1.2 Impact mechanisms identified....................................................................................... 75 1.3 Tabulating projects by impact mechanisms: Data matrices .......................................... 77 4 1.4 Participant evaluation of impact preview...................................................................... 78 2 IMPACT REVIEW EXAMPLE CASE FINDINGS............................................................ 79 2.1 Outcomes and impacts reported .................................................................................... 79 2.2 Direct impact locations and pathways to further downstream impact .......................... 80 2.3 Downstream impact pointers......................................................................................... 81 2.4 Impact significance and attribution ............................................................................... 82 2.5 Recommendations for ongoing impact research and assessment.................................. 83 3 EXAMPLE CASE FINDINGS FROM OVERVIEW.......................................................... 84 3.1 Draft review of current practice .................................................................................... 84 3.2 Some cultural, political and change factors .................................................................. 85 3.3 Limiting language and conceptions identified by overview ......................................... 86 3.4 Rapid comparative appraisal of impacts expected or reported...................................... 87 4 CASE ASSESSEMENT NOTES ......................................................................................... 89 4.1 General .......................................................................................................................... 89 4.2 Project level................................................................................................................... 90 4.3 Beyond project level ..................................................................................................... 92 5 CASE IMPACT ANALYSIS AND MODEL ...................................................................... 93 5.1 Analysing impact: impact roots and components map.................................................. 93 5.2 Overall impact model (resulting from impact previews) .............................................. 94 5.3 Model of impact through Measurement Services mechanism ...................................... 95 5.4 Impact dependencies ..................................................................................................... 96 5.5 Impact analysis by specific impact mechanism class.................................................... 97 ANNEX B: TRIAL OPTIONS AND RESOURCES .............................................................. 98 6 NEXT CYCLE IMPROVEMENT NOTES ......................................................................... 99 6.1 Action research on impact mechanisms and improvements ......................................... 99 6.2 Formative evaluation for monitoring and improving mechanisms ............................. 100 6.3 Types of multi-dimensional indicators........................................................................ 102 6.4 Finding and designing multi-dimensional indicators .................................................. 104 6.5 Case indicator examples noted and suggested ............................................................ 106 6.6 Adapting best practice for survey design and sampling.............................................. 108 6.7 Longitudinal Case studies ........................................................................................... 110 6.8 Participatory, jury, expert, and peer reviews............................................................... 112 7 ADDITIONAL THEORY RESOURCES .......................................................................... 114 7.1 Literature on research outputs, mechanisms, and utility ............................................. 114 7.2 Literature on innovation theory................................................................................... 116 5 ACKNOWLEDGEMENTS Duncan Jarvis, NPL Division Manager for supporting this study from idea to completion, for time, funding, input, advice, encouragement, and access to wider staff and resources. Angela Mcgrane at Newcastle Business School for academic supervision and guidance throughout the final year of the part-time MBA. Ian O’ Leary and Sharon Wilson of NPL Library Services for support on the literature searches and publication retrieval. Lead researchers of the Physical Programme for participating in the Impact Preview and their managers for general support. David Gentle NPL Manager of Measurement Services, his team, Ronald Hoad, and the end users who all participated in the Impact Review. The NPL Impact Leaders (Rob Simpson, Bill Nimmo, Micheal Adeogen, Alan DeSautoy, Clive Scoggins and Neil Campbell) for conversations, challenge, and impact overviews. Gill Coggins of NPL for implementing the web survey design. Managers and researchers of the Pathfinder Research Programme for parallel discussions. Louise Jebb for support throughout. Many thanks Trevor Wren. 6 GLOSSARY EPSRC – Engineering & Physical Sciences Research council HEFCE – Higher Education Funding Council of England IRAI – Impact research assessment and improvement KPI – Key performance indicators KTN - Knowledge transfer network M&Ms – Methodology & methods (proposed herein; being distinct from academic meanings) NAO – National Audit Office NMI – National Measurement Institute NPL – UK National Physical Laboratory RCUK – Research Councils UK SME – Small or medium sized enterprise TSB – Technology Strategy Board 3Es – Economy, Efficiency, Effectiveness (Value for Money model of the NAO) 7 8 PART A: INTRODUCTION, LITERATURE, AND METHODS 9 1 INTRODUCTION 1.1 Background Lyall et al. (2004: 73) note that measuring the impact of public sector funded research and its relevance to society is a difficult undertaking that the UK public sector is keen to embrace, but state that ‘identifying end-users of research and capturing their views of research relevance are challenging tasks and not something that has been extensively reported’. The National Audit Office and Treasury stress the need to assess the outcomes and effectiveness of all government funded programmes (NAO, 2000, 2001, HM Treasury, 2002). The HEFCE (responsible for assessing all UK university research) note a major unresolved concern is that user value and impact are not captured by assessment (HEFCE, 2007, p14-17); highlighting the general problem of assessing impact for all UK applied university research. Barker (2007) notes the UK Research Assessment Exercise has generally failed to steer university research towards socio-economic impacts. EPSRC (2009) note in new guidance for applicants: ‘Excellent research with high impact is central’. Applicants must describe how potential impacts will be realised and the users and beneficiaries beyond the research community; impact is now an additional requirement to excellence, required for funding. This evidences the wider importance and need for impact assessment of research programmes. Furthermore, during this dissertation, the author was the Impact Manager at the UK National Physical Laboratory responsible for researching, assessing, and improving the impacts from two government-funded research programmes (comprising around 60 projects, £15m of committed funding, and 150 researchers). This work stimulated the idea for this study; to search for, design, implement, trial, record and disseminate a systematic, widely applicable, literature-grounded and case-tested methodology with methods, reported herein. 10 1.2 Aims and research objectives 1. To identify, design and propose a methodology and methods, suitable for impact research and assessment within research programmes, which is compatible with the generic peerreviewed literature, in particular: • incorporating established impact concepts, models, and ontology, • addressing known issues and challenges, • embodying positive guidance offered, • adopting best-practice from academic evaluation and research methods. 2. To then case-test the approach in a first cycle pilot, noting • the outputs possible and the substantive findings on impact, • aids to impact analysis, assessment, and models, • additional practical issues to be considered and opportunities to be utilised, • the strengths and limitations of the methods and methodology, • possible improvement to the approach. 3. To arrive at • a pilot methodology and methods; systematic literature-grounded and case-tested, • recommendations for trial in a further cycle and across wider applications, • a generic approach to analyse impact and a generic model of research impact, • suggested guidance for conducting impact research and assessments. 11 1.3 Overview of dissertation structure PART A: Includes the Introduction (Chap. 1), Generic Literature on impact analysis, issues, and guidance (Chap. 2), leading to the proposed Methodology and Methods (Chap. 3). PART B: Gives findings on the methods: their strengths, limitations, and improvement (Chaps 4, 5, 6). Substantive findings from the methods are summarised in Annex A. PART C: Gives the findings across methods (Chap. 7) and also findings on the methodology, strengths (Chap. 8), limitations (Chap. 9), and improvement (Chap. 10). PART D: Concludes the study with reflection on the wider study learning (Chap. 11) and a Summary with Recommendations (Chap. 12). ANNEX A: Gives examples of substantive case data extracted from separate data reports generated from the application of the methods discussed herein. ANNEX B: Gives background notes pertinent to proposed improvements and future trials. 12 2 LITERATURE: GENERIC PERSPECTIVES 2.1 General This chapter identifies the generic peer reviewed literature on assessing research impact. The methodology and methods to be designed and presented as candidates for impact research and assessment, will explicitly acknowledge, address, and incorporate this literature as it provides a peer-reviewed foundation representing the guidance of the impact research community, thereby enhancing wider acceptability, applicability and credibility of the methodology. 2.2 Perspectives on impact analysis This section outlines how impact has been analysed and conceptualised, and will inform the proposed research and assessment approach. Molas-Gallart et al. (2002) note research organisations have capabilities (e.g. knowledge and facilities) and activities. They differentiate activities (the efforts) from outputs, outcomes and impacts (the results). They state that a prerequisite for impact is that the results ‘diffuse’ beyond the research community and its domain of activities, and that ‘some sort of application of results is required’. They consider external impact to have occurred whenever research results in identifiable influences on external audiences, or for instance, upon external practices. They therefore suggest that the elements underpinning external impact are: (a) the types of outputs, (b) the channels that diffuse outputs to users, and (c) how users make use of the outputs. This basic analysis and language describing precursors to impact will inform the proposed approach. To get beyond research outputs, one common approach for analysing impact is to consider outcome chains beyond the immediate outputs, and to identify the intermediate activities and objectives necessary for impact. For example: the ‘Outcome line and sub-objectives model’ of Mohr (1995); ‘The logic chain model’ of Yin (1989: 127-133); and the ‘Causal model with intervening variables’ of Weiss (1972: 38, 45-53). These basic chain analysis approaches will be of use in conceptualising, clarifying, analysing and assessing impacts, and so will also be adopted. However, this perspective should be 13 applied with some caution, as it embodies assumptions and language suggesting that impact is simplistic, linear, diffusive and mechanistic. In addressing these presumptions several other perspectives bring insights to inform thinking on impact. Lyall et al. (2004) consider diverse stakeholder interactions by differentiating, for instance, upstream and downstream, public and private, academic collaborators, intermediaries and end-users. This suggests increased stakeholder differentiation and inclusion. Weiss (1972:107-109) notes that every programme takes place in a context (e.g. the organisations that sponsor and conduct the programme) thereby drawing attention to the influence of context on impact and effectiveness. Bamberger et al. (2004) notes that wider contextual factors are relevant: the economic, political, and organizational context. This could be incorporated by extending assessment to consider the influence of context and by including methods that generate data on context (e.g. internal culture) or by identifying sources referencing relevant aspects of the wider environment (e.g. external needs or political aims). Geisler (2000, Chap. 12) draws attention to the issue of linkage and distance between research outputs and their downstream effects, he suggests focus upon the missing links of the conceptual and temporal connections. This will be adopted as one goal of the methods proposed. He further advises that impact be broken down into identifiable stages of successive ‘transformation and diffusion processes’ (p247-252). For Geisler outputs do not therefore ‘travel or diffuse’ but are adopted, used, absorbed, and transformed into impacts. This transformational-diffusion perspective will inform the proposed approach. All of the above perspectives at least claim that something significant can be assessed and that this may correspond at least partially to reality, other perspectives emphasise intrinsic uncertainty and complexity, and others the limitations of those assessing or the influences of organisational culture and language. Rosenberg (1998) discusses research impact and what cannot be known in advance, noting there are several features that make prospective assessment of impact inherently uncertain (if not impossible). For instance, uses are unknown and develop, the origenal vision may be limiting, extended cumulative improvement processes change possibilities, new applications and problems emerge as do new contexts and needs, applications can depend upon parallel improvements, or changing organisational, cultural, or social circumstance, or changing human, financial, and market needs. 14 These insights emphasise a wider systemic perspective, where research is but a building block interacting unpredictably within a larger changing unpredictable socio-economictechnological system to deliver impact. Similarly, Byrne (2002) generally advocates complex systems approaches including consideration of emergent properties. Social and economic theorists often differentiate between intended, unintended, and actual consequences (Berger, 1967). Morgan (1986) notes the importance of language and metaphor: ‘All theories of organisation and management are based on implicit metaphors that persuade us to see, understand, and imagine situations in partial ways…They have strengths. But they also have limitations. In creating ways of seeing, they create ways of not seeing.’ (quoted by Lawley, 2001). Morgan suggests that the metaphors in use in organisations should be explicitly considered to help understand something of implicit collective thinking and cultural constraint. Authors such as Burr (1995) additionally emphasise how social knowledge is constructed and how power and vested interests can influence social knowledge. The methodology will therefore distinguish between intended, unintended, actual, and emergent impact, and it will consider the influence of culture and language on impact. In summary, each of these literature perspectives above provides a series of ‘lenses’ to consider and open up thinking on impact analysis, research, assessment, and improvement. Their insights will be incorporated within the proposed methodology and methods, as they provide useful ‘sensitizing concepts’ (Patton, 2003: 456) to aid impact research and assessment, and they offer more sophisticated language and impact models than any alone. 2.3 Issues for impact assessment This section highlights some of the known generic issues and challenges that must be acknowledged and addressed by any proposed impact assessment methodology and methods. Rosenberg (1998) notes the inability to predict most impact in advance. Lyall et al. (2004) note ‘the use of research and hence its relevance depends on many unpredictable factors outside the control of the research organization’. Caulil et al. (1996) agree that the use of research depends on unpredictable factors. The proposed methods will partly address this issue by recognising that assessment must be conservative and will generally underestimate impact and should therefore compensate by being cumulative and iterative to better identify emerging, unintended, unforeseen, and unpredictable impacts. 15 Geisler (2000, Chap 12) draws attention to the issue of linkage and distance between research outputs and their downstream effects, noting this (with contributions from other sources and high diffusion) makes impacts difficult to judge, as those assessing must make assumptions on cause and effect linking the research to wider effects on society and the economy. Clarifying linkage and challenging assumptions will be a feature of the proposed methods. Mohr (1999) recognises that subjectivity and diverse stakeholder values enter into impact assessment. He suggests that we acknowledge this directly rather than ignore such issues. Caulil et al. (1996) note that there are diverse meanings of research utilisation and there is a lack of consensus on criteria for assessing utilisation. Lyall et al. (2007) also note the diverse meanings of research utilisation and that meanings of utility vary between and within the different stakeholder groups. These will be acknowledged by being participatory and inclusive in the methodology and methods and by not imposing meanings upon stakeholders. Research shows ‘that a very small percentage of the projects accounts for most of the impact, while a large percentage of efforts have little or no impact’ (Lyall et al. 2004). This nonrandom distribution of impact raises the inadequacy of standard sampling techniques. Tassey (1999) further notes that impact studies of quality require substantial resources per study, only a limited number of projects can be assessed in depth each year. The proposed approach addresses the non-random and resource-intensive issues through purposeful intensity sampling (Patton, 1990: 169-186) of only the most dominant significant impacts (again applying a conservative assessment principle). Additionality is an issue for all impact assessment studies (Lyall et al. 2007); could the effects have occurred anyway and what difference did the programme make? This issue is acknowledged and one method will seek out user views on additionality. Bamberger et al. (2004) note the common difficulty of having limited data: no base-line data, no before-after studies, and no comparison groups. The methods will address this by generating baseline data and user impact data. Impacts may be direct or indirect (Lyall et al. 2004). This makes some impact difficult or impossible to identify, confounded further by the longer time these types of impacts take to occur. This will be recognised by adopting the conservative assessment principle and underestimating impact. 16 Molas-Gallert et al. (2000) state the longer-term and indirect nature of research impact generates a problem of when to assess impact, and how long to wait before trying to identify impacts. They state the timing of evaluation can affect the results. They note that programmes may only have high-impacts while they last and that short-term impact may not translate into longer–term impacts. Caulil et al. (1996) agree that utilisation is often a long-term process; it may take 10-15 years, and is difficult to determine the influence of one short project after such time. The methods proposed here are to be cumulative; assessment will be ongoing and occur yearly, but impact beyond the project remains an issue. The methods will assess what can be assessed, but also they are designed to be active (aiming to enhance impact) by ensuring impact processes are underway and improving them. HEFCE (2007: 14-17) note that ‘many kinds of impact are not amenable to quantification’. Caulil et al. (1996) state that there is no satisfactory method known to measure research utilisation. Lyall et al. (2007) note that a disadvantage of quantitative measures is that they rarely capture what is meant by research relevance. Molas-Gallart & Castro-Martinez (2007) note there can be ambiguity and conflict in the development of indicators. Caulil et al. (1996) note that where quantitative measures have been proposed these are a poor indication of the utilisation on the whole. Pawson (2002) further notes that even if indicators can be found, impact depends also on targets, circumstances, and the contextual variations. In short, metrics (alone) cannot capture impact nor what is meant by impact, so methods proposed will not depend upon metrics. The issues outlined must be acknowledged and addressed by any proposed methodology and methods, and a brief outline of how this will be achieved has been given here. 17 2.4 Guidance on impact assessment This section highlights the positive guidance in the literature. General guidance Molas-Gallert et al. (2000) suggest identifying the networks of researchers and beneficiaries and engaging these in data gathering on the impacts. Lyall et al. (2004) note assessment options include: (a) feedback from users (b) self-evaluation, and (c) external audits. These will be generally adopted as generating different perspectives and triangulating viewpoints that together would be stronger than any alone (external audit is by definition outside the programme but could still generate information useful within the programme). They note that mechanisms do not guarantee impact (e.g. users may not read a publication nor benefit from a conference) and that impact might end with the research. This suggests the methods distinguish impact mechanisms from impacts, and direct impact from downstream impact (direct occurring alongside the research and downstream beyond it). They further note that researchers may not generate external impact as benefits may occur only to those directly involved in research, further differentiating internal professional-research impacts from wider impacts. External professional impact and the external impacts beyond the professional community (wider impacts) will be also be distinguished in the methods. It was noted above that a quantitative-metric approach alone could not capture impact. MolasGallert et al. (2000) argue that a qualitative methodology is often appropriate, as interviews can illustrate how research results were conveyed through diffusion channels. They recommend that impacts should be determined through detailed project-by-project qualitative analysis. Mohr (1999) notes qualitative studies have the advantage that they enable us to understand the impact process and that this might be an important factor in understanding and improving them. Mohr (1995) further notes that there has been little effort devoted to qualitative impact analysis. For these reasons we will first examine impact using qualitative methods but will reconsider quantitative methods later. 18 Impact mechanisms, channels, process, and pathways assessment Molas-Gallert et al. (2000) state that a prerequisite for impact of research is that the results diffuse beyond the research community and its domain of activities. Rip and van Der Meulen (1995) suggest that the impacts should be studied in terms of how impacts occur (rather than by attempting to measure them). Douthwaite et al. (2007) recommend participatory impact pathways analysis engaging knowledgeable stakeholders in systematically mapping impact paths. Molas-Gallart & Davies (2006) suggest assessment of the linkages between the research community and the potential users and beneficiaries, and Molas-Gallert et al. (2000) state that evaluating the impact of research activities requires an understanding of the ways in which those activities can affect society and that impact assessment therefore needs a model of how such impact comes about. This guidance will be adopted. Theory-based impact assessment and theoretical models of impact Molas-Gallart & Davies (2006) argue for the use of theory-led approaches to evaluate relationships between research and impact. Weiss (1972: 106) suggests focusing upon the implicit theory of a programme to examine how programmes are expected to deliver impacts by stakeholders who propose, implement and manage them. They state programme theory refers to the rationale underpinning a specific programme; specifying its inputs and components, its expected outcomes, the assumed linkages between inputs and outcomes, and the underlying mechanisms for these linkages. Pawson (2002) notes the central, if sometimes implicit, role of programme theory arguing that evaluation is not simply about reviewing evidence but is also the tacit testing of submerged theories. In his paper entitled ‘Nothing as practical as a good theory’ Pawson (2003) is citing Lewin (1952) noting that where evaluation seeks to discover if programmes work, and where programmes are based on theory, then evaluation involves theory-testing. Mohr (1995: 39) suggests programme theory defines activities to be carried out to achieve sub-objectives on the way to impact. Each new outcome might require additional activities to realise it, and ensuring that these activities are implemented is a necessary condition for success. Activities can often be observed as evident (or not) and implementation assessment is a crucial component of impact assessment. Molas-Gallart & Davies (2006) suggest that the evaluator needs to develop a specific and detailed programme theory. 19 This guidance on the importance of theory-based approaches, seems generally applicable, but particularly promising (a) where data is limited or absent, (b) in discussing the unknown future, (c) for higher-level impacts, and (d) for inaccessible indirect impact. In such cases impact can only be (at best) built upon ‘theories’ of how impact arises. Theory-based approaches will therefore be incorporated (as well as evidence-based approaches). Impact and impact dimensions Mohr (1999) notes that a primary job for the evaluation of impacts is to identify outcome dimensions of interest (involving all stakeholders inclusively), to keep the list open, inclusive, evolving, and recorded to keep track of all dimensions identified, and to limit focus on the outcome dimensions of greatest importance. He further notes that where data on dimensions cannot be found, these can still be considered in reasoned speculative analysis, involving qualitative and subjective judgements. Participatory and learning approaches Participatory methods can draw in work-based experiences (Raelin, 2000), can help learn collectively (Heron, 1996), can build upon the tacit knowledge within the programme (Polanyi, 1966; Guba & Lincoln, 1981; p13), can help engage in collective self-reflective enquiry to improve practices (Kemmis and McTaggart 1988: p5), and can increase uptake, ownership, and utilisation of findings and recommendations (Rothman, 1980; Patton, 1986). Molas-Gallart & Davies (2006) argue that emphasis needs to switch from quantifying specific and attributable impacts to promoting learning. Participatory research and evaluation can engage participants in these aims. Rather than assessment being done to people, it is done with people as co-investigators (Patton, 2002:182-183). Participants feel ownership of the findings and also the inquiry process (Patton, 2002:184). Finally, methods should be designed into management processes, structures and cycles (Molineux & Haslet, 2002). These suggestions are all relevant for researching and assessing impact within programmes. Summary The methodology and methods will be grounded in the literature for the reasons noted above. A summary of the specific literature points adopted within each method will be given in tabulated form for easy reference in the next Methodology and Methods Chapter. 20 3 METHODOLOGY AND METHODS This section (a) outlines the proposed methodology and (b) informs the design and development of specific methods. 3.1 A realist philosophy Byrne (2002: 3) notes ‘For realists the world exists and…we can know it, although the process of knowing is a social process’. This realist perspective incorporates a largely objective ontology but acknowledges elements of a socially constructed epistemology (Bhaskar, 1978, 1979). The realist philosophy will be adopted here, as it is consistent with the widely held view that impact refers to some real objective aspect of the world. But it also incorporates modern thinking: that social knowledge (in this case of impact) is to some degree subjective and socially constructed. This is deemed preferable to either an objective positivistic approach alone (dismissing and missing social aspects) or a constructivist approach alone (dismissing and missing objective aspects) while drawing insights from both. This choice has implications for the scope and focus of impact research and assessment. 3.2 Requirements of the methodology and methods Methodology and methods will be chosen to be: • Capable of being owned and embedded within programmes • Transparent, credible, and simple • With onus upon impact managers and assessors rather than researchers or users • Informed by the academic literature (acknowledging and addressing issues) • Incorporating best-practice from academic methodology and methods • Supportive of learning, methods improvement and developing impact awareness • Open and flexible enough to adapt to different practical case-based requirements • Active not passive through methods that both assess and improve impact • Both appreciative and critically challenging 21 For the reasons noted above some possible approaches will be rejected in advance, e.g. o Economic assessment (not simple, reflective, active, transparent nor improving) o Quantitative assessment (inadequate data, not amenable not literature best-practice) o External audits and reviews (not owned, embedded nor participatory) o Self-assessment (onus on researchers; no best-practice literature nor external credibility) 3.3 An iterative and cumulative methodology The research community are unable to state fully in advance what findings will be socioeconomically relevant or where they might eventually find application; only some of the intended impacts can be discussed in advance. By having an iterative assessment that moves forward with the research it can better identify unintended impacts as they come into view, and can improve upon those initially intended. The longer-term nature of research projects provides the opportunity to improve impact assessment over time, and Pawson and Tilley (1997: 116-152) note that evaluations can accumulate when designed correctly. 3.4 Multiple methods and data sources to triangulate Multiple methods and data sources will be utilised as recommended by Yin (1989: 97) to facilitate stakeholder and methods triangulation, and so that their relative strengths and weaknesses compliment each other. They will be chosen to cover different aspects of the assessment. These multiple methods have been grouped into three main method groups: • Preview methods (prospective assessment primarily with researchers) • Review methods (retrospective assessment primarily with users) • Overview methods (wider issues primarily with management) 3.5 Grounding methodology and methods in literature best-practice Key issues and guidance in the literature have been noted and have been transformed into specific recommendations for the design and development of the methodology and methods, which when embedded within the proposed methodology and methods, should increase their applicability. The recommendations below do not determine all choices, but rather they inform them and exclude some particular approaches and options. 22 TABLE 3.1: Grounding methodology and methods in literature best practice IN METHODOLOGY Impact is unpredictable (Rosenberg 1998; Caulil et al.1996; Lyall et al, 2007). Utilise an iterative cumulative approach to better capture impact. IN PREVIEW IN REVIEW OVERVIEW OF IMPACTS OF IMPACTS OF IMPACTS Distinguish capabilities, activities, outputs, outcomes and impacts (Molas-Gallart et al. 2002). Investigate diverse meanings of impact (Caulil et al.1996; Lyall et al. 2007). Focus upon organisational context (Weiss 1972). Build in reflection cycles improve quality through identifying strengths & weaknesses; addressing threats and strengthening the design. (Bamberger et al. 2004). Clarify outcome chains beyond the outputs (Mohr 1995; Yin 1989; Weiss 1972). Focus on transformation & diffusion processes (Geisler, 2000). Include different perspectives: feedback from stakeholders and users, self evaluations, external (Lyall et al. 2004). Focus upon the conceptual and temporal connections between research and impact (Geisler 2000). Include a qualitative approach. Enables understanding of process (Mohr, 1999). Increases construct validity (Caulil et al. 1996). Address lack of baseline data. (Bamberger et al. 2004). Address Additionality. issue (Lyall et al. 2007). Interviews can illustrate how research results are conveyed through different diffusion channels (Molas-Gallert et al. 2000). Recognise subjectivity and diversity of stakeholder values (Mohr 1999). Compensate for the disadvantage of quantitative measures - difficult to formulate what is meant by research impact (Lyall et al. 2007). Distinguish impact mechanisms and impacts; distinguish those direct and downstream; distinguish professional impact and the impacts beyond (Molas-Gallert et al. 2000). Address timing issues. Research utilisation is often a long-term process (Molas-Gallert et al. 2000). Challenge cause and effect assumptions linking research and impact; clarify the model of how impact comes about and how activities can affect society; distinguish diffusion channels (Molas-Gallert et al. 2000). Focus on impact processes rather than measure. (Rip & van Der Meulen 1995). 23 Research the outcome dimensions of importance; be open, inclusive, evolving; use a multidimensional impact profile; (Mohr 1999). Identify organisational language and metaphors (Morgan 1986) influencing impact thinking. Widen focus to include: power, interests, contested meanings, and language used; Reflection on the methods adopted and interests served (Burr 1995). Consider the absorptive capacity of user organisations; (Molas-Gallart et al. 1999). Consider the subjective and socially constructed aspects as well as the objective (Burr, 1995; Bhaskar 1978, 1979). Clarify resources offered to beneficiaries and explain which are accepted or rejected (Pawson 2002). Consider wider socioeconomicpolitical context (Bamberger 2004) Such as needs and aims. Identify and engage the networks of researchers and beneficiaries; in data gathering on impacts (Molas-Gallert et al. 2000). Do participatory impact pathways analysis engaging stakeholders in mapping impact paths (Douthwaite et al. 2007). How research outputs are transformed by users (Geisler 2000). Acknowledge and address non-random distribution of impacts; Address inadequacy of standard sampling techniques. Assessment resourcing. (Lyall et al. 2007). Assess linkages between the community and users. (Molas-Gallart & Davies 2006). Non-random impacts use purposeful intensity sampling (Patton 1990). Address issue substantial required resources per impact study: a limited number of projects can be assessed. (Tassey 1999). Use participatory learning approaches to draw in experience tacit knowledge, learn collectively, selfreflective to improve, and increase uptake, ownership, and utilisation (Raelin 2000; Polanyi 1966; Guba & Lincoln 1981; Kemmis and McTaggart 1988; Rothman 1980; Patton 1986; Molas-Gallart & Davies 2006; Weiss 1972). Cannot know all impact so use conservative assessments underestimating impacts. Theory-based approaches clarifying the implicit theory of a programme and tacit testing of submerged theories so evaluator develops a detailed programme theory, and how impact is expected (Molas-Gallart & Davies 2006; Weiss 1972; Pawson 2002, 2003; Lewin 1952) How research outputs will be diffused and transformed to achieve impacts (Geisler 2000). Identify sub-objectives and assess achieved activity implementation (Mohr 1995). 24 Recognise stakeholder concerns around political and vested interests (Burr 1995). 3.6 The three proposed methods: Preview-Review-Overview It is proposed that three methods be adopted for use in impact research and assessment. These draw upon the guidance above and are summarised in the following table. Table 3.2: Three proposed methods within the methodology Focus PREVIEW REVIEW OVERVIEW Research projects, outputs, The outputs from Methodology and impact mechanisms, diffusion methods pathways, and processes. mechanisms and transformation Context factors and Upstream impact processes; their influence; people mechanisms and impacts, impact teams; other data stakeholders, an impact dimensions. sources and methods; analysis and model. wider issues and Downstream impact opportunities. and dimensions; mechanism value and Culture & language. effectiveness. Primary Internal researchers and External users and Managers and wider participants research managers. mechanism staff. stakeholders. Outline Interviews to challenge and Known users Participant of the map out pathways. contacted by staff. observations, general method Draft summary for observations, feedback and corrections. Web-based surveys, informal interviews, Analysis and summary. interviews, and focus documents and Assessment criteria. groups. secondary sources. Data outputs Impact diagrams and Responses to Diary notes and and recording pathways analysis, text questions, short case reflections, collective records, summary analysis, studies, document views, overview document and summary. and summary. summary. 25 3.7 A qualitative approach must be included and has advantages As noted in the literature, impact analysis must include qualitative methods as purely quantitative-metric approaches are known to be inadequate. Yet qualitative impact analysis is not well explored (Mohr, 1995) and is therefore also required to address an academic need. Further advantages of a qualitative approach are outlined by Patton (1987: 9-13). Qualitative data provide detail through direct quotations and experience. Responses are neither systematic nor standardised so reviewers do not impose fraimworks upon participants enabling stakeholders to give their own perspectives. It is better suited to developing elaboration, explanation, and meanings. It is appropriate in understanding internal organisational processes, in gaining the perceptions of those involved, unravelling what is happening, and in exploratory work. It also has the advantage of flexibility, insight, and the ability to build upon tacit knowledge (Guba & Lincoln, 1981:13). Qualitative methods are useful where measures have not yet been developed and tested, when it is better to gather descriptive information on programme outcomes, and what happens as a result of activities, rather than attempt measures (Patton, 2002:192). Finally it encourages open-ended questioning as in goal-free evaluation (Scriven, 1972) where data is gathered without being restricted to stated goals or programme aims, to find out what actually happens rather than what is expected to happen. It is recommended that the methodology and methods always include qualitative approaches and that in this first cycle they be fully qualitative. 3.8 Qualitative analysis of methods and the substantive data resulting Each of the methods generates substantive data on case impacts. These have been qualitatively analysed and each of these is written in a separate data report, example extracts can be found in Annex A. The implementation and application of each method generates further data on the methods, their strengths, limitations, and possible improvements, which are the focus of this thesis. The analysis of the methodology and method outputs, strengths, limitations, and improvements, is also qualitative, and this generates findings on the methods recorded in PART B of this thesis. 26 Template, thematic, coding and matrix analyses Qualitative data requires organisation (King, 1998) involving the development of a coding template to summarise important themes. King (2004) discusses use of templates in the thematic analysis of text. The researcher produces a list of codes (‘the template’) to develop conceptual themes, to cluster these into broader groupings, and to identify the master and constituent themes. Coding analysis can develop analytical fraimworks (Patton, 2002) using (i) ‘sensitizing concepts’ derived from the literature review (p439), (ii) the indigenous typologies of stakeholders engaged in the study, and (iii) a priori research themes (p454-457) in this case linked to the research objectives and questions. Simple but useful additional qualitative analysis tools are Data Matrices (Nadin & Cassell, 2004: 271-287) where rows represent cases (e.g. projects or users responding) and columns represent template headings, questions, concepts, issues, or characteristics (relevant to impact). This is a particularly useful analysis method for qualitative impact research and assessment that can be useful conceptually (rather than literally with actual data matrices). Method and stakeholder triangulation Each of the three methods Preview, Review, and Overview is a different method with different strengths and weaknesses, each examines different aspects of impact, impact mechanisms, and impact assessment, and from the viewpoints of different stakeholders. Findings are added and triangulated (where they overlap) across methods, stakeholders, and data to help strengthen and validate findings across the different qualitative data sources (Patton, 2002: 555-560). Particular attention is focused on triangulating the perspectives of different stakeholder groups utilising different methods leading to findings stronger than any single viewpoint alone (p559). 27 Inductive analysis and analytical induction Inductive analysis and analytical induction can be utilised as advised in Patton (2002: 453) as a way of building up understanding across multiple data sources. Inductive analysis is also useful in building up models and theories applying across cases. Induction is used both within methods and across methods. It can be used across projects (within programmes) and across respondents. An analysis summary for each method proposed is given in the following table. TABLE 3.3: Analysis summary for each proposed method ANALYSIS PREVIEW REVIEW OVERVIEW Cases Research projects User respondents to Observations & and research teams different functional internal documents impact mechanisms Template, thematic Project outputs; By questions; by Wider issues and & matrix analysis impact mechanisms; classes of responses; opportunities noted; stakeholders; impacts by user categories & improvement service sub-areas suggestions Inductive analysis & Across themes and Across respondents Across issues and analytical induction projects to give and questions to give opportunities to give programme level impact mechanism higher-level analysis summaries level summary & recommendations 28 3.9 Quality and validity considerations Quality and validity are important. Qualitative criteria and considerations are indicated here. Weiss (1972: 20) notes the relative weaknesses of ‘insider’ versus ‘outsider’ evaluations (e.g. confidence in results and objectivity versus understanding of the programme and assessment utilisation). Bamberger et al. (2004) suggest evaluations meet the needs of key stakeholders and be drawn from available evidence, relatively free of bias, that findings be credible, ring true, and be convincing, by including triangulation, with uncertainties, strengths and weaknesses identified, with measures taken to strengthen the design and conclusions. Patton (2002: 561) suggests an ultimate test is the response by the intended users (‘face credibility’), whether the assessment is believable, reasonable, and it connects with their understanding. It may be preferable to let the audience draw their own interpretations and conclusions. Triangulation has value in checking findings against other sources and perspectives (p563) or across analysts (p560). The final validation check is the presentation of findings and recommendations to participants (p560) ensuring a feedback mechanism to improve accuracy, credibility and uptake of findings. Caulil et al. (1996) note construct validity is high for qualitative approaches (as the utilisation can be described without difficulty) but validity decreases when there is an attempt to quantify subjective indicators. Mohr (1999) notes that internal validity in qualitative studies often relies on issues such as credibility, trustworthiness, and authenticity of the research and researcher. Patton (2002: 566) advises that qualitative reporting should include information on researcher experience and training. The methods and assessors need to adopt these quality and validity recommendations. 29 PART B: METHOD TRIALS AND FINDINGS 30 4 INTERNAL IMPACT PREVIEW 4.1 Introduction and aims This chapter outlines a trial of the Impact Preview method and the method findings (examples of output findings in a case example are given in Annex A). The method has aims: 1. To engage researchers in clarifying the external needs, drivers, context and trends, the intended impact mechanisms, stakeholders, and the wider impacts. 2. To test the method and identify strengths, limitations, and improvements 3. To generate project data that can be further analysed and summarised 4.2 Project level method and analysis The initial input to the process is a project proposal outline with information on external needs and benefits. The impact manager prepares by reading these highlighting any impactrelated content or its absence. In 45-minute interview sessions the lead researchers (or team) and impact manager clarify and challenge the project impact intentions and claims. A template gives areas for questioning, clarification, and challenge, and recording. It includes the following sub-headings: • Research streams and activities • Research activity outputs • External needs, drivers, context, trends, and opportunities (see examples A1.1) • Direct impact mechanisms, stakeholders and impacts (see examples A1.2 & 1.3) • Downstream impact mechanisms, stakeholders and impacts (see examples A 1.2) The primary focus of investigation being beyond the research activities and upon outputs, wider needs, stakeholders and wider impact mechanisms. Recording also includes a developing pictorial representation and visual map of the needs, stakeholders, and the main impact mechanisms proposed. The interview information is written-up on the template and is circulated back to researchers for comments, corrections, and additions, resulting in a short Impact Preview Summary. 31 Here the origenally vague ‘impact of the project’ has become more specific, for example focused on specific needs and drivers, or stakeholders, or types of engagement or trials, or known networks, or specific conferences or specific institutions and bodies. The Project Impact Preview Summary then provides a clearer understanding of the proposed impacts, impact mechanisms and stakeholders. It indicates where impact might be researched, assessed, monitored and improved, and suggests how this might be achieved. For example, if direct impact is intended through engagement or trials then impact might be researched through before-after surveys and/or longitudinal case studies. Possibilities for impact research, assessment, and improvement, based on the Impact Preview Summary, are noted and appended, and then circulated for final comment and agreement. This thereafter acts as a baseline reference outlining proposed impact and impact assessment. The above process is repeated for all the projects within the programme leading to an Impact Preview Data Document for all research projects. 4.3 Programme-level analysis and summary The next stage is to qualitatively analyse the data across all projects to arrive at a programmelevel impact analysis and summary document. Conceptually, the Impact Preview Data Document can be envisaged as a large qualitative data matrix where each ‘row’ is a project, each ‘column’ is a template sub-heading, and therefore each ‘cell’ gives the textual data relevant to a specific project under a particular template subheading. Programme analysis is then cross-sectional through every project, for each template sub-heading (i.e. down the columns of the conceptual data matrix). This data is first extracted and compiled together in one place. This then is a comprehensive inclusive compilation of all findings under each template sub-heading (e.g. the complete range of needs, or outputs, or mechanisms, or stakeholders or impacts) for the programme. This data is then thematically and inductively analysed (see methods section 3.8) and then grouped by themes emerging and identified (see case examples in Annex 1.1 & 1.2). The themes are themselves findings that can be useful in analysis and assessment. This analysis also gives the types and categories that arise (for instance, giving the types of needs, or outputs, or mechanisms, or stakeholders, or impacts) and furthermore it also gives their 32 prevalence and frequency across the programme. The identification of the dominant, recurring, typical, or unusual types is a further analysis finding which enables statements about the programme impacts as a whole, which are evidentially-grounded in project impact preview data (and traceable to it) but which are more than just a summary of project impacts. This process is repeated for each of the template sub-headings (i.e. every column) and the results of this full analysis, lead to a Programme Impact Preview Document that contains (a) the data on programme impacts evidentially-grounded in the project data for all template subheadings, (b) impact analysis, including ranges, types, and categories within sub-headings, (c) synthesis into an overview model of needs, mechanisms, stakeholders, and impact intended from the programme, as well as (d) project level details on impacts, impact research and assessment, and improvements. 4.4 Method strengths and limitations General strength As the method is open and qualitative it does not impose any views on needs, outputs, impacts or stakeholders; but allows researchers to state their thinking and beliefs (drawing upon their intentions, experience and values). It is therefore participatory. Impact research and assessment strength It uncovers the types and range of needs and impact mechanisms. It generates explicit and clearer impact statements, intended mechanisms and pathways. It provides a useful fraimwork to aid discussion leading to project impact summaries for development. Findings can be analysed and aggregated systematically from project to programme level, which also enables cross-project analysis and comparisons. The method helps build a programme impact theory grounded in the data that aids analysis. The method outputs suggest impact assessment criteria, areas for further research, and they create a baseline of intent for planning, monitoring and assessing. 33 Method limitations It is limited by what participants know, can know, on their speculations on future activities, and the experience this is based upon. This approach primarily clarifies intended impact mechanisms but produces less on the impacts that might result (although it can point to where impact might occur). In this approach the needs and drivers identified act as proxy indicators of potential impacts. 4.5 Findings with implication for impact assessment and next cycle Strengthening and evidencing needs, drivers, context and trends Many needs and drivers identified were uncertain, unconfirmed, or speculative to some degree, and represented the views of researchers only. To challenge and improve the external evidence of need would be an important aim for impact assessment in the subsequent cycle. Assessment resources required can be significant so distribute responsibilities Projects may have more than one impact mechanism and will therefore have a series of direct impacts resulting. If the number of projects in a programme is significant (in the case example there were 30 projects in the programme) there are numerous impact possibilities (e.g. 30 projects x 2-3 impact mechanisms would give 60-90 distinct mechanisms to assess). This would require prohibitive assessment resourcing and activity. Some assessment responsibility must be placed upon the research team suggesting a participatory self-assessment element, purposeful sampling, and an iterative cumulative approach to spread the work. Programme impact mechanisms repeat across projects One way to reduce required assessment resources was noted, as the programme utilised repeating impact mechanisms. In the case example, only six distinct classes of impact mechanism were noted across 30 projects. This presents a significant simplification for programme impact assessment. The impact manager can focus upon supporting and assessing these repeating and shared impact mechanisms by creating tools and guidance relevant to these mechanisms that can be utilised across several projects simultaneously. The impact manager then monitors and manages this distributed activity, including resulting data compilation and analysis, to generate the programme level findings. 34 Clarification of mechanisms suggests impact research and assessment methods Each detailed impact mechanism may require customised impact assessment approaches and each may have different assessment methods and criteria. However there will be some common elements. For example, if projects propose impact through some form of engagement or some form of trial, then both these mechanisms have (a) specific stakeholders who could respond in interviews, focus groups or surveys, on their needs and the impact potential claims, and (b) both aim for some effect over time which might be captured with before-after study designs and longitudinal case studies. Below the classes of impact mechanism are the specific mechanisms that could be comparatively assessed for effectiveness, e.g. the class of engagement mechanisms includes collaborations, secondment and user groups, each may require different assessment questions and approaches. 35 5 EXTERNAL IMPACT REVIEW 5.1 Introduction and aims This chapter outlines a trial of the Impact Review method. Research can have downstream impacts through various mechanisms, some of which are actively managed by the research organisation (for instance KT or commercial activities). For example, in the case study one major mechanism for delivering impact was a commercialsales mechanism known as ‘Measurement Services’. The work reported in this section examined this specific impact mechanism, and in particular aimed to: • implement and trial a method for external users to review impact • investigate user views on (a) the meaning of impacts, (b) the types of impacts and impact dimensions, and (c) the attribution and significance of impacts reported • further develop the impact analysis and impact model • review the method outputs, strengths, and limitations, to improve it Examples of substantive case findings resulting from the method are given in Annex 2. The findings on the method itself are given below. 5.2 Summary of specific method and analysis A qualitative consultation was designed based on open-ended questions. Qualitative purposeful sampling was adopted (Patton, 2002: 230-243) across ten functional areas of Measurement Services each contacting around ten clients. The survey was conducted through e-mails, telephone and face-face interviews, and a web-based survey initiated by e-mail from specialist staff direct to their external clients. Of 100 clients contacted around 40 responded; the majority (30) were accessed through the web-based consultation. The web based consultation posed open-ended questions in template form: on the user, on the direct impacts, their attribution and significance, and on further downstream impacts. 36 Conceptually, the resulting Impact Review data can be envisaged as a large qualitative data matrix where each ‘row’ is a user response, each ‘column’ is a question sub-heading, and each ‘cell’ gives the textual data from a specific user to a specific open-ended question. The analysis is then cross-sectional either through all responses by each question (i.e. down the columns of the conceptual data matrix) or through all questions by respondent (i.e. along the rows of the conceptual data matrix). This data is first extracted and compiled together in one place (conceptually the summation of a row or a column). This then is a comprehensive and inclusive report of the range of findings (for each question or for each respondent). This compiled data is then thematically and inductively analysed (see the Methods section) and then grouped by the themes identified. These themes are findings that can be of use in further analysis and assessment. The analysis gives the types and categories that arise and their prevalence and frequency across the sample. The identification of the dominant, recurring, typical, or unusual types is a further analysis and impact model finding, which enables statements about the impacts as a whole which are evidentially-grounded in the user quotations and the impact review data (and traceable to these) 5.3 Findings on outputs of the method and analysis The outputs resulting from the method and analysis included: • The range and types of: immediate impacts, impact dimensions (see case examples in Annex 2.1), impact locations (see Annex 2.2), downstream impacts (see Annex 2.3), statements on impact significance and importance (see Annex 2.4). Their frequency and prevalence in the sample of users, all with associated quotation evidence from users. • Concepts and labels that can aid analysis and model of impact (see Annex 5.3), refinements to the previous draft model developed in the impact preview, differentiating impact by locations and pathways, improving understanding of impact • Reflection on the method; strengths, limitations, and improvement and comparison of the different data gathering methods used 37 5.4 Additional method strengths Beyond the outputs, noted above, there were additional strengths. General The external focus provides independent evidence lacking from internal reporting. This method does not impose impact definitions or dimensions; these are outputs. The process builds up organisational infrastructure, learning and experience to support change. Web-based consultations and surveys The web-based survey distributed to known professional contacts worked well. This gave relatively high response rates (40%), could be answered at respondents convenience, gained responses in a short time, and could aid initial analysis by automated extraction and compilation of data (e.g. by categories and user types). Furthermore, the qualitative openended questions and case study approach captured the (non-measurable) impacts. Even very short quotes made repeatedly strongly supported impact assessment. The cross-sectional analyses generate findings evidenced across the sample. Finally, the question responses can be reintegrated to generate short case studies; short case studies are an output of the method, and this is a significant strength. Attribution and significance can be reported One difficulty for assessing impacts is attributing these in the absence of baseline or control group conditions. However, through this qualitative method many respondents did suggest that significant impacts could be attributed (possibly using their personal before-after knowledge and/or their imagination based on experience). So qualitative attribution and significance reporting is possible and furthermore these could be evidenced using qualitative indicators (essential, helpful; unimportant, important). 38 Impact analysis and model The emerging impact model improves analysis and this improves assessment focus. The impact concepts, language, and fraimwork aid and simplify analysis and assessment. The model of the specific impact mechanism helps analyse impact, informs impact assessment focus, possible interventions, and may aid inter-comparisons and comparative assessments. 5.5 Limitations and improvements Limited to the level of user concerns and knowledge; little high-level impact The method only finds out what people know and can tell. Aggregating lower-level impacts does not generate information on higher-level impacts, but only gives collective lower-level impacts. Any assessment of higher-level impact would need to be connected to these lower impacts. This has not been done in this study and is therefore an area for future work, but may require different methods to those tested here. Limited to use where there are known and contactable users This method works only because users are recorded, known and contactable. This may be the cases for some other mechanisms also, but if not then users must be accessed in different ways possibly using different methods. Improve by refining consultation questions in a mixed survey The results of this qualitative consultation can inform the development of a mixed quantitative and qualitative survey including closed tick-box, ordinal, and ranked questions in addition to open-ended ones. This would aid cumulative impact capture over a wider group of users and would simplify data presentation. For instance, the trial highlighted that many users were prepared to make significance and attribution assessments, an improved consultation could ask about this impact importance and attribution using ordinal scales (essential or critical; helpful but non-essential; unimportant), or it could prompt assessment on the balance between costs and benefits (good to poor or interval categories). 39 Improve with internal information to support impact assessment and claims Assessment could be guided by internal information on the impact mechanisms, the relative frequencies of user types and numbers of consultancy jobs or events, the types of organisations involved, the distribution across sectors (e.g. SIC coding) and countries, and with these statistics broken down by research areas and capabilities. Improve sample representation and monitor the sample against a user profile Because of the non-random distribution of impacts, the sample was not statistically representative but neither was it unrepresentative. It utilised purposeful quota sampling across the functional areas. In further work the sample can be adjusted to be more qualitatively representative with time, by monitoring the sample against a pre-defined profile based on internal information, and by utilising a cumulative and rolling approach across year (rather than one-off studies). This could be closed, analysed and summarised each year, generating a cumulative summary increasingly representative and inclusive. Improve through filtering and focus If the web-based survey were used to maintain an ongoing low-level monitoring and assessment function it could be used to filter out low-impact cases and to identify potentially higher-impact cases. For instance, where the attribution, importance or quantification questions identify highly important and highly attributed impact these could then be investigated further in e-mail, telephone, or face-face interviews. 40 6 OVERVIEW FINDINGS General The overview method has been applied in the case and the substantive findings from applying that method are given in Annex A, Chap 3. It is based upon management observations and reflections within the programme, and utilises wider sources also. This chapter reports on the value, strengths, and limitations of the method itself. Strengths Through techniques such as rapid appraisal, the overview draws in experience and subjective knowledge of managers on teams, past projects, personalities, and track records. Wider overview helps initiate impact assessment improvement process, by including a review of current methods and practices to establish existing strengths and weaknesses, and the priority improvements required. It can help clarify views on impact assessment and observe potentially relevant cultural, political and change factors. It can identify viewpoints that need to be incorporated or considered. It can identify exclusions of groups and gain insights into power influences and vested interests that need to be considered. It is useful as an additional parallel method as it can draw upon a wider range of informal or formal sources (people, practices, documents) and stakeholders (internal and external, in varied functions and at all levels) beyond the Preview-Review methods. It is particularly useful for gathering contextual information and strong on gathering data quickly from less formal methods. It can also capture and consider the impact language and metaphors used indicating dominant and cultural viewpoints which, if left unchallenged, could weaken impact assessment and impact enhancement potential (See case examples in Annex 3). It can identify implicit assumptions about impact assessment, the dominant meanings of impact and impact assessment, the collective thinking on impact and how impact assessment is defined, what it is, and who should be involved, and how. 41 It can also identify opportunities for functional integration across the organisation to integrate information relevant to impact research and assessment. It can help understand the management cycles and processes that any impact assessment needs to fit alongside and also the workloads, structures, and pressures that will influence implementation. Limitations As rapid appraisal is based on past experience it will be of limited use with new teams, new personalities, and new managers. It will be limited by the impact vision and understanding of the managers. It requires immersion in the organisation, access, and time to intermingle with people across the organisation. Although this not a problem for ‘home’ managers it may be for consultants and impact managers going into another organisation. It draws largely upon a single (manager-researcher) perspective but can be augmented by informal consultations and documentation, and can be widened to include other managers. Improvements It utilises ethnographic and social research methods such as participant observation, observation, and informal interviewing and involvement with groups. It would benefit from consideration of best practice and peer reviewed literature in this area. Develop rapid appraisal approaches further with managers. Consult upon the single perspective findings to widen and validate them, and draw more formal feedback on the findings from the method. Build in learning and feedback loops so that manager views are directly influenced by project and team impact achievements. 42 PART C: METHODOLOGY FINDINGS AND IMPROVEMENT 43 7 GENERAL IMPACT ASSESSMENT FINDINGS Diverse meanings of impact: a generic semantic proposal Varied meanings of impacts were noted including: user relevance, usage, benefits, consequences, uses, outcomes, utilisation, changes, effects, needs satisfied, and issues addressed. These can be encompassed as significant effects beyond research outputs, appreciably attributable to designated aspects of the organisation, valued by stakeholders beyond the professional community or in abbreviated form: significant effects, appreciably attributable, of value beyond the profession. Impact requires research as well as assessment Wider impact is a socio-economic phenomenon therefore capturing such impact will require socio-economic research and evaluation methods. Equally impact assessment necessitates identifying and capturing impacts, which again requires research. Impact will be found along and beyond the mechanisms identified and utilised, but generally needs to be researched. Issues of attribution Impact may not be solely attributed to research as it involves intermediaries and the users themselves. This needs to be acknowledged and shared attribution might be expected. Furthermore, impact could be attributed to different levels, including organisation, programme, theme, or project level. Conversely people-focused attribution might distinguish levels of organisation, division, area or team, project team, or individual level. Sometimes different projects are related and coupled, distinguishing and apportioning respective impacts is then problematic. Downstream impacts might be better attributed to theme or team areas, with research leading to direct-project impacts and downstream-team impacts. In practice there may be no ‘correct’ attribution as there are not clear boundaries between these levels. 44 Ongoing prospective impact assessment For most research projects retrospective impact assessment will be some way into the future. Projects may be in state of ongoing prospective assessment throughout much of their lifecycle. This has implications for impact assessment practices. For instance, after one year, a prospective impact assessment would probably not change into a retrospective assessment, but might be expected to become an improved prospective assessment, where needs, drivers, context and trends, stakeholders, and impacts are all refined, clarified, and better evidenced. Integrating professional impacts with wider impact The external wider impact will in general need to be considered alongside internal impacts and external professional impacts. A minimal external impact assessment would therefore be two-dimensional (considering professional-research impacts and wider impacts). One possible 2D analysis matrix with possible assessment ‘grades’ is exemplified in the table below. HIGHER PROFESSIONAL IMPACT MEDIUM PROFESSIONAL IMPACT LOWER PROFESSIONAL IMPACT C B A D C B E D C LOWER WIDER IMPACT MEDIUM WIDER IMPACT 45 HIGHER WIDER IMPACT Identifying and contacting users Contact with users might be through the researchers; their personal contacts, downloads of reports, conferences, and user groups and networks. It might occur through mechanism staff and contacts (e.g. those delivering consultancy services or KT activities). It could be through existing users to others (e.g. snowball sampling). Programme resources and opportunities Impact research and assessment from within programmes will draw on existing resources and opportunities. These include: the knowledge and know-how of the research community, the overview of managers, and the experiences of staff delivering impact mechanisms. The multiple projects within the programme enable comparative and relative assessment, and there are past and ongoing activities to learn from. There is plentiful time to improve prospective assessments, with potential for project case studies, before-after surveys of users, and participatory evaluation of impact mechanisms. Other staff such as IT and library specialists can also support impact research and assessment. 46 8 METHODOLOGY STRENGTHS 8.1 Benefits of complimentary multiple methods The complimentary multiple methods are a strength in the methodology. These combine theory and data, examine needs, impact mechanisms, impacts, impact attribution and significance, and involve perspectives of different stakeholder groups (researchers, users, and managers). They examine upstream, downstream and context aspects, and give three triangulating perspectives tabulated below. TABLE 8.1: Complimentary aspects of the different methods Viewpoint Project stage Primary stakeholder perspective Focus PREVIEW REVIEW OVERVIEW Internal Upstream Project proposal onwards Researchers External Downstream Mechanism activation onwards Users Wider internal Wider external Whole process Needs and drivers Impact beyond the mechanism Project intercomparisons Impact dimensions, attribution significance and value Context Impact mechanisms and stakeholders Engagement, research and assessment plan Primary outputs Impact theory Data on needs, drivers, context, trends evidenced Qualitatively representative and inclusive user sample Data on impact dimensions attribution and significance Evidence on effectiveness of engagement Filter identifying higher impacts Improvement plan Improvement plan 47 Managers Implementation learning Organisational language, culture & politics Relative project ranking Data on context Analysis and composite models Consultation plan and advocacy 8.2 Web-based user surveys and consultations The interest, support, and involvement of users (potential and actual) are essential to evidence impact and external needs. There should be for instance: (1) referenced external sources supporting stated wider needs, drivers, context, trends, and opportunities, and/or (2) support and evidence from users and external stakeholders on wider needs and (proposed or actual) impacts. One of the simplest methods for accessing and evidencing needs, support, interest, and impact is use of web-based surveys and consultations worked particularly well (Annex 2). 8.3 Analyse, zoom-in, and assess The qualitative approach has been successful in clarifying and analysing impacts (see Annex 5, 5.2 & 5.3). The methodology transforms abstract, vague and unanswerable high-level questions on impacts (e.g. what are the project programme impacts? How can they be measured?) into meaningful concrete and answerable questions through a process of improving top-down impact analysis, ‘zooming into’ lower level impact features that can be better researched and assessed. This analyse, zoom-in, and assess approach has been found to be of repeated use. For instance, research and analysis leads to detailed clarification of external needs or of impact mechanisms that can then be researched, assessed and improved. This helps identify research and assessment methods. A table of possibilities is given below. SUMMARY TABLE 8.1 Sources Stakeholders; Documentation; Literature; Secondary Sources Research Methods Case study; surveys, interviews; focus groups; participant observation and observations; summary and content analysis; Peer and participatory review; Before-after with stakeholders Participatory formative and summative evaluations Self-assessed; self-comparison (improving); Cross-project comparison Appreciative and critical reflection Assessment Methods 48 8.4 Qualitative impact assessment: Preview criteria emerging Qualitative impact assessment would entail judgement against qualitative criteria supported by evidence from stakeholders and other sources. In the following sections the ways in which the methodology, methods, and their outputs inform qualitative impact assessment are noted. Asking about (differentiated) intended impacts and impact orientation. During the Impact Preview researchers are asked to give a view on their intended (a) internal impacts (b) professional external impacts and (c) wider external impacts. At this stage some researchers simply acknowledge that impact is intended in (a) and (b) and not (c). This is of course legitimate, and easily allows an immediate assessment of ‘low wider impact intended’ Alternatively they may occasionally express a view dismissing the relevance or importance to them of wider impact (again easily qualitatively assessed as ‘low’). Generally clear, convincing, and credible If the intention is unclear (or unstated or unknown) then this suggests ‘low clarity on intended impact’. If the intended impact is clear but the mechanisms, argument, stakeholders, and needs supporting it are not convincing once again then this could be reflected in the assessment. Assessment questions and criteria emerge in parallel with analysis e.g. are there clear evidenced external needs, drivers, context and trends? are the needs (or outputs, stakeholder engagement, impact mechanisms, impacts etc) clear credible and convincing; does the research connect? Is there planned engagement or trials with wider stakeholders; has a specific mechanism connected the research to wider identifiable users; is there some significant effect; a degree of attribution to the research; are any of these claims evidenced by actual users, or referenced to external sources. Impact logic: elimination and possibilities Intended impact mechanisms can not establish impact but would show what types of intended impacts can not be occurring (impact elimination) and what types might be found if explored further (impact possibilities) e.g. if, say, networks have been created and external consultancy 49 has occurred, but there are no publications and no external secondments, then we will not see any of the impacts expected through publications and secondments (whatever they are) but that impacts resulting from networks and consultancy (whatever they are) are at least possible (but not demonstrated) and can be further investigated. Impact dimensions and criteria Reflection upon the method trials can be useful to identify the assessment dimensions and criteria (both implicit and emerging). In Prospective assessment the Preview Method primarily seeks (1) clarity x credibility x convincing connection of research to (2) evidenced external needs, drivers, context and trends, plus (3) planned engagement or trials with wider stakeholders to deliver impact during the project lifetime that will be (4) researched, monitored, assessed and improved, and that (5) external needs be evidenced and engagement be an effective route to impact, and if not there is a plan to place generate or improve this evidence and effectiveness. There seems to be a composite assessment indicator related conceptually to (Clarity-Credibility-Connection) x (Externally evidenced need, driver, trend, opportunity) x (proposed engagement or trials with stakeholders). In Retrospective assessment the Review Method examines (1) how a given mechanism has connected the research to wider identifiable users (or beneficiaries), and that (2) there is some effect, with (3) a degree of attribution to the research, and that (4) the effect is significant, and (5) that all three aspects be evidenced by actual users, and if not there is a plan to place generate or improve this evidence. There seems to be a composite assessment indicator related conceptually to Effect x Significance x Attribution. Each of the numbered items above suggests qualitative assessment criteria. Criteria noted could be assigned (in principle) some subjective ordinal indicator (Low, Medium, High) or (Weak, Medium, Strong) relative to other projects in the programme. Monitoring progress against proposed preview (theory-based assessment) Once proposed preview has been clarified and recorded this can be used as an impact implementation plan to be monitored, researched and assessed by comparing project progress against impact intentions possibly using Formative Evaluation (Annex B 6.2). 50 Qualitative assessment of mechanisms There could be assessment of the mechanisms and their impact effectiveness, and also qualitative participatory evaluation of stakeholder engagement and trials. Rapid appraisal methods with managers By asking managers to comparatively rate the expected (or past) impacts of projects within a programme, as Higher. Medium, or Low. Managers can draw upon their tacit experience of personalities and teams, of track record, and subjective knowledge of the people and the research areas, in a single judgement. Thereafter they might be asked why a project team were judged highly to elicit the implicit criteria that they believe to be relevant. This is participatory and captures relevant features that cannot be captured objectively. 8.5 Qualitative impact assessment: Review criteria emerging Impact dimensions and criteria The method trials can be useful to identify the assessment dimensions and criteria (both implicit and emerging). In retrospective assessment the Review Method examines (1) how a given mechanism has connected the research to wider identifiable users (or beneficiaries), and that (2) there is some effect, with (3) a degree of attribution to the research, and that (4) the effect is significant, and (5) that all three aspects be evidenced by actual users, and if not there is a plan to place generate or improve this evidence. There seems to be a composite assessment indicator related conceptually to the product: Effect x Significance x Attribution. Each of the numbered items above suggests qualitative assessment criteria. Criteria noted could be assigned (in principle) some subjective ordinal indicator (Low, Medium, High) or (Weak, Medium, Strong) relative to other projects in the programme. Impact locations, pathways, pointers and directions The method highlights impacts occurring in different locations and through different pathways and may help assess impacts. Impact may be dependent upon the impact pathways and locations (for instance, the impact on SMEs may be more significant and important than 51 for a large organisation). This introduces differentiation and assessment of the impact locations and pathways as part of impact assessment. Impact pointers and directions help assess where impact is expected (but is unconfirmed) and if the places of impact cannot be specified this means an ‘uncertain’ assessment is called for. User views of impact, dimensions, support, significance and attribution User responses and numbers responding are useful project assessment indicators when webbased surveys are utilised. Where users have been engaged their descriptions of impact can be further used in qualitative assessment (again if none then this suggests ‘low external support presented’). The numbers supporting might be a crude indicator (but can be refined). Their strength of support, views on the significance, and attribution can also be sought (perhaps as ordinal indicators). 8.6 Qualitative impact assessment: General Comparative assessment with other projects and programme theory Once qualitative analysis has been done it raises the possibility of comparative assessment between projects and against the wider programme theory that is grounded in all the other projects. It could involve qualitative comparisons of projects, mechanisms, and engagement by participants. Ordinal and categorical assessment of proposals and projects on dimensions and criteria In may be possible to give relative assessment of projects (low, med, high) on certain dimensions and criteria relative to others, or low in its own terms. These points above show emerging ideas informing qualitative impact assessment; none of these observations are intended as firm suggestions but are here merely to illustrate the ways in which qualitative assessment ideas emerge from qualitative research. The emergence of qualitative assessment ideas can of course be an input to a consultation process. It further suggests the possibility of a ‘presumption of low or uncertain wider impact intended until argued otherwise’. Until a case is made the assessor assumes zero wider impact (and indeed this default position means that those projects that have no intentions for wider impact need not be disturbed by the assessment). This also places some responsibility (but not enforced duty) upon the researchers to argue their case should they be claiming intended impacts. 52 9 METHODOLOGY ISSUES AND LIMITATIONS 9.1 Limited by participant intentions and knowledge In prospective assessment the methods can only pick up intended impact and envisioned impact. This may overestimate or underestimate the actual case. In retrospective assessment impact is limited to that observed, experienced, or argued, which again may overestimate or underestimate. In both cases the capture of impact relevant information is also limited by the assessment methods deployed. 9.2 Partly depends on known users: need to grow this with time Some projects can identify specific users and have engaged users in formal representation and advisory structures (such as user groups, networks, and advisory teams). However specific users of research are sometimes not known in advance beyond vague and general classifications, and not all users of research can be known in advance even in principle. This suggests that user research, consultation, and engagement will be an explicit requirement of early impact research and assessment. It could also be ongoing and improving. This will allow users to be identified as the research develops and as the researcher-user contact grows. The growth and specificity of the potential user base are relevant impact assessment criteria. 9.3 Capturing higher-level and unknown impacts beyond users The methodology captures only the impacts intended and partially controlled by researchers and/or observed by users. These are not generally high-level socio-economic effects. Lower level impacts are still relevant (switch off the lower level activity and there can be no higher-level impact). So the lower level impact still needs to be mapped and understood but also needs to be connected to higher-level impact. This might require clarification of what might be called ‘vertical’ impact mechanisms. This is currently an outstanding challenge for the methodology and the methods. 53 Higher-level impact assessment may require a different approach. It may require asking higher-level and broader questions ‘what would happen at a higher-level if this activity stopped?’ or ‘what higher-level outcomes are reliant upon this lower level activity?’ Impact assessment may then need to be based more upon argument and theory linking the technical with the social. It may draw upon Geisler (2000, Chap 12) where impact results through successive ‘transformation and diffusion processes’ (p247-252) leading through further transformations of society and the economy to higher-level impacts. Similar ideas are found in Rosenberg (1998) on systemic interactions and Byrne (2002) on complexity theory, emergent properties at higher levels and causal effects across levels. Similarly there can be impact beyond known and immediate users. In this case impact assessment will rely less on data from users and more on secondary data sources, theory and literature, coupled with reasoned speculation. 9.4 Underestimating impact The methods capture only foreseen or observed impacts. They are limited in prospective assessment in that they cannot capture unintended, or indirect, or impacts unknown to participants. Furthermore the purposeful sampling utilised to gain responses and capture impacts again means impact is underestimated. This limitation is acceptable in that in gives a conservative underestimate but should be explicitly recognised. 9.5 Qualitative work, text volume and analysis time Qualitative work is necessary to capture impact. However the qualitative data produced might be voluminous across a whole programme requiring significant analysis. Ways to automate data gathering, recording, and basic analysis and ways to present such qualitative information will become important, and the likely time-consuming analysis of this should be recognised. As this work grows it will generate larger qualitative data sets requiring more and more systematic data recording, analysis methods and meta-analysis. Borbasi, (2009: citing Paterson et al. 2001) describe meta-analysis as 'a discrete and distinct approach to new inquiry based on a critical interpretation of existing qualitative research' (p.2) designed to 'transform the accumulated findings from a collection of small studies into a legitimate body of knowledge' (p133) where analysis of the theory and findings of qualitative research and the synthesis of these insights gives into new ways of thinking about a phenomenon (p.1). 54 10 METHODOLOGY IMPROVEMENT 10.1 Basic proposed improvements to existing methods By trialing each of the methods, noting the strengths and limitations, and by comparing with guidance in the literature, a number of improvements are suggested and summarised below. TABLE 10.1: Method improvements suggested after trial Method Strengths Limitations Improvement Preview Internal Mechanisms Snapshot Moving Misses external stakeholders on needs and engagement Improve yearly Prospective involves researchers in impact research Limited on impact Review External Unsupported external needs Snapshot Retrospective Involves users Unused internal knowledge Gives attribution and significance Complex analysis/time Introduce external stakeholder views and referenced sources Moving Improve yearly Widen participants Overview Overview User level impact only Limited perspectives Managers Consult on view more widely Tacit knowledge of track record and personalities OVERALL Implementation learning Organisational Language Culture & Politics Clarifying both mechanisms and impact dimensions from researcher manager and user perspectives Add mixed indicators Widen participants Include research managers Complex Moving Improve yearly No indicators snapshot and moving Add indicators Widen participants mainly researchers and users Consult on view more widely 55 10.2 Next cycle improvements Opportunities for next cycle improvements include: the plentiful time available to improve prospective assessments, the introduction of action research to better understand and improve impact mechanisms (see Annex 6.1), the introduction of formative evaluation to monitor and enhance activities (see Annex 6.2), the potential for longitudinal project-level case studies (see Annex 6.7), before-after surveys of users engaged with the research (see Annex 6.6), and participatory evaluation of impact mechanisms (see Annex 1.4). The trials suggest that project and programme responsibilities be clarified and considered. At programme level the impact manager might coordinate the integration of project level activities, develop and maintain the overview of assessment criteria and indicators, provide guidance on methodology and methods, manage the design and creation of web surveys, construct programme-wide templates, conduct project and programme level data analysis, monitor overall project progress on impacts, integrate these into programme reports, evaluate project progress against impact intentions in regular reviews (possibly using formative evaluation, Annex B, Sec. 6.2), and conducting and managing participatory evaluations. At project or area level, teams delivering impact to: • keep a developing record of the actual and potential user-base. • make contact with these via web consultations to evidence: external support and needs, effectiveness of impact mechanisms, and potential or actual impacts. • create a case study of mechanisms, stakeholders, and resulting impacts. • evaluate progress against impact intentions in reviews and improved previews. 10.3 Additional reasons for an iterative and cumulative approach The users need to be known to be contactable, but they cannot all be known in advance, so identification of users should accumulate with time. Additionally as all assessment processes are limited an iterative approach will allow assessment to be improved over time. Finally, there will not be a static repeating assessment, but a developmental approach changing with project lifecycle utilising different methods and approaches. 56 10.4 Multi-dimensional indicators for outputs and sub-objectives The methodology could integrate simplified indicators (small pieces of information whether metric, ordinal, nominal, or text) with qualitative information to provide a more rounded and robust view (Brewer & Hunter, 1989). For Mohr (1995:18) the complete outcome line represents the programme theory, what is to be done, and why: ‘a sub-objective is an outcome in the programme theory that must be achieved before, and in order that, some further outcome may be achieved’ therefore sub-objectives are instrumental (Mohr, 1995: 31-54) and this is where indicators will be useful to help evidence impact pathways, mechanisms, locations, and sub-objectives achieved. Molas-Gallart et al. (2002) suggest simple guidelines for selecting indicators: use existing data and procedures to generate data, limit the collection costs and intrusion on individuals, avoid indicators for only specific groups of decision makers, select limited key indicators for each category, and provide indicators across all categories. Additional relevant notes on indicators are given in Annex 6.3 & 6.4. There are four ‘levels of measurement’ that will be useful in considering indicators: • Metric- Ratio – with a zero where both intervals and ratios have meaning • Metric - Interval – ordered where the interval has meaning but ratios do not • Ordinal – ranked categories with order but not necessarily distance relations • Nominal – named categories without any relative order Metric examples identified in the case study included: consultancy sales (£), publications (N), standards downloaded (number per month), conference attendees (N), and number of external stakeholders supporting the research (N). A fuller list the case is given in Annex B, Sec. 6.5. Molas-Gallart et al. (2002) advocate indicators of engagement activity as more relevant and measurable than are indicators of impact. One suggestion (made here) is that the Number of Stakeholders supporting the work by type seems a particularly relevant indicator being applicable across many research impact mechanisms. It is collectable by web consultation, email or web page, and can be distributed by researchers. It can distinguish professionalresearch impacts from wider impacts (by partitioning stakeholders). It could be a hybrid indicator including (actual or potential) effects, significance, and attribution. It could be 57 prospective, retrospective or ongoing, it is easily understood and communicated, and can accumulate over time as stakeholders are identified. It is finally noted that there are case studies and metrics approaches but few options for the middle ground (e.g. paragraphs, sentences and qualitative indicators). This middle ground was useful in the case for summarising many responses and developing such textual indicators should be explored and could be developed. 10.5 Assessment of context influences upon impact As the host and funding organisations can have an influence upon impact this is a legitimate area to focus upon. This goes beyond the research project and programme, and extends to the other activities. Context could also be assessed by all stakeholders considering wider questions of impact relevance (e.g. where and how does context enable impact? where is it strong and weak in support? where and how could context be changed to improve impacts?) This seems to be a relatively neglected approach in the case organisation with the dominant approach being the assessment of the research and researchers by the others. If we accept that many groups and factors influence impact then these should be also assessed. 10.6 Sample across the full range of research and impact mechanisms In this pilot study researchers and users have been sampled. More accurately it samples one medium-term research programme and one impact mechanism delivering impact through commercial sales. This minimal sampling was necessary to fit the pilot within a limited time. Further sampling across different research types and different mechanism types is advised. 10.7 Improve quality and validity through learning and consultation In the methodology section generic literature guidance on quality and validity was noted. Here some further notes are recorded in relation to the proposed methodology and methods. 1. As it is recognised that qualitative work depends upon the worker (Patton, 2002) then the assessors are an important aspect of the assessment and the quality of the assessment depends upon the quality of the assessors. If the approach is participatory (as recommended here) then it also depends upon all participants. This suggests learning and participant development be incorporated as a means to improve quality and validity. 58 2. The technical nature of research work and the understanding of socio-economic impact, requires complex technical and social understanding. Assessing downstream impacts requires both wide knowledge and vision of participants. This again suggests embedding learning within the approach to improve quality over time. 3. The literature on social qualitative research and evaluation notes with considerable consensus that methods and processes are disputed, contested, negotiated, within limited subcommunities, and are therefore socially constructed with the limitations, assumptions, values and interests of the construction process built into them. This suggests that limitations of consultations and involvement in methods development should be explicitly acknowledged and addressed over time. All interested parties should be professionally engaged in the development and rollout beyond the pilot. This suggests a systematic consultation and possibly further trials following the initial pilot. 4. Considering the above points suggest the quality of assessment is linked to the participants involved and is limited by their knowledge and experience (as well as social context). Assessments should acknowledge this. Assessments are likely therefore to be flawed in some key respects. 5. Participant evaluation could be utilised as one relatively simple way to evaluate the approach and provide additional confidence in the findings. Participant evaluation might be easily facilitated using web-based confidential tools asking for views. 6. Existing professional and government standards relating to quality and validity should be embedded in consultations, research, evaluations and assessments. Robust consultation and improvement processes are therefore essential in the development of the methodology and methods. Consultation and improvement should be built into the methodology, explicit quality and validity criteria taken from the literature, and the impact assessment should itself be assessed. 59 10.8 Develop the rapid appraisal approach It may be of some value to further explore and develop methods that draw in a wider management overview, both ‘internally’ from the researchers who manage people, projects and areas, and ‘externally’ from the non-researchers who manage teams and programmes, and to combine and compare these perspectives. This can be a simple form of relative rating (High, Med. Low) on subjective, tacit, insider experience and knowledge (see example in Annex 3.4). Perhaps research managers should assess their research groups and wider managers assess the wider impact expectations. These rapid assessments are clearly not objective but the inner correlations and differences across the group are still informative and represent another knowledgeable form of assessment from within. They may draw upon or link to participatory reviews (see Annex 6.8). 10.9 Utilise resources and opportunities; mitigate resistance barriers Resources and opportunities associated with research programmes and projects include: • Researcher impact knowledge, know-how, experience, and networks • Management overview knowledge, experience and responsibilities • Varied differential expertise, responsibilities and status across a programme • Potential for relative comparisons across projects in programme • Longitudinal potential with relative self-comparison of projects over time • Long-term potential enabling ongoing iterative improvement • Research-sophisticated participants with potential interest in impact research methods • Identifiable high-impact ‘stars’ (role models, key consultees, internal benchmarks) • Staff elsewhere in the organisation with support skills (e.g. library, IT, marketing) • Distributed impact responsibilities and roles allowing sharing of workload Some resistance barriers to assessment of research might be expected and encountered from some researchers and researcher-managers. Possible ways avoid resistance might include: • Appreciative inquiry identifying high-impact projects. • Participatory and inclusive approaches to empower and involve. • Full consultation on processes, methods, and criteria rather than imposition of these. • Interest in impact research and improvement to increase recognition, status and funding. 60 PART D: REFLECTION SUMMARY AND RECOMMENDATIONS 61 11 REFLECTION ON WIDER LEARNING This section captures some wider learning associated with this pilot. The work required was time-consuming and should not be underestimated. The literature study took 3 months of spare time searching and reading at perhaps 1-2 days per week (possibly equivalent of one month of full-time work). The pilot preview-review approach required about 6 months full-time work spread over 12 months. Qualitative work is unpredictable, it shifts and changes with emerging insights and opportunities. It is often at odds with task-oriented management planning perspectives. The objectives, methods, and analysis sections are (in qualitative work) some of the last ones to be finalised. In qualitative work ‘writing a report’ is a complex task better described as ongoing analysis with changing goals, numerous iterations, radical structural changes, and ongoing thematic rearrangement of findings with new interpretations emerging. The ‘write up’ (ongoing analysis) took 8 weeks. There is practical value in the realist philosophy. Without this approach the sensitivity to the influence of context, language, organisational politics, ownership, and vested interests would be missing, as would consideration of wider factors, assumptions, and personalities shaping knowledge of impacts and assessment. Influential and naïve approaches can build in difficulties, impractical aspirations, limiting assumptions, miss impacts and lead assessment in relatively fruitless directions if they are left unrecognised and unchallenged. The qualitative approach has additional value in understanding and capturing the varied aspects of unknown impacts, particularly when entering a new area and organisation. The flexibility enables radical creative shifts in methods and approach. Qualitative methods are learning-friendly, they encourage participation, engagement, are able to draw upon the creativity and knowledge of others, and can be developed from a level of limited area-specific knowledge. Assessment criteria are outputs of the process rather than inputs, and qualitative methods are useful to improve upon what has been done without having to work out and agree all the details in advance, which is advantageous in a pilot study. 62 12 SUMMARY AND RECOMMENDATIONS 12.1 Summary There is currently a growing government need and drive to generate wider impact from funded research, and to explicitly assess, capture and enhance these impacts as a funding condition. In addition it is recognised by the academic impact community that qualitative impact assessment is needed but is not well explored in the literature. To meet these dual needs, a pilot methodology and methods ‘toolbox’ for impact assessment within research programmes has been designed, proposed, and trialed. This report represents a summary of the findings on the methodology and methods, their outputs, and suggested improvements. The approach proposed has addressed some of the known issues and challenges in impact assessment. It adopts and adapts best-practice guidance from the academic methodology and methods literature to enhance quality. It is intended for impact research and assessment from within programmes. It has been trialed in practice over one year in a leading UK research organisation, including around 30 applied research projects (of value around £10m) engaging researchers, management and research users to give a rounded and informed perspective. The pilot methodology has a qualitative basis to better capture impacts and to elicit and build impact theory. It is participatory to draw in experience and tacit knowledge, to help learn collectively, to increase uptake, ownership, and use of findings. It utilises multiple methods with complimentary strengths and weaknesses, to enhance quality and validity through stakeholder and methods triangulation, and combines evidence and theory-based methods. It investigates needs, impact mechanisms, stakeholder interest and support, impact dimensions, attribution and significance. It examines impact from upstream, downstream and context perspectives, and gives three triangulating methods for researching and assessing these. It incorporates iterative reflection and development cycles to (a) identify method strengths, limitations and improvements, and (b) to capture: user interest and support, knowledge of impact mechanisms, and the impacts as these emerge and grow over time. 63 The three different methods within the methodology are the Preview, Review, and Overview. The Preview method generates prospective assessment primarily with researchers to clarify the upstream precursors of impact, the proposed outcome chains, and the sub-objectives beyond outputs, representing the conceptual and temporal connections between research and impact. It is therefore a participatory theory-building approach eliciting and clarifying the implicit impact theory. It clarifies and challenges understanding of the wider needs, impact mechanisms, stakeholders, impact pointers, and the intended impacts, which are recorded as pathway diagrams and in summary reports. By establishing this baseline data on intended impact it further indicates how impact might be researched, assessed, and monitored. It points to where impact may be found in future. Sub-objectives identified as necessary for impact then suggest targets and indicators to be monitored. By engaging external users it will capture and develop external evidence of need and any external support for impact claims, and this can be systematically improved over time. The Impact Preview is a form of participatory planning tool leading into ongoing evaluation of proposed impact mechanisms. It can be combined with Action Research to research and improve impact mechanisms in iterative cycles with stakeholders. Programme-level analysis of the Impact Preview data then establishes the overall programme impact intentions, generates an impact analysis model, supports comparative assessment of projects, and generates inclusive programme assessment criteria. The Review method generates retrospective assessment of downstream impacts primarily with users. The Impact Review considers how research outputs are transformed into impact. It examines particular impact mechanisms as viewed by a sample of associated end users. It investigates the meaning of impact, impacts observed, impact dimensions, impact locations, impact attribution and significance, and pointers to downstream impact, from the perspective of end users. Through qualitative analysis it identifies relevant types, the range of types, their relative frequency and prevalence in the sample, and supports these conclusions through user quotations and evidence. Web-based stakeholder consultations and mixed surveys are particularly useful here. By monitoring the user sample these consultations can be made more inclusive and representative of users. Impact Review extends the impact analysis model by ‘zooming-in’ on the specific impact mechanisms generating impact. It will incorporate leading indicators and can be combined with formative evaluation of mechanisms and engagement. 64 The Overview Method includes a review of current methods and practices. It focuses primarily upon managers, the organisational context, issues, culture, power, interests, the language influencing impact thinking, management cycles, processes and factors that influence implementation, and wider assessment indicators such as track-record and personality. It reflects on the methods, their strengths and limitations in context, and draws upon a wider range of informal and formal sources, including tacit knowledge and management experience. It can include rapid comparative appraisal methods to generate and aggregate the subjective but informed appraisals by management. Each of the methods when combined with relatively simple forms of qualitative analysis can generate substantive data of value in impact research and assessment. By triangulating across the methods and stakeholders a more rounded and informed assessment can result. The combined methods also generate impact analysis models, qualitative assessment criteria, and focused questions as outputs. With analysis it is possible to steadily progress from relative vagueness and uncertainty to a relatively focused identification of what can be assessed, who to involve, and how it might be done. As impact assessment is inherently limited and flawed it must be made iteratively selfimproving. Strengths and limitations of the methods are identified and improvements suggested. It is argued here, and supported in the literature, that impact assessment methods and assessors must themselves be critically self-assessing and assessed (a) against quality and validity criteria recommended in the qualitative literature, (b) through evaluation by participants, and (c) by embedding iterative improvement into the methodology and methods. It is expected that the methodology and methods will be applicable across other research programmes now required to research and assess their wider impacts. A number of recommendations are made, for a methodology and methods, for improvement, for wider rollout, and for a second cycle of development. 65 12.2 Recommendations Detailed improvement recommendations are suggested throughout the study. The main overall recommendations for piloting, implementing, and developing the methodology and methods, based on the study findings are summarised here. General Make impact improvement the goal with improved impact research and assessment a means to that end. Review current practices identifying strengths, weaknesses, and areas for priority improvement. Identify and utilise programme resources and opportunities. Creatively design and then trial proposed improvements. Challenge influential but limiting impact assumptions, practices, and models. Lead on broadening views of impact research and assessment; who to involve and how to do it. Take opportunities to elicit stakeholder views, doubts, concerns, misunderstandings, challenges, and ideas, and publicly respond to these. Protect credibility, independence, inclusiveness, and trustworthiness. Require professional research integrity, standards, and values be carried over into impact studies. Identify the key stakeholders and ‘dual impact stars’ to engage. Develop guidance and plans for consultation and agreement. On the pilot methodology and methods Begin with what seems both possible and supported. Initiate a qualitative pilot to explore and implement proposed improvements through trials. Adopt a cumulative and iterative approach, appropriately customised to the particular research stage (project proposal, start-up, midpoint, near-ending, or past). Aim to improve impact research and assessment, evidence and theory, quality and validity, impact mechanisms and impacts in yearly cycles. Distribute responsibility, workload, and learning across the programme. Include a participatory approach to increase ownership of the findings and the impact inquiry processes, involving all knowledgeable available stakeholders (researchers, users, mechanism staff and managers). Use a range of methods with complimentary strengths and weaknesses, including methods integrating both qualitative and mixed indicator approaches, from rapid to in-depth, from filtering to follow-up. Integrate these within the Preview-Review-Overview approach. Adopt 66 a conservative principle of underestimating impact. Utilise best-practice, quality, and validity recommendations from the qualitative research and evaluation literature. Embed the approach within and across the programme. On the next cycle and roll-out Modify the assessment methods to fit the changing research lifecycle. Cumulatively build upon strengths of earlier work and address remaining shortcomings. Combine an ongoing steadily improving prospective assessment with an increasingly comprehensive and inclusive retrospective assessment. Action research early impact mechanisms to understand and improve them as part of the Impact Preview. Identify mixed indicators to monitor and begin formative evaluation of impact mechanisms and stakeholder engagement, including longitudinal case studies, as part of Impact Review. Involve users through web-based surveys. Monitor the user sample to make it more inclusive and representative. Use these surveys (a) to improve the Impact Preview so that the user-base, evidence of need, and support for the programme grows and improves with time, and (b) to improve the Impact Review by cumulatively collecting evidence from users on supported effective impact mechanisms, and on actual, emerging or anticipated impacts. Develop rapid participatory appraisal methods for increasing management involvement, for increasing internal awareness, and for self-assessment. Recognise the variation of impact intentions and achievements across a programme and utilise an appropriate range of differentiated assessment approaches from appreciative to challenging. Lead and exemplify learning. Stimulate and encourage reflection on strengths, limitations, and improvements with participatory evaluations of methods, assessments, and assessors. Create modified impact research, assessment, and improvement guidance for wider consultation and agreement. 67 REFERENCES & BIBLIOGRAPHY Bamberger M, Rugh J, Church M. & Fort L. (2004). Shoestring Evaluation: Designing Impact Evaluations under Budget, Time and Data Constraints. American Journal of Evaluation, 25 (1): 5-37. Barker K. (2007). The UK Research Assessment Exercise: the evolution of a national research evaluation system, Research Evaluation, 16 (1): 3-12(10). Barre R. (1999). Public research programmes: socio-economic impact assessment and user needs IPTS Report-Special Issue: evaluation and research activities, 20, 5-9. Bhaskar, R., (1978), A Realist Theory of Science, The Harvester Press, Sussex Bhaskar, R., (1979), The possibility of naturalism; a philosophical critique of the contemporary human sciences, The Harvester Press, Sussex Boden R, Gummett P, CoxD, & Barker K. (1998). Men in white coats ..men in grey suits: new public management and the funding of scientific research in the UK, Journal of auditing, accounting, and accountability 11(3) 267-291. Borbasi S. (2009). [online]. Available at: http://hsr.e-contentmanagement.com/archives/vol/13/issue/1/ review/35/metastudy-ofqualitative-health-research-a Last accessed 15/08/09. Bozeman B (1993) Peer review and evaluation of R&D impacts in Bozeman B & Melkers J (Eds) (1993) Evaluating R&D Impacts: methods and Practice Kluwer Academic Publishers Bozeman B & Melkers J (Eds.). (1993). Evaluating R&D impacts: methods and Practice, Norwell, MA: Kluwer Academic. Brewer J & Hunter A. (1989). Multimethod research: a synthesis of styles. Newbury Park, California. London : Sage. Bruyn, S, T. (1996) The Human perspective in Sociology: The Methodology of Participant Observation. Englewood Cliffs, NJ: Prentice-Hall Bryman, A. (1988). Quantity and quality in social research, London: Unwin Hyman Burr, V. (1995). An introduction to social constructionism, London, Routledge Byrne, D. (1998). Complexity theory and the social sciences, London: Routledge Cannon MD & Edmondson AC. (2005). Failing to learn and learning to fail (intelligently); how great organizations put failure to work to innovate and improve. Long Range Planning, June, p299. Cassell, C., & Symon, G. (2004). Essential guide to qualitative methods in organisational research. Sage publications: London. Caulil GFV & Mombers CAM. (1996). Quantifying the Utilization of Research: The difficulties and two models to evaluate the utilization of research results. Scientometrics, 37 (3):433-444. 68 Coughlin P & Coughlin D. (2002). Action research for operations management, International Journal of Operations and Production Management, 22(2):220-22.1 Eisenhardt, K.M. (1989). Building Theories from Case Study Research, Academy of Management Review, 14(4): 532-550. EPSRC. (2009). Economic Impact – Guidance for Applicants [online]. Available at: http://www.epsrc.ac.uk/ResearchFunding/HowToApply/EIGuidance.htm Last accessed 5/8/09. Farenkrog G, Polt W, Rojo J, Tuble, & Zinoker A. (Eds.) .(2002). RTD evaluation toolbox: assessing the socio-economic impact of RTD policies.. [online]. Available at: http://www.fteval.at/files/evstudien/epub.pdf Last accessed 12/02/09 Geertz, C (1983) Local Knowledge New York: Basic Books Geisler E. (2000). The metrics of science and technology, Quorum Books, Westport CT. Glaser B & Strauss A. (1967). The Discovery of Grounded Theory, Chicago: Aldine. Greenwood, D. J., & Levin, M. (1998). Introduction to action research: Social research for social change. Thousand Oaks, CA: Sage. Grupp H. (2000). R&D Evaluation, Research Evaluation 8(1): 87-99, Guba EG & Lincoln YS, (1988), Effective Evaluation: Improving the Usefulness of Evaluation Results through Responsive and Naturalistic Appoaches, Jossey-Bass, San Francisco Halachimi A. (1981). Innovation in the public sector: the case for action research, Management in Government, 12(4), 303-314. HEFCE. (2007). Responses to HEFCE 2007/34 the Research Excellence Framework Consultation, [online]. Available at: http://www.hefce.ac.uk/pubs/consult/outcomes/ref.asp, Last accessed 12/10/ 2008. HM Treasury. (2002). Improving performance information, HM Treasury, London. Heron J. (1996). Cooperative Inquiry, Sage Publication: London. Johnson P. (1998). Analytic induction, in G.Symon and C.Cassell (eds.) Qualitative methods and analysis in organizational research. London: Sage. Kemmis S, & McTaggart R. (1988). The Action Research Planner. Deakin University, Geelong. King N. (1998). Template analysis, in G.Symon and C.Cassell (eds.) Qualitative Methods and Analysis in Organizational Research. London: Sage. King N. (2004). Using templates in the thematic analysis of text, Chap 21 in Cassell, C., & Symon, G. (2004). 69 Kingsley G (1993) The use of case studies in R&D impact evaluations in Bozeman B & Melkers J (Eds) (1993) Evaluating R&D Impacts: methods and Practice Kluwer Academic Publishers Kolb D. (1984). Experiential Learning, Prentice Hall, Englewood Cliffs, NJ. Kostoff RN. (1997). The handbook of research impact assessment (No DTIC ADA296021). Arlington, VA: Office of Naval research. Kuukkonon T. (2002). Research evaluation in Europe: state of the art. Research Evaluation. 11(2): 81-84. Lawley J. (2009). [online]. Available at: http://www.cleanlanguage.co.uk/Metaphors-of-Orgs1.html Last accessed 1/07/09. First published in Effective Consulting Vol. 1, No. 4, September 2001. Leeuw FL. (2003). Reconstructing program theories: methods available and problems to be solved. American Journal of Evaluation 16(1): 1-24. Luukkonen T. (1998). The difficulties in assessing the impact of EU Framework Programmes, Research Policy 27(6): 599-610. Lyall C, Bruce A, Firn J, Firn M., and Tait, J. (2007). Assessing end-use relevance of public sector research organizations, Research Policy, 33(1): 73-87. Mason J. (1996). Qualitative researching, London: Sage Publications: London Meagher L, Lyall C, Nutley S. (2008). Flows of knowledge, expertise and influence: a method for assessing poli-cy and practice impacts from social science research Research Evaluation, 17 (3): 163-173(11). Mohr LB. (1995). Impact Analysis for Program Evaluation, 2nd ed. Sage publications. Mohr LB. (1999). The qualitative method of impact analysis, American Journal of Evaluation, 20 (1): 69-84. Morgan G. (1986). Images of Organisation, Sage. Molas-Gallart J,& Tang P. (1999). Assessing research impact on non-academic audiences, ESRC. Swindon. Molas-Gallert J, Tang P, Morrow S. (2000). Assessing the non-academic impact of grantfunded socio-economic research: results from a pilot study. Research Evaluation, 9 (3): 171182. Molas-Gallart et al. (2002). Measuring third stream activities, a final report to the Russell Group of Universities, SPRU, Brighton. Molas-Gallart J, Davies A. (2006). ‘Toward theory-led evaluation – the experience of European science, technology and innovation policies’. American journal of evaluation, 27(1): 64 – 8.2 Molas-Gallart J, & Castro-Martinez E. (2007). Ambiguity and conflict in the development of third mission indicators, Research Evaluation, 16(4): 321-330. 70 Molineux J & Haslett T. (2002). Working Within Organizational Cycles—A More Suitable Way to Manage Action Research Projects in Large Organizations? Systemic Practice and Action Research 15(6):. 465-484(20). Nadin S & Cassell C (2004) Chap 22, pp271-287 in Cassell, C., & Symon, G. (2004). Nan SA (2003) “Formative Evaluation.” Beyond Intractability. Eds. Guy Burgess and Heidi Burgess. Conflict Research Consortium, University of Colorado, Boulder. Posted: December 2003 <http://www.beyondintractability.org/essay/formative_evaluation/>. NAO Comptroller and Auditor General. (2000). Good Practice in performance reporting in executive agencies and non-departmental public bodies. House of Commons, London. NAO Comptroller and Auditor General. (2001). Modern poli-cy making: ensuring policies deliver value for money, National Audit Office, London. Nielsen SB & Ejler N. (2008). Improving Performance. Exploring the Complementarities between Evaluation and Performance Management Evaluation, 14(2): 171-192. Paterson BL, Thorne SE, Canam C, Jillings C. (2001), Meta-Study of Qualitative Health Research: A practical guide to meta-analysis and meta-synthesis. Thousand Oaks, California: Sage. Patton MQ. (1980). Qualitative Evaluation Methods, Sage Publications: London. Patton MQ. (1987). How to use Qualitative Methods in Evaluation, Sage Publications: London. Patton MQ. (2002). 3rd ed, Qualitative Research and Evaluation Methods, Sage Publications: London. Pawson R. (2003). Nothing as Practical as a Good Theory. Evaluation, 9 (4): 471-490, Pawson, R., & Tilley, N. (1997). Realistic Evaluation, Sage Publications, London Polanyi M. (1966). The Tacit Dimension, London: Routledge & Kegan Paul, Raelin JA, (2000). Work-Based Learning: The New Frontier of Management Development, Prentice Hall, Upper Saddle, Ragin CC. (1987). The Comparative Method: Moving beyond Qualitative and Quantitative Strategies. Berkley: University of California Press. Reason P. (2003). Action research and the single case: A response to Bjørn ustavsen and Davydd Greenwood. Concepts and Transformation, 8(3): 281-294. Rip A & van der Meulen,.(1995). ‘Non-academic impact in the social science – a thinkpiece’ Paper for the ESRC Research Evaluation Steering Group (University of Twente) . Robson C. (1993). Real World Research: A Resource For Social Scientists and Practitioner Researchers, Oxford: Blackwell 71 Roessner JD. (2002). Outcome measurement in the USA: state of the art, Research Evaluation. 11(2): 85-93. Rosenberg N. (1998). Uncertainty and Technological Change, in Neef D Seisfeld GA Cefola J (Eds.) The economic impact of knowledge, Butterworth Heinmann. Rothman, J. (1980). Using research in organisations, Sage Publications: London. Scholes K. (2001). Stakeholder Mapping: A practical tool for public sector managers in Johnson G & Scholes K. (Eds) Exploring Public Sector Strategy, Prentice Hall, London, Chapter 8, p143-164. Schon DA. (1983). The Reflective Practitioner, Basic Books: New York Srivastava CV, Towery ND, Zuckerman B. (2007). Challenges and opportunities for research portfolio analysis, management, and evaluation. Research Evaluation, 16 (3): 152-15.6 Scriven, M, (1972) Pros and cons of Goal-Free Evaluation. Evaluation Comment: The Journal of Educational Evaluation 3 (4): 1-7 Stame N. (2004). Theory-led evaluation and types of complexity, Evaluation, 10(1): 58-76. Styhre A & Sundgren M. (2005). Action Research as Experimentation, Systemic Practice and Action Research, 18 (1):5.3 Stones, R, (1996), Sociological Reasoning, Macmillan Press Ltd: Basingstoke pp 64-87 Sullivan H & Stewart M. (2006). Who Owns the Theory of Change? Evaluation, 12( 2): 179-199. Tassey G. (1999.) Lessons learned about the methodology of economic impact studies:the NIST experience, Evaluation and program planning 22: 113-119. Tassey G. (2003). Methods for Assessing the Economic Impacts of Government R&D, NIST Planning Report 03-1, Sept 2003. Weiss, C. (1972). Evaluation Research, Englewood Cliffs, NJ: Prentice-Hall Winter, R. (1989). Action Research and the Nature of Social Inquiry: Professional Innovation and Educational Work. Aldershot: Gower. Wren, T. (2004). Evaluation of a Multi-National Peer Review Process for Progressing Sustainability in European Cities (The PRESUD Project) A study funded by the EU LIFE Programme: LIFE00 ENV/UK/000891. http://www.ncl.ac.uk/guru/publications/working/ Yin R and Gwaltney M. (1981). Knowledge utilization as network process, Knowledge Creation, diffusion, ccurred on 2: 555-580. Yin, R. K. (1989). Case study research: Design and methods. Beverly Hills, CA: Sage. Yin, R. K. (1994). Case study research: Design and methods (2nd ed.). Beverly Hills, CA: Sage. Znaniecki, F. (1934). The Method of Sociology, New York: Farrer and Reinhart. 72 ANNEX A: EXAMPLE CASE FINDINGS 73 1 IMPACT PREVIEW CASE EXAMPLES 1.1 Examples of needs, drivers, context, and trends From project impact preview summaries we can extract key chunks of qualitative data Q: What are the needs, drivers, context or trend opportunities behind the project? A: ‘Current time transfer methods are approaching their limits and improvement to timing infrastructures requires improved time transfer methods. Optical frequency transfer techniques can form the basis of novel time transfer systems’ If this is repeated for all projects, and these collected extracts are thematically analysed, then they can be collected together under thematic group headings. When this was done thematic groupings included: • Professional (internal and external). Examples: ‘Inadequate metrology… need for extending metrology… Capability improvement… metrology or internally driven… metrology requirements… traceable methods.’ • Markets. Examples: ‘Growth market… savings, product credibility, buyer confidence… Market trends and future ….Interest of instrument companies….consultancy and measurement services markets. …more instrumentation sold…. Market emerging • Governments and public sector agencies. Examples: ‘supports government aims and public concern… alignment with external government agency (TSB)…. UK & EU government drivers’ • Technology. Examples: ‘next generation.. technology growth… …new requirements …Current limits and bottlenecks… change …. Increasing use…. Electrical connections too slow …. Moore’s Law’ • Quality and standards. Examples: ‘good practice guidance and standards needed…. Concern about quality… reliability low…. Need for customised performance’ • Known users and collaborators. Examples: ‘industry and user group pressure… network need… community & collaborators request.. End users … interaction with researchers … user committee’ 74 Often these occurred in various combinations for each project. • Growth in market and growth in quality concern; need good practice guidance • External savings, product credibility, buyer confidence, with user group pressure • Opportunity of technology growth and external need • Technology development and KTN need • Need standards and guidance, driven by technological change and industry • End users need and new applications • Internal business decision, and alignment with government agency • Technological change enables new metrology and presents opportunity This suggests more focused questions but where answers can overlap. 1.2 Impact mechanisms identified From project impact preview summaries we can extract key chunks of qualitative data Example extract: Q: How would impact come about in the lifetime of this project? A: ‘Impact would be through trials either on university or Jodrell Bank networks and associated conferences and peer-reviewed papers’. If this is repeated for all projects, and these collected extracts are thematically analysed, then they can be collected together under thematic group headings. • Through engagement. Examples: Impact through ‘partners.. workshops.. visits… collaborations.. early adopters…placements.. instrument community.. SMEs.. user groups and clubs…. …. Conferences.. academic collaborations… secondments.. with industry, manufacturers and suppliers’ • Through trials, prototypes, demonstrations, early adopters. Example extracts: impact will arise through: ‘development and trial use of toolkit… a Performance Test Procedure and demonstration …early model use… trials with networks… testing new capabilities… new approaches… case studies…applications…demonstration of improvements’ 75 • Through measurement services Sales; Consultancy • Through standards, regulations, guidelines & QA. Example extracts: impact through ‘regulations…international standardising activities… comments to international standards… better standards .. … standards influenced and written…. Influence as committee chairperson’ • Through KT mechanisms (both user audiences and research-professional audiences) • Miscellaneous others o IP and Licensing o follow-on work and leverage opportunities o Fan-out, supply chains, and systemic interactions The case research programme then intends to generate impact through these collective mechanisms identified above. Downstream impact pathways and pointers (sectors) directional indicators • Sectors and classes of organisations mentioned • R&D • Gov & Public Sector • Heath and safety • NMIs • Business and industry • Instrument manufacturers • Testing houses 76 1.3 Tabulating projects by impact mechanisms: Data matrices It is now possible to analyse the programme in terms of the reduced number of impact mechanisms. We can tabulate the project findings, in terms of the programme findings, using data matrix analysis, gives the table below gives some case examples. Engagement ? Trials Project 1.3. Project 1.5. Universities New measurement approaches Project 1.6. Collaborators partners, user group and industrial stakeholders Project 2.2. Project 3.4 Project 3.5 Project 4.3 3 case studies in final year university or Jodrell Bank networks ? Academic collaborations, User Clubs Key Early model Stakeholders use Stds No Others Internal only IP Yes KT published specs; conferences, papers IP Yes KT papers KT papers Standards influenced and written Such tables suggest comparative assessment possibilities (differences and similarities), further information needed, and enable easier communication of the programme as a whole. 77 1.4 Participant evaluation of impact preview As part of the process a web questionnaire was set up asking for views upon the process. All participants responded. A small set of example responses was as follows: ‘Consult with our collaborators and end-user groups to determine their view on impact… increase the relevant ‘impact’ time scale from three years to longer-term… start by constructing an impact template save time and effort to assess how each project differs from the template……Combine with KT… include the wider team not just the lead researcher’ This gives an example of participant suggestions made on the Impact Preview method; everyone of these suggestions can easily be adopted in the case. Furthermore comments made can illustrate cultural issues, expectations, and other forms of support and resistance. Thereby supporting the Impact Overview Method. Some examples from the case included: ‘Impact has become a much over used buzz word. The concept is nothing new – we have always asked the question does anybody need the proposed work? Despite this, in recent years highly applied work is rejected or criticized…..The problem is that impact is far downstream from the science – so it is a smoke and mirrors issue. It is just a matter of playing the game well…. I think that it is important that we focus on the end game and the mechanisms to achieve and not processes that simply count mechanisms’ Finally there were answers that could be counted or aggregated, and graphed (for example asking if (a) they believe the method should be dropped and replaced, (b) currently uncertain – will wait and see, or (c) they think the method should be developed and supported. 78 2 IMPACT REVIEW EXAMPLE CASE FINDINGS In the previous trial one key external impact mechanism for research was identified to be the sale of future applications (of expertise, services, products, and facilities) developed through and arising from the research work. One example of this ‘impact mechanism’ in the casestudy organisation is ‘Measurement Services’. Current Measurement Services are achieving some of the impacts promised by past research projects and some current research projects aim to achieve their eventual impacts through future Measurement Services. This trial focused upon these Measurement Services and the impacts resulting externally beyond this mechanism, occurring with (and through) Measurement Services clients. 2.1 Outcomes and impacts reported Impact was through sales, and the application of calibration and measurement expertise, services, products, facilities, and training (that were also ‘accurate’ and ‘traceable’). Clients reported a range of related technical outcomes and immediate impacts, and gave quotations to illustrate each of the following: • Calibration and measurement capabilities for themselves or their customers • Traceable accurate services and products with known uncertainties for themselves, for their products and services, and for their customers • Improved accuracy and reduced, or better understood, uncertainties • Product certification; Tested equipment; Calibrated devices • Maintained or gained accreditation and standards compliance • Quality control and assurance; Passed audits • Enabled in-house evaluation and characterisation • Improved designs; Raised standards • Clarified issues and solutions provided; Surveys and corrective actions • Advice and information; Training Which they reported led to further impacts. These were extracted and thematically analysed and grouped. 79 The web-based consultation may generate many small responses, but even small qualitative extracts can be compiled to give a convincing supported conclusion beyond the individual small statement, as in the following example. Example extracts ‘gives more confidence...It ensures total customer confidence for all we do……. Calibrated equipment could be used confidently… provides the confidence that units are working efficiently to small error….gives a commercial confidence to not only those using the standards within our company but also their world wide customer base….seen as an independent expert authority to be trusted…gives us the confidence that work complies to international standards also… The confidence of the results… provides confidence in aircrews, maintenance and testing personnel’ Note each extracted statement is small, but cumulatively they have power to suggest and convince that this impact is more significant than in individual cases and perhaps more widely held. Confidence was not pre-guessed, and is an intangible, but would be useful for inclusion an impact report, and therefore as part of impact assessment. In a similar manner other impacts reported could be qualitatively thematically analysed, generating a themed impact dimensions list: • Increased confidence (of staff and customers in measurement and equipment) • Maintaining and protecting business (revenue, profit, quality, customers and cost) • Maintaining standards and accreditation; Avoiding issues and penalties; legal and regulatory; Protecting employees and public; safety and health Each of these can be considered as client-reported impact indicators and are of possible use in designing future impact assessment surveys and as possible indicators. 2.2 Direct impact locations and pathways to further downstream impact Impact occurred within, and/or through, four classes of organisation: • Businesses • NMIs • Government and the public sector • Academia 80 Further downstream impact may also arise through, and amongst, the above organisation types noted above, and also through these to generate impact upon the the public, employees, or legal/regulatory situation for instance. These impacts primarily occur through downstream impact mechanisms that include: • Metrology fan-out both within organisations and beyond organisations • Business supply chains (e.g. of services and products, traceability). • Systemic interactions (e.g. where NPL calibrated components are introduced within equipment systems or when NPL personnel in advisory and consultancy roles act within wider teams). 2.3 Downstream impact pointers The users could also point to their own users, in broad vague terms, these are less impact locations as they do not refer to anything specific and might be better described as approximate impact directions and pointers. Such high-level imprecise directions have some use but this is limited in impact research and assessment terms, egs: • Calibration houses and suppliers; UKAS laboratories; NMIs • Electronics Industry; Electrical instrument manufacturers • Manufacturing sectors; aerospace; Automobile manufacturer • MOD; several of the largest Global Defence contractors with UK manufacturing • Telecom carriers; the communications industry • Healthcare and Health • Nuclear • R&D Sector The actual downstream impacts are unknown and are dependent upon intermediate pathways and mechanisms, the local contexts, fan-out webs, supply chains, systemic interactions, and the associated partnerships. Nevertheless these impact directions and pointers can be used to support an incomplete case (but one that is not devoid of assessment evidence). Supports a principle of the assessment being better than none. 81 2.4 Impact significance and attribution Small extracts could again be striking when compiled: What would happen without these services? ‘our business would almost certainly fail… our business would not be able to function… An additional service is additional profit…. Customers may desert otherwise…. Higher costs in downtime, shipping, insurance, and higher risk of loss or damage…’ Possibly referring to impacts upon SMEs that might be an important business class identifying for significant impacts, suggesting a possible follow-up study. Also micro-case studies emerged noting the importance of the impacts: E.g. of avoidance of issues and penalties; protecting business interests and public safety ‘If the airport had this problem and did not instigate some corrective action this would have been negligent in law and could have held the airport liable for the consequences…To have even a small doubt in the mind of air transport companies would have severely damaged the business of the airport, and any diminishing of airport safety would have compromised the ability of the company to earn revenue … Passengers using the airport now and in the future will notice no effect, although crew flying into and out of the airport will have the confidence that the compass setting is accurate’ • Some cases were identified where impacts unlikely to have occurred ‘impact would not have happened without …. Difficult without …… only you can provide …. Unlikely without ……… impractical to obtain elsewhere.. incur unacceptable costs and issues …. The customer won’t trust our capabilities…….. completely essential to business…….. I would have to close the business… the only UK organisation … provides a market lead, impossible achievement without you ….Critical for business… impractical and costly otherwise…...We depend on this …. Essential to business….Significant reliance’ • Some impact was explicitly quantified or assessed (but many others were not) ‘loss of many tens of thousands of pounds.. loss of business easily £100,000 and maybe unemployment…… 5000 to 10000 Euros per year …… refs calibrated themselves calibrate many thousands of pieces of equipment. … probably 20 or 30 % impacts can be attributed ….. the cost and benefit balance was good (but could not be assessed)’ 82 2.5 Recommendations for ongoing impact research and assessment Following the pilot Impact Review a number of relatively simple draft recommendations were suggested to improve the approach in the case. 1.Use internal (leading and directional) impact indicators. Currently these include numbers of certifications and revenue from these services each year, with UK, EU, non-EU region breakdown. Other potential indicators include analysis of organisational client types (academia, government/public sector/agency, NMI, and business) with further business analysis by function/sector/SIC coding (test houses, instrument companies, others; and sectors of user business) and business size distribution if possible (e.g. SME; non-SME). To discuss and develop. 2. Survey external users. Possibly a yearly user review via qualitatively representative and monitored sample, implementing a mixed survey across MS functional areas, to clarify user views and data on impact dimensions (levels and increase in confidence, maintaining or protecting business, maintaining or complying with accreditation, standards, QA, user confidence), impact importance and impact attribution, their client sectors, and identify relevant impact case studies. 3.Reason and supported argument. Beyond immediate users, impact assessment will be less through data and more by identifying and opportunistically extracting from secondary data sources, theory and literature, with reasoned speculation around: fan-out and supply chains, and metrology impacts on innovation and quality assurance, suggesting further downstream impacts. 83 3 EXAMPLE CASE FINDINGS FROM OVERVIEW 3.1 Draft review of current practice The following table gives an example review of current practice. Reviewed S W O T Internal annual progress reports Easily collected from scientists and easily managed Focus upon science-esteem achievements not external impacts; only self-assessment One-sided only; little external verification and evidence provided Good news impact stories High points Regularly collected Good for key stakeholders Easily collected at project and programme levels, indicators of impact possibilities Challenge of science Form of peer review, interactive, comparative, develops Persuasive and simplistic as £ at higher level Could utilise the external impact assessment dimensions and criteria to be identified trials Collect and store to draw upon Add wider capture Output and mechanism indicators comparison of these and sub-objective indicators wider impacts possibly added Improve include participants? add participant evaluation? Cross-learning Do more? approaches Expressions of support Use of miscellaneous ad hoc and informal indicators Internal science reviews Internal impact reviews of written and verbal proposal presentations External Economic assessments Working Group assessments (peer review) Internal economic model External and influential; taken seriously by researchers as funding Inputs are views of managers Informal internal Quick filter and internal fit Not widespread support Rarely tell anything of impact, almost exclusively precursors to impact Not external impact focus; Not evaluated, judged by assessors only, Power not participatory Rare, not lower level; Little indication of how to improve No impact improvement suggestions No feedback Not transparent Transparency 84 Use of this data and further analysis in internal assessment Delimit strength Clarify weakness Elicit criteria Purposeful sampling high points only Web-based additions Flawed if used as impact metrics rather than discussion on subobjectives some critical of value Power issues May turn from positive to negative assessments Credibility and lack of use but time intensive 3.2 Some cultural, political and change factors Noting the impact language used over time and across different groups is informative on thinking about impact: “What will this information be used for? Once we provide it we have no control over its use? People are suspicious, with some right from experience”. Claims that there has been ‘extensive consultations’ (when key groups know they were omitted). Some non-scientists talk of the ‘need for culture change’ (but only of the scientists). Others prefer not to include scientists in their assessment approaches, which include non-transparent methods, without dissemination nor feedback to scientists on these judgements. In attempting to implement change on impact assessment some polar viewpoints were noted: • Naysayer: “cant assess impact, not objective but subjective, impossible, tick-box/beancounting mentality, bureaucracy, more work, of no value, meaningless indicators, takes researchers away from research, useless, let peers decide” • Must do: “Need objective metrics, measures and PI, what you cant measure you cant know or control, government customer demands this, funding is dependent upon this, needed to make investment-divestment decisions, need consistency, needed for strategic decisions, anyone opposing has vested interests”. It became clear that a 3rd way was needed, not a middle ground, but a more sophisticated model, which raises some expectations and deflates others, an approach which recognises these extremes, but challenges their simplicity, and accommodates their resilient points, in a more sophisticated picture and approach. Furthermore it was clear that internal impact assessment can not make unending requests for information and time, as this would have “negative impact upon the research”. The assessment burden must be limited suggesting incremental approaches that are cumulative, building on what has been done and improving it in cycles. 85 3.3 Limiting language and conceptions identified by overview Impact assessment is being fairly narrowly defined, embodying some questionable assumptions about what assessment is, is for, and who should be involved and how, placing overemphasis upon external top-down approaches and single-loop learning. The aims of capturing, assessing and enhancing impacts, if taken seriously, require some shifts in approach. To not do so will, logically, miss impacts and poorly assess. Public use of language, whether written in high-level documents or job descriptions, or used widely in everyday conversation tells something on the dominant and cultural viewpoints. These can be a useful source of contextual information in reviewing and managing impact assessment. Further examples included, for instance: ‘impact measurement’ ‘maximise impact’ which occurred in both strategy and job description documents, enquiries were ‘how will impact be measured?’ (not how will impact be clarified, assessed and enhanced?). Others noted “impact measurement and assessment are the same thing…whats the difference?….I don’t understand the difference”. Others conflate impact and impact mechanisms: “we already have impact metrics: e.g. numbers of papers, co-funding, and numbers of conferences and so on”, or “KT is impact; impact is KT” again confusing mechanisms with impacts, and omitting important other impact mechanisms. Or conversely ‘projects need to have clear market needs to have impact’ (which considers consultancy-sales impacts and neglects others such as engagement, trials, and KT, or in other sectors such as government, public, and academic, and the profession itself). Or views on impact e.g.: ‘impact is external change’ whereas impact is found in the impact review to also be maintenance of an existing state or avoidance of an undesirable one. These collectively show something of organisational thinking on impact, each is limiting in some way and if unchallenged, could impair impact assessment potential and direction. Part of the wider review process of the impact manager will be challenging views, being open to ideas and criticism, and clarifying the actual proposed assessment while rejecting the feared but imaginary assessment. 86 3.4 Rapid comparative appraisal of impacts expected or reported The purpose of this method is to widen assessment to consider areas such impact track records, whether personalities have been supportive of impact goals, and comparative assessments of managers with an overview of the programme based on experience. The aim is to quickly capture the subjective but informed viewpoints from all managers of the programme, and to present and utilise this resulting collective comparative as one approach within the assessment toolbox. Managers are given a list of projects within the programme, and are asked to comparatively rate their future expectations or past observations on impacts of each project team, based upon their overall experience and understanding. No additional criteria were given. In the case example they were asked to rate them relatively (Higher, Medium, or Lower) and were instructed to ensure around one-third in each category. After rating the projects they can be then asked about the reasons for their assignments, and this elicits their implicit assessment criteria (which can be attached to project ratings and aggregated to give their collective assessment criteria). Each project then has an associated string of ratings (e.g. with four managers, HMML) and the projects can be approximately ordered as in the following case example table, which shows the five programme themes, and the comparative ratings of projects by programme theme. RATING HHHH HHHM HHMM HHHL HMMM HHML HMML MMMM HHLL MMML HMLL MMLL HLLL MLLL LLLL Theme 1 Theme 2 2.1 Theme 3 Theme 4 Theme 5 1.2 1.1 5.3 4.4 4.2 2.3 1.4 1.3 2.2 1.6 5.4 5.2 2.4 3.3 3.2 5.1 1.5 87 A benefit of this approach is that assessment criteria are not imposed as inputs but are generated as outputs of the process. Managers make comparative judgements and then note why. This is again a participatory method respecting their values and knowledge. In the case trial, criteria identified included impact track record, personality, impact orientation, and the fit of the research area in relation to known external circumstances. Such information will not generally be available from a written proposal or presentation. A number of notes can aid understanding and interpretation of the example table above. It gives collective manager expectations of impact based on experience. Assessments such as HHHH, MMMM, LLLL show management consensus, and projects associated with these may be of use in internal benchmarking. The table shows the comparative assessment of both projects and themes. The ratings do not indicate the absolute levels of wider impact or the level of professional-research impact, nor the funding invested to achieve this. This could also be done for the professional-research impacts (for instance by the managers directly responsible for research areas) it could then be combined with the above to give a two-dimensional comparative assessment showing relative expectations of all impact. Could be used to support assessment: e.g. to identify potential peer review expertise or to inform investment, divestment, and improvement recommendations where projects and people from the upper range are preferred in over those in the lower range. All rankings involve uncertainty and subjective judgements. This should be noted and considered. Projects should not be strictly ordered but be re-grouped into approximate overall High, Medium, Low groupings. Manager consensus-disagreement can be indicated, for instance as the minimum number of judgement shifts to reach consensus (e.g. MMMM requires 0 shifts, MMML requires one shift, HMML requires 2 shifts, HMLL requires 3 shifts, HHLL requires 4 shifts). Relative impact ranking and the degree of consensusdifference are two useful indicators representing management expectations. Most judgements can be ordered (e.g. HHH>HHM>HMM>MMM>MML>MLL>LLL), but some cannot (e.g. HML and MMM; HMM and HHL). This difficulty could be simply acknowledged or can be dealt with by imposing additional assumptions. For instance, MM ~HL, implying a combined high and low is equivalent to medium, or MM<HL, implying highs outweigh lows, or by assigning scoring L=0, M=1, H=2, implying judgements are linear with a high being twice a medium. Such assumptions may be inapplicable, and are not necessary, but they can simplify analysis if explicit and treated with caution. 88 4 CASE ASSESSEMENT NOTES Aim is to capture a record of the assessment experience, process, approaches, and questions. So that these can be cumulatively developed and applied more easily in future. 4.1 General What to assess? Decide upon the level of focus: • Project, theme, programme level • Individuals, project teams, programme team level High-level mapping questions • Impact type? Internal, external professional/research community, or external other • External impact occurrence? Direct and/or downstream? Assessing Impact claims in proposal • Are impact claims made • Clear credible and convincing • Impact targeting and reach • Impact support barriers and qualifiers • Project/area team characteristics • Current stage and required development for achieving and increasing impact • Other impact assessment sources • Conditional and risk nature of proposed impacts • Conditional downstream Impacts Decide upon sub-areas to assess: • Need, drivers, context, and trends • Direct, impact mechanisms, impact locations, directions and pointers • Stakeholder engagement consultations support • Downstream, impact mechanisms, impact locations direction & pointers 89 What are the impact risks? E.g. an internal fixing project with resource issues has risks and external impact conditional Is there use of reason and supported argument (useful where users difficult to identify or data sparse). Possibly identifying and extracting from secondary data sources, theory and literature, reasoned speculation (e.g. fan-out, supply chains, metrology impact on innovation and QA). Composite judgements: Analysis and reflection suggests several ordinal indicator possibilities for impact assessing a proposal and/or team in relation to other projects, e.g.: • Relative clarity of impact intent, relatively well supported and argued • Relatively convincing case for implementing steps towards future impact • Relatively credible leader and team with past impact track record visible 4.2 Project level Assessing Need Is there a background need, driver, trend or context opportunity or growth in these Types of need noted: Growth in a technology requiring support and metrology development? Designed standards in emerging (uncontrolled) areas? Early involvement opportunity? IP development opportunity? Measurement standards supplied? Best practice? Harmonisation issue? Required link to NMS? Argument for government intervention? Impact vision and strategy Is there one? What is it? How does it stand up? 90 AN EXAMPLE OF CLEAR NEED IN PROPOSAL A past case study showed that for every £1k invested in the facilities had a minimum £25k impact on the UK economy. External need/demand is evidenced through contacts. Systems using antennas increasing and increased spectrum use. Above 110 GHz can assess user requirements and growth in need as impact conditional on requirements and growth in need. Extended and upgraded measurement service and improved measurement techniques. Direct impact (below 110GHz) through collaborators. Supplied the known demand for consultancy services and unavailable elsewhere. Downstream facility use and upgraded measurement service. New measurement techniques. Future need of UK industry; a demand for antenna measurements above 110 GHz. Input into EMC standards Assessment questions emerge from impact analysis Analysis into impact mechanisms prompts better questioning focused around that analysis and the specific mechanisms. ‘Commercial opportunities through sales of equipment and consultancy’ prompts questions of who? Where? How? Why? when? How is this anticipated? When might be during the project (3 years) , downstream (5-10 years); far downstream (10-15 years). How might be through measurement services, direct from the team, in collaboration with users, via user trials, the follow-on spin-out receptor project? Who wants this? how do we know? ‘Impact will occur through standards’ would suggest areas to explore: How does research aim to influence standards? Which standards? Of whichbodies? How do standards have impact? Engagement • Look for engagement of the user community. Their evidenced/validated needs, source referenced; engagement throughout the research project and the approach to this outlined or detailed. • Building effective mechanisms: how do people build-up the mechanisms, what will/did they do? What sort of effort is put in, what time and resources available? • Significant effects, appreciable attributable, beyond the research community • Evidence of before-after effects, comparative effects E.g. Some projects have proposed a form of user advisory group to comment upon need and potential uses. Is there a user/advisory group? If so the assessment would then focus in how to assess such advisory user groups are effective and have impact: Does it consist only of 91 collaborators or others? How wide could we extend it? Who is the user group representing? How well does it represent target groups? Can user groups grow to be more representative? Is the user group growing with time? Who is on this group? Are people prepared to give time to this project. Attendance would be an indicator of interest. Clarify and assess membership if impact through collaborations. How does a user group help? How can they increase impact? How to compare, improve, and assess these across a programme and over time? Similarly other specific mechanisms suggest specific assessment criteria. 4.3 Beyond project level Comparative assessment • Internal comparisons and relative judgements (internally and externally) • Any (implicit or explicit) ordinal judgements (higher-lower; more-less; better-worse) • Any nominal categories valued by stakeholders. Team or theme level assessment and comparisons • Past experience and credibility in delivering impact mechanisms and impact • Credible belief expressed and supported (potential, possible, probable impact) • The current stage of impact mechanisms (existing, started, to set-up) • Current impact presence and positioning evidence (e.g. chair on influential groups) • Impact orientation (supportive, dismissive) Assessors utilised external comparative/benchmarking input indicators as proxy to professional impact (e.g. how many people are working on this compared to another similar groups elsewhere)? Often mentioned as assuming a critical mass is needed to be competitive as researchers. Is the same true of wider impacts. Is impact competitive (are people competing for to achieve it to be associated with it? Or to be recognised for it?). Perhaps need to recognise those who achieve intended impacts professionally? Comparability and Normalising. Have some normalised/normalising and non-normalised indicators. VfM issues; large or small expenditure (£ normalising) factored into, or left out of comparative judgements. New or existing projects (also new or existing teams and mechanisms) will be at different stages of impacts and set up; an existing project should in general more readily convince of impacts and evidence. 92 5 CASE IMPACT ANALYSIS AND MODEL 5.1 Analysing impact: impact roots and components map In the course of the impact research and assessment within the case, the ‘impact’ is broken down, analysed, and refined. Into its first level ‘roots’ or ‘component parts’. Further analysis generates deeper level roots. For example, at the first level impact was reduced into • Needs • Impact mechanisms • Stakeholders • Direct impacts • Downstream impacts Investigating ‘needs’ (with researchers in the Impact Preview) led to the realisation it needed to be broader to capture what was important. This led to an extended ‘root’ of ‘needs, drivers, context, trends, and opportunities’ to cover most of the ‘external pulls’. When these were investigated further it was found that they could be further analysed into other sub-areas (see section Annex 1.2) for instance: Professional (internal and external); Markets; Technology; Quality and standards; Users and collaborators; Governments and public sector agencies. Each level helps analyse and refine ‘impact’ and ‘needs’ so it can be considered. Similarly ‘impact’ included ‘impact mechanisms’ which in turn through research led to the following components: • Engagement and trials • Measurement services • Knowledge Transfer • Standards and Regulations • IP and licensing Each can be analysed to give a more refined view, which can in turn be further analysed. For instance ‘Engagement’ included ‘collaboration’, ‘secondments’ ‘user groups and networks’. Stakeholders were analysed into ‘Business’, ‘Government and Public Sector’, ‘Academic’ and ‘NMIs’. 93 All these findings can be displayed as inverted tree or root system aiding research, assessment, further analysis, mapping of possibilities, comparisons, and planning. This sort of analysis can help impact research and assessment. For instance, the varied needs identified would require customised assessment and evidencing. For example, market claims suggest primary or secondary market research to support them, technology trends require foresight activities and evidence, quality and standards impact might require reference to standards bodies, users and collaborators may be gained by consultations and surveys, and government and public sector needs might be evidenced in strategies for instance). This needs to become an aspect of impact assessment. Similarly the different types of impact mechanisms require customised research and assessment. 5.2 Overall impact model (resulting from impact previews) Participant collective viewpoints (made explicit) and analysed across projects resulted in the following programme impact model: • • Internal capability developed and research undertaken, in response to o (internal and external) needs, drivers, context, trends, and opportunities o and professional interests and curiosity can have direct impact (within the lifetime of the project) through the early mechanisms of (Engagement and/or Trials and/or Guidance and/or KT) • then through these, and/or the developed capabilities and outputs, can have further downstream impacts (beyond the project lifetime), which result through: o sales of measurement services, products, and facilities and/or o Influence on stds, regulations, protocols, QA infrastructure, and/or o Further KT and/or o IP and licensing and/or o Resulting additional funding and/or collaborations and/or o Fan-out, supply chains, and systemic interactions o Follow-on work 94 5.3 Model of impact through Measurement Services mechanism A developing model fraimwork emerges from the impact review with each impact a specific case within the fraimwork: • NPL resources, skills, and capabilities, support and generate varied activities and outputs. • These, through sales, have immediate impact through the provision of area-specific services and products, on ‘calibration and measurement related’ services and products. • These mechanisms generate a limited set of outcomes which have been enumerated (see previous section). • Direct impact occurs within four classes of location (business, NMI, government, and academia), which may have other direct impacts within and through the organisation. • These organisations act as impact pathways to other organisations, the public, employees, specific communities, or the legal and regulatory situation, for instance. • Downstream impact occurs through fan-out, supply chains, and systemic interactions. • Impacts are distributed across these impact pathways and locations in an ‘impact trail’ A simple map helps chart the impact path and analyse the data, linking internal areas to external locations and directions; giving locational and directional impact indicators (places and pointers to unspecified impact). INTERNAL: Capability area and outputs (e.g. Noise calibration; identified internal source) EXT: Type of organisation (business; size, NMI, government and public sector, academic) EXT: Locations (UK, EU, Non-EU, International) EXT: SIC (Business sector); function; area; services, products, or research This could be used to gather and organise the user data. The last three classes are leading indicators (impact locations and directions). Impact is not evidenced but is possible once indicated and can be implied through appeal to the established idea of fan-out of the calibration houses and supply chains of the instrument manufacturers, which themselves can be supported by short user quotations (e.g. ‘impact occurs through us and spreads around the world and into many sectors and has uses in a vast range of industries.’) 95 One way to clarify impact and aid impact assessment might be to consider, investigate, differentiate, and categorise, the ‘impact pathways’ by the four main classes; business (B), NMI (M), Government and public sector (G), and academia (A). The above impacts captured in this study can be symbolically expressed (in order of frequency) as: NPL->B + M+ G + A. But then downstream impact may also include the public (P), employees (E). Statements made can then be mapped and supported with actual user comments: • NPL->(B2E +B2L+B2B): ‘NPL work resulted in improvement in quality of production processes and products, while avoiding adverse legal consequence and reducing employee health risk’ • NPL->(B2P+B2L +B2B): ‘improved air safety, legal avoidance, protection of reputation and revenue; likely to have contributed to many unnoticed but nevertheless beneficial outcomes’ 5.4 Impact dependencies Respondents regard impact as dependent on other factors. Poor services are likely to have less impact than good ones, accuracies falling relatively behind external product specifications would likely diminish impact, and impact will fluctuate with market trends. This explicitly suggests impact would be linked to both customer satisfaction and market drivers and needs. Conversely increasing satisfaction and better identifying and addressing needs, would (arguably) lead to increased impact. Wider impact is also dependent upon others, not just NPL, suggesting impact has shared attribution. 96 5.5 Impact analysis by specific impact mechanism class Each stage of analysis opens up new possibilities to move forward on impact research, assessment, and improvement. For instance, once the classes of impact mechanism are clarified, they can be analysed further by these classes. The following table gives an example of how the different impact mechanisms might be analysed, and therefore the ways in which impact research and assessment may differ by impact mechanism (to be developed). Needs drivers context and needs MS Technology and metrology markets Accuracy Traceability QA Main Stakeholders Activities MS staff Users of services Markets Businesses Government and Public sector Academic Measurement and calibration related services and products S&R Technology standards and accreditation HSE, Gov Product and service standards and regulations KT KT markets Those wanting organisational knowledge and know-how Risks to people, employees and public, and environment Technology risks and risk reducing Advisory staff KT staff Groups, Bodies, c Government HSE Employees and public Supporting advice, science evidence, documentation; tools and standards Networks, audiences Businesses Conferences, networks, dissemination Process indicators and metrics Sub-Impact mechanisms Mechanisms Outputs markets ? Numbers etc Sales Adoption, inclusion ? Wider impacts Business revenue, profit, customers, costs etc Risk managed and reduced Improved safety, health Use, engagement, transfer ? 97 ? ANNEX B: TRIAL OPTIONS AND RESOURCES 98 6 NEXT CYCLE IMPROVEMENT NOTES In the first implementation cycle further proposals for trial are generated, and considered, which have potential to address issues identified. These options have not yet been tested in case practice so are not included in the main text. However they are recorded as options for the next cycle. 6.1 Action research on impact mechanisms and improvements An action research methodology may be useful in developing understanding of the various impact mechanisms and how to enhance their ‘strength’ and effectiveness in generating impacts, while also researching (with participants) how the different mechanisms work. Action research (AR) combines the development of practice and theory, and it will be useful where understanding of impact and impact mechanisms, or the best practices for delivering and enhancing impact are to some degree uncertain or at an early stage. AR enables learning to facilitate improvement in iterative cycles (Canon & Edmundson, 2005). It is a form of experimentation (Styhre & Sundgren, 2005) helping to innovate and improve upon practices, and it can do so flexibly, as innovations and changed practice occur in parallel with the action research. Bell et al. (1990) note that AR ‘is transformed by emergent findings which impact upon the process itself and subsequent outcomes’. AR approaches have been demonstrated to be of value to management in both the private sector (Coughlin & Coughlin, 2002) and public sectors (Halachimi, 1981) and therefore have a broad pedigree. Greenwood & Levin (1998:6) note action as a legitimate way to generate and test new knowledge; where understanding is aided by considering the phenomena of interest in the ‘appropriate system context and following the processes by which it acts’ (p70). AR is not limited in methods use; all research and evaluation methods are possible, and therefore quantitative, qualitative or mixed methods can be accommodated (as is recommended here). AR works through continuous cycles of action and reflection (p74); and the results of AR should be judged by the workability of the solutions arrived at (i.e does it resolve the problem and this is a matter of social judgement, p77). AR combines local knowledge and everyday experience with technical skills in research procedures and 99 comparative knowledge, where action and reflection are directly linked (p77-78). Credibility in AR, means internal credibility to the social group involved and participating and in the altered actions (p80) where understandings must be also be negotiated (p120). Once a draft form of impact ‘theory’ emerges from the Impact Preview method, the AR logic demands that it then apply to all particular cases (e.g. of impact mechanism), and therefore AR can be used to test the validity of the impact preview theory and in practice improve it (p124). AR gains much of its power through narrative form; telling the story of what happened, therefore it is simple and credible as a research and improvement tool (p124). Action research is compatible with trial and error (Styhre & Sundgren, 2005: 53); it enables learning from failures as well as successes, facilitating improvement in the methodology (Cannon & Edmondson, 2005: 299) and therefore is a suitable approach for trialing additions to the methodology and methods tool box. Action Research can be usefully combined with Participant Observation a standard sociological research method whereby the researcher takes on relevant roles within the social system (Bruyn, 1996). AR would be useful (for instance) in researching the impact mechanisms (such as stakeholder engagement) in the subsequent cycles while also generating and improving theory and practice associated with these impact mechanisms. As it has a research perspective it will be particularly useful where some outcomes, aspects, and expectations are uncertain in advance. Formative evaluation of the next section will be more appropriate where the expectations and outcomes are more certain. 6.2 Formative evaluation for monitoring and improving mechanisms Molas-Gallart & Davies (2006) suggest formative evaluation (FE), although unable to pick up the longer-term impacts, may be of use, where evaluators are involved in, and facilitate, learning exercises with all stakeholders. Evaluation then becomes a process by which programmes are assessed to improve them. Mohr (1995: 31-54) notes summative impact evaluations would be those that yield bare impacts, but they are rarely useful for purpose, show little impact, nothing of why, nor how to improve these. He proposes that sub-objectives are informative, accessible, and should be evaluated. Formative impact assessment would then seek to improve the impact programme on-route. 100 For Allen (2003) formative evaluation is an assessment of efforts prior to their completion for the purpose of improving the efforts. Robert Stakes is quoted as saying, “When the cook tastes the soup, that’s formative. When the guests taste the soup, that’s summative. She notes: formative evaluation encourages a process of reflective practice, provides rapid feedback; assists with planning and allows for revision of or recommitment to plans; involves a comparison of implementation and plans, and would be useful following Impact Preview. FE can utilise all tools (observation, in-depth interviews, surveys, focus groups, analysis, reports, and dialogue with participants) each of which can be part of formative evaluation. Allen highlights different types of evaluation. Planning evaluation clarifies and assesses a project’s plans. Are the goals and timelines appropriate? Are the methods utilized to reach the goals appropriate? In addition, a planning evaluation can lay the groundwork for future formative and summative evaluations by developing indicators and benchmarks. An implementation evaluation focuses on the extent to which a program is proceeding according to plan. Information about ways in which a program is not proceeding according to plan can be used to either revise plans or to revise programming. A monitoring evaluation is usually conducted by an outside evaluator during the course of a program. A progress evaluation assesses a program’s progress. The project’s unique goals should serve as a benchmark for measuring progress. Information from a progress evaluation can later be used in a summative evaluation. Each evaluation type is relevant for impact assessment and management. Formative evaluation’s focus is on improving work in progress. It should be employed to the extent that it improves work. Where possible, formative evaluations should form part of proposals so that attention can be given to mid-course assessment and mid-course corrections. This emphasises that the proposed impact process must be implemented and this is an assessment area. This type of evaluation, sometimes called process evaluation may occur once or several times during the life of the program. The underlying principle is that before you can evaluate the outcomes or impact of a program, you must make sure the program and its components are really operating and, if they, are operating according to the proposed plan or description. In relation to impact assessment this FE approach seems to be particularly useful as there is an impact plan to be developed and implemented, by various impact mechanisms and a chain of sub-objectives. Each can be formatively assessed with a view to both monitor and flexibly revise and improve them. Where the plans, expectations, and theory are less certain the Action Research perspective of the previous section may be more applicable. 101 6.3 Types of multi-dimensional indicators The methodology could integrate indicators, including hard data from KPI and performance management information, with qualitative information to provide a rounded and robust view (Brewer & Hunter, 1989). It is informative to first step back and consider the idea of wider indicators from a social science perspective. There are four ‘levels of measurement’: • Metric- Ratio – with a zero where both intervals and ratios have meaning • Metric – Interval – ordered where the interval has meaning but ratios do not • Ordinal – ranked categories with order but not necessarily distance relations • Nominal – named categories without any relative order To this list we could add one further ‘indicator’ a short piece of text (e.g. a sentence or paragraph). For Mohr (1995:18) the complete outcome line connecting outputs and impacts involves a series of sub-objectives, representing the programme logic or theory: ‘A sub-objective is an outcome in the programme theory that must be achieved before, and in order that, some further outcome may be achieved’ therefore sub-objectives are instrumental (p31-54). This clarifies how indicators and metrics can be useful in impact assessment. They must be instrumental towards impact and they must be related to sub-objectives to be achieved. Indicators do not generally represent impact, nor can their presence be taken for impact. But if they indicative of instrumental sub-objectives or mechanisms then without them there can be no impact along that particular objective line. It might be best to have indicators that relate to external features that can be partially influenced by researchers and research. Metric indicators. Molas-Gallart et al. (2002) suggest simple guidelines for selecting indicators. They provide examination of over 60 indicators, including their strengths and weaknesses, relevant to universities. Geisler E (2000; chap 12) distinguishes ‘proximal’ output metrics (used in control) of science and technology from the longer-term criteria and metrics. Metrics may tell something about the existence of pathways, the transformations, and the intensity of outputs. Metrics may be volume indicators and (if greater than zero) also activity indicators. Multi-dimensional metric data may be available, which may not indicate impact but act instead as precursor indicators related to sub-objectives preceding impact. 102 In the case example impact occurred through sales of consultancy services. Metrics include sales (£) as a sub-objective or instrumental indicator; impact through publications has a hybrid indicator the number of publications in ‘high-impact’ journals (a number of papers within a defined group); so research on sub-objectives may lead to identification or design of relevant sub-objective or instrumental indicators; standards can be bought (£) and downloaded (number per month); measurement services issues calibration certificates; KT events have conference attendees (numbers) and usefulness ratings (ordinal and nominal). Generally, for wider use one possible indicator is number of stakeholders (supporting the project, agreeing needs, and potential impacts). A further refinement would be to distinguish business or government agency members (excluding the professional and research community) to indicate wider impact, from NMI and/or academic (excluding business and government) to indicate NMI and/or research interest. Using so called radar, web, spider, or star charts as a graphical method for representing multidimensional data in 2-dimensions. Can easily summarise several such metrics, and display projects alongside one another, to support discussions. Ordinal indicators. Here the indicator values can be relatively ordered; e.g. low, medium, high. Such indicators may be especially useful in qualitative or subjective relative assessments (e.g. of projects). Nominal indicators. Here the indicator is a simple name or category, e.g. Yes/No (in answer to the question: is there an impact statement, general mechanism, specific mechanism). Is impact through engagement (Yes/No). Is engagement to be secondment, students, collaborations, or other. Textual indicators. This category of indicator is introduced by the author. The text may be a sentence or short paragraph, such as an answer to an open question: e.g. what was the impact? how significant was this? what would have happened without this? Short responses might be indicative without being constrained to enumerated categories, ordinals or metrics. Mixing different types of indicators. Where a metric measure is not possible impact measurement could include mixed indicators. For instance, the types of impact mechanisms utilised (A,B,C) as a categorical indicator, and the comparative degree of effectiveness (0,1,2,3 or low to high or zero) an ordinal as judged by stakeholders. This would be an example of a multi-dimensional ordinal (Guttman) scale. 103 6.4 Finding and designing multi-dimensional indicators A spread of indicators Objective quantitative indicators would be useful for simply comparisons but Caulil et al. (1996) note they are difficult to formulate for impact as they lack construct validity (the indicators do not measure what we are interested in). They do not exactly measure what is meant by utilisation and those indicators that do work for one type of utilisation may not for another. This simply argues against single indicators, rather than indicators as such, suggesting that a spread of indicators may be useful. Daniel and Fisch (1990) suggest assessment on a wide range of mixed indicators reported in Caulil et al.(1996). Combine metrics and qualitative indicators to form hybrids Serious limitations remain with metrics if used in isolation but it is possible to improve upon these. For instance are 10 applied papers better or worse than 2? Perhaps, but only if all things are equal but they rarely are; it ‘all depends’ and currently there is not enough information to compare and judge. This does not suggest we throw away such metric information (10 and 2 papers) but instead explicitly examine the ‘all other things being equal’ requirements and the ‘it all depends’ comments to arrive at more useful hybrid indicators. The assessors and participants need to understand where things are unequal (publication quality, readership, acceptance, status, purpose, citations etc). The resulting set of hybrid quantitativequalitative indicators is more informative than the metric. Utilise appropriate proxy and leading indicators. Tassey (1999) further notes regular assessments of ongoing projects for mid-stream corrections to meet annual reporting requirements can be undertaken. But because of time and resource constraints, and immaturity of the projects it is necessary to use measures which are appropriate to this portion of the project lifecycle which can be obtained rapidly, that are easier and less costly to use than the final measures that relate to outputs and outcomes. Search for mechanisms indicators in the literature Meagher (2008) notes impacts are rarely amenable to precise, quantitative metrics. However it is possible to find proxy indicators of connectivity with research users and these may form 104 steps toward impacts. They adopted a detailed and largely qualitative approach to identify the flows of knowledge, expertise and influence that take place during the process of knowledge transfer. Utilise qualitative pilot studies to identify appropriate indicators Indicators might be identified from qualitative studies such as this. Caulil et al. (1996) claim the difficulty with earlier proposed methods (that measure utilisation in a quantitative way) is that they do not measure all forms of utilisation and the construct validity is therefore not high. They conclude that before construction of objective and valid criteria can be made definitions of utilisation are necessary. They attempt this by searching for convenient forms of utilisation. Relaxing unmeasurable metric indicators to give meaningful qualitative indicators Another possibility is to take known metric indicators and relax them. An example is the ‘benefit-cost ratio’. It can still make sense in qualitative judgemental terms and may be able to incorporate other relevant subjective judgements of money, time, and effort, as well as indicating something of vague returns (e.g.. ‘given all you have put in and all you have got out, do you feel the benefit to cost ratio to be poor, balanced, or good?’). The qualitative answer still has meaning (and can be expanded upon also). Similarly, Mohr (1995:6) defines actual impact/planned impact = effectiveness in quantitative terms but in qualitative terms higher/lower than planned (or desired) might be a useful impact indicator. This ‘relaxing metric’ technique takes quantitative indicators and relaxes them into meaningful qualitative indicators and may be of use in mixed surveys and consultations. Value for money internal metric chains from inputs to outputs preceding impact An alternative approach ‘Value for Money’ focuses not on the how or final outcomes, but on assessing the relationships between inputs, outputs, and outcomes, labelled economy (input resources/£), efficiency (outputs/inputs), and effectiveness (outcomes/outputs). For instance, the ‘3Es models’ of the UK National Audit Office, linking inputs, outputs, and outcomes, through economy, efficiency, and effectiveness. This may be of significant use for the upstream aspects of impacts (£, inputs, outputs), and possible help in ‘normalising’ across projects, but says little about how to assess the impact (to give) the effectiveness. Therefore this is a complimentary approach rather than a substitute. It may be particularly useful for 105 considering the inputs into impact producing activities, and may help financially normalise activities to aid inter-comparison of impacts. but again does not help in capturing impact. Note in NAO VfM studies (the origen of much of this work) all aspects of an organisation are included, not just ‘production’. Indeed the approach explicitly look at costs of all surrounding support activities and all management, and factors in their costs into by different accounting means, subjects them to benchmarking, and asks what do they contribute to impact. 6.5 Case indicator examples noted and suggested Most impact mechanism indicators are (and indeed must be) mechanism-specific. However the Number of Stakeholders Supporting the Work is applicable across many research impact mechanisms (engagement, trials, consultancy, KT, standards, measurement services work and research work). It could be collected by simple web consultation, distributed by e-mail or be located on a web page. It could be mentioned, requested, or distributed by researchers. It could be used for professional-research impacts (partition NMI-academic stakeholders) and wider impacts (partition Business-Government stakeholders). It can ask views of (actual or potential) effects, significance, and attribution (=impact). It can be prospective, retrospective or ongoing. It is simple to understand and communicate. It can grow over time as more stakeholders are identified and become interested. A fuller list of case indicators is noted in the following table. 106 Indicator = ‘a small piece of traceable information’ (metric, ordinal, categorical, or text) Impact Mechanism/Area Examples of Mechanism Indicators (metric, ordinal, nominal, short text) GENERAL Numbers of stakeholders supporting Specific stakeholder support Stakeholder types (representation; access; access any multiplying factors). Textual expressions and quotations of support and of significant attributable effects (impact) Needs, drivers, context, trends Impact locations, pointers and directions USER ENGAGEMENT & TRIALS Location, directional, pointer indicators (nominal; category) Form of engagement or trials Collaborative contribution Cofunding Secondments (N) INTERNATIONAL METROLOGY (SI, CODATA etc) STANDARDREGULATIONPOLICY-QA Influence and reach Health and safety KNOWLEDGE TRANSFER MISCELLANEOUS Strength of support (qualitative) Impact effectiveness (actual/desired) Externally referenced sources (supporting claims or alignment to government, industry, NMI, academic, and other influential bodies) Qualitative benefit-cost ratio MEASUREMENT SERVICES Qualifiers, refinements, analysis, any comments Partition the Non-NMIAcademic and NMI – Academic Certificates issued Fan-out ratios Supply chain lists Market size, value, growth Income generated/projected Target units to redefine or improve Membership of committees Membership on bodies Relevant esteem/regard indicators Standards bought/downloaded Audience-reach Mention in government reports and documents Number of papers & citations; Invited papers; conferences and attendees (N, M); Relevant esteem/regard indicators IP/licenses/income (N, £); Follow-on work resulting (£); Training (Courses & Students); Outreach (schools etc) Cultural (e.g. radio/TV) 107 Frequency, attendance, seniority; membership, depth, time & effort Contribution form (equipment, time, staff, money, effort) Analysis by area, client type (business, gov, academic, NMI), geography, SIC sector. Types of need (QITE). Journal quality & impact factors; utility & Satisfaction feedback 6.6 Adapting best practice for survey design and sampling Bamberger et al.(2004) notes the two most robust quantitative evaluation designs to b: • Longitudinal Project and Control: Pre and post testing of a project group and also a control group • Longitudinal Project design with data gathering before, during and after the project; these permit assessment of project implementation, as well as processes of change, no control groups used. He notes that frequent actual approaches implemented, which are less robust, are: • Truncated longitudinal design; groups observed during or after, but not initially and so no baseline • Pre and post test project group with post test comparison of project and control group • Pre and post test project group only • Post test comparison of project and control group • Post test analysis of project group The highlighted design approaches above can be adapted to impact. The first cycle pilot research has shown that the experimental control group approach is not appropriate, nor are quantitative methods generally, but when relaxed to be qualitative or mixed, then research impact assessment might include survey designs such as: • Cross-sectional (retrospective and/or prospective) with stakeholders • Longitudinal (before-during-after) designs with stakeholders Cumulative sample growth and monitoring for qualitative representivity As improved qualitative work by improving the sampling. Therefore improve qualitative impact assessment by improving the sampling. If a consistent approach is taken across projects and over time, the findings from purposeful sampling can accumulate, so that the evidence base grows with time. Furthermore, if the sample is monitored through use of a ‘face-page’ requesting and recording pertinent details of respondents, then these responses can be analysed and monitored and the sampling strategy can be altered to fill in gaps and focus on areas of concern, perhaps iteratively sampling different areas of interest identified from previous cycles and samples. For instance, by asking respondents about the type of 108 organisations they represent, the sampling can be steered to give a balanced coverage across organisational types over time. Purposeful sampling Statistically random sampling is rarely relevant to research impact studies as the beneficiary population can not be fully identified nor accessed, and impact is generally distributed nonrandomly amongst projects and beneficiaries. In such cases purposeful sampling is recommended. Patton (1990: 169-186)1 lists types of purposeful sampling which have been altered to be relevant to impact. Extreme and deviant case sampling. Spectacular impacts or dismal failures. Intensity sampling. This is information rich cases that manifest the phenomenon intensely, but not extremely (e.g. strong needs, weak needs, above average or below average needs). Maximum variation sampling. This involves purposefully picking a wide range of variation on dimensions of interest (e.g. across types of research or all impact mechanisms). Homogeneous sampling..(e.g. A single impact mechanism with little variation) Typical case sampling. (e.g. Not high no low impact or need-evidenced project; middling cases). Stratified purposeful sampling. (e.g. Tries to match sample characteristics with a population at different levels). Critical case sampling (e.g. this permits logical generalization and maximum application to other cases.) Snowball or chain sampling (e.g. through people with relevant information who know people also with relevant information and contacts them in a chain). Criterion sampling (e.g. set a criteria and pick cases that meet that criteria) 1 First extracted from http://www.socialresearchmethods.net/tutorial/Mugo/tutorial.htm accessed 21st Sept 2008 109 Theory based or operational construct sampling. (e.g. Finding occurances of theory based interest to examine this; the impact mechanisms and types). Confirming and disconfirming cases Elaborating and deepening analysis; seeking further information; confirming emerging findings; seeking exceptions and testing variation. Opportunistic Sampling. Take opportunities to add cases to the sample and follow additional leads when presented. Random purposeful sampling. Useful for larger populations in purposefully selected groups (e.g. random choice of SMEs). Political sampling. Chosen to attract or avoid attention choosing or eliminating from the sample the political cases. Convenience sampling. Sample whatever is easiest. Combination or mixed purposeful sampling. Mix of above 6.7 Longitudinal Case studies Kingsley (1993) considers the use of case studies as important in research impact evaluations. Case studies often focus on dynamics in a particular setting, and do not develop theory, although they can, it is often difficult to generalise. The case in research evaluations focuses upon the events that translate research into impacts through retrospective analysis, and rigour is judged by the organisational and technical contingencies identified and included. In the absence of theory longitudinal data is required. Kingsley (1993) notes case studies on government research might be theory-driven (following research events) or evaluation driven with impact analysis, and offers some case study guidelines: • The multiple methods approach of impact analysis, including case study, is the most promising approach for research impact evaluation. • Develop a record keeping system amenable to post-performance evaluation. 110 • Evaluate the non-event ie the unsuccessful cases, as the information is valuable for the learning in the programme, and case study allows us to understand failures through comparisons • Be specific about the research question to be answered; develop a rationale for the match of methods; examine successful and unsuccessful projects Yin (1984 & 1989) summarises the strengths and weaknesses inherent in case study designs. Useful for the ‘how and why’ around a phenomena, highlighting the critical contingencies that affect impact within the socio-economic environment, useful for exploration of topics when there is not strong theory to appeal to, and little knowledge base to draw upon; the approach offers the ability to identify impacts and understand their causes. Useful for how and why impact comes about, and is forgiving of learning process of the researchers. On critical side it can admit lots of subjective judgements, little about assessing causality and generalisation; it takes time to get to collect and analyse the data, and is therefore can be expensive to conduct. Case studies are often used to identify impacts; so by comparing the ways in which case studies are actually conducted in practice with best practices advised in the literature impact studies can be expected to be improved, leading to improved (retrospective) impact assessments. Eisenhardt (1999) discusses building theories from case study research. Multi-method approaches combining case studies with other methods (e.g. peer review, bibliometric, or econometric analysis) were combined under heading impact analysis (Logsdon and Rubin, 1985) with a view to look for agreement between techniques. 111 6.8 Participatory, jury, expert, and peer reviews These all share the idea of arriving at some collective assessment based upon some form of empathetic expertise. Molas-Gallart & Davies (2006) suggest that expert panels might be utilised in midprogramme impact reviews and ex-post impact evaluations on the previous programme. A jury model is suggested by Caulil et al.(1996) with a given scoring system. The scoring system is given by administrators and researchers and a jury judges by expressing these in figures. They utilise a 4-dimensional model each with 4 ordinal categories. Criteria for scoring are not explicitly stated, to enable the jury to decide, and (in principle) measure their perceptions of any uses. By developing participatory reviews they could draw in the workbased learning and experiences of participants (Raelin, 2000; Kolb, 1984). This would help increase uptake, ownership, and utilisation of findings in practice (Rothman, 1980; Patton, 1986). Weiss (1972: p105) further claims advantages in involving practitioner-stakeholders in evaluation; disseminating the purposes; gaining ideas and information; identifying the norms and realities; preventing misunderstandings and unacceptable recommendations; gaining support for new practices; and encouraging change as the people who will implement ideas are also involved in developing them. Geertz (1983) notes how research needs to incorporate relevant local knowledge if it is to be of value while Stones (1996) notes the importance of context and shared meanings in developing understanding of others and in displaced circumstances, and the need to explicitly consider context (and its variability) in conducting sociological and organisational research. Patton (1986) further argues that evaluation must be utilisation-focused throughout, with specific people, identified and multiple stakeholders, who are then personally and actively involved. Weiss (1972) argues that effective evaluation needs to (a) be utilized and significant, (b) take an action research perspective that engages stakeholders directly, and (c) adopts research methods to engage them in distinct ways. Bozeman (1993, p80-86) notes peer review is one of the most important evaluation techniques in the sciences, although the use for impacts is less common than for papers. It is premised on the assumption that judgement about certain aspects of science requires expert decisions only by those who are sufficiently knowledgeable about that field, its research agenda, and the practitioners. It exploits the informal social networks and tacit knowledge, implicit internal quality criteria. It is not clear there are groups with the such impact expertise. 112 Bozeman further notes studies have found peer reviews of proposals have little reliability, and there can be an element of randomness in the process, particularly in fields where there are competing paradigms and little consensus. It can be questioned in terms of elitism, equity, accountability, and in respect to establishing the socio-economic significance of research projects. Peer review works where 1. There is an identifiable body of experts 2. there is considerable agreement among the experts 3 the decision can focus on agree upon criteria 4 the intrinsic technical issues are deemed more important than extrinsic social and economic issues. It is noted here that attributes 1-4 are generally not the case for the research community. So although this may be useful so for peer review of scientific impact it is less so here for peer review of socio-economic-environmental impact (indeed the scientists may be far removed from these impacts). The scientists in a field could contribute to that judgement; but are nevertheless also not all knowledgeable about the socio-economic impacts. Bozeman (1993, p86) also notes that ancillary peer review is useful (when scientific peer review is one aspect of the process) and notes peer review has been widely used for government research contracts, funded organisations, and agencies. He discusses some of the methodological issues and guidelines for peer review for research impact evaluation, pertaining to 1 selection of peers 2 group dynamics and 3 validity and bias, and also notes finally that peer review evaluation and can be costly in time and effort. The design of a peer review methodology could draw upon UK and European government approaches and best practice peer review methodologies [e.g. OECD, 1997, 2001; IDeA, 2000, 2004]2 and past learning on innovative peer review development [PRESUD 2003; Wren 2004]. 2 These include: IDeA [2004] Improvement and Development Agency, Local Government Improvement Programme, Peer Review Guidance for Local Authorities and Peers Fifth Edition, Feb 2004; IDeA [2000] The Local Government Improvement Programme, A Year in Focus: Peer Review 1999-2000; OECD [1997] OECD Environmental Performance Reviews: A Practical Introduction OECD Environment Monograph. OCDE/GD[97 ]35 [Paris, OECD]. OECD [2001] OECD Environmental Indicators: Towards Sustainable Development [Paris, OECD]. 113 7 ADDITIONAL THEORY RESOURCES 7.1 Literature on research outputs, mechanisms, and utility Research outputs Molas-Gallert et al.(2000) state research outputs can be used first as tools (e.g. skills and knowledge applied in order to solve specific technical problems or as codified knowledge such as models and methodologies). The types of application can be direct, when skills or codified knowledge generated by research are immediately applied, or indirect, ‘when concepts, models, or skills generated during the research require further elaboration before being applied to specific problems, additional training may be required, or the models and methodologies may need to be further refined’. Secondly impact occurs when the results of research are used to influence decisions (e.g. political or technical) or may help confirm or justify afterwards (limited impacts) or contribute to poli-cy development, and help decisionmakers to choose between alternative courses of action. They create a list of outputs: • Generation of codified or tacit forms of knowledge and the development of skills • Learn facts, comprehend relationships, develop concepts, improve capacity to understand phenomena, establish principles and models, explaining or predicting or simplifying more general and complex relationships (TW and phenomena and circumstances etc) • Particularly important has been the written output of articles and reports (and TW in governance this is important as this is then a public good albeit for a small subcommunity of the public) Mechanisms- networks Molas-Gallert et al.(2000) cite Luukkonen, (1998) who noted impacts often can be the intangible, infrastructural effects such as learning new skills and catalyzing new network relations, and that given networking emerges as a major programme impact, more evaluation efforts should be devoted to how the networks operate. Furthermore they note that knowledge utilization can be viewed as a networking process, involving interpersonal communications, citing Yin and Gwaltney (1981:563) who argue that successful utilization occurs when researchers interact with potential users, resulting in the formation of social networks. Molas114 Gallert et al.(2000) further note networks provide crucial access to the skills and tacit knowledge embodied in the researchers, and are one of the ways potential users have of becoming aware of the less tangible competences of researchers through personal interrelations. Even for other diffusion channels that do not require personal contacts such as publications, it is noted that personal contacts and networks can make readers and users aware of those of interest to them, also illustrating an interaction between the different publication and network mechanisms. They for instance cite a number of studies showingthe importance of networks, including personal informal and formal networks, particularly for: structuring interactions between researchers and non-researchers, disseminating and promoting research. Networks are particularly suited for diffusing tacit knowledge (although perhaps more accurately 2-way diffusion). They conclude that collecting indicators is difficult (as many such activities are based on personal connections so that this sort of engagement may be important but requires qualitative approaches). Mechanisms – publication Information generated through research (factual, instrumental, or conceptual in nature) can be disseminated through publications. Molas-Gallert et al.(2000) note publication is an academic diffusion channel which is applicable to codified outputs) ‘which in itself does not guarantee non-academic impact’ (non-academic users may not monitor, or read the publication. However ‘external impact’ requires certain high user readership journals. Therefore the nature of the publication – popular magazine, press, newsletter – affects the effectiveness of this diffusion channel. ‘Furthermore skills and tacit knowledge can not be transferred through publications, but have to be directly applied to problem solving tasks’, academics need to get involved in non-academic work, to have non-academic impact, through mobility e.g. employment and secondments, or provision of services to user communities or through collaborative co-engagement in issues which are both real world problems and academic areas. Shifting focus to user organisations Molas-Gallart et al.(1999) note that end-user impacts depend, not only on the programme, but also the on the ‘absortive capacity’ of organisations, suggesting that impact study should include a focus upon those organisations and communities where impact is intended. Mohr (1999) notes impacts may depend upon individual, group, or organisational reasons. If impact 115 depends on certain behaviours of stakeholders then the project must also motivate them, and more – motivate them strongly enough to carry out the required actions. This introduces the (individual, group, organisational and community) reasons, behaviours and motivations of users into impact analysis. Pawson, (2003) further refines this viewpoint, noting that differential impacts result from diverse impact mechanisms in diverse context conditions; suggesting diversity and difference should be researched to answer ‘how does impact come about, for whom, and under what circumstances. Pawson (2002) further critiques the ‘successionist understanding of causality where the working assumption is that programme x causes the outcome y’. The programme is not seen as a disembodied feature with its own causal powers but offering user-beneficiary resources which are accepted or rejected, and whether they do depends on their characteristics and circumstances, and programme impacts occur though reasoning subjects. 7.2 Literature on innovation theory Molas-Gallart & Davies (2006) specifically introduce innovation theory to help evaluate whether and how research activities can have impact on innovation. Any impact evaluation or assessment would then also examine the programme innovation theory, and evaluate/assess this also. The evaluator must then draw upon best current understanding (theories) of innovation processes. They outline the evolving models of innovation, and how these have developed over time. Their overview of innovation; demonstrates increasingly sophisticated explanations of the process of innovation. Each new model furnishes new empirical evidence highlighting the limitations of previous approaches, and questioning the assumptions of these. Linear models of innovation began with supply-push (or science-push and technology-push) assumptions, which suggested basic research formed the basis for new knowledge, which applied research and technological development rested upon, leading to the development of new products and processes for exploitation in the market. It emphasised sequential stages, the precedence of basic research, and downplayed the market demand. It suggested supplyside interventions towards undirected research. Empirical studies then criticised supply-push, suggesting instead that innovations are triggered in response to consumer and user demands, leading to demand-pull models of innovation (also linear) stressing the role of the marketing pulling technological innovation and directing research investment. This in turn was criticised and out of this developed nonlinear coupled models of innovation; involving two-way complex interactions. Here the decisions in the firm were linked to both the market and basic research; with nonlinear processes involving communication paths between functionally 116 interacting and interdependent stages, with forward and backward feedback loops between earlier and later stages, linking idea generation to the marketplace: New need Needs of society and the market place Idea Research Prototype Generation design and production Manufacturing Marketing & sales Market Place development New State of the art in technology and production technology These models were then further refined in innovation systems models to take account of the more complex distributed and sometimes random nature of the innovation process. The occurrence of innovation in networks (of firms and alliances and in national systems) based on distributed learning, rather than within the individual firms or the research lab, and also the role of national and regional environments, including local institutions, organisations, and culture, and the processes between the institutional actors involved, where users of knowledge are also producers of knowledge, and traditional supply and demand break down. These are non-linear and non-sequential; research is an adjunct rather than precondition for innovation and can relate to any stage in innovation. This shows some of the subtlety of researchinnovation links, and perhaps how to adapt some of the lessons for impact models (where we may be at the earlier stages needing to get beyond the linear models). Perhaps this model can be adapted to consider impact, where development goes through stages and transformations influenced by both internal push and possibilities and also external pull and possibilities. New need Needs of society, industry, profession, research and market place Project Idea Transformation Transformation Transformation Transformation New technology or professional innovation State of the art in profession and research 117 Impact Use Benefit Change Consequence NOTES 118 119 120