Testing

Innovation in
the Real World
Real-world testbeds
Siri Arntzen, Zach Wilcox, Neil Lee,
Catherine Hadfield, Jen Rae

October 2019
Acknowledgements

This report has come about from the insight and expert, Nicole Mather, Previously Head of Strategy
guidance of a variety of professionals in a range of in the Offices for Life Sciences (OLS) and SRO of
institutions. We would like to express our gratitude testbeds, Michael MacDonnell, Previous Head
to the following for their participation, advice, of the Testbeds Programme for NHS England,
insights and time through interviews, workshops, Brian Matthews, Head of Transport Innovation,
feedback and other similar activities. Milton Keynes City Council, Luis Manoz, Project
With thanks to Kirsten Bound for original Coordinator and Research Lead, Smart Santander,
inspiration and thought leadership throughout the Filip Kjeldgren, Administrative Officer, Vinnova,
research, and Tom Bridges for his expert advice Thomas Sunden, Project Lead, Energy Efficiency,
and project leadership of the Arup team. Harry Sustainable Innovation, Professor Carlo Ratti,
Armstrong, Olivier Usher and Nancy Wilkinson for MIT, Roboat , Maria Strömberg, Chair of Testbed
their comments on drafts. Gothenburg, Lars Bern, Area manager for
innovation for Testbed Gothenburg programme,
Workshop and interview participants Augustin Boey.

Will Cavendish, Former Senior Responsible Project team

Officer for NHS testbeds, Tim Armitage, Associate
Director at Arup, specialising in testing new Tom Bridges, Arup, Zach Wilcox, Arup, Siri
transport technologies, Lynne Miles, Associate Arntzen, Arup, Professor Neil Lee, LSE, Catherine
Director, Arup’s City Economics team, Evaluation Hadfield, Arup.

About Nesta

Nesta is an innovation foundation. For us, innovation means turning bold ideas into
reality and changing lives for the better.

We use our expertise, skills and funding in areas where there are big challenges
facing society.

Nesta is based in the UK and supported by a financial endowment. We work with

partners around the globe to bring bold ideas to life to change the world for good.

www.nesta.org.uk

If you’d like this publication in an alternative format such as Braille or large print
please contact us at: information@nesta.org.uk

Design: Green Doe Graphic Design Ltd

Testing Innovation
in the Real World
Real-world testbeds
Foreword 4
4.6 Make public services better and 34
Executive summary 6 more efficient
4.7 NHS testbeds – using testbeds to 36
1 Introduction 8 make health services better and
more efficient
1.1 Real-world testbeds as innovation 8
4.8 Addressing market failures and 37
policy
solving grand challenges
1.2 Purpose of the report 9
4.9 Capturing competitive advantages 40
from frontier technologies
2 Methodology 10
2.1 Literature review and bank of 10 5 Designing real-world 44
examples testbeds – things to consider
2.2 Case studies 13 5.1 Clearly define the purpose – and 44
whether a real-world testbed is
3 What are real-world testbeds? 14 the right means to achieve them
3.1 Overview 14 5.2 Collaborate with the right 45
3.2 The literature on testing and 14 stakeholders and ensure
experimentation tools coordination
3.3 Defining and positioning 15 5.3 Organising and funding the 46
real-world testbeds real-world testbed appropriately
3.4 Why, and for whom, are 20 5.4 Analyse risk and embed in 48
innovation testbeds important? real-world testbed design
3.5 Real-world testbeds to watch 22 5.5 Embed evaluation early 49
3.6 Main findings 23 5.6 Market your testbed and include 51
the right innovators
4 Why use a real-world testbed? 24 5.7 Working with regulators 52
4.1 Introduction 24 5.8 The transition from testing to 53
commercialisation
4.2 Six use cases for real-world testbeds 24
4.3 Testing innovation for verification 26 6 Conclusions and policy lessons 54
and proof of concept
4.4 Marketing and local economic 28 Appendices 56
development
4.5 Creating an appropriate regulatory 32 Endnotes 84
environment
Testing Innovation in the Real World: Real-world testbeds

Foreword
Testbeds are common in business. Companies try out new products and services
in real-world conditions to see if they work, if consumers like them or if they throw
up unexpected problems. What works on a screen or on paper often doesn’t work
in reality. Insights come in rapidly once an idea is being tried out in the messy and
demanding conditions of everyday life.
But what about testing bold ideas and solutions that have the potential to solve some of
the biggest challenges we face as a society? This report provides a survey of how testbeds
are being used in areas that involve a public benefit or value – and also where there are
gaps. It shows how many are being run to explore the technologies of the fourth industrial
revolution: the Internet of Things, new forms of mobility, energy and housing. But it also
shows that they are not being used much for mainstream public services, from schools and
public health to police forces and welfare offices.

We think that testbeds could and should be used more widely, and that much more of
society could be a living laboratory for new ideas and technology. This can help avoid
unnecessary mistakes and wasted spending, as these ideas are tried out before they are
implemented more widely. It can also ensure that innovations not only serve the public but
are seen to serve them too.

Testbeds have become an important area of practical work for Nesta. We are involved in
testbeds for drones (through our Flying High programme), for educational technology (through
our EdTech Innovation Testbed) and for uses of Artificial Intelligence (AI) in the labour market
and other fields, alongside many other experiments and sandboxes through programmes like
the Innovation Growth Lab, which runs dozens of RCTs linked to industrial policy.

We commissioned this research to get a sense of the landscape of testbeds around the world.
How are they designed? What works for different purposes? How could they be used more
actively? And what can we learn from these examples to spread good practice? The answers
to these questions will help both public and private sector to design more effective testbeds.

4
 Testing Innovation in the Real World: Real-world testbeds

In some fields, such as transport technology, they are becoming mainstream. But testbeds
should be more integral to major streams of public research and development aimed at
solving some of the world’s global challenges. Action to mitigate climate change is an
obvious one: we need much more active experimentation around promising applications,
from zero carbon buildings to new ways of motivating the public to change their behaviour.

AI is another one. So far too much research is being done in the lab, and with too little
understanding of how AI can support human decision-makers in real-world conditions.
There are increasing numbers of testbeds in healthcare, such as the UK’s NHS innovation
testbeds (a case study in this research) and in technologies to support healthy ageing. But
it’s still remarkable that whereas there are very expensive systems for testing out new drugs,
there is still little comparable machinery to test out public health innovation such as digital
apps that might help with mental health conditions, or new models of peer-to-peer support.

More active experimentation in all of these fields should pay big dividends. But the biggest
value will come when the knowledge that is gained from testbeds is synthesised and made
useful for everyone else. One of the key findings from this research is how little emphasis has
been placed on evaluation as part of setting up testbeds. This orchestration of ‘what works’
is still too often an afterthought, or no one’s job. We need more testing; but we also need
much more intentional learning too, to truly harness the power of experimentation.

Geoff Muglan
Chief Executive,
Nesta

5
Testing Innovation in the Real World: Real-world testbeds

Executive summary
Testing out how new ideas and technologies work in practice is a critical part
of ensuring that they are both fit for purpose and safe before they reach a
wide audience. Scientists, businesses and – increasingly – governments, look
to test innovative products and services before they enter our homes and
cities, or are applied to our public services.
This experimental approach to testing innovation is an opportunity to make
innovation safer whilst also maximising real-world positive impact. It offers a way of
finding out how new ideas and technologies can be applied to solve society’s biggest
challenges, such as climate change, healthy ageing and inequality. It should also
provide a setting for establishing how governments, businesses and citizens can all
benefit from innovation.

In this report we focus on testing environments which are both bounded and
real-world settings, or close to real-world settings, and explore examples of
experimentation which have taken place in this environment. We term these real-
world testbeds, and we set out to learn more about how they are being used across
the world.

Based on a review of 95 real-world testbeds, this report investigates how they are
being used to stimulate innovation and draws out lessons for policymakers who are
considering setting one up.

The report will help improve understanding of how real-world testbeds work and
how they fit within the innovation process and landscape. We hope it will be used by
public sector stakeholders, regulators and private sector firms to design effective real-
world testbeds.

Real-world testbeds: What are they and why do they matter?

In the report, we begin by explaining different categories of testbeds, but our focus is
on the specific category of ‘real-world testbeds’, which we define as:

“Controlled or bounded environments for testing innovation in

real-world, or close to real-world, conditions in the manner (or
close to the manner) in which they will be used or operated.”

6
 Testing Innovation in the Real World: Real-world testbeds

Our research shows that developing real-world testbeds can help people and places to:

• Strengthen collaboration within a clear and • Promote the local area as a good place
structured framework between the public to invest and develop knowledge-intensive
sector, business, universities and other functions, giving potential investors and
research-intensive organisations. existing firms in the area confidence that
there is a supportive and enlightened local
• Focus and attract investment and resources in
innovation ecosystem.
innovation in specific technologies, sectors and
research areas where the local area is seeking • Improve the delivery of (or reduce the demand
to develop and strengthen a competitive for) public services by creating the right policy,
advantage. governance and regulatory systems.
• Reduce risk in the process of developing new • Maximise the economic potential and value of
products and processes for firms, providing a research done locally, and of other assets such
safe space for them to iterate, fail, influence as public-sector data.
regulatory and policy change and support
• Make better use of publicly-available
them to develop to an investment-ready stage.
infrastructure.
• Provide a framework for innovation policy that
enables effective evaluation.

To take advantage of the opportunity that real-world testbeds can bring, our
recommendations from this research are:

1. Real-world testbeds are an important policy 4. Governments should compile a national

tool that can increase innovation and achieve overview of the infrastructure available to test
the aims of national and local industrial and demonstrate innovation. As demonstrated
strategies. Our research shows that they can in our example from Sweden, this would allow
play multiple roles for national and local these tools to be marketed and help coordinate
government, set out in the use-cases in this resources and learning from them.
report.
5. Relevant governmental and/or innovation
2. Governments should experiment with applying agencies should agree on a shared
real-world testbeds as a tool for solving grand terminology for testing and demonstration
societal challenges. Real-world testbeds offer tools as confused terminology prevents
an opportunity to incentivise and co-ordinate learning. Our research aims to provide a basis
across public, private and third sectors to work for a shared terminology.
together in directing their efforts at solving
6. Real-world testbeds should not be stand-alone
challenges. This use of real-world testbeds is
policies, but part of a strategic approach. The
currently under-represented in our research.
best examples from our research embed the
3. Public engagement should be central to the testbed in a wider strategy based on research
design of testbeds and considered from the strengths.
start. A framework for how the public will
7. Evaluation and learning should be considered
be engaged and involved should be set out
from the start of real-world testbed design,
from the start of the design phase of a real-
to understand the impact of the innovation
world testbed. Real-world testbeds can raise
being tested, but also the process of testing.
significant ethical questions, including those
Evaluation is too often an afterthought rather
around the type of technologies which are
than a core aspect of the testbed design,
tested and the consent of those who are in the
leading to the loss of important learning
testing environment.
and evidence. Our report highlights several
examples of approaches that could be taken.

7
Testing Innovation in the Real World: Real-world testbeds

1 Introduction
1.1 Real-world testbeds as innovation policy

Testbeds are an increasingly used innovation policy tool. As a common term they refer
to testing and demonstrating infrastructure within environments such as laboratories,
simulated or constructed environments, and real-world environments. The Swedish
innovation agency Vinnova refers to these three environments as Level 1 (laboratories), Level
2 (simulated or constructed environments) and Level 3 (real-world environments used for
testing innovation). This report mainly focusses on Level 3: real-world testbeds.

One of the core challenges in bringing innovation to market, and securing economic and
societal benefits from it, is enabling innovators to test new technologies and their use
cases in the real world. Innovation may work well in laboratory conditions, but they could
fail or behave differently when introduced into the environments they are intended for –
often complex, adaptive, real-world systems. For example, autonomous vehicle technology
may work seemingly well in laboratories or test tracks but may be unable to cope when
confronted with the unpredictable human traffic environment.

To address this challenge, policymakers and institutions are increasingly enabling

the testing of new technologies in bounded, real-world environments. New financial
technologies have been tested with flexible regulation on small groups of users, for example,
and sensors have been deployed in a city district to test various smart city solutions. There
is no consistent definition for this sort of policy instrument, and they are known by different
names including ‘demonstrators’, ‘sandboxes’ or ‘testbeds’. This report focusses mainly on
testbeds operating in real-world environments, and we term these ‘real-world testbeds’.

Real-world testbeds share two common features. First, they are environments designed to
contain risk. Their bounded nature has several advantages: it allows the management of
risk, reduces the challenges associated with the regulation, provides access to specialist
infrastructure, and secures access to specific user groups or associated data. Second, they
enable the testing of new technologies in real-world conditions, providing a framework
for learning, evaluation and adaptation in a controlled process. Their real-world nature
distinguishes them from laboratories or simulations, which tend to reduce the complexity
of real life. Real-world testbeds are designed to test technologies in the complex systems in
which they will ultimately be used.

Facilities for testing are an increasingly important part of the innovation infrastructure,
required to keep up with the pace of technological development. With part of the real world
available to innovators – be it a road, an airfield, a neighbourhood or an elderly care centre
– technologies can be introduced relatively safely and tested in various use cases when they
are close to commercialisation. Understanding how to facilitate real-world testbeds can also
help public, private, and other stakeholders collaborate, and ultimately be better positioned
to solve critical, societal challenges together through the innovative process.

The public and private sectors can use real-world testbeds for a variety of purposes – each
of which shapes the requirements and design of the bounded environment. This report
discusses these purposes and their impact on the use of testbeds.

8
 Testing Innovation in the Real World: Real-world testbeds

1.2 Purpose of the report

This report is for those developing and implementing testbeds, including policymakers
in national and local governments, private and third-sector developers of technology,
researchers, and agencies tasked with promoting and facilitating innovation. The report
provides insights to help them understand, design and implement successful testbeds in the
real world and learn from previous examples.

Based on a review of academic and other literature, a review of global examples of

testbeds, and in-depth case studies, we investigate and clarify what testbeds are, distinguish
them from other similar innovation tools, and illustrate different ways of using testbeds for
various purposes and actors.

Based on the research findings, this report sets out points to consider when designing a
testbed, to ensure the safe and effective introduction of new technologies in ways that
maximise potential economic and social benefits.

The report should be read and used by:

• Anyone interested in understanding real-world testbeds and how they fit within the
innovation process and landscape.
• Public sector stakeholders or others with access to real-world environments wanting to
explore opportunities to develop real-world testbeds.
• Regulators seeking to explore opportunities of getting involved in real-world testbeds.
• Private sector firms seeking to use real-world testbeds.

9
Testing Innovation in the Real World: Real-world testbeds

2 Methodology
2.1 Literature review and bank of examples

A review of academic and other literature was undertaken. This facilitated a greater
understanding of the terminology and the type of tools used for testing innovation and
identified main principles and lessons valid for real-world environments.

Comprehensive desk-based research was undertaken of relevant examples of testbeds from

across the world, developing and using a set of criteria to classify these examples. These
have been brought together in a ‘bank of examples’ for this study. Because there is already
literature on laboratories (which test new technologies in closed conditions), the focus of this
research was on testing facilities which tested innovation in the real world. Box 1 sets out the
criteria for inclusion of case studies in the bank of exampless for this report.

Box 1. Criteria for inclusion of testbeds in the bank of examples for

this study
• Does it test a new technology or an existing • Is it in a real-world (or close to real-world)
technology in a new or innovative context? environment?
• Does it lower barriers to testing through one or • Does it allow failure, learning and evaluation in
more of the following: a testing phase?
·· Risk management. • Has it got some form of public sector
·· Regulation. involvement?
·· Access to user-groups.
·· Access to physical/technical infrastructure?

Over two hundred testing and demonstration facilities were reviewed, informed by our
literature review and various terminologies in different languages. This resulted in a sample
of eighty examples of real-world testbeds (available in Appendix 1).

These examples exclude closed facilities for testing Connected Autonomous Vehicles (CAVs)
or laboratories testing various technologies. The sample was by no means exhaustive.
Thousands of testbeds exist across the world, under different terminologies and across
sectors. The full list of real-world testbeds included in the Bank of Examples can be found in
Appendix 1.

The geographic spread of the sample of examples can be seen in Figure 1.1

10
 Testing Innovation in the Real World: Real-world testbeds

Figure 1: Geographic distribution of real-world testbed examples considered for

this study

1-3 examples 4-9 examples 10+ examples

Figure 1 illustrates the spread across sectors and technologies identified in the examples for
this research.

11
 Testing Innovation in the Real World: Real-world testbeds

Figure 2: Testbed examples considered for this project by sector and technologies

Smart city/transport tech

Health and social care
CAVs
Energy and climate change
Maritime sector
5G networks
Internet of things
Construction and building
Drones
Blockchain
Financial technology
Education technology (ICT)
Social development
Environmental development
Cyber security
Agriculture

0 2 4 6 8 10 12 14 16 18

The majority of the identified testbeds operate within smart city and transport technology,
health and social care, CAVs, Energy, 5G technology and Internet of Things (IoTs). This is
not necessarily reflective of the full global spread, it represents the sample of 95 real-world
testbeds identified through this research.

Some transport and smart city real-world testbeds are very specific, such as ElectriCITY
in Gothenburg, focusing solely on testing public transport solutions. Others, such as Smart
Kalasatama in Helsinki (Finland), take a more open approach, making a new neighbourhood
available for testing various technologies that contribute to the overarching objective of
“saving the inhabitants one hour per day in efficiency solutions.”2

Some of the examples are projects with a set of pre-identified partners from across sectors,
while others are open to continuous enrolment to provide innovators with access to the
testbed.

12
 Testing Innovation in the Real World: Real-world testbeds

2.2 Case studies

Based on an in-depth review of the bank of examples and discussions with a network of
experts listed at the start of the report, six case-studies were chosen for detailed review.
Table 1 provides an overview of the case studies.

The case studies are in Appendix 2 with full descriptions and an overview of the main
findings. These findings are used, along with insights from the broader bank of examples,
to inform the framework, arguments, and recommendations in this report. External and
internal experts with various experience have contributed to the report in several workshops
and feedback sessions.

Table 1: Case study overview of detailed case studies (see Appendix 2 for the case
studies)

Testbed Description Method

(geography)

NHS testbeds A national testbed programme for health technology deployed with a Desk-based
(England) range of different patient groups in hospitals and local NHS bodies. research and
three interviews

Roboat An autonomous vessel testbed in the canals of Amsterdam, testing Desk-based

(Netherlands) various use cases such as freight and passenger transport, environmental research and
monitoring and temporary infrastructure. e-mail interviews

National A national strategy set to coordinate existing test and demonstration Desk-based
Testbed Strategy facilities across Sweden, aiming to ensure collaboration and shared research and
(Sweden) learning between them, a holistic approach to innovation testing policy three interviews
and mitigating for lack of investment in less mature technologies.

Testbed The city of Gothenburg’s local initiative to coordinate testbeds and Desk-based
Gothenburg market the city based on their existence. research and two
(Sweden) interviews

Milton Keynes A testbed facilitated by Milton Keynes City Council, being open to testing Desk-based
transport technologies in the urban environment, including autonomous research and two
vehicles and delivery robots. interviews

Smart Santander An IoT testbed facilitated by the EU and the city of Santander, deploying Desk-based
(EU: focus on sensors in the urban environment for smart city purposes. research and two
Spain) interviews

Digital AV A concept for the ‘Cross-Border Digital Test Bed’: A real-world testbed Desk-based
testbed aiming to test AV technology and adapt the regulatory environment research and one
of autonomous vehicles on various types of roads between France, interview
Luxembourg and Germany.

13
Testing Innovation in the Real World: Real-world testbeds

3 What are real-world

testbeds?
3.1 Overview

The terminology around testbeds is fuzzy and unclear. Tools enabling experimentation,
testing, and the managed introduction of new technologies are sometimes called
‘demonstrators’, ‘sandboxes’, ‘testbeds’, and ‘living labs’. While there is increasing use of
these terms, there is little consensus about their precise definition. This report uses the term
‘real-world testbed’ because of the focus on the real-world element. This section of our
report defines real-world testbeds, explaining how they are distinct from other similar policy
tools, and argues why they are important to various stakeholders.

3.2 The literature on testing and experimentation tools

Providing virtual or real-life spaces for experimentation and testing is an increasingly

important part of innovation policy. Trial and error learning has always been essential for
innovation, but this has generally been in environments which simplify real life, reducing the
number of parameters, rather than being a real-world situation.3

The increased importance of real-life testing has been driven by two changes. The first is
the growing importance of smart city type technology. This has made it easier to evaluate
the performance of new technologies, as the increasing use of real-time monitoring and
sensors allows authorities to monitor what is happening – a phenomenon sometimes called
‘test-bed urbanism’.4

At the same time, regulators have been looking for new strategies to manage the pace of
technological change.5 While many interventions can be tested immediately in real-world
situations, this is more challenging when testing in a complex system where the “challenges
of complexity that arise in any interactions among institutional, technological, and human
systems” are obvious.6

The Organisation for Economic Co-operation and Development (OECD)7 classifies testbeds
as “testing environments (or test beds), where new technology developments can be tested
in controlled but near to real-world conditions.” They distinguish these from regulatory
sandboxes, which provide waivers from regulation, allow companies to trial new products
and allow regulators to learn as well.

The use of real-world testbeds is also on the rise, specifically in the Nordics and the UK.
In Sweden, a country with highly developed testing and demonstration infrastructure, the
number of real-world testbeds now represents 40 per cent of all the available tools for
enabling testing of innovation, in addition to being at the top of the Swedish government’s
list of innovation priorities. Real-world testbeds have received growing interest across the
UK through projects such as UK Auto drive, NHS testbeds and numerous 5G testbeds. Nesta
has also published research on how EdTech testbeds are being used around the world,
suggesting four models for how these could be used in the UK.8

14
 Testing Innovation in the Real World: Real-world testbeds

3.3 Defining and positioning real-world testbeds

Based on an in-depth review of the Bank of Examples and discussions with a network of
experts listed at the start of the report, six case studies were chosen for detailed review.
Table 1 provides an overview of the case studies.

The case studies are in Appendix 2 with full descriptions and an overview of the main
findings. These findings are used, along with insights from the broader bank of examples,
to inform the framework, arguments, and recommendations in this report. External and
internal experts with various experience have contributed to the report in several workshops
and feedback sessions.

The definition for this research

‘Testbeds’, or ‘test and demonstration infrastructure’ in their broadest form include all
different types of environments. The Swedish Innovation Agency, Vinnova, categorises them
as Level 1 (laboratories), Level 2 (constructed/simulated environments), and Level 3 (real-
world environments). Real-world testbeds, the focus of this report, provide environments
where innovation is tested in collaboration between users, innovators and, often, regulators
and other institutions in the environment where the technology will be implemented. From
the public sector perspective this can, for example, be opening up bounded environments for
testing specific infrastructure, public systems such as health care, or real districts of a city.

Real-world testbeds are defined as follows in this report:

Box 2: Definition of real-world testbeds

“Controlled or bounded environments for testing a testbed allows the management of risk,
innovation in real-world or close to real-world experimentation, evaluation, and failure.
conditions in the manner (or close to the manner)
• Because the focus is on testing in real-
in which they will be used or operated.”
world environments, they are suitable for
• They reduce the barriers to testing by helping higher technological readiness levels, where
manage risk, changing regulation, allowing technologies have already been tested in
access to user groups, or providing a specific simulated environments.
physical environment. • Because they allow technologies to be tested
• They are particularly suited for technologies in a manner in which they would not otherwise
which operate in complex systems or in be tested, they encourage investment in
challenging ethical contexts. The use of innovation.

How real-world testbeds overlap with innovation testing tools

A categorisation of innovation testing tools was developed, informed by the literature
review, expert interviews and case studies. Testing and experimentation tools vary in two
main ways. First, they vary according to the degree of control in the environment in which
they operate. Laboratories are attempts to control the environment in which testing and
experimentation occurs, reducing the number of variables which might interfere with
outcomes. Other technologies can only be tested in real-world environments, in which cases
real-world testbeds might be more suitable.

15
 Testing Innovation in the Real World: Real-world testbeds

Second, these tools differ according to the relevant stage of the innovation process. Some
of these tools, such as living labs, tend to be designed to create new ideas.9 Others are used
for technologies which are near to market, with proven or near-proven technology such as
demonstrators or use cases.

We use this division to map out the different type of innovation and testing tools. Figure
3 plots the innovation stage on the horizontal axis, from idea generation through to
development, testing, implementation and market. The vertical axis represents the amount
of control in the testing environment, spanning from completely ‘controlled’ (a laboratory) to
real-world environments.

Figure 3: Real-world testbeds placed among other test and demonstration

terminologies

Real-world
environment Demonstrator

Real-world
Sandbox
testbed
Living lab Proving-ground

Simulated/constructed

Laboratory
Controlled
environment

Idea Development Testing Implementation Market

generation

Level 1 testbed Level 2 testbed Level 3 testbed

While there is no settled definition in the literature and terminology is often fuzzy,
our research situates real-world testbeds as being those which are at the testing or
implementation phase, but which involves them being in a real-world environment. The
concept of real-world testbeds overlaps with the concepts of demonstrators, sandboxes and
proving grounds, but are relatively analytically distinct from living labs and laboratories.
In particular, there is considerable overlap with the term demonstrators, a term which was
originally used to refer to products which were very close to market (therefore needing
‘demonstration’ rather than ‘testing’).

16
 Testing Innovation in the Real World: Real-world testbeds

The following section provides explanations for each of the terms, underpinning their
placement in Figure 3. In future, it is important to distinguish these as different tools to
avoid confusion and improve the understanding of when they should be applied. Although
they will continue to overlap to a certain extent, as they all operate in similar areas of the
innovation process, in future it is important to improve our understanding of what makes
them different.

Laboratories (Level 1 testbed10) are strictly controlled test sites where innovators
can test specific technical properties in isolated, artificial and heavily controlled
circumstances. They are often limited to part-functions and components rather
than systems or production methods. They tend to be clearly distinguished from our
definition of real-world testbeds, but they are commonly referred to under an umbrella
‘Testbed’ term which encompasses all environments that allow testing of innovation.

Simulated/constructed testbeds (Level 2 testbed11) are areas for testing and verifying
systems, process and product levels. The environments are closed-off areas and
test facilities (e.g. closed-off tracks for testing autonomous vehicles). They often
offer a simulated or constructed version of reality, still closed off and able to control
by the testers. Figure 3 places these testbeds in spanning from development to
implementation in the innovation process, between fully controlled and real-world
environments. Simulated testbeds often overlap with real-world testbeds as they
sometimes occur in combination (a testbed offering both simulated and real-world
environments).

Real-world testbeds (Level 3 testbed12) are controlled or bounded environments for

testing innovation in real-world conditions in the manner (or close to the manner) in
which they will be used or operated. This report’s definition is based on Vinnova’s Level
3 definition.

Sandboxes are described as follows by Ribiere and Tuggle:13

“The idea behind the innovation sandbox is to create a ‘space’ (often

virtual) in which developers can play around with an innovation to get
a feel for how it might be used, add/delete features to it, comment
to one another about it, ask a variety of ‘what-if’ questions of others
playing in the sandbox about the innovation and related innovations…
(and)…show it virtually to prospective purchasers/users of the
innovation to get their reactions, ideas, and suggestions.”

Some sandboxes operate in close to real-world environments, for example through

testing regulation in closed-off elements of financial systems, while others such as the
CAA sandbox operate as an advisory service for innovators wanting to operate in the
commercial drone landscape. As illustrated in the figure, they stretch across controlled
and real-world environments, available in the development phase and the testing
phase of various innovations.

17
Testing Innovation in the Real World: Real-world testbeds

Living lab is the term that is most fuzzy and difficult to pin down into one, agreed
definition. The European Network of Living Labs defines them as “user-centred open
innovation ecosystems based on a systematic co-creation approach, integrating
research and innovation process in real-life communities and settings.”14 Based on this
and other definitions,15 Living labs operate in simulated or real-world environments
focussing on the co-production of products. This is in an earlier stage of the innovation
process than real-world testbeds.

Proving ground is the tool often used to test products within transport, to ensure they
fulfil industry requirements and standardisations. Definitions are often found within the
specific industries, for example by General Motors:16 “proving grounds exist to enable us to
establish indisputable facts about automobiles in general. In other words, it is necessary
to provide facilities for the comparison of all cars; a place where we can evaluate GM cars
against all other products.” We have placed Proving Grounds in the ‘implementation’
stage of the innovation process, in simulated or real-world environments.

Demonstrator is the term that most closely overlaps with real-world testbeds and is
sometimes used interchangeably. The European Commission described the purpose
of a Demonstrator “to validate the technical and economic viability of new or improved
technology, product, process, service or solution in an operational (or near to operational)
environment, whether industrial or otherwise, involving where appropriate a larger scale
prototype or demonstrator.” Examples of Demonstrators suggest that the most apparent
distinction from real-world testbeds is that Demonstrators focus on more mature
technologies and can thus operate in more real-world environments, demonstrating
their market-readiness in the stage before market implementation.

When might different tools be used?

As seen from the previous section, there are sometimes subtle differences between various
tools used throughout the innovation process. Despite these differences, the various tools
are required at different times in the innovation process for specific reasons.

Which environment is used and the level of control of the circumstances is directly
associated with how safe and mature the innovation is. Taking autonomous vehicles as an
example, the technology needs to be highly mature and proven to be safe before they are
tested alongside the many unpredictable elements of real traffic. If there are risk mitigations
that cannot be properly evidenced, a more controlled environment should be selected
before allowing testing in the real world. If enough evidence has indeed been provided
that the technology is safe (evidence that recognised risks have been tested and mitigated
in labs and closed environments previously with successful and robust results), real-world
environments can be considered.

To illustrate this, Figure 4 illustrates how and when various environments were used during
the project UK Autodrive, deployed in Milton Keynes. UK Autodrive was one of the transport
innovations tested in and around the city of Milton Keynes. The testing was designed based
on a risk register listing all the potential aspects that could go wrong on various levels. The
majority of these risks (except a residual that was accepted in collaboration with insurance
companies), required evidence of mitigation that needed to be proven in different testing
environments. Some risks could be mitigated through modifying the technology or the
autonomous systems, others through limiting what the pods could be used for.

18
 Testing Innovation in the Real World: Real-world testbeds

Figure 4: Example of an innovation process using different tools

Step 1 Step 2 Step 3 Step 4

Identifying Identifying risk Test Design of
technological, mitigations specifications testing
system, use case to verify and process
and other risk Technological validate
requiring
mitigation Autonomous
systems
The use case
100 risks Accepted risks
Residual

Risks verified Risks verified Risks verified Risks verified

and validated and validated and validated and validated

Test 1 Test 2 Test 3 Test 4 Test 5

Testing Testing in a Testing brakes Testing in Block Testing in
mecanical warehouse etc. on closed- 4 – building the city:
aspects in a designed as a off track: area with road Level 4 testbed
Level 1 testbed film-studio: Level 2 infrastructure
(laboratory) Level 2 in MK:
Level 2/3

The mitigations guided the design of the multiple tests required to prove the safety of
the pods. Those relating to the technology needed to be tested in completely controlled
laboratories (described as a ‘treadmill for cars’). Once mitigations had been sufficiently
proven, the next step was testing the pods in various weather conditions and controlled
obstacles in a simulated testbed (similar to a film set). The third step was to test the
vehicles on closed-off testing tracks before moving on to real-life conditions without users
(close to real-world conditions). Finally, the pods were tested in the city of Milton Keynes.
Before reaching this point, the testing would move back and forth between the previous
testing environments if it was deemed necessary to retest or reassess adjustments to the
technology before testing in a more real-world environment.

19
Testing Innovation in the Real World: Real-world testbeds

Should a demonstrator be used in this case, it would, according to our definition, come
after Test 5 when the pods had been proven safe enough to ‘demonstrate’ their ability to
operate in real-world traffic. A living lab would not be compatible with testing such a risky
technology, as it deals with co-development of technology with users. A living lab could
be useful to test how the AVs were to be used, for example either through a shared model
or privately-owned cars. A sandbox would be suitable throughout the process if the main
purpose was to develop AV regulation, and it could be represented through, for example, the
Department for Transport facilitating sessions with AV innovators to help them understand
how to safely operate the rules.

3.4 Why, and for whom, are innovation testbeds important?

There are a series of actors leading, participating in, funding, or facilitating real-world
testbeds. This report focuses particularly on why real-world testbeds are key for public and
private sectors. They can be especially important as the rapid development of increasingly
complex goods and services continues.

Actors typically involved in real-world testbeds

A range of actors are often seen to be involved in real-world testbeds.

• Innovators, either from the private sector or research institutions, are naturally always
involved. However, anecdotal evidence points to large, resourceful companies or
departments being involved more often. Small and medium enterprises (SMEs) are often
harder to engage, although their needs may be as high.
• The public sector is often involved in testbeds, especially on the funding side. There is a
strong case for the public sector to take greater ownership in real-world testbeds as this
could influence the outcomes and objectives to become more socially beneficial.
• Local and regional authorities increasingly use testbeds for economic development
purposes, as a piece of the innovation ecosystem.
• Regulators are involved in some testbeds, however, often in the least ‘invasive’ manner.
They could be increasingly involved to allow a more experimental and anticipatory
approach to regulation.
• Universities and other research institutions are somehow involved in the majority of real-
world testbeds we have identified. It represents a great way of spreading the value of
the researcher’s expertise and promoting linkages with private innovators and the wider
public sector.

Why real-world testbeds are useful for the public sector

It is increasingly important for public sector actors to develop new ways of delivering, or
reducing demand for, public services to improve outcomes and reduce costs. They have
access to various environments that innovators are keen to test in. Real-world testbeds
have a potentially valuable role to play to ensure safe and effective use of new technologies
across health systems, schools, mobility services, cities and regions.

For nations, cities and regions, real-world testbeds can play a role in supporting more
productive economic growth, including attracting and retaining investment from knowledge-
intensive firms, increasing the rate of start-ups and scale-ups amongst innovation-driven
enterprises, and securing greater investment in R&D from existing large firms.

20
 Testing Innovation in the Real World: Real-world testbeds

National and local governments and their agencies are aiming to maximise the diffusion of
innovation – and to increase the stickiness of this locally – from research-intensive anchor
organisations such as universities and other research bodies, hospitals and some public
sector functions. They are also working to build stronger and larger angel investment/
venture capital networks, particularly in areas with low levels of risk capital investment. The
public sector is also seeking to promote innovation to develop new ways of delivering, or
reducing demand for, public services to improve outcomes for people and to save money.

These cities and city regions are seeking to put in place the right conditions and build
the ecosystems to support innovation-driven growth and/or to improve public services,
bringing together universities and other research-intensive institutions, entrepreneurs,
corporates, venture capital investors and the public sector. To do this, local policy makers
and stakeholders need to consider the range of tools and interventions at their disposal.
Innovation testbeds have an important role to play alongside other policy tools such as
living labs, pilots, open data and so on.

In particular, developing real-world testbeds can help places with:

• Strengthening collaboration between the public sector, business, universities and other
research-intensive organisations within a clear and structured framework, which is
necessary for developing an innovation-driven growth ecosystem.
• Focusing and attracting investment and resources in innovation in specific
technologies, sectors, and research areas where the local area is seeking to develop and
strengthen competitive advantage.
• De-risking the process of developing new products and processes for firms, providing a
safe space for them to iterate, fail, influence regulatory and policy change needed, and
supporting them to develop to an investment-ready stage.
• Promoting an area as a good place to invest and develop knowledge-intensive
functions, giving potential investors and existing firms in the area confidence there is a
supportive and enlightened local innovation ecosystem.
• Improving the delivery of (or reducing the demand for) public services through creating
the right policy, governance and regulatory systems.
• Maximising the economic potential and value of research done locally, and of other
assets such as public sector data.
• Providing a framework for innovation policy that enables a more systemic approach to
evaluation of effectiveness.
• Making better use of publicly available infrastructure.

Why testbeds are important for the private sector and other innovators
For innovators, testing and verifying technologies or other innovations is vital to ensure the
innovation is ready for the market. Several private companies run singular tests or pilots
of their products, however this can be an expensive and complicated process, especially if
resources are scarce. Testbeds can help businesses in:

• De-risking the process of developing new products and processes for firms, providing a
safe space for them to iterate, fail, influence regulatory and policy change needed, and
supporting them to develop to an investment-ready stage.

21
Testing Innovation in the Real World: Real-world testbeds

• Building relationships with regulators and public sector, of which the latter could be
either local policy enforcers or potential customers.
• Attracting venture capital investment by providing robust evidence that the innovation
is safe and fit for purpose in the environment where it is intended to be used.
• Providing access to infrastructure, users (or their data), regulators and other key
institutions, and professionals in the environment they seek to innovate in. This is
particularly important if the sector is highly complex and challenging to access, such as
the financial sector, rail, or health care.
• Gathering robust evidence that technologies are suitable for the real-world environment
and are well-received by the target audience, easing the road to commercialisation. This
is naturally dependent on allowing the opportunity to involve users and/or ‘customers’ of
the technology in the testbed design.

3.5 Real-world testbeds to watch

Real-world testbeds can naturally be as startlingly different as the real world itself, as
reflected in our sample of 95 real-world testbeds spread across sectors, environments and
scale. This section highlights some recent examples from our research that illustrates real-
world testbeds’ diversity and their sometimes controversial and surprising uses.

Unmanned Warrior Testbed: In 2016, unmanned systems were tested in conjunction with
a major multinational naval exercise in Scotland and Wales. Among the many military
vessels tested was the MAST (Maritime Autonomy Surface Testbed) system, a vessel able
to operate at a fully autonomous level, and unmanned aerial vehicles. The six-week long
test in the real world integrated 40 vehicles from land and sea in numerous scenarios, to
allow the Royal Navy to see how autonomous systems and sensors could be integrated into
future operations. This example illustrates how existing traditional formats such as naval or
military exercises, normally conducted for testing and practising operations, can be used to
test technology.

Sidewalk Toronto: Sidewalk Toronto is a partnership between the City of Toronto and
Sidewalk Labs, an Alphabet subsidiary, intending to develop the 12-acre Quayside area
along the Toronto waterfront. Sidewalk Labs’ vision is to ‘develop a fully connected
neighbourhood and a testbed for digital innovations’.17 Although not active yet, concerns
have already been raised by the public regarding privacy and data harvesting, with fears of
Quayside developing into a surveillance city instead of a smart city. This real-world testbed
illustrates the immense importance of transparency and citizen participation when new
technologies are to be tested and deployed in the real world.

Growth Corridor Finland: Home to a third of Finland’s residents and approximately half
of its businesses, Growth Corridor Finland functions as a real-world testbed for smart
accessibility provided by convenient transport links and digital infrastructure. As a network
of over 20 cities and municipalities, three regional councils, four chambers of commerce
and four ministries, the real-world testbed aims to promote experiments and introduce
new series and mobility practices and become the leading testing platform for new, smart,
sustainable mobility. This real-world testbed illustrates the potential scale of the real-world
environment operating as a testbed, and the opportunities for collaboration between
stakeholders this method entails.

22
 Testing Innovation in the Real World: Real-world testbeds

EnergyBlock (EG): EG is a real-world testbed aiming to explore the potential of using

renewable energy sources in an existing urban neighbourhood and connecting them to
an open blockchain for energy. The ambition is to validate the technology and the proof
of concept to investors, citizens and decision makers with an aim to further scale the
technology. Developed from a sustainability perspective, the real-world testbed will test
and analyse the benefits of the local energy production alongside local food production
and a social agenda of local job creation. This illustrates the opportunity of using real-world
testbeds to test technologies combined with social or environmentally friendly approaches
to urban life.

3.6 Main findings

• Real-world testbeds are part of a set of policy tools associated with experimentation and
testing innovation. They provide environments for testing with two key features – they
are real world and they are bounded. The combination of these two features means they
can be used to manage the risk of introducing new technologies.
• Real-world testbeds are often confused with other, similar tools in the innovation process.
Real-world testbeds are used when the innovation is deemed safe and mature enough to
be tested in the real world. This distinguishes them from living labs, which are more often
associated with the development of new ideas, although many living labs are described
similarly to real-world testbeds. Some of the real-world testbeds this report focuses on
are termed demonstrators.
• There are few limits to the extent stakeholders could benefit from being involved in
real-world testbeds. The sample for this research shows that large firms and research
institutions are most often involved in testbeds, while SMEs can struggle to get involved.
The public sector could potentially take a stronger lead on some real-world testbeds to
gain more socially beneficial innovation.
• Real-world testbeds are important to public sector actors due to their potential to
support innovation and economic growth, develop better and more efficient public
services, strengthen collaboration and make better use of existing infrastructure. For
private sector stakeholders (innovators), the access to real-world environments is
often essential to progress the product or process, as it de-risks commercialisation,
provides evidence of their functionality and safety, and provides access to users and
environments.
• An innovation process will often move in between the different innovation tools,
depending on the learning obtained from testing. The example of UK Autodrive shows
that different risk mitigations guided the choice of which testbed was required at what
time, before moving to a real-world testbed.
• Real-world testbeds can be highly useful to the public sector, for example through de-
risking procurement processes, as a tool for economic development, and as a framework
for necessary collaboration with a range of institutions in the innovation ecosystem.
• For the private sector, real-world testbeds can provide vital access to environments
where their innovation is intended to be used; they can facilitate learning and
evaluation in a pre-commercial stage and provide the necessary evidence that makes
commercialisation more convenient.

23
Testing Innovation in the Real World: Real-world testbeds

4 Why use a real-world

testbed?
4.1 Introduction

The primary purpose of any testbed, regardless of whether it is in a laboratory or in the real
world, is to test innovative products, technologies, services or processes. However, they are
being used around the world to achieve a wide-ranging number of aims such as attracting
investment, gaining competitive advantages, or making better and more efficient public
services.

This section presents a framework of six different use cases to explain how real-world
testbeds are being used and can be used in the future to achieve wider aims beyond
facilitating testing. The framework provides information about the most common ways
of using real-world testbeds, as well as guidance and lessons learned from established
examples, aimed at those who plan to initiate or consider using real-world testbeds
themselves.

4.2 Six use cases for real-world testbeds

This research – which involved expert interviews, workshops and analysis of a bank of
examples (presented in Appendix 2) – identified six exemplar use cases for real-world
testbeds, presented in Figure 5. The use cases are not necessarily exhaustive, and they are
not mutually exclusive; Many real-world testbeds can be used by participating stakeholders
seeking to achieve different objectives. For example, one stakeholder may use a testbed to
capture a competitive advantage from a new technology (e.g. a nation wanting to become
a world leader in a technology), while another uses it for ‘local economic development
and inward investment’. Accordingly, real-world testbeds are also tools for coordinating
stakeholders in the innovation process.

Each of the identified use cases in the framework is presented in more detail in this section.
The distinction between them is important, as they have implications for:

• How real-world testbeds are designed.

• Who could be involved.
• How they are organised.
• How they are funded.

A core challenge is to balance the many objectives involved, and to make informed
decisions in the process of designing, setting up and running a testbed in a real-world
environment.

24
 Testing Innovation in the Real World: Real-world testbeds

Figure 5: Six use cases for real-world innovation testbeds

Making Capturing
public competitve
services better advantages
and more from frontier
efficient technologies

Testing
innovation for
verification and
Creating an proof-of-concept Local economic
appropriate development
regulatory and inward
environment investment

Addressing
market failures
or grand
challenges

The following section describes common traits of each use case, which actors tend to be
involved, and the implications this use case has for designing and managing a real-world
testbed. Each section also presents a case study with lessons learned and focus elements.
Full case study reports can be found in Appendix 2.

25
Testing Innovation in the Real World: Real-world testbeds

4.3 Testing innovation for verification and proof of concept

Testing and verifying the properties and functionality of new products, services, technologies
or processes is a vital component of nearly any innovation. It is at the core of all testbeds,
regardless of whether the environment is fully controlled, simulated or within the real world.

Description of the use case

Real-world testbeds are developed to be safe settings to ensure that innovations work as
anticipated, solve a problem, establish the use case, and provide the intended benefits and
outcomes for the target audience. The purpose of this tool always contains some form of
testing innovation for verification and/or proof of concept.

Real-world testbeds often provide access to relevant stakeholders (such as citizens and
potential customers) or the ‘target system’ the innovator seeks to enter (such as a public
road or a hospital). Learning from the interaction of the tests and the target user or system
is vital, as the tests can uncover mistakes, malfunctions, or raise questions that have not
been previously considered. In turn, feedback from users can ensure more robust products
and provide evidence of the functionality, safety, and usefulness of the innovation to
support market deployment.

Verification and validation also provide the potential customer or user (for example a local
authority or a public body) with essential information before making decisions regarding
procurement, uptake, or investment. The evidence that comes from testing in the real world
can contribute to de-risking procurement of new technologies or other innovations.

Typical actors involved in this use case

There are some actors to whom this use case is of specific importance or is more prominent.

• Private sector companies or research institutions that require validation and verification
for technologies, processes or services before getting to demonstration stage or
commercialisation stage.
• Public bodies or similar public stakeholders that require proof of concept before
considering procuring or deploying an innovation.
• Sectors that have low accessibility entry-levels for innovators, or very strict
standardisation barriers, such as rail industry and aviation.

Implications for design and lessons learned

This section provides some important lessons learned that could have an impact on
the design of a real-world testbed specific to verification and proof of concept. The
recommendations are drawn from literature review, insights from the bank of examples, and
interviews.

Intrinsic knowledge of the technology and the system is required: Designing a real-world
testbed aimed to verify proof of concept and the functionality of a technology requires
close collaboration between those with knowledge of the environment it should be
deployed in (e.g. a health system), and those with intrinsic knowledge of the technology (e.g.
the artificial intelligence used to improve the health outcome of a specific patient group).
This will ensure that the right questions are asked throughout the process and that the
design of the real-world testbed can answer these.

26
 Testing Innovation in the Real World: Real-world testbeds

Design with risk in mind: Along with the proof of concept and merely testing whether an
innovation works as it should, it is important to design the real-world testbed based on
which risks it needs to mitigate. The tester will need to manage which risks can be mitigated
and ensure the residual risks can either be accepted or insured against. Testing will provide
evidence that can make the innovation easier to commercialise.

Coherent evaluation standard across users: When evaluating the proof of concept or the
validation of the innovation, it is important to use recognised and accepted criteria. This
will help align the evidence gathered with the evidence required by the future customer or
regulator.

Using real-world testbeds to test various uses of a new technology: Many emerging
technologies, such as autonomous systems or IoT, can be used for a range of purposes.
Testing such technologies in real-world environments can allow the tester to pinpoint in
which areas of use the technology can best fulfil its potential. Such real-world testbeds
would benefit from being designed with a higher range of flexibility.

Case study

Roboat – using urban canals to test use cases for autonomous vessels

Geography: Amsterdam, Netherlands

Facilitators: Massachusetts Institute of

Technology (MIT) and Amsterdam Institute
for Advanced Metropolitan Solutions (AMS)

Public stakeholders: City of Amsterdam,

Waternet – the public water utility of
Amsterdam).

Timeframe: 2017-2022

Description

This real-world testbed is part of a five-year innovators were first able to verify that the
research programme on autonomous floating technology – which had previously been tested in
systems (‘Roboats’) in Amsterdam, designing the closed-off pools – functioned in the unpredictable
world’s first fleet (potentially) able to move people environment of a lively city canal. They were
and goods, function as portable, temporary also able to investigate how urban waterways
infrastructure, and gather data. Through testing in combination with the vessel can be used to
the vessel in the canals of Amsterdam, the improve city functions and quality of life.

27
Testing Innovation in the Real World: Real-world testbeds

Key success factors and lessons

• Proving safety and effectiveness supports • The Roboat was used for a wide range of
testing and demonstrating in other cities. solutions to meet the city’s and technology
developer’s respective needs. These include
• The real-world testbed showcased and
demonstrating technology such as 3D mapping
improved control systems for autonomous
of the urban infrastructure using artificial
navigation in urban environments and showed
intelligence and ‘laserscapes’; functioning as
that new technology can revitalise 17th century
temporary bridge infrastructure; water quality
infrastructure. The technology was first tested
and pollution data gathering; and clearing
in Amsterdam before moving on to other
the canals of waste. This partnership of
similar canal and waterways cities such as
testing new technologies – many of which had
Cambridge, UK, sharing the initial learning
additional public service use cases – allowed
across the different urban environments.
for verification of multiple technologies in close
• Use the real-world testbed to prove multiple collaboration with public bodies such as the
uses and functions. water company and the city council.

4.4 Local economic development and inward investment

Testbeds can be used by the public sector, inward investment organisations or business
representative organisations (such as chambers of commerce) to market an area to
investors, talent, and businesses, and support local economic development.

Description of the use case

When a real-world testbed is used for these purposes, it can, for example:

• Have a strong inward-investment component to attract firms to take part in activity in

the area, and/or capacity-building for local firms.
• Be part of cluster-related policies, enterprise zones or other economic development tools.
• Be a tool to enhance local strengths, such as natural assets, skills or environmental
conditions.
• Be used to attract talent.
• Reduce spending and risk of taking a chance on an innovation for the purpose of
economic development.
• Lead to more innovative institutions through a better understanding of the innovation
process and close collaboration with the innovators.

28
 Testing Innovation in the Real World: Real-world testbeds

Using real-world testbeds for local economic development and inward investment is also
useful for innovators. They often depend on access to real-world environments (e.g. roads,
schools, care homes, or other publicly-managed or regulated spaces) to carry out their tests.
Providing access to this infrastructure through real-world testbeds can open doors for firms
in need of testing grounds, while simultaneously supporting the local economy through
investment.

There are naturally risks to this approach and, like any other economic development
policy, it requires planning and tailored approaches suitable to the place in question.
There is a higher likelihood of benefiting from real-world testbeds if they are tailored to
local challenges. The learning outcomes from a real-world testbed can provide intrinsic
knowledge about the system that is tested (e.g. transport systems), and ways they could
solve local economic challenges through innovative approaches. Involving local government
and regulators can increase the chances of innovation being better fitted to solve local
challenges.

Who initiates the testbed, and who else is involved: a section on the typical
stakeholders
As a place-focused use case, the actors involved could include:

• Local, regional or urban authorities seeking to market their place, solve local challenges
and drive local economic development.
• Economic development or inward investment agencies using testing in the real-world
environment as a means to achieve their aims.
• Local institutions such as banks, Chambers of Commerce, Local Enterprise
Partnerships (UK).
• Supranational institutions such as the European Regional Growth Fund can often be
involved as funders of such activity to contribute to sectoral stimulation or economic
development.

Implications for design and lessons learned

The approach for designing real-world testbeds for place-based purposes should be
adapted to the local economic, social and environmental conditions as well as the
institutions involved. The following recommendations are drawn from literature review,
insights from the bank of examples, and interviews with real-world testbed stakeholders:

• Build potential real-world testbeds through focussing on locational advantages that

could be attractive to innovators and investors. They will be attracted to testing in real-
world environments rooted in a particular advantage.
• Real-world testbeds initiated by local institutions require the involvement of neutral
specialists to enable fit-for-purpose design of testbeds that will benefit the place in
question.
• To avoid potential ‘lock-ins’ into specific sectors, real-world testbeds could be open,
flexible and able to respond to changing market demands. Choosing participants,
sectors and technologies for a real-world testbed should be done in collaboration
between local government and business, in line with local economic development
strategies.

29
Testing Innovation in the Real World: Real-world testbeds

• If a company reaches out to use a place’s environment for testing, local stakeholders
should try to find ways that the testing activity can benefit the area. This can be secured
in the design process and can lead to benefits of building local knowledge or supply
chains.
• To increase the local benefits, the real-world testbed should seek to include local SMEs,
not only the ‘usual suspects’ of large companies. This can contribute to more sustainable
outcomes through capacity building and knowledge spillovers between actors.

Case study

Milton Keynes – a testbed for innovative transport solutions

Geography: Milton Keynes, UK

Facilitators: Milton Keynes City Council and

various stakeholders

Timeframe: 2014 – ongoing

Description

Milton Keynes (MK) is the largest ‘New Town’ in • City systems data management: MK: Smart
England, founded in 1967 as a commuter hub supports acquisition and management of data
situated between London, Birmingham, Oxford related to city systems such as energy, water
and Cambridge. The city council recognised that and transport management.
it needed to look beyond traditional transport
• Starship Delivery Robots: Testing robots to
solutions to keep up with population and jobs
deliver food and goods.
growth and to manage challenges of expanding
the current road network and a dispersed housing • Lime Electric Bicycles: Developing a dockless
pattern. To meet their future transport challenges e-bike share scheme in MK’s urban areas.
and support long-term economic growth, MK’s
urban infrastructure has been made into a real- MK is not continuously operating as a testbed,
world testbed, including testing of: but the city council uses the infrastructure as a
testbed when opportunities arise. In addition,
• An electric bus fleet: A testbed aimed at
MK City Council recognises in their economic
delivering a clean and commercially viable
development strategy that the status of being a
electric bus fleet using inductive opportunity
forward-thinking, smart city should be used to
charging.
brand the city and enhance the sense of place.
• Self-driving vehicles: The UK Autodrive This could create a virtuous cycle of attracting
consortium-based project, facilitated by a UK firms to grow the local economy and using
government-backed competition to support testbeds to solve the challenges of growth.
the introduction of self-driving vehicles.

30
 Testing Innovation in the Real World: Real-world testbeds

Key success factors and lessons

• Use real urban challenges to design testbeds: the autonomous pods were set to try to
MK City Council has been focussed on solve a particularly congested area. This has
choosing technologies and co-designing influenced the anticipatory planning of a
testbed objectives and environments to transport system fit for the future.
be in line with locally-specific challenges
• Real-world testbeds can ease budget
(as opposed to ‘just’ opening the urban
constraint challenges for public services and
environment for any kind of technology).
infrastructure: Procuring unproven technologies
Interviewees claimed this ensured benefit for
can be both risky and expensive. The MK
Milton Keynes and mitigated risk of not gaining
approach has been to provide some of its
such benefits.
available urban infrastructure for testing. In
• Attract technologies based on local strengths: return, companies get to verify their products
The council is attracting testbeds that use and develop their business case, which is less
their unique assets. For example, with a more risky and costly than directly funding such a
recently designed road and footpath system, solution.
autonomous pods and delivery robot testing
• Using the ‘brand’ of being a city open for testing
has been easier to carry out. The city further
to attract investment and activity: MK has
plans to use the proximity to an Amazon
built a reputation as a city open to testing as a
distribution centre and a university with a
direct result of the real-world testbed activity.
world-class aeronautical faculty to test the use
This can be seen through media narratives and
of drones in the city.
an increased number of businesses reaching
• Use real-world testbeds to improve the city: out to the council. This reputation, along with
Through co-designing the properties of the the networks from previous projects, has led to
real-world testbed – what the innovators companies seeking out MK as a location for
should test in an area – MK has learnt valuable further testing and investment – a proactive
lessons not otherwise possible to anticipate inward investment strategy.
through the testbed activity. For example,

31
Testing Innovation in the Real World: Real-world testbeds

4.5 Creating an appropriate regulatory environment

Technological change often outpaces the regulatory frameworks by which technology

and businesses are governed, creating issues for proper regulation, protection of rights
and law enforcement. Real-world testbeds can involve regulators to support change
in line with technology. Real-world testbeds also allow regulators themselves to test
different approaches to regulation, learning and evaluating different effects as the testing
progresses. This section is about using real-world testbeds for the purpose of creating
appropriate regulatory environments through temporary changes in bounded areas such as
experimentation clauses and special permits.

Description of the use case

The demand for more agile regulation is increasing due to the rapid acceleration of
new, ground-breaking technologies.18 Regulatory mismatch can occur if technological
advancements move at a faster pace than the regulation. This is often the case with
technologies such as drones, data use and artificial intelligence.

The format of testing regulation in a real-world testbed context provides several benefits.
real-world testbeds aimed at creating appropriate regulatory environments can:

• Ensure involvement of regulatory bodies at an early stage rather than ad-hoc.

• Provide an opportunity for regulators to test different approaches to regulation in a
secure environment, collaborating closely with innovators. This also builds necessary
expertise within the regulatory body in question, which is necessary to provide safe and
appropriate regulation.
• Design a manner of regulations that incentivise innovation while still assessing risks and
ensuring safe implementation.
• Allow regulators to see the potential effects of an innovation in the real-world
environment or with its intended users, increasing the probability of more appropriate
regulation.
• Be used to test regulation on technical innovation, process innovation and the use of new
business models to deliver services.
• Build trust and increased understanding between innovators and regulators.

Main actors involved

Depending on the innovation, the actors involved can be:

• The relevant regulatory bodies for the innovation in question, for example the Civil
Aviation Authority (CAA) in the UK related to testing of drones.
• Relevant governmental bodies responsible for the sector/cross-sectoral approach;
• International regulators such as relevant EU bodies.
• Local authorities in the location of the testbed with regulation on the local level such as
local plans, transport regulation and other devolved responsibilities.
• Large and small innovators operating in the testbed.
• External technology experts, for example from universities and research institutes.

32
 Testing Innovation in the Real World: Real-world testbeds

Implications for design and lessons learned

Based on interviews with experts and real-world testbed practitioners, this section presents
some important elements to consider for this use case:

• When testbeds involve regulatory bodies, it is important to build a trusting environment.

With a high level of trust and clear guidelines, testbeds can be ‘safe spaces’ that offer
mutual understanding of each other’s perspectives and knowledge through dialogue and
co-design, as well as the ability to ask questions.
• Creating a structured manner of collaboration is crucial to allow an iterative process with
many stakeholders such as policy labs (see section 5.7 for more detail).
• It is important to allow for sufficient time and to identify the suitable ways of working
so that regulators can be involved and contribute to the process. Regulators may not
be accustomed to working iteratively and flexibly, and innovators may need to be
more patient than usual, respecting the level of knowledge required to ensure safe
implementation.
• ‘Regulators’ can also be local authorities with specific devolved powers, or international
bodies. It is important also in this case to allow for co-design, thorough risk analysis and
collaborative methods.

Case study

Cross-border testbed to test Connected

and Autonomous Vehicle (CAV) regulation 19

Description

In April 2019, France, Germany and Luxembourg regulatory perspective is to:

opened the cross-border testbed for autonomous
• Facilitate a joint exchange of experience
vehicles. The environment involves a 206 km
based on experiments, results of industry and
circuit between Schengen, Saarbrücken and Metz,
academia, especially regarding judicial and
allowing for a wide range of digital technologies
technical issues encountered during the testing.
to be tested for CAVs under real-life, cross-border
conditions. It is backed by partners from industry • Identify challenges related to traffic
and regulation in the EU and the respective management and safety impacts, interactions
countries. with infrastructure, and the generation,
processing, storage, dissemination and
The goal of the real-world testbed from a
exploitation of data for CAVs.

Lessons learned

Coordinating with other countries will save costs highly beneficial for all stakeholders involved. Not
and increase benefits: Large-scale real-world only does it make the experimentation cheaper
testbeds are costly and complex. In the case of (and thereby perhaps easier to accept from a tax
the highly integrated border system facilitated payer’s point of view), it also facilitates integrated
by the EU, sharing knowledge, risk and costs is thinking in an integrated traffic system.

33
Testing Innovation in the Real World: Real-world testbeds

Organising process to consolidate objectives and testbed. Some aspects of introducing disruptive
aims – but based on the regulatory perspective: technologies is challenging to imagine through
The real-world testbed was initiated by public scenarios and conversations, so seeing AVs ‘in
authorities. The aims, priorities and thematic action’ can bring these issues to the surface at
test areas were established by national, federal, an earlier stage, enabling a more appropriate
regional and local authorities in the participating regulatory response.
countries. It was later subjected to scrutiny and
Maximising learning through designing a multi-
challenge by industry and academia to establish a
purpose testbed: The Digital Motorways testbed
consolidated set of testbed priorities.20
does not simply consist of a strip of road; it
Real-world testing provides concrete use cases encompasses different categories of road, from
for analysis: The regulatory bodies which initiated motorways to more rural settings. This takes into
this real-world testbed recognised the opportunity consideration the varieties of contexts where AVs
to explore the approach to regulating AVs in may be used and highlights the multiple challenges
a less abstract manner through a real-world that may occur from different use cases.

4.6 Make public services better and more efficient

Innovation such as emerging technologies can have profound effects on delivering better
and more efficient public services. It can also pose a challenge to integrate the most useful
technologies due to risks (perceived and actual), lack of knowledge and lack of evidence
of the effect to prove the business case. This use case is about real-world testbeds for the
benefit of public services.

Description of the use case

In most areas, public services21 such as health and social care, transport, education and
energy provision have potential for increasing the uptake and spread of innovative measures
across the delivery of public services. Using real-world testbeds to achieve this can:

• Reduce the risk of procuring technologies that are not fit for purpose.
• Include public service end users (such as patients or passengers) in the testing to
understand the effect the innovation has on them and their experiences.
• Ensure safe and controlled access to users (and often their data), public professionals
and systems for innovators, reducing barriers to innovation in the public sector.
• Ensure an evidence-based approach to spreading innovation across systems or public
bodies.
• Break down barriers for collaboration between the private and public sector.
• Capture the learning from being involved in the innovation process, potentially
contributing to cultural or institutional change and more openness to innovative
solutions within public service bodies.

34
 Testing Innovation in the Real World: Real-world testbeds

Real-world testbeds also allow public and social infrastructure (such as hospitals, schools or
roads) to be better utilised for innovation purposes. It can be challenging for private firms
to gain access to such systems to develop and test products or processes that benefit these
services.

In the face of rising public service costs and/or budget cuts, testbeds can safeguard and
improve high-quality public services. Testing can prove whether the innovation reduces
costs, increases local revenues, improves service delivery, improves outcomes or reduces
risks compared to existing services or technology.

Main actors involved

Using real-world testbeds to make public services better and more efficient usually includes
some of the following actors:

• Government departments and other public bodies seeking to innovate in their area of
responsibility.
• Regulatory bodies if required, often ensuring ethical considerations such as privacy and
other sensitive issues.
• Public service providers at the local level.
• Public service users and/or their data concerning specific public services.
• Innovators with suitable solutions needing to test and validate their product, process or
service in the environment where it is intended to be used.

Implications for design and lessons learned

When designing real-world testbeds for improving public services and making them more
efficient, the following aspects are important to consider in the design process:

• As real-world testbed activities are often additional to public servants’ day-to-day jobs,
it is key to involve individuals who are passionate about implementing innovation and
change, otherwise the testbed may struggle to see results.
• As evidence is particularly important when dealing with public services, designing a
good, unbiased evaluation process is key to obtaining the best possible evidence. See
Section 5 for more detailed guidance on this.
• As real-world testbeds for public services are often tested in one place and meant
to spread across larger infrastructures (for example from one school throughout
an educational system), finding agreed measurements and criteria for success in
evaluation is also important.
• There are often several stakeholders with different ways of working involved in these
real-world testbeds. Getting objectives in line has been noted as vital to ensure
everyone pulls in the right direction. The time it takes to achieve this should not be
underestimated.
• There are often ethical considerations and risks connected with real-world testbeds
dealing with public services and use of citizen data. This needs to be discussed with
regulators and anyone involved in the real-world testbed as users. Data and information
management plans should be part of any trial or real-world testbed.

35
Testing Innovation in the Real World: Real-world testbeds

4.7

Case study

NHS testbeds – using testbeds to make health services

better and more efficient

Geography: England

Facilitators: Department of Health and Social

Care, Office for Life Sciences, NHS England

Timeframe: 2016 – ongoing

Description

The primary purpose of the National Health and evaluation along the way. The first wave
Service (NHS) testbeds in the UK was to improve commenced in 2016 and included seven testbeds
services delivered by the NHS and make them across England with 40 innovators and over
more cost efficient through using technology. 250,000 patient participants. The technologies
It was also important for the programme to tested included predictive algorithms to manage
contribute to changing the culture of innovation patients at risk of developing conditions,
within the NHS – a large public sector aggregation of data to improve clinical decision-
organisation, which by nature can be difficult to making, and technology to monitor risk of crisis
drive and implement innovation across services. in clinical pathways at individual homes or care
homes. Testbeds brought together partners of
The programme is organised into ‘waves’,
senior government officials, academia, industry,
each lasting two years, to enable learning
patient groups and charities.

Key success factors and lessons

• Evaluation is a fundamental part of the across the NHS nationwide, and enabling
testbed: A thorough evaluation provides robust improvement of the process itself in future
evidence of what works well and areas of waves of testbeds. A handbook was created for
improvement across core objectives. The NHS further learning from Wave 1.
evaluated whether the intervention improved
• Ensure an evaluation plan: An evaluation
patient outcomes, lowered health care costs
plan to be implemented by quality personnel
and supported partnerships. Evaluation was
was required when partners applied to join
necessary to create evidence that eased the
the testbed. Qualified personnel were often
process of potentially adopting the innovation
external, to avoid bias in the evaluation.

36
 Testing Innovation in the Real World: Real-world testbeds

• Make learning easy: Structuring the testbeds individuals who were sufficiently engaged and
into two-year waves allowed the learning interested in implementing digital technologies
from the first wave to be implemented was core to the success of the testbeds.
into the second wave. Handbooks for data Ensuring quality of leadership with experience
management and evaluation were amongst and resources to fully engage with the testbed
the tools created to ensure that lessons was also essential. Testbeds chosen without
learned were disseminated and incorporated. fully fulfilling this criterion struggled the most.
• Finding and coordinating the right partners • Clarify and reconcile the (often competing) aims
and collaborators takes time: Setting up the of stakeholders: With a range of stakeholders
testbed took longer than anticipated, both from diverse sectors involved, competing
from the private and public sector point of objectives were an issue. While the firms aimed
view. The following elements were identified to access data and verify their products, others
as particularly time-consuming: i) getting wanted to use the testbeds to incentivise SME
the core team in place, ii) determining terms innovation, or help local NHS bodies, with little
of collaboration and shared objectives, iii) tradition of R&D work, ‘catch up’. It was vital
information governance and decision-making, to clarify which aim was most important and
and iv) testing and evaluation. stick to it throughout. Managing the number of
stakeholders and number of technologies tested
• The importance of quality of leadership
supported clarity of the aims.
and ownership: Finding local bodies with

4.8 Addressing market failures and solving grand challenges

Real-world testbeds can be used to incentivise economic activity or steer innovation in

directions that respond to challenges which the market does not necessarily solve on their
own. This use case is about addressing market failures and solving grand challenges.

Description of the use case

Both market failures and grand challenges represent a broad range of issues facing society.
On the one hand, testbeds of all levels can be used to solve market failures and challenges
on the supply side, such as lack of R&D22 investment in private companies. To this end,
publicly (or otherwise) funded testbeds can incentivise R&D activity across different firms,
leading to more innovative activity and investment-ready firms.

Testbeds can also be used to incentivise development and testing of solutions to ‘grand
challenges’ or ‘moonshots’: important national or global problems not being pursued (or not
being sufficiently pursued) by actors in the market. Typical ‘grand challenges’ of our times
are responding to climate change through renewable energy, capturing carbon, or changing
transport options and behaviours.

Using real-world testbeds to achieve responses to market failures and grand challenges can:

• Contribute to creating demand for products and solutions that represent better
alternatives to what the market currently offers.
• Be used as a tool to respond to lack of risk capital being invested in unproven
technologies – government funding can share the risk of investment with the private
sector through real-world testbeds.

37
Testing Innovation in the Real World: Real-world testbeds

• Incentivise more socially and environmentally-beneficial innovation activity, through

earmarked investments targeting specific sectors or aims.
• Facilitate collaboration between sectors, academia and relevant institutions being
incentivised to achieve specific aims.
• Provide incentives to invest and carry out more testing of innovation by the private
sector, potentially improving the business case for investments in otherwise uncertain
and unproven technologies.

Key actors involved

The actors involved in this use case will naturally vary, based on the exact nature of the
testbed purpose. Generally, actors can be:

• Public and private research institutions aiming to solve grand challenges or

commercialise products.
• Institutions responding to market failures on regional, national or supranational levels.
• Start-ups and spin-outs from universities.
• Private sector companies, NGOs, philanthropists, or non-profit organisations seeking to
solve grand challenges.
• Financial institutions or investors, either oriented towards solving specific challenges or
more mature business cases.
• Public agencies or innovation agencies seeking to incentivise innovation or R&D
investments in general, or within a problem segment or sector.

Implications for design and lessons learned

The implications for design and the establishment of these real-world testbeds is dependent
on the nature of the specific purposes. There are, however, some overarching lessons and
implications worth considering for those aiming to use testbeds for these purposes:

• Both when designing to mitigate for market failures and/or grand challenges, the
products and processes tested often have no guarantees of success. It can be difficult to
balance the priorities between choosing the right technology to test and staying flexible
and responsive to changes in the market. Real-world testbeds should bring in market
experts to assess competing technologies as well as the market response.
• Real-world testbeds involved in solving market failures or grand challenges both
require cross-sectoral and/or multi-stakeholder approaches due to the complexity of
these issues. Like other real-world testbed use cases involving multiple actors, allowing
sufficient time to align aims, develop working methods and build trust is crucial for
success.
• This use case requires a common understanding of the long-term horizon of
results. Grand challenges and market failures are seldom a ‘quick fix’, and success
is often dependent on patient capital and space for mistakes or wrong directions.
Communicating progress frequently, transparently and effectively to involved
stakeholders is key to building the required patience and understanding.

38
 Testing Innovation in the Real World: Real-world testbeds

Case study

Testbeds in Sweden – a national strategy responding to

multiple challenges

Geography: Sweden
Funding

Facilitators: The Swedish Government, The VINNOVA support State funding hopefully
Innovation Agency Vinnova for test and
demonstrations
attracts private
investments

Access to state Need for

Timeframe: 2016 – ongoing R&D funding private capital

Idea/technical opportunity Innovation/commercialisation

Description

The national strategy ‘Testbed Sweden’ was opportunities for SMEs to test their products and
set up to provide a common framework across processes in a real-world environment despite
Sweden, coordinating the many existing test and lack of resources; and narrow the ‘valley of
demonstration facilities across the country. Some death’23 for firms, caused by the lack of risk capital
of its aims include to: coordinate existing testbeds and decrease in business R&D investments (as a
better and establish new facilities that can solve percentage of GDP).
challenges for society; engage and facilitate

Key success factors and lessons

• A coordinated network of testbeds can • Public investment in testbeds can contribute

contribute to solving complex challenges: to closing the funding gap for innovative firms:
Countries across the globe are likely to Testbed Sweden is partly developed to close
already have a range of testbeds operating the widening ‘valley of death’ for firms. As the
individually. Coordinating them into networks, picture above depicts, the theory states that
both nationally and internationally is, public funding and coordinating testbeds
according to the Swedes, necessary to solve will incentivise firms to bring their ideas and
the many complex challenges facing modern technical opportunities closer to successful
society. The Swedish National Testbed Strategy commercialisation. This progress then
aims to facilitate better coordination between incentivises private capital investment when
the many individual testbeds operating across the technologies are more mature, narrowing
the country. Having them operate as an the valley where many innovators struggle.
integrated system is meant to enhance the
• Engaging SMEs in testbeds is challenging: A
quality of each testbed, increase the number
challenge of many testbeds in Sweden has
of innovators using the testbeds, and improve
been to support SMEs to participate. SMEs
Sweden’s opportunity to market the country
often struggle with resources to be able to
with an ‘innovation supply-chain’. They are
test their technologies and commercialise
also looking to integrate them into a European
their innovations or access testbeds if they are
system of testbeds.

39
Testing Innovation in the Real World: Real-world testbeds

not customised. Some testbeds are providing as an integral part of the innovation system,
access to expertise and wider mentoring and allowing innovators to advance their solutions
business development advice, although they in a secure environment. An evaluation of
are still rare. testbeds within health and social care has
shown that several tested innovations have
Testbeds are a critical part of the innovation
gone on to implementation, while others have
and commercialisation process: The Swedish
been identified as less mature. Interviewers
Government is convinced that investing in
also claimed that testbeds contributed to
testbeds supports economic growth of the
better coordination of private and public
country by being open and providing the right
stakeholders, one of the key aims of the
environment for investors. It is recognised
strategy.

4.9 Capturing competitive advantages from frontier technologies

With the speed of advancement within new segments of technology, opportunities to solve
problems in innovative manners open on several levels. Regions, nations and institutions can
gain competitive advantages through facilitating for testing and implementation of such
technologies. This use case illustrates how to best capture these advantages.

Description of the use case

The Fourth Industrial Revolution is defined by technological breakthroughs such as artificial
intelligence and machine learning, robotics, 3D printing and the Internet of Things.24 They
offer a multitude of opportunities to reimagine how we approach tasks and systems. These
types of technologies have paved the way for innovation in the realm of into ‘waves’, each
lasting two years, to enable CAVs, smart city technology, and a range of automated services
previously carried out as manual tasks.

Technologies such as these are dependent on being thoroughly tested in real-world

environments to ensure (amongst others) safe operations, functionality, usefulness and citizen
acceptance. Real-world testbeds are crucial tools to achieve such testing, and the areas
facilitating it can in return achieve competitive advantages. Real-world testbeds operating in
this realm can:

• Ensure relevant systems are developed and prepared for the next wave of technological
advancement, being ahead of the wave instead of lagging behind.
• Represent both national strategic opportunities for future development and/or
commercial opportunities for relevant sectors or firms.
• Capture first-mover advantages for stakeholders involved.
• Coordinate the necessary stakeholders through a testbed to foresee potential issues and
challenges of implementing the technologies.
• Allow users to participate in the testing, which also leads to familiarisation with the
technology and direct feedback and learning.
• Facilitate collaboration between large and/or foreign companies seeking real-world
testing environments with local firms and SMEs, which could potentially lead to
knowledge spillovers.

40
 Testing Innovation in the Real World: Real-world testbeds

Key actors involved

The core actors involved when using real-world testbeds for capturing competitive
advantages of frontier technologies can be:

• National or supranational governments or institutions seeking to gain advantages.

• Internal test experts (testers) and external stakeholder (advisers) technology experts.
• Relevant regulators (often more than one).
• Economic development agencies or similar institutions.
• SMEs and large companies.

Implications for design and lessons learned

Based on interviews with experts and relevant real-world testbed practitioners, the following
design implications and lessons learned can be drawn:

• Real-world testbeds developed for this use case are often large-scale infrastructure
or projects with similarly large costs suitable for large partnerships and collaborative
funding.

• As this use case explicitly deals with (often) large-scale new technology, it will almost
always require regulatory adaptation and thereby collaboration with regulators at an
early stage of the design process. Designing real-world testbeds to allow for regulatory
testing can add to the benefits, as there are no definite answers regarding how to
regulate new technology.

• The real-world testbed will be stronger if it involves the end user/potential customer of
the technology from an early starting point, to work with attitudes and adaptation to
preferences.

• Large-scale technological opportunities often require long-term funding and sufficient

time to allow for progress and results. Patience in results and capital is important.

• These types of real-world testbeds are often suitable at a global scale – attracting
particularly good research environments and large companies with capacity to innovate
at this level.

41
Testing Innovation in the Real World: Real-world testbeds

Case study

Smart Santander – facilitating a competitive advantage for IoT

technologies in cities

Geography: Santander (Spain)

Partners: European Union Future Internet

Research and Experimentation, City of
Santander, Santander Local Economic
Development Agency, 25 partners from
across Europe and Australia

Timeframe: 2010-2014

Description

Smart Santander was an Internet of Things (IoT) study focusses on the city of Santander in Spain,
testbed operating across Europe from 2010-2014. where they deployed 12,000 censors across the
At the time, it was the most extensive urban IoT city for various purposes. The testbed facilitated
infrastructure in the world, comprising thousands testing of approaches to and building blocks of
of sensors communicating with the deployed IoT IoT architecture, and the interaction with device
architecture. IoT is and was a frontier technology technology and their use in delivering public
with big rewards for the first cities and firms that services.
could crack the technology and its uses. This case

Key success factors and lessons

• Testing in a real-world environment allowed • Involving citizens boosts credibility and

for better research and collaboration: The adaptation levels: As citizens were brought
real-world environment provided a more along on the journey of embedding sensors
practical approach for researchers, allowing within the city using the services during testing
them to learn and improve products quickly. stages, it allowed for a status change from
It also facilitated collaboration between ‘testing’ to ‘operational’ following the testbed
research institutes, citizens and private sector period. The public continued to use the
actors, creating a framework for more rapid technology in their daily lives. A survey carried
development of the IoT technology that has out by McKinsey showed that the citizens of
moved beyond the testbed phase. The number Santander were both aware of the sensors
of sensors deployed across the city increased across the city, and highly satisfied with the
from 12,000 during the testbed period to services they contributed to.
over 20,000 that are now used in day-to-day
management of the city.

42
 Testing Innovation in the Real World: Real-world testbeds

• Clarity of purpose is key to innovating ahead learning and improving, well-run partnerships
of the wave: The purpose stated by the Smart with win-win outcomes, and sufficient long-
Santander project was two-fold: 1) to facilitate term funding. Funding was also spread over
real-world experimentation of IoT technology several aspects of the IoT, reducing the risk
to achieve a leading role for European cities in of investing in ‘future trends’ and facilitating
IoT technologies, 2) to provide useful services learning between projects.
tailored to the cities during the testbed
• A successful testbed can lead to changes in the
operation. It has also achieved the aims set by
economic environment: Santander, originally
the EU, one of which was to ‘achieve a leading
a heavy service economy, has seen the smart
role in Europe in the IoT technologies’. The
city investment enabling a revitalisation of the
European Commission stated that “The project
business and entrepreneurial ecosystem. This
has successfully achieved all its objectives,
is also due to an integration of the smart city
even exceeding initial expectation.”25
project into the municipality’s strategic plan,
• Patience and scale are necessary to gain the masterplan for innovation, and the Smart
real advantage: To achieve the competitive City Plan. Santander is now, according to
advantages on a supra-regional scale (as McKinsey, among the top ten strongest cities in
envisioned by the EU), the testbed area(s) Europe regarding their smart technology base,
require adequate scale, length of time to allow better than Berlin, Paris and Moscow.

43
Testing Innovation in the Real World: Real-world testbeds

5 Designing real-world
testbeds – things to
consider?
Real-world testbeds are increasingly popular and are frequently considered by local and
national institutions with various motivations. For real-world testbeds to be successfully
set up or run by stakeholders such as national and local governments, it is essential to
understand how to best design the real-world testbed to achieve their respective aims and
objectives. This section outlines eight critical considerations for any stakeholder, private or
public, seeking to establish and design successful testbeds in real-world environments. It is
informed by findings from literature, lessons learned from case studies, and interviews with
experts and real-world testbed actors.

5.1 Clearly define the purpose – and whether a real-world testbed

is the right means to achieve it

Although obvious, a clearly defined purpose with achievable and measurable objectives
is imperative for a successful real-world testbed. There are often a wide variety of
stakeholders involved in real-world testbeds who might have different, sometimes
conflicting aims for participation. If such aims fail to be prioritised or clarified, it may harm
the outcome and make the real-world testbed harder to govern. Aim for simplicity, clarity
and a hierarchy of objectives early in the design process. As the purpose gets clearer, ask
the question of whether real-world testbeds are the right tool.

Clarifying what any intervention aims to achieve is vital in any investment, particularly
involving spending public money. Although seemingly obvious, most real-world testbed
actors involved in this research underlined the importance of clear objectives. real-world
testbed aims are often too broad, making measurement of success and design of testbed
requirements challenging.

When Milton Keynes City Council made their infrastructure available to testing autonomous
vehicles, they ensured that the aims were specific to local challenges. Instead of having
a general aim such as ‘we want to test the use of autonomous vehicles in the urban
environment’, they required that the testbed should contribute to solving local traffic
challenges such as a specific congested area or a crossway with high frequency of
collisions. By asking more specific, local questions of the real-world testbed, the council
were guaranteed valuable outcomes and lessons learned from the presence of Autonomous
V, alongside the innovators and regulators involved.

44
 Testing Innovation in the Real World: Real-world testbeds

Recommendations
• Ask the questions ‘Why do you want to establish a real-world testbed?’ and ‘What do
you seek to achieve with it?’ The answers to these questions will also better inform the
decision of which tool to use and whether a real-world testbed is the most suitable
option.
• Different stakeholders will have diverse objectives; ensure that they do not conflict,
prioritise objectives rigorously and ensure that the key actors have a common
understanding of these priorities. This process can take time and should not be
underestimated.

Key resources
Design implications from the use cases in Section 4.

5.2 Collaborate with the right stakeholders and ensure

coordination

Real-world testbeds can be tremendously useful to break up silos and unite stakeholders
across various institutions required throughout the innovation process. A real-world testbed
can, for example, include national and local government, innovators, citizens, regulators and
researchers. In the design process, it is essential to consider which stakeholders will add value
to the real-world testbed in question and when they should be involved. We recommend not
underestimating the time it takes to coordinate these actors and establish common ways of
working, building a clear line of decision-making and avoiding ‘too many cooks’.

Most testbed experts and actors who contributed to this report mentioned the importance
of taking time in building the testbed team. First, partners need to find the right team
combination and build trust amongst the team members, and second, they need to agree
on working methods and facilitate agreements on intellectual property rights and other
sensitivities.

It can be tempting to include a wide range of stakeholders in a testbed, as many could

potentially contribute with knowledge, financial resources or other capacities. This tends
to be a balancing act, where simplicity is often key to success. One of the partners in the
Testbed Sweden strategy, Tilvaxtanalys, stated:

“When a specific task and few stakeholders characterise a project, the

implementation was fairly easy to manage as both the testbed operator
and the stakeholder tended to better understand their role in the project.
More complex and stakeholder-rich projects can be subject to more severe
challenges involving coordination and administration.”

45
Testing Innovation in the Real World: Real-world testbeds

Future users of the tested technology are important stakeholders who should not be
forgotten in the development of a real-world testbed. Public acceptance is a key ingredient
for the ‘real world’ element of the testbed. The public need to interact with the testbed
and understand its aims. Because of this, all other stakeholders need to have a common
message for the public and to engage them at an early stage.

Recommendations
• Create a broad stakeholder map of who might be involved in the real-world testbed.
Follow the steps of identifying relevant stakeholders, analysing their perspectives and
interests, mapping the relations between the team and other stakeholders, and prioritise
their relevance.
• Avoid ‘too many cooks’ unless the capacity of the real-world testbed is designed to
handle it. Use the objectives and purposes to ensure that any participant contributes
toward accomplishment.
• Ensure quality of leadership and passionate individuals with capacity to engage with the
testbed activities throughout the given period.
• Ensure a suitable organisation, a good framework of collaboration, a clear decision-
making strategy, and clear division of responsibilities.
• Where possible, involve citizens or users in the real-world testbed. Of the real-world
testbeds interviewed that did this, they underlined its added value to the investment and
a much easier road to implementing and disseminating the innovation after the testing
process.

Key resources
Stakeholder engagement tools, for example: www.bsr.org/reports/BSR_Stakeholder_
Engagement_Stakeholder_Mapping.final.pdf

5.3 Organising and funding the real-world testbed appropriately

Organisation and funding of a real-world testbed can be conducted in many ways. The
testbed can be formal or informal, long-term or short-term, and run as a business or a
private-public partnership. A conscious decision needs to be made related to the context in
question. We recommend looking to similar testbeds, not over- or under-funding, taking a
step-wise approach, and seeking to maximise the output.

Real-world testbeds are diverse. Some are ‘formalised’ as continuously available and
specialised environments, for example in Norway, where a real-world testbed continuously
offers a piece of the fjord to innovators who wish to test autonomous shipping technology.
Others are more ‘informal’, for example the testing of transport technology in Milton Keynes.
They are open to innovators approaching them to request using the urban environment for
testing, however only the projects that are deemed most beneficial to Milton Keynes are ‘let
in’. Other real-world testbeds can be run like a business, with employees running the facility,
testbed customers (the innovators), and capacity management.

46
 Testing Innovation in the Real World: Real-world testbeds

In terms of funding, real-world testbed costs vary widely, depending on the scale and
maturity of the innovation that is set to be tested. Whether short-term or long-term funding
is required should be aligned with the purpose and objectives. The real-world testbed may
be relatively inexpensive if the real-world testbed is short-term, if the innovator carries much
of the cost, if the technology is highly mature, or if using the environment is inexpensive.

More uncertainty is associated with the real-world testbed if the purpose relates to, for
example, introducing disruptive and/or less developed technologies or deploying them in
complex environments. Achieving such aims often requires more patient funding, which
allows flexibility in design and learning from experiences along the way.

There are different examples of funding models. Testbed Sweden is a national programme
funded through the national budget to facilitate testbeds and innovation around the
country, based on the Industrial Strategy. The NHS testbed programme was a centralised
pot of funding that potential participants were able to bid for funding. And, in the Singapore
Autonomous Vehicle testbed, the private sector funded the cost of the testbed equipment
and design, while the public sector provided access to the environments.

Recommendations
• Seek the appropriate funding linked closely to the aims and objectives of the real-world
testbed in question.
• If public money is part of the funding mix, there should be requirements attached that
secure public benefits.
• If the real-world testbed seeks to enable testing of disruptive technologies or solving
grand challenges, funding needs to be sufficiently patient.

Key resources
Funding strategies such as:

Deloitte Smart Cities Funding and Financing Strategies: https://www2.deloitte.com/us/en/

pages/public-sector/articles/smart-cities-funding-and-financing-strategies.html

How to develop a funding model: https://ssir.org/articles/entry/finding_your_funding_model

47
Testing Innovation in the Real World: Real-world testbeds

5.4 Analyse risk and embed in real-world testbed design

Innovation that requires testing in real-world testbeds is associated with various levels of
risk, be it related to health and safety, usefulness, finances, regulation or sustainability.
Whatever the innovation tested, it is imperative to assess the potential risk of testing in
the real world and introducing it to the market. The real-world testbed provides an arena
to build evidence of risk mitigation and should directly influence the real-world testbed
requirements, and thereby the design. We recommend strict review that previous testing
has been carried out, a wide range of experts to ensure risks are identified from various
perspectives, the use of rigorous methodologies, and robust evidence before giving the ‘all
clear’ for commercialisation.

The level and nature of the risks are naturally reliant on the purpose, the traits of the
product, and the environment in which the testing is taken place. For example, one of the
NHS testbeds introduced new technology for self-management of diabetes patients, which
poses different risks to testing financial technology or clean energy in a local energy system.

Whichever innovation is proposed to be tested in a real-world environment requires a

strict standard that it is sufficiently safe. A risk register is required to understand what can
go wrong and the mitigations provided to avoid this. Several safety-critical risks must be
proven mitigated before tests can be carried out in the real world.

A risk assessment and mitigation register can also determine the requirements of the
real-world testbed’s design. For example, a drone that is meant to deliver blood samples
from one hospital to another needs to be able to test that the drone can navigate to the
right address, not damage the sample, deal with obstacles and handle various weather
conditions. A well-designed testbed should include the infrastructure to test these elements
and provide evidence that risks associated with this are either fully mitigated, low enough
to be acceptable, and are insurable. This ensures technologies can be safely introduced into
the market.

There will always be residual risks that are not possible to mitigate fully, even in a well-
designed testbed. However, someone still needs to accept the residual risks, which can
seem off-putting to many institutions uncertain about new technologies. UK Autodrive, the
programme testing autonomous vehicles, worked with insurance companies with wide-
reaching experience of risk assessments to understand which risks they would be able to
accept. This is an approach recommended in other testbeds as well.

Recommendations
• Involve a broad range of experts to create a risk assessment and mitigation register
covering risks related to health and safety, finance, regulation, operation, technical issues
and public perception.
• Ensure that the innovation is safe enough to operate in the real world, e.g. that safety-
critical elements have been tested in laboratories or closed-off environments.
• Ensure robust methodologies for risk assessment and mitigation are utilised.
• Use the risk and mitigation register to inform the real-world testbed design, what the
real-world testbed is required to be able to test and provide evidence for.
• If necessary, work with insurance companies or similar institutions to discuss which
residual risks can be deemed acceptable.

48
 Testing Innovation in the Real World: Real-world testbeds

5.5 Embed evaluation early

Evaluation is a core necessity of any real-world testbed to understand both the impact of
the innovation and the outcome of the process. Too few testbeds are robustly evaluated,
leading to loss of important learning and evidence. It is imperative to start any form of
evaluation early and ensure a baseline of the current environment before the intervention
is introduced. Although there is no ‘one-size-fits-all’ approach to evaluation of impact
or outcome, there are a range of supportive frameworks available to guide various
approaches.

Evaluation is a necessity in testbeds to understand how the processes and technology

affected its outcomes. Process evaluation relates to understanding how different
implementation processes affect the successful outcomes of the technology. Conducting
process evaluation can be just as important as outcome evaluation, as certain aspects of
implementation could affect replicability of safe and effective use of the technology in
demonstration or commercialisation phases.

To understand the impact of innovation in process, product or policy, testers need to

compare the outcome of the innovation with what would have happened anyway. This is
usually not straightforward because what would have happened is almost impossible to
know for sure. There are often many factors at play at once, and this can complicate simple
comparisons. Can the tester be sure that an observed change is because of the action
they took (in this case: the innovation)? Or is some other factor responsible for part of that
change? The further testers move from a controlled test environment, the harder it can be to
be sure of causality.

The most important first step in testing impact is to be clear about what it is that must be
measured. Sometimes this will be the direct output: “Were patients served more quickly
or cost effectively?”; “Did people travel in our autonomous vehicles safely?”; “How many
businesses took advantage of new smart technology opportunities?” In this case, causality is
less in question.

To understand outcomes and impacts more broadly, testers need to establish what would
have happened anyway: “Did patient outcomes improve?”; “Was there less congestion in the
city centre?”; “Did firm level productivity increase?”

Accepted methods for establishing causal impacts include a wide range of techniques. The
‘gold standard’ is randomised control trials, which are quasi-experimental approaches which
exploit elements of randomness in timing, allocation or placement. But simple difference-
in-difference based approaches are also helpful: these either define a comparison group
or use regression techniques to control for the other factors that might simultaneously
influence change.

49
Testing Innovation in the Real World: Real-world testbeds

Recommendations
• Start early. Familiarise yourself with and decide the method that will be adopted and the
scope of the evaluation.
• Identify the right, unbiased evaluation team. Ensure they have a sustainable governance
of evaluation throughout the testbed process. Some real-world testbeds may consider
procuring evaluation from an external actor, or collaborate with researchers.
• Identify the right questions to ask and collect the necessary data before the intervention,
so there is an established baseline for comparison.
• Ensure that the outcome of the evaluation is agreed as a good enough standard to
further deploy the innovation if deemed to have positive impact (for example, across a
health system, across necessary geographies, etc.).
• Spend time thinking about whether the comparison method you have established is
appropriate – are you comparing ‘apples with apples’? This is a matter of common sense
and knowledge of the specific policy and locality rather than of mathematics.
• Identify and collect the learning from the evaluation in a succinct manner and ensure
good dissemination is in place to ease the spread of evidence to the right actors.

Key resources
The NHS testbed evaluation handbook has an overview and description of the various
methods used for evaluation, as well as a full overview of the evaluation process: https://
www.england.nhs.uk/wp-content/uploads/2018/11/test-beds-programme-evaluation-
learning-from-wave-1.pdf

The ‘How to Evaluate’ section with the What Works Centre for Local Economic Growth
(WWCLEG) is very useful to better understand methods of evaluation:

• What to Evaluate: https://whatworksgrowth.org/blog/how-to-evaluate-what-to-evaluate

• Define Success: https://whatworksgrowth.org/blog/how-to-evaluate-define-success
• Start Early: https://whatworksgrowth.org/blog/how-to-evaluate-start-early
• Get Everyone on Board: https://whatworksgrowth.org/blog/how-to-evaluate-get-
everyone-on-board
• Plagiarise: https://whatworksgrowth.org/blog/how-to-evaluate-plagiarise
• How Long: https://whatworksgrowth.org/blog/how-to-evaluate-how-long
• Collect Data: https://whatworksgrowth.org/blog/how-to-evaluate-collect-data
• Find a Control Group: https://whatworksgrowth.org/blog/how-to-evaluate-find-a-
control-group

50
 Testing Innovation in the Real World: Real-world testbeds

5.6 Market your testbed and include the right innovators

Regardless of how the real-world testbed is organised, there are a range of innovators that
can be involved. However, as the bank of examples shows, large and resourceful firms are
often easier to engage, while SMEs and third sector actors require more targeted marketing.
Using marketing and building networks to find the right innovator for the testbed can
uncover new firms and emerging talent in the industry.

Especially when real-world testbeds are long-term stable environments open to innovators,
they require a plan for marketing. This would allow facilitators to reach out to suitable
innovators, highlighting the opportunity it represents. Most real-world testbeds involve large,
well-resourced firms because of their market share, economies of scale or networks, but
they might not always have the best new ideas for testing.

SMEs, third sector organisations, international or non-local actors can miss out on such
opportunities, either through lack of information or resources. In many cases, disseminating
the opportunity to these and other relevant innovators is crucial to ensure the real-world
testbed is inclusive and that the state-of-the-art technologies are represented. A mix of
larger and smaller innovators could also lead to knowledge spillovers and improved local
economic impacts (should the innovators be local).

Key resources
Resources for marketing are widespread and difficult to generalise for real-world testbeds
specifically. In terms of including SMEs, which many real-world testbeds share, the table
below illustrates some frequent challenges based on the shared difficulty of including SMEs,
and potential solutions.26

Challenge for SMEs Potential solutions

The testbed is too costly for Subsidising the participation of SMEs either directly or through challenge
participation. funding earmarked for a certain type of innovator.
Providing SMEs with free access to the real-world testbed.

Lack of knowledge about real- Having part of the real-world testbed budget earmarked for marketing
world testbed opportunities purposes.
for relevant SMEs, or doubt in
Creating support structures for SMEs that increase their potential benefits and
the worthiness of the testbed
outputs.
participation.

SMEs not the primary target As many real-world testbeds may prefer to work with large companies, those
group. that involve SMEs, or plan to do so, can be rewarded or gain preference in
funding applications.
Smaller innovators can be involved through working methods such as
workshops or labs concerning the real-world testbed topic.

Fear of exposing their products/ Having clear frameworks on Intellectual Property (IP) implications and building
processes. trust and transparency within the testbed.

51
Testing Innovation in the Real World: Real-world testbeds

5.7 Working with regulators

In many cases, especially with disruptive technologies, regulation may be less developed or
in some cases not exist whatsoever. Real-world testbeds could be a valuable tool for both
regulators and innovators to build a dialogue, and for regulators to build the necessary
knowledge to regulate and get answers to their questions regarding the innovation. Early
involvement of regulators allows them to co-design the real-world testbed, build a trusting
relationship and find structured ways of working.

Regulation is a key element of the introduction of innovation into the real world. Regulators
should be brought along at every stage of the process and should be a collaborator rather
than an organisation causing barriers to innovation. At the beginning of a project, a
stakeholder analysis should be conducted that will identify regulators in the field.

Recommendations
• Be clear about what is being tested and the relevant laws, regulations and policies which
may be affected, then clarify whether the testbed will need authorisation or involvement
from more than one regulator.
• The earlier the involvement of the regulator, the better. This provides the opportunity to
better understand the regulatory frameworks and include this into the design. Regulators
may be able to help to create a ‘safe space’ for testing technologies and get to test
methods of regulation themselves.
• It may be possible, or even necessary, to involve the regulators in the design and
governance of the testbed. This should be regarded as an opportunity, not a barrier.
• Learn from other regulatory testbeds operating in similar sectors and environments;
there are potentially opportunities to share lessons learned and avoid making similar
mistakes.
• Policy labs can be a good approach to integrating regulation and other required
competencies within a testbed, or across several testbeds. A policy lab can be a
temporary initiative or a more long-term and permanent investment. This is a format
that has the potential to work well with testbeds, as it provides an additional feature
of seeing the regulation ‘in action’. In Gothenburg, policy labs have been introduced
as a means of tackling some of the more complex issues around innovation, such as
autonomous vehicles and their integration into society.

52
 Testing Innovation in the Real World: Real-world testbeds

5.8 The transition from testing to commercialisation

Once the technology has been proven safe, effective and ready for demonstration, there
are some key considerations in the transition to demonstration and commercialisation. One
key differentiator is that in the testbed it is acceptable to fail, while in the demonstrator
failure is not accepted. A testbed is a place for testing, evaluation, learning and re-testing.
Demonstration is a phase in which the innovation – as proven safe – can be shown to
stakeholders, including regulators and the general public, to build acceptance and support.
A plan for disseminating evidence to the market and building business cases and financial
cases for market exposure should be developed alongside the testbed.

Planning for demonstration and commercialisation phases should begin in the testbed. The
learnings, improvements and relationships developed in the testbed should be built upon to
deliver demonstration and attract further investment for scale-up. The early business case
developed at the start of the testbed should be built out with the evidence from the testing,
to build the case for further investment in the development and roll out of the technology.
Using evaluations in the testbed in this phase can inform the economic and financial cases,
and any process evaluation should inform the management case for future implementation.

53
Testing Innovation in the Real World: Real-world testbeds

6 Conclusions and policy

lessons
This report has dealt with the question of what real-world testbeds are, why they are useful,
and how they can best be utilised. The findings and framework of this report is based on
desk-based research, a bank of eighty examples from across the world, case studies and
interviews with testbed actors, and expert insights.

This research defines real-world testbeds as:

“Controlled or bounded environments for testing innovations in real-world

or close to real-world conditions in the manner (or close to the manner) in
which they will be used or operated.”

Amongst other qualities, they can reduce the barriers to testing by helping manage risk,
change and test regulation, allow access to real-world environments and user groups or
their associated data, which is often a necessity for innovators before commercialisation.
Real-world testbeds are particularly suited for technologies operating in complex systems,
requiring evidence and interactions with the real world to prove their safety and usefulness.
The format of testbeds allows the management of risk, experimentation, learning,
evaluation and de-risked failure.

There are examples of established and new real-world testbeds across most sectors.
They include innovators, local and national governments, public sector bodies, research,
charities and other institutions. With technology moving at such fast pace, it is crucial for
policymakers to facilitate testing trailblazing innovation in the real world, to learn about
their effects. It is also vital for innovators to gain access to testing their products, processes
and services in the environments they are intended to be used in.

Public sector involvement in this kind of innovation infrastructure is important to incentivise

the sector to innovate in a safe and controlled manner. Using environments such as
hospitals, roads, schools, land, university campuses or neighbourhoods as testbeds also
makes more use of such infrastructure, adding to their value to society.

Real-world testbeds can be used for many different purposes and objectives, depending on
the stakeholder and context of the testing. At the core of every testbed is the need to test
new innovation, often new technologies, for verification and proof of concept. They can also
be used in a place-focused manner, serving as a tool to solve local challenges, market a
place to businesses and talent, and contribute to local economic development. They can be
used to respond to market failures and grand challenges or facilitate gaining competitive
advantages in frontier technologies. Real-world testbeds can also be used to cut costs
and improve public services, testing which technologies serve this objective better. Finally,
this report has covered the use case of exploiting real-world testbeds to adapt regulatory
frameworks.

54
 Testing Innovation in the Real World: Real-world testbeds

Lessons for future testbeds

Testbeds should be considered as a tool to incentivise innovation and the spread

of good solutions
With all their potential benefits, testbeds should be considered as a tool used in innovation
policy across sectors and geographical levels of authority. If designed and operated in the
right manner, they can facilitate much-needed collaboration and coordination of private
and public stakeholders with aligned objectives. They can de-risk the process of procuring
innovative solutions to complex problems for the public sector and incentivise socially
beneficial use cases of innovation.

Testbeds should not be stand-alone policies, but part of a more extensive strategic
approach
Real-world testbeds will have a greater impact if co-ordinated with other innovation
policies in parallel. Real-world testbeds and other test and demonstration facilities such
as labs and simulated environments should be part of a broader innovation infrastructure,
enabling learning from one another’s approaches. Other innovation policies along the
innovation process, such as supporting idea generation and development, are still vital and
should be part of a strong and integrated innovation system on a national and local basis.

A coordinated testing and demonstration infrastructure could deliver additional

benefits
As various types of testbeds and demonstrators are established across the world, innovation
agencies should gain an overview of these programmes in order to coordinate, market
and share learning across the various initiatives. This is carried out successfully in Sweden
through a national strategy for testbeds – a process that other countries should seek to
learn from.

Regulators should be involved early and actively

Real-world testbeds represent a unique opportunity to test regulatory approaches for
trailblazing technologies. If regulatory bodies are involved at an early stage and throughout
the process, there is a higher probability that proper and safe regulation can happen.

Relevant governmental and/or innovation agencies should seek to agree on a

shared terminology for innovation policy tools
To ensure a shared understanding of the many terms and tools within innovation policy,
relevant government bodies and institutions should agree on shared terminologies. This is
important to avoid confusion and create clarity and standards for testers, regulators and
investors.

Ethical considerations should always be front and centre

Many testbeds deal with complex ethical situations, including use of citizen data and
introduction of health technology. These considerations should always be front and
centre when testing innovation, and the time spent to deal with them should not be
underestimated.

55
Testing Innovation in the Real World: Real-world testbeds

Appendices
Appendix 1: Bank of examples

Name Geography

The Canary Islands Oceanic Platform Canary Islands (Spain)

Autonomous shipping testbed Norway

I-STREET USA

NHS Testbeds programme England

Singapore AV initiative Singapore

Maritime Autonomy Surface Testbed UK

Smart Kalasatama Finland

Living Lab Bus Finland

TRL UK Smart Mobility Living Lab UK

Growth Corridor Finland

UK Autodrive UK

Smart Santander EU/Spain

EnergyBlock Denmark

Copenhagen Street Lab Denmark

ROBOAT US/Netherlands/UK

LaMiLo – Last Mile Logistics EU

Digital Motorway Testbed Germany

US Autonomous Vehicles Proving Ground US

Urban Drone Testbed Sweden

Jurong Lake District Singapore

FinTech Regulatory Sandbox Singapore

Number 9 – Unmanned and manned aircraft testing Finland

Sohoja and Nordic Way Projects Finland

ICT testbeds project UK

Pervasive Nations Ireland

Smartbay – Test, validate, innovate Ireland

Concept House Village Netherlands

56
 Testing Innovation in the Real World: Real-world testbeds

Hållbarheten – Smart Homes Sweden

IoT Lab France

DRIVEME Sweden

ElectriCITY Sweden

LA Cyber Lab USA

FED – Fossil Free Energy Districts Sweden

Chalmers Five Star Campus Sweden

Innovation Platform VGR Sweden

Test site Akvamarin Sweden

PICTA – Perhospital arena, mobile testbed Sweden

Testfield Niedrichstrassen Germany

Project Wing Australia

GATEway project UK

Midlands Future Mobility UK

Nordic Testbed Consortium Nordics

Sunnaas Testbed Facilities Norway

Centre for Welfare Technology Denmark

Centre for regional development, research and innovation unit Denmark

The health and care administration testbed Denmark

Intervention Centre, Oslo University Hospital Norway

Operating room of the future US

Sheridan Elder Research Centre Canada

Save the peaches Argentina

Queen Elizabeth Olympic Park UK

Mistel Sweden

Drone Centre Sweden Sweden

UK 5G testbed UK

The Sharing Cities Programme EU

Riksbyggen Positive Footprint Housing Sweden

Elmob – electric vehicles for sustainable travel Sweden

57
Testing Innovation in the Real World: Real-world testbeds

iDag – innovative test environment for distance learning Sweden

Carbon Credits from the woods Sweden

SICS ICE Sweden

High Voltage Valley Sweden

Experio Lab Sweden

Karolinska – Testbed for telemedicine Sweden

The national testbed for innovative radiotherapy Sweden

KTH Live-In-lab Sweden

iZone USA

MindCET Israel

Digital Promise USA

European Schoolnet Europe

EDUlabs Estonia

Qinzhou Intelligent Urban Water Supply cloud service testbed China

Manufacturing quality management testbed China

HarborNet – A real-world testbed for Vehicular networks Portugal

World Cup Russia – Testbed for Sports Technology Russia

5G testbed, Department of Telecommunication India

Cross-border regulatory testbed for AVs France/France/Germany/

Luxembourg

Digital Greenwich UK

FCA regulatory sandbox UK

Krafla Magma Testbed Iceland

5G RuralFirst Orkney

Marriott Hotel Testbed Middle East and Africa

58
 Testing Innovation in the Real World: Real-world testbeds

Appendix 2: Full case studies

2.1 Milton Keynes as a testbed – key facts

Description

Milton Keynes is the largest ‘New Town’ in England. Founded in 1967, the city was originally
conceived as a commuter hub, situated roughly equidistant between London and
Birmingham and Oxford and Cambridge. It is currently one of England’s most economically
vibrant towns, with one of England’s most productive workforces, generating more start-ups
than any city outside of London.

Milton Keynes has a rich tradition of testing technology in the urban environment. The
first testbed in Milton Keynes was the ‘MK: Smart’, a project to develop capability for
management and use of big data. This was a £16 million collaboration between academia,27
industry28 and MK Council. The collaboration created a data hub based at the Open
University, focussing on energy, transport and water monitoring and data collection. The
city has subsequently collaborated with testbeds on electric bus schemes, rapid electric
vehicle charging, CAVs, delivery robots and sensor technology.

Purpose: Solving local urban challenges through testing new technology, building on the
‘brand’ of being a New Town through being innovative and testing the validity of new
technologies.

What is being tested

• Electric bus fleet: A testbed aimed at delivering a clean and commercially viable electric
bus fleet.
• UK Autodrive: A three-year, consortium-based project facilitated by a government-
backed competition to support the introduction of self-driving vehicles into the UK.
• MK:Smart – MK data hub supporting acquisition and management of big data relevant
to city systems in the areas of energy, water and transport management; A smart city
education programme providing training in advanced technology; Engaging citizens
through a citizen lab.
• Starship delivery robots: Starship, a company using robots to deliver food and products.

Core lesson – using testbeds for marketing, local challenges and economic
development purposes
Milton Keynes has operated as a real-world-testbed for a number of years, providing areas
within the city for projects and companies seeking to test their technologies and solutions.
This is claimed to be important both to creatively solve urban challenges and reinvent itself
for the 21st century. In their Economic Development Strategy, Milton Keynes City Council
states that MK is at the forefront of new and innovative technologies, and that the status
as a forward-thinking, smart city should be used to further brand the city and enhance the
sense of place.

59
Testing Innovation in the Real World: Real-world testbeds

“It gives the city a competitive edge, as a location of choice for business
development both nationally and internationally, and as a city where people
want to live, work and visit.”
Milton Keynes Economic Development Strategy 2017-2027

The council has several designated roles working with testbeds, including Head of Transport
Innovation and a Director of Strategies and Futures.

• Using real city challenges to design the real-world testbeds: One key principle on
which the MK Council base their testbed activity is whether the technology tested can
contribute to solving local challenges. Many of the MK testbeds have been focussed
on transport as the city is planned to double in size by 2050. With this ambition comes
a requirement to adapt the transport system for more passengers, whilst avoiding
worsening air quality, an increase in carbon emissions, and expensive investments in
infrastructure.
In several cases, MK has also designed the challenge for which the technology can be
tested to solve. For example, in the UK Autodrive project, the challenge was set to create
a more liveable city centre less dominated by cars, and solve some of the safety issues
such as their problems with sideway collisions: a real problem causing the council a lot of
anxiety.

• Using testbeds as a response to budget constraints: Using testbeds to trial technology

also contributes to solving urban challenges in a world of public sector budget
constraints. Procuring unproven technologies to solve challenges can be both risky and
expensive. The MK approach has been to facilitate companies’ opportunity to test their
products and develop their business case. The reward from this is that the technology
is verified and can be implemented into other markets more rapidly, which is a better
alternative to funding such a solution, which the council on their current budget would
struggle to do on a similar scale. The testbeds also provide a stronger case for whether
this is ‘the right tech-horse to gamble on’.
• Attracting investment and activity: The testbed activity and overall brand of Milton
Keynes has led to companies ‘lining up’ to test their technology in the city. This has often
come as a result of participation in large projects, such as UK Autodrive. The projects
are prioritised based on which problems they can solve for the city, or which represent
other clear benefits. Not all projects materialise, but it provides opportunities for inward
investment and activity, an aim in the city strategy.
• Playing to the local strengths: In addition to responding to local challenges, Milton
Keynes has also played to local economic and locational strengths. The robot delivery
and autonomous pods testing has been easier to carry out in the city as it is designed
more recently than many other cities with, for example, wider footpaths. Another
example is the idea of facilitating drone testing in the area, as the city is located in close
proximity to an Amazon distribution centre.

60
 Testing Innovation in the Real World: Real-world testbeds

Other lessons learned

Using the testbed to verify the technology

• Testbeds can be useful not only to solve urban challenges and promote economic
development, but also as a useful validation method to see whether the technology
is all that it claims in the sales brochure. Testing technologies in the local context will
contribute to fewer unviable investments, and also verify whether the technology in
question functions in the local environment with a true demand from citizens.

Regulatory challenges

Testbeds can also contribute to solving regulatory challenges associated with new
technologies, for example in cases where the technological development has moved beyond
the appropriateness of current regulation.

• The Milton Keynes Transport Plan further states one of the desired outcomes as ‘a
modern regulatory system that works to improve the way transport regulation supports
improvements in the transport system’. Testbeds can provide an opportunity to test and
adapt current regulations, developing the framework alongside the tests and avoiding
being ‘on the back foot’ once the technology is ready to be deployed.
• Milton Keynes has also contributed to regulatory discussions with the Department
of Transport, together with some of the companies testing technology in the city. An
example is the testing of electric cycles with Lyme and electric scooter companies, which
are currently prohibited on footpaths in the UK. The experiences from testing these
services in MK could better inform a potential adaptation of such regulation.

Incorporating learning and allowing flexibility

Testbeds can, if designed accordingly, be a great way of learning how to improve urban
design and development.

“There are outcomes that could never have been anticipated, new questions
that come up along the way, and we learn from that as much as the
technology developer. It gives us insight on how to develop our city for the
future of transport.”
– Brian Matthews, Head of Transport Innovation at Milton Keynes Council

• Several outcomes from testing technologies can provide valuable learning that could
not have been obtained outside of a real-world environment. One example is the testing
of the Starship Delivery Robots. The expectation and scepticism prior to the project
concerned whether it would make people lazier, stopping them cycling to shop for
groceries. Instead, the scheme removed about 1000 cars off the road, as the project
illustrated that trips to the grocery shop were carried out by car, an unexpected outcome
and a positive surprise.

61
Testing Innovation in the Real World: Real-world testbeds

• The evolution of testing means that the right questions will not always be obvious at the
beginning of the testbed period. It is important to stay flexible to develop the testing as
the questions and assumptions change.
• Another unexpected outcome from the Starship Robot testing came from a citizen in
Milton Keynes:

“We received a letter from a mother who had a child that refused to eat
fruit and vegetables…Until the day it was delivered by a robot.”

Involving citizens

Involving citizens is a core priority in Milton Keynes’ testbeds, as they are the primary users
and beneficiaries of the technology being tested.

• Testing new technologies with citizens depends on dissemination. If the testbed is carried
out for the right reasons, it is much easier to get the citizens on board. When MK Council
tests autonomous vehicles to respond to the need for a better transport system, the
citizens are more willing to accept the testing.
• Most of the technologies tested in Milton Keynes have had a positive reception from
the public. There can often be initial concerns which relate to ‘fear of the unknown’,
however this largely disappeared with a planned and managed way of introducing the
technology.

Collaboration with the private sector

• Aligning objectives can take time when there are many participants in a partnership,
however frequent meetings with the same representatives and an open approach tends
to build the necessary trust. With UK Autodrive in Milton Keynes, it took ten months to
establish the collaboration agreement across 16 partners.
• It is also important to listen to private sector partners to create as much of a win-win
situation as possible. Jaguar Land Rover was part of the UK Autodrive partnership, and
it took flexibility on both sides to achieve a shared objective. Milton Keynes originally
wanted them to provide services to more deprived areas with their shared car service,
however this was too far away from Jaguar’s market, which is more focused on high-end.
The council ended up agreeing to the objective of reducing the number of cars on the
road and looking for a different partner to achieve their original objective.

Concerns about data

A concern raised in Milton Keynes is the need for continued research and insights into data
management and privacy issues.

• Schemes that focus on data issues should be carried out alongside local testbeds to
improve the general environment and to stay on top of the development.

62
 Testing Innovation in the Real World: Real-world testbeds

Impact

• Milton Keynes is recognised, largely due to the testing activity in the city, as one of
Britain’s top ‘smart cities’.29
• The main impact from the high-profile testing activity in Milton Keynes is the brand
obtained by the city as the ‘place to go’ for testing innovative solutions. This can be seen
through national and international media narratives.30, 31
• The testing activity brings several companies with innovative transport solutions to
Milton Keynes and the city council. The city has managed to accept the testing of only
those solutions that are suitable to the local challenges of the city, thereby increasing the
likelihood of benefits for the city.32

References

Interviews with

Brian Matthews, Head of Transport Innovation, Milton Keynes Council

Other literature

Milton Keynes Futures 2050 Commission. Making a Great City Greater. https://drive.google.
com/file/d/1Zi-hD-RVwS_9Gz50C5m6hNhHTpcEtheN/view

Milton Keynes Council (2017) Economic Development Strategy 2017-2027 https://www.milton-

keynes.gov.uk/assets/attach/46815/FINAL%20Economic%20Development%20Strategy%20
2017-2027.pdf

Milton Keynes Council (2018) Mobility Strategy for Milton Keynes 2018-2036 – Mobility for
all. https://www.milton-keynes.gov.uk/assets/attach/51006/2018-03-21%20Mobility%20
Strategy%202018-2036%20(LTP4)%20ADOPTEd%20by%20Council%2021%20March%20
FINAL%20submitted%20copy.pdf

Simpson, Paul (2018) Smart cities special: Why Milton Keynes is more than just roundabouts.
https://www.cips.org/en-GB/supply-management/analysis/2018/september/smart-cities-
special-more-than-just-roundabouts

UK Autodrive (2018) About: http://www.ukautodrive.com/the-uk-autodrive-project

63
Testing Innovation in the Real World: Real-world testbeds

2.2 NHS testbeds – key facts

Facilitator

A cross-government venture funded by NHS England, the Department of Health and Social
Care and the Office of Life Sciences (OLS).

Purpose: Make public services better and more cost-efficient; Spreading useful innovation
throughout the NHS; Changing the NHS innovation culture.

What is being tested

Wave 1 tested three different types of innovation:

1. Predictive algorithms to manage patients at risk of developing a condition.

2. Aggregation of data into one place to inform operational and clinical decision making
and improve an individual’s ability to manage their condition.
3. Technology to monitor risk of crisis in clinical pathways or an individual home or care
home.

The testbeds in Wave 1 were:

• Lancashire and Cumbria Innovation Alliance – Improving support for those over 55
with Chronic Obstructive Pulmonary Disease, heart failure and dementia. Integrating
technologies and linking them to new care models supporting self care at home.
Participants: 1,600 Innovators: 8
• Long-term conditions early intervention programme – Promoting early intervention
to reduce the burden of ill health by developing a predictive algorithm, redesigning
pathways and training health professionals.
Participants: 214,700 Innovators: 2
• Diabetes digital coach – Providing people with type one and type two diabetes with a
selection of integrated IoT digital tools to manage their condition. Participants: 1,000
Innovators: 8
• Perfect patient pathway – Improving pathways for asthma, diabetes, falls and frailty by
increasing access to technology and facilitating information sharing.
Participants: 1,300 Innovators: 7
• RAIDplus – Developing a demand and capacity tool that shows patient flow in real time
and a predictive algorithm to identify when people are going to experience a mental
health crisis.
Participants: 33,000 Innovators: 2
• Care City – Testing a combination of digital devices and software alongside new
approaches to service delivery and patient participation.
Participants: 4,100 Innovators: 6
• Technology integrated health management – Providing people with dementia and their
carers with: wearables, monitors and other devices which will combine into an IoT to
monitor their health at home.
Participants: 1,400 Innovators: 7

64
 Testing Innovation in the Real World: Real-world testbeds

Organisation

Each testbed brought together partners from senior government, academia, industry,
patient groups and charities. Organised in Waves, the second wave commenced in 2018
and will run to 2020.

Budget

£9.5 million from NHS England, the Office for Life Sciences (OLS) and Department of
Health. £15 million was leveraged from industry.

Figure 6: Timeline and programme of Wave 1

Concept Generating Two Testbed Announcement

Testbeds interest matchmaking selection Testing in
described in the 32 applications events between 14 potential the city:
NHS Five Year from NHS sites, NHS and testbeds Level 4 testbed
Forward View 370 innorvators held industry interviewed
express an
interest

Winter 2014 Summer 2015 Autumn 2015 Winter 2015/16

Core testbed
teams recruited Implementation Legacy tools Programme Learning and
Programme 51 technologies Guides to end new wave
governance are rolled out collaboration, Wave 1 end Wave 2 is
established information initiated
Participants
Mapped out recruited to governance and
pathways testing evaluation
published
Building
relationships

Spring/Summer 2016 Autumn 2017 to Summer 2018 Summer 2018 Autumn 2018 Autumn/Winter 2018

65
Testing Innovation in the Real World: Real-world testbeds

Core lesson – evaluation

Evaluation was described as fundamental to the testbed programme. The aim was to
provide robust evidence of what worked well and areas of improvement across the following
four questions:

1. Did the intervention improve patient outcomes?

2. Did the intervention lower health system costs?
3. What changes were made to the intervention or implementation and why?
4. Have NHS-innovator partnerships worked and why?

The NHS testbeds programme was designed in waves. The first wave consisted of seven
testbeds over a period of two years. Each of the testbeds were required to have an
evaluation plan, including qualified personnel to carry it out. Many chose to collaborate
with a local university to get an unbiased evaluation. The structure of waves also allowed for
a structured implementation of the learning obtained from the first wave into the second.
In terms of evaluation, a handbook was published with detailed description of methods and
learnings, designed for future testbed designers and operators.

Figure 6 illustrates the steps taken throughout the evaluation process of the first wave.
There are a number of transferable lessons from this approach.

The NHS approach to evaluation

Step 1 – Be clear on the scope of evaluation

• Define the intervention to be evaluated, technology to be used and services to be

changed.
• Articulate the intervention and what would have happened otherwise.
• Articulate the context in which it is being implemented and the technological maturity.
• Define what is out of scope.
• Agree percentage of budget to be used on evaluation.

Step 2 – Agree what you want to learn from the evaluation

• Decide what kind of evaluation is required (process/impact/economic).

• Work with key stakeholders to articulate questions to facilitate the evaluation.
• Develop and implement learning and dissemination plans.

Step 3 – Identify who needs to be involved

• Consider who needs to use the evaluation findings.

• Identify who should input to ensure a comprehensive and robust evaluation.
• Facilitate input from funders, patients, clinicians, delivery teams, data security teams etc.
Ensure alignment with governance of the wider intervention.

66
 Testing Innovation in the Real World: Real-world testbeds

Step 4 – Set up an appropriate governance process for project close down

• Follow best-practice guidance and form an advisory group with external experts, with
clear reporting lines.
• Ensure appropriate involvement form patients and the public.

Step 5 – Get the right evaluation team

• If commissioning an external evaluation team, ensure an open and fair process.

• Ensure the team has the required skills and experience in evaluation.

Step 6 – Agree ways of working with evaluation team

• Facilitate clear communication channels between the evaluation and implementation

teams.
• Set up appropriate information governance processes.
• Agree regular engagement between the teams.

Step 7 – Ensure joint ownership of evaluation approaches

• Ensure a logic model framework for the evaluation is prepared.

• Ensure the most robust methods are selected, given data and practical constraints.
• Ensure methods are appropriate, given the maturity and context of the interventions.
• Ensure methods can be integrated to maximise learning.

Step 8 – Be an intelligent customer for the evaluation

• Play a scrutiny role. Ask questions of the team to ensure you fully understand the
activities.
• Monitor patient recruitment activity and be prepared to suggest adapting to maximise
numbers.
• Test and challenge findings to ensure appropriate interpretation.

Step 9 – Ensure preparation of a clear evaluation protocol

• Agree accountable representatives across evaluators and implementors with the relevant
ethics clearance.

Step 10 – Ensure findings and context are clear in final reporting

• Ensure context for the interventions is clearly conveyed (the system in which the
interventions were implemented).
• Ensure findings identify conditions under which outcomes were observed, for whom
and why.
• Sense-check results and findings, including limitations and unintended consequence.

67
Testing Innovation in the Real World: Real-world testbeds

Step 11 – Take stock of learning and embed it in future action

• Identify key points of learning from the evaluation.

• Identify practical ways to put this into action for future interventions.
• Translate evaluation findings for policy makers to use.
• Set up framework to continue to monitor ongoing interventions.
• Make appropriate learning experiences available to relevant stakeholders.

Step 12 – Ensure dissemination plan is put into practice

• Continue appropriate sharing of learning throughout evaluation.

• Ensure dissemination materials are prepared with clear messages.
• Ensure frontline staff, public and patients are among those with whom learning is shared.

Other lessons learned

Further lessons learned were obtained from a series of interviews with key stakeholders
involved in the first wave of testbeds.

Finding and working with the right partners and collaborators

The NHS testbeds programme had several stakeholders involved in different ways of
working. The seven testbeds in Wave 1 were established based on the technology tested and
the patient groups they were working with. Some were naturally more comprehensive than
others. However, some overarching lessons were valid across all:

• Do not underestimate the time required to develop mutually beneficial partnerships,

recruit patients and operationalise the innovation. Getting the testbeds set up took
longer than anticipated, both from the public and private sector point of view. These
were areas that the first wave identified as particularly sensitive to time management:
·· Getting the core team in place
·· Determining terms of collaboration and work
·· Testing phase
·· Evaluation
·· Information Governance and decision-making

• The NHS programme involved several stakeholders from national and local levels of the
NHS and government, large and small private companies, academia and the voluntary
sector. The more partners were involved, the harder it became to align competing
objectives. For example, one testbed had 19 partners, leading to difficulties in the
decision-making process. One of the interviewees claimed that it would have been
much easier to disappoint a further ten partners, than to work with such a high number
of partners. Smaller partnerships with fewer stakeholders within the group were more
successful.

68
 Testing Innovation in the Real World: Real-world testbeds

Clarity of purpose and simplicity of tasks

• With the private sector, national and local public sector representatives involved, there
will be a series of competing objectives. The companies aimed to partner with the NHS,
access patient data and verify their products. Some wanted to use the testbeds to allow
SMEs access to innovate in the NHS, while others wished to use them to make local NHS
bodies, with little tradition for R&D work, ‘catch up’. It was vital to clarify which aim was
most important and stick to this aim. Although this was very difficult, the core aim of
establishing cost-efficient methods of delivering quality care was set as the primary goal.
• Clarifying what technologies will be tested is also key to success. It can be tempting to
test a range of different products, however those that had a more straightforward design
with fewer technologies were also the most successful.
• Asking the industry the right questions about how their products fit into existing
pathways is critical for the long-term success of the technology implementation.

The importance of qualified leadership, local ownership and hand-holding

• All of our interviewees mentioned the importance of collaborating with local NHS bodies
that were actively engaged and interested in implementing digital technologies. Without
this, they claimed the programme would struggle to function. Some also emphasised the
importance of senior involvement, in both national and local bodies, to ensure this.
• One of the significant factors that determined the applications of the local NHS bodies
was the evidence of strong leadership with proven experience in similar projects.
Testbeds that were chosen despite this were also those that struggled most in the
process.
• A lot of ‘hand-holding’ was required from the facilitators throughout the process.
Each testbed had a Senior Responsible Officer (SRO) who held monthly and quarterly
meetings with the testbed. This was key to identify and deal with challenges along the
way and ensure progress.

Funding

• If possible, try to obtain a longer-term funding stream that ensures the stability of the
programme in a political landscape with much insecurity.
• The NHS testbed programme funded several operations on behalf of companies. There
is a need to be strict to provide funding to those who need it the most. Does the testbed
need to fund large companies with stable incomes, or should the funding for the private
sector instead be used to provide access to SMEs?
• Some testbeds are not too expensive to run. These are often simple with more mature
technologies.

Information governance and risk management

With a complex system of technologies, data, different stakeholders and geographic

locations, a robust information governance framework was essential to the NHS testbeds
programme.

• The information governance process also included identifying and managing risks
related to Data Protection. It was key to create an overarching log of all possible risks
and to review and log new risks as the programme went along. The NHS testbeds
programme recommends monthly risk review meetings.

69
Testing Innovation in the Real World: Real-world testbeds

• Early and transparent communication of the testbed’s approach to using patient data,
including benefits and outcomes, is key.
• Future testbeds should not overlook or underestimate the GDPR33 implications to some
testbed objectives.

Impact

• The clearest evidence of success from Wave 1 was the substantial, continued investment
of Wave 2 by the funding partnership.
• Programme leaders claimed that, whilst not everything was effective, there had also
been huge successes. This has particularly been seen in the use of IoT devices monitoring
patients with complex conditions such as dementia and diabetes. The devices tested
showed a decrease in the number of necessary visits to health service providers, and a
better way for health professionals to monitor the condition.34
• The number of participants in the testbed programme, and the data generated from the
use of new technologies, allowed NHS health professionals to develop their services more
accurately.35
• The testbed in Lancashire and Cumbria saw impacts of a 2.7 per cent decrease in
probability of being admitted to hospitals, a 10 per cent reduction in primary care
community and emergency services, as well as an 86 per cent increase in patient
confidence on their own health, and a 68 per cent increase in indicated knowledge and
skills enabling self-management of long-term conditions.36
• Data from Phase 1 indicated that during the testbed programme, there were
improvements in experience of healthcare, resulting in a reduction of emergency care
and hospital admissions in the long term.

References

Interviews with

Will Cavendish, Previously Senior Responsible Officer (SRO) of the NHS Testbeds
Programme, Department of Health.

Nicole Mather, Previously Head of Strategy in the Offices for Life Sciences (OLS) and SRO of
testbeds.

Michael MacDonnell, Previous Head of the Testbeds Programme for NHS England.

Literature

NHS England Test Beds Programme: Evaluation Learning from Wave 1 (2018). https://www.
england.nhs.uk/wp-content/uploads/2018/11/test-beds-programme-evaluation-learning-
from-wave-1.pdf

Test beds – The Story So Far (2018) NHS England. https://www.england.nhs.uk/wp-content/

uploads/2017/09/test-beds-the-story-so-far.pdf

NHS England Test Bed Programme – Information Governance – Learning from Wave 1
(2018) https://www.england.nhs.uk/wp-content/uploads/2018/09/test-beds-programme-
information-governance-learning-from-wave-1.pdf

70
 Testing Innovation in the Real World: Real-world testbeds

2.3 Smart Santander37 – key facts

Description

Smart Santander was an IoT testbed funded by the EU which included the cities of
Santander (Spain), Lubeck (Germany), Belgrade and Guilford (UK). At the time, the most
extensive urban infrastructure in the world was deployed: over 20,000 sensors across
these cities communicating with the deployed IoT architecture. The testbed is regarded
as a success and Santander, in particular, is now recognised as a trailblazer of Smart City
technology deployment. Since the official testbed ended, many of the tested technologies
were implemented into the city and are still developed and used.

Figure 6: Sensor deployment in Santander

What was tested

Overall, what was tested were the approaches to, and building blocks of, the IoT
architecture; evaluation of the IoT interaction with device technologies; and critical services
such as discovery, data management and services. Santander was the city testing most
citizen services through over 12,000 sensors, amongst others:

• Environmental monitoring of temperature, Co2, noise, light and car presence.

• Remote dimming of street lamps: Street lamps are dimmed while streets are empty or
when other light sources, such as the full moon, are present.
• Efficient garbage collection: The garbage collection system is made more efficient
through sensors identifying which bins are not full – reducing needless trips.
• Outdoor parking area management: 400 parking sensors buried under the asphalt
manage and communicate availability.
• Parks and garden irrigation: Making irrigation as efficient as possible through measuring
moisture, humidity, the temperature in relevant parks and gardens.
• Augmented reality features on tourist attractions: Providing useful information about
several buildings and attractions in the city.

71
Testing Innovation in the Real World: Real-world testbeds

The sensors and their uses were installed in phases between 2010 and 2013. Each stage
contained different types of sensors depending on the final services. Some were static,
buried in the asphalt or placed within street lamps or boxes, others were mobile, situated on
the city’s public transport network, including buses, taxis and police cars. By downloading
an app to their smartphones, citizens could also become moving sensors in their own right.

Main actors involved: The Smart Santander project was funded mainly by the European
Union’s Future Internet Research and Experimentation (FIRE), which consists of the
creation of facilities to support experimentally driven research in the field of Information
and Communication Technologies (ICT). The €6 million grant funded the consortium of
25 partners from across Europe and Australia. The cities involved were Santander (Spain),
Belgrade (Serbia), Lubeck (Germany), and Guilford (UK).

Purpose

The purpose stated by the Smart Santander project was dual: 1) to facilitate real-world
experimentation of IoT technology to achieve a leading role for European cities in IoT
technologies, 2) to provide useful services tailored to the cities during the testbed operation.

Core lesson – capturing competitive advantages in a frontier technology

Santander is a city of around 180,000 people, most known as a city of tourism. Through
the Smart Santander programme, it has been transformed into one of the most recognised
smart cities in the world. The EU-funded testbed has, according to participants, diversified
the economy of Santander and placed European cities at the forefront of using IoT
technology in urban planning and development.

It has also achieved the aims set by the EU, one of which was to ‘achieve a leading role
in Europe in the IoT technologies’. In their evaluation report, the European Commission
stated that “The project has successfully achieved all its objectives, even exceeding initial
expectation.” The following elements were key to this success:

• The testbed as a tool was highly beneficial for the involved partners. Professor Luiz
Muñoz, the lead coordinator of the partners, stated that the testbed format was a
much more practical approach for the researchers, allowing them to learn, but also be
expected to produce results.
• The testbed format in the real-world environment served as an excellent catalyst
and necessary critical mass for IoT research. It also facilitated collaboration between
research institutes, citizens and private sector actors to operate their tests and develop
their products. The testbed created a framework for collaboration that led to the more
rapid development of IoT technology.
• Using a testbed that involved citizens in facilitating the testing boosted credibility
and made adaptation easier. Some of the sensors already installed in the city stayed
there after the testbed period was finalised, simply changing its status from ‘testing’ to
‘operating’ within the real world.
• Testbeds that are built around accurate and locally specific urban challenges, not merely
‘to test a sensor’, also increases the opportunity of a positive testbed legacy.
• To achieve the competitive advantages on a supra-regional scale such as that
envisioned by the EU, the testbed area requires adequate scale, length of time to allow
learning and improving, well-run partnerships with win-win outcomes, and sufficient
funding.

72
 Testing Innovation in the Real World: Real-world testbeds

• The EU FIRE project is a large funding programme that spreads funding across several
aspects of the future of the internet, such as AI, cloud services and other wireless
technology. This spreads the risk of investing in ‘future trends’ and facilitates learning
between the projects.

Other lessons learned

Involving citizens early

There is no smart city testbed without citizen involvement. If you put technology into the
city that people cannot use, or do not like or accept it, then the city risks not being ‘smart’,
only technology-led. A testbed is an excellent way of testing services and sensor presence
with citizens at an early stage.

• Santander used the data coming in every two minutes from the 12,000 sensors, to create
a ‘Pulse of the city app’. The app makes the data accessible to the citizens and puts
them at the heart of public service delivery. Through the app citizens could, for example,
point their device towards any bus stop to obtain information about when the next bus
will arrive. They could also use the app to report everyday annoyances such as potholes
or other damages to the public realm, forwarded to the relevant authority.

Involving SMEs

It was important to the EU and the cities involved to allow SMEs to be involved in the
infrastructure. The solution from Smart Santander was to incentivise their involvement
without expecting financial contributions. This was done through, for example:

• Workshops with SMEs on city challenges and IoT responses.

• Providing challenge-led funding that SMEs could apply to, with bespoke solutions.
• Facilitate close collaboration with the SMEs, the research institutes and large companies
involved to facilitate the increased capacity of contribution.

Local economic development and inward investment

The close coordination of the city council and the local economic development agency in
Santander was vital to harness the local competitive advantages. According to Luiz Muñoz,
the city has now diversified its economy through attracting companies and researchers
wanting to learn from the Santander approach, use the open data available, or offer
services attached to the IoT Architecture. Some elements stood out as being particularly
important in harnessing economic development benefits for Santander:

• The involvement of the economic development agency in Santander from an early stage
allowed for a long-term marketing plan, and participation of local economic actors
throughout the process.
• The success of the programme created a new source of marketing for the city. From
being a tourist destination, the city is now known for being one of the pioneers of smart
city development. The city council, researchers and companies involved are often invited
to share their model and experiences with cities worldwide. Although the success of
Santander generated a lot of free marketing, this can be achieved by others through a
conscious marketing strategy by relevant stakeholders.

73
Testing Innovation in the Real World: Real-world testbeds

Impact

• The most important results of the real-world testbed activity include improvement in the
quality of citizens’ lives, optimization and reduced costs of the urban services provided,
as well as positioning the city as a world leader in the field of innovation.38
• A report by McKinsey39 lists Santander among the ten strongest cities in Europe
regarding its smart city technology base,40 higher than Berlin, Paris, and Moscow.
• It also finds that citizens were both aware of the available technology and highly
satisfied with it.
• Santander is considered a pioneer smart city due to its widespread deployment of
mobile and fixed IoT devices, which has come as a direct result of the real-world testbed.
• The sensor deployment was also considered a success regarding integration between
public and private sector. The use of the deployed sensors has now been integrated into
the municipality’s strategic plan,41 the Santander Masterplan for Innovation and the
Santander Smart City Plan.
• The city is considered a unique area where the simultaneous deployment of
technological devices has created an atmosphere where experimentation and service
delivery can coexist.42
• It is also claimed (by McKinsey) that the development of Santander as a smart city
(enabled by the real-world testbed) has enabled the business and entrepreneurial
ecosystem to be revitalised. It has led to increased internationalisation, attraction of
investment, consolidation of research and development activities, and more efficient
urban services such as waste management.43

References
Galache, J., Guiterrez, V., Santana, J., Sanchez, L., Sotres, P., Casanueva, J., Munoz, L. (2014)
SmartSantander: IoT Experimentation over a Smart City Testbed. University of Cantabria,
Lancaster University, University of Surrey, Ericsson Serbia, Research Academic Computer
Technology Institute of Greece, University of Lubeck.

Smart Santander – Telefonica (2015) https://iot.telefonica.com/multimedia-resources/smart-

santander

Smart Santander Tutorial (2014) http://www.smartsantander.eu/downloads/Deliverables/

smartsantander-tutorial.pdf

Smart Santander Website: http://www.smartsantander.eu

Interviews

Luiz Muñoz, Head of Coordination of SmartSantander, University of Cantabria.

74
 Testing Innovation in the Real World: Real-world testbeds

2.4 Testbed Sweden – key facts

Description

Sweden ranks highly on global rankings for innovation.44 A central element of the
country’s ability to innovate is the rich ‘test and demonstration infrastructure’, where firms,
researchers and other users can test, verify and demonstrate new processes, products and
services. Approximately 40 per cent of the 19145 registered testbeds are real-world testbeds,
which is also the area that is seeing most growth. The testbeds are mainly spread across
topics such as product and production methods, energy, transport, environmental science,
construction, health, automation, ICT, life sciences and community development.46

The Testbed Sweden strategy is a focal area within the overarching Smart Industrialisation
Strategy for Sweden from 2016. The aim is to coordinate the existing testbeds better
and establish new facilities that can solve societal problems. Testbeds are reviewed as a
potential contribution to closing ‘the valley of death’ for firms and incentivising investments
in R&D. It also aims to contribute to lessening institutional barriers to testing and
demonstration through smart policy development. Vinnova, Sweden’s innovation agency,
has a mission from the government to establish a national function to coordinate Testbed
Sweden.

Facilitator

Swedish Government, carried out by Vinnova, Business Sweden and the Research Institute
of Sweden (RISE).

Purpose

The Testbed Sweden Strategy aims to narrow the ‘valley of death’ for firms, caused by the
lack of risk capital and decrease in business R&D investments (as a percentage of GDP).
The strategy also aims to strengthen the competitiveness for Swedish industry/academy/
institutes and make a ‘bulk’ testbed ecosystem that provides mutual learning and better
services for users.

Actors involved in testbeds: In the Swedish case, there are a great variety of actors involved
in testbeds. The private sector is the largest category, investing the most in testing and
demonstration facilities. The Research Institute of Sweden (RISE) is the largest single actor
in terms of ownership, operation and participation in testbeds, along with other public
bodies involved in a variety of ways. Vinnova, the state innovation agency, spends about
100 million SEK (£82 million) annually on test and demonstration purposes. 60-70 per cent
of Vinnova’s funding goes to projects involving test and demonstration activities, and 2-3
per cent of the funding goes into these facilities themselves. Tillvaxtverket, the agency for
economic and regional growth, contributes to establishment, use and accessibility funding
for testbeds, for example through the ‘Rural Testbeds’ with a budget of approximately £85
million. The National Energy Body is also a core funder of small and large testing facilities,
as is Business Sweden, who received a mission from the government in 2016 to market
testbeds and facilitate inward investment.

75
Testing Innovation in the Real World: Real-world testbeds

Core lesson – responding to market failures

The idea to take a national approach to testbed funding and coordination in Sweden
came about for several reasons. According to our interviewees, the primary purpose of this
approach was to respond to two specific market failures:

• Closing the ‘valley of death’ for firms and incentivising R&D spending, which had been
decreasing

Figure 7: Closing the valley of death for firms, adaptation of Vinnova model (2018)

Funding

VINNOVA support State funding hopefully

for test and attracts private
demonstrations investments

Access to state Need for

R&D funding private capital

Idea/technical opportunity Innovation/commercialisation

As Figure 7: Closing the valley of death for firms, adaptation of Vinnova model (2018) shows,
the idea is that public funding and coordinating testbeds will incentivise firms to bring their
ideas and technical opportunities closer to innovation and commercialisation. This progress,
made possible by public funding, is then meant to incentivise private capital investments
at an earlier stage, with more mature ideas and technologies. The ‘valley of death’ will
still exist, however with a narrower gap. The reason for selecting to invest increasingly in
testbeds is due to their capability to support several firms and technologies. Coordinated,
they are also capable of learning from one another and being marketed more efficiently.

The testbed strategy also aims to contribute towards several other benefits and secondary
purposes, amongst others:

• The need for quicker iterations of trends through real-world testing facilities.
• Better connection between large firms and SMEs/Startups.
• Better coordination of the ‘problem-owners’ and the industry/academy/institutes.
• Achieving more cross-fertilisation of sectors.
• Knowledge transfer between testbeds.
• Attract investments and more testing and demonstration activities.
• Creating a common framework.

76
 Testing Innovation in the Real World: Real-world testbeds

Other lessons learned

Involving the users of testbeds

Involving users is fundamental for testbeds to ensure demand and usefulness. The lack of
dialogue with potential users risks design of unsustainable testbeds by not responding to
demand.

• Flexibility: There are strong indications that attractive testbeds rely on flexibility, taking
the user’s needs into account in addition to collaborating in the design, development and
operational phase. The type of involvement, and indeed who the user is, depend on the
testbed function and characteristics.
• Long-term funding and engagement: The attraction and success of a testbed further
depend on a driving partner (project coordinator) investing significant time and
resources in creating complex and heterogenous constellations of actors, to ensure
engagement and co-funding. Creating these constellations can take time, which needs
to be accounted for.
• Clear objectives: Insecurity in terms of the expectations of use and the clarity of the
objectives can be barriers to sustainable development of testbeds.
• Ownership from users: Real-world testbeds see the participation of users in the
operational phase as a core element. The ambition of the Testbed Sweden Strategy is to
facilitate this category in particular, focussing on collaboration.
• Setting requirements: Public support of testbed development should set requirements
for the user participations, with varied engagement depending on the purpose of the
individual testbed.
• Ensuring the capacity of the testbed is fully exploited: Many Swedish testbeds only
utilise about 50 per cent of their full capacity. This is an area that can often be
neglected: make sure plans exist for full exploitation of the testbed to make the most of
the investment.

Engaging SMEs

Involving SMEs in testbeds is also an important element; however, this does not necessarily
happen naturally, and they are not always in the target group of testbeds. SMEs often
struggle with resources to be able to test their technologies and commercialise their
innovation or access testbeds if they are not customised.

There are several factors that influence a testbed’s accessibility for SMEs. Table 2 illustrates
some known challenges and possible solutions to SME engagement.

77
Testing Innovation in the Real World: Real-world testbeds

Table 2: Challenges and solutions for SME engagement with testbeds

Challenge for SMEs Potential solutions

The testbed is too costly for Subsidising the participation of SMEs either directly or through challenge
participation. funding earmarked for a certain type of innovator.
Providing SMEs with free access to the real-world testbed.

Lack of knowledge about real- Having part of the real-world testbed budget earmarked for marketing
world testbed opportunities purposes.
for relevant SMEs, or doubt in
Creating support structures for SMEs that increase their potential benefits and
the worthiness of the testbed
outputs.
participation.

SMEs not the primary target As many real-world testbeds may prefer to work with large companies, those
group. that involve SMEs, or plan to do so, can be rewarded or gain preference in
funding applications.
Smaller innovators can be involved through working methods such as
workshops or labs concerning the real-world testbed topic.

Fear of exposing their products/ Having clear frameworks on Intellectual Property (IP) implications and building
processes. trust and transparency within the testbed.

Public sector involvement

One of the objectives of the Swedish Testbed Strategy is to increase the number of public
sector bodies that own and operate testbeds. The reasoning behind this is that testbeds are
tools that can lead to better public services and that publicly run testbeds have a higher
probability of retaining and spreading the knowledge produced in them. Currently, only
about 10 per cent of Swedish testbeds are publicly driven. However, they are nearly always
involved in real-world testbeds. Barriers to public sector involvement are recognised to be
resistance to change, laws and regulation, time and resources, and knowledge of testbeds
as a tool.

Vinnova points to two distinct roles the public sector plays in the development of real-world
testbeds:

• Providing access to public infrastructure (Schools, hospitals, land, roads, water

infrastructure etc.).
• Facilitating solutions to public sector related challenges.

78
 Testing Innovation in the Real World: Real-world testbeds

Vinnova further argues that:

• Doing testbeds within the public sector with a distinct focus on learning can facilitate
more public innovation.
• Testbeds can increase the use and usefulness of already available public infrastructure.
• Public sector driven testbeds can lead to an improved regulatory environment.

Coordination of existing testbeds funded by the public sector is also key. Sweden already
has a rich ‘innovation infrastructure’ and aims to coordinate them better to allow sharing
of knowledge and resources. For countries/places with fewer testbeds, early facilitation of
such collaboration, before set patterns of behaviour and organisation are set, can make
the innovation system more robust in the future. Coordination could also be done at an
international level.

Smart policymaking

Policymaking and regulation is a key part of the innovation system, and having regulatory
bodies understand their part is vital for a flexible and well-functioning regulatory system in
a fast-changing technological world. The use of ‘policy labs’ or ‘smart policymaking’ is an
important part of the Swedish Testbed Strategy.

A policy lab is a group of actors with different skills aiming to develop regulation. The policy
lab uses a set of user-centred methods and competencies to test, experiment and learn in
policy development. These can be integrated within testbeds for specific cases or sectors, or
they can be long-term constellations working with policy development over time.

Impact

• In a review of the testbeds involved in health and social care, several of the testbeds
involved could showcase several implemented products and services within their
respective areas as a direct cause of the testing activity. Other products or services
have been deemed unready for implementation, due to incompatibility with the current
system, and are envisioned not to be implemented for decades. These are both valuable
results that would perhaps have been misjudged without testing in the environment in
which the innovation was meant to be used.
• Evaluation of testbeds in the health and care sector is collectively deemed to show
a high degree of coordination between the actors involved, and that it is focused on
tackling mutual challenges.
• Many testbeds were evaluated to have found a good balance between the need of the
public and the private sector, allowing both participants to advance further and achieve
objectives and aims.

79
Testing Innovation in the Real World: Real-world testbeds

• In testbeds operating in construction and environmental technique, several testbed

operators emphasise the non-material value of the testbed’s function as a meeting place
and collaboration platform, where participating innovators and actors can exchange
ideas and experiences or find new collaborations and business opportunities. They also
provide innovators with an opportunity to show themselves, their products or services
through a testbed’s wider marketing – acting as ‘storefronts’ to SMEs. Some also offer
extended support to SMEs, such as Hammarby Sjorstadverk, which provides expertise in
onsite cleaning technology, or Solar testbed, which provides support for developing ideas
and writing applications, mentoring and idea development – although this is rare.

References

Interviews with

Fillip Kjeldgren, Head of Testbeds at Vinnova

Thomas Sunden, Project Manager, Sustainable Innovation

Literature

Ministry of Enterprise and Innovation (2018) Why testbeds are important for attraction?

Vinnova (2017) Testbeds within health, social and elderly care – a portfolio analysis.

Swedish Testbeds (2019) https://swedishtestbeds.com/en/about-swedish-testbeds/

Vinnova (2015) Mapping and needs audit of test and demonstration infrastructure. (Report in
Swedish)

Vinnova (2019) A study within the creation of the national function of Testbed Sweden – a study
of the foundations of suggestions made to promote Testbed Sweden. (Report in Swedish).

Vinnova (2019) Testbed Sweden – Filip Kjellgren

https://www.government.se/498615/contentassets/3be3b6421c034b038dae4a7ad75f2f54/
nist_statsformat_160420_eng_webb.pdf

80
 Testing Innovation in the Real World: Real-world testbeds

2.5 Testbed Gothenburg – key facts

Description

Testbed Gothenburg is part of the Testbed Sweden strategy as a response to the Smart
Industrialisation Strategy for Sweden. Testbed Gothenburg is an initiative that enables
an overarching initiative to bring together industry, academia, institutes and the city. It
encompasses actors across the city in trade and industry, the public sector, academia,
research institutes and science parks.

In recent years, Gothenburg-based companies have invested heavily in the region and in
developing new products and services. One-third of Sweden’s private R&D investments are
made in Västra Götaland, a county in Gothenburg. Global brands such as AstraZeneca
and the Volvo Group have invested heavily in the region and have contributed to Testbed
Gothenburg.

Facilitator

Testbed Gothenburg was initiated by the City of Gothenburg through Business Region
Göteborg, RISE Research Institutes of Sweden, Chalmers University of Technology
and Gothenburg University. Business Region Göteborg is responsible for the business
development of the 13 municipalities in the Gothenburg region.

Purpose

Testbed Gothenburg aims to accelerate economic development and build capacity and
ability for testing innovation for firms and research institutes, as well as freeing up resources
to proactively address tomorrow’s challenges. Testbed Gothenburg advertises the city as an
open environment in which companies can test their products before investing in full-scale
development.

What is being tested

There are four different ‘types’ of testbeds and facilities advertised within the Testbed
Gothenburg Programme:

1. Academic environment: (E.g. universities/research institutes) who own and fund the
testbeds.
2. Virtual testbeds: (Funding from academia and industry) e.g. Volvo car and Volvo group
use virtual testing environments to launch new vehicles;
3. Living labs: Areas within the city that work with citizens to co-develop products and
services.
4. Policy labs: Where infrastructure of knowledge and competence is used to address
policy issues (e.g. autonomous driving) that will require regulatory intervention.

The target industries for Gothenbrug are the automotive industry, life science, urban
development and logistics.

81
Testing Innovation in the Real World: Real-world testbeds

One of the most well known testbeds is Electricity – a bus route that offers quiet, exhaust-
free buses that pick up passengers indoors and are powered by electricity from renewable
sources.

Core lesson – create a strong network

Feedback from Testbed Gothenburg has emphasised the importance of ensuring clarity
and a strong, shared message for marketing between all partners. The success of the
testbed programme is owed to the strong relationships between Gothenburg City Region
and influential private sector partners. Business Region Göteborg has created a business
directory on its website, which allows users to find and connect with partners they may be
interested in working with.

Companies such as AstraZeneca and the Volvo Group have a strong sense of pride in the
city region and are happy to invest and grow their offices in Gothenburg. This, coupled with
the City’s openness to invest and facilitate innovation, has built a strong network between
the private and public sectors and academia.

Other lessons learned

Clarity of purpose

Through interviews, it was made clear that cities should be clear on their target industries
and areas of growth. Gothenburg has advertised its key areas of opportunity as automotive,
life science, urban development and logistics. Also, these target industries should be rooted
in key sectors of economic strength or growth in the area.

The automotive industry in Gothenburg employs around 30,000 people directly and, when it
comes to vehicle development, is one of the most knowledge-intensive per capita. Business
Region Göteborg employs an expert in this sector to facilitate links between potential
investors, the public sector and academic institutions.

Engagement with SMEs

While engaging SMEs is an aim of the testbed programme, anecdotal evidence suggests
that engagement with SMEs has been a challenge for Business Region Göteborg. Due
to the number of high-profile large organisations already in the region, the majority of
collaboration has been with those already ‘set up’ to invest in the region.

Through our interviews, it was made clear that engagement with SMEs is a key target area
for Gothenburg. The website includes resources for starting and growing a business, as well
as providing networking opportunities for SMEs to meet potential investors and business
partners. Business Region Göteborg offers services such as finance support and seminars to
support business growth. The next step is to facilitate a route to get involved in the testbed
programme, which may require significant capital investment up front.

Some of the key challenges for SMEs that Testbed Gothenburg is trying to solve are:

• Funding to participate in a testbed.

• Lack of networking opportunities.
• Lack of experience (compared to established global businesses).

82
 Testing Innovation in the Real World: Real-world testbeds

Challenge: taking the testbed international

Evidence from interviews suggests that the industrial system in Sweden makes it challenging
to take an idea tested in Gothenburg to a global scale. It appears that funding may run out
once a testbed is deemed as completed, leaving a lack of funding to market the product
on a global scale. As described in the Testbed Sweden case study, products require capital
to ensure commercialisation on a larger scale. This is mainly a problem for SMEs and less
internationally-established companies.

References

Interviews with

Maria Stromberg, Programme Manager and Head of Department Clusters and Innovation,
Business Region Göteborg

Literature

Invest in Gothenburg website – an outline of current testbeds and learning https://www.

investingothenburg.com/target-industries

Business Region Göteborg exhibition at MIPIM 2019 https://www.gothenburgatmipim.com

Business Region Göteborg website (business support) https://www.businessregiongoteborg.

se/en

83
Testing Innovation in the Real World: Real-world testbeds

Endnotes
1. This is a sample of testbeds operating in the real- 11. Ibid.
world environment from around the world. It is by 12. Ibid.
no means exhaustive, and it is based on a number
we have been able to identify in our desk-based 13. Ribiere, V. and Tuggle, F. (2010) ‘Fostering
research. There are most examples from the innovation with KM 2.0.’ VINE, Vol. 40 Issue: 1,
Nordics (Sweden in particular) and the UK, both pp.90-101.
due to better knowledge of relevant institutions, 14. European Network of Living Labs: https://enoll.org/
and the level of promotion of the testbeds. about-us
2. Mustonen (2016) quoted from City Digest: https:// 15. See Appendix 3 for an overview of collected
www.citiesdigest.com/2016/12/19/smart-city- definitions from literature.
kalasatama-a-smart-district-aimed-at-saving-
16. General Motors (1928), ‘The Proving Ground for
citizens-1-hour-per-day/
the Products of General Motors.’ https://www.
3. Thomke, S. and Bell, D.E., 2001. Sequential testing gmheritagecenter.com/docs/gmheritage-archive/
in product development. Management Science, historical-brochures/GM_Pr oving_Gr
47(2), pp.308-323.
17. Sidewalk Toronto Project Update (2019) https://
4. Halpern, O., LeCavalier, J., Calvillo, N. and Pietsch, sidewalktoronto.ca/wp-content/uploads/2019/02/
W., 2013. Test-bed urbanism. Public Culture, 25(2), FEB14-SWTO-Business-Case-Overview.pdf
pp.272-306.
18. Nesta (2019) ‘Renewing regulation: anticipatory
5. Armstrong, H. and Rae, J., 2017. A working model regulation in an age of disruption.’
for anticipatory regulation. London: Nesta.
19. Due to the recent history of this RWT, there is no
6. Dodgson, M. and Gann, D., 2011. Technological full case study in Appendix 2.
innovation and complex systems in cities. Journal
20. https://meco.gouvernement.lu/dam-assets/
of Urban Technology, 18(3), pp.101-113.
publications/brochures---livres/2018-05-08-
7. OECD (2019) Digital Innovation: Seizing Policy concept-digital-test-bed-ger-fra-lux-v1.pdf
Opportunities. OECD, Paris.
21. We recognise that these are public services in
8. https://www.nesta.org.uk/report/edtech-testbeds/ some countries, and privately run by others. This
Batty, Florescu, Wong and Sharples (2019): EdTech report focuses on these services run by public
testbeds: models for improving evidence. London: bodies, however, many of the recommendations
Nesta. and descriptions can be useful to service providers
9. Bulkeley, H., Marvin, S., Palgan, Y.V., McCormick, in the private or not-for-profit sector also.
K., Breitfuss-Loidl, M., Mai, L., von Wirth, T. and 22. Research and Development.
Frantzeskaki, N., 2018. Urban living laboratories:
23. The ‘valley of death’ is a common term with
Conducting the experimental city? European Urban
innovators, referring to the difficulty of covering the
and Regional Studies, forthcoming.
negative cash flow in the early stages of a start-up,
10. Vinnova: https://www.vinnova.se/en/m/testbed- before their new product or service is bringing in
sweden revenue from real customers.

84
 Testing Innovation in the Real World: Real-world testbeds

24. United Nations (2018) ‘Frontier Technologies for 35. Findings from interviews with Testbed Programme
sustainable development in Asia and the Pacific.’ SROs.
25. https://www.telefonica.com/en/web/public-policy/ 36. NHS Test Beds: Testing innovations in real world
blog/article/-/blogs/the-european-commission- settings (2018) Health and Care Innovation Expo.
considers-that-the-smartsantander-project-has- 37. This case study mainly focusses on experiences
exceeded-expectations- from the city of Santander, Spain.
26. Based on interviews with Vinnova and Testbed 38. Interview with Prof. Luis Muñoz, Project Coordinator
Gothenburg. and Lead from University of Cantabria.
27. The Open University, University of Bedfordshire, 39. McKinsey Global Institute (2018) ‘Smart Cities:
University of Cambridge. Digital Solutions for a more Livable Future.’
28. E-On, Huawei, Samsung, Tech Mahindra, 40. Sensors, communication, open data portal.
ThngWorx, Privitar, Comarch.
41. Strategic plan for Santander 2010-2020.
29. UK Huawei ‘Smart Cities Index.’ (2018).
42. McKinsey Global Institute (2018) ‘Smart Cities:
30. ‘Smart Cities Special: Why Milton Keynes is more Digital Solutions for a more Livable Future.’
than just roundabouts.’ (2018). https://www.cips.org/
en/supply-management/analysis/2018/september/ 43. Ibid.
smart-cities-special-more-than-just-roundabouts 44. Second place in ‘The Global Innovation Index.’
31. ‘The future of mobility? In Milton Keynes it’s already (2016). ‘Sixth in the Global Competitiveness Report.’
here.’ (2018). https://spectrum.ieee.org/cars-that- (2016-2017) and First in ‘The European Innovation
think/transportation/self-driving/model-town- Scoreboard.’ (2016).
plans-next-steps-beyond-uk-autonomous-car- 45. There are more than 191 test and demonstration
project facilities, however, the 191 fall within the
32. Interview with head of Transport Innovation, Brian three categories of testbeds developed by
Mathews. Vinnova (Laboratory, simulated and real-world
environment). There are several others that e.g.
33. General Data Protection Regulation. facilitate testing only for one company or that test
34. ‘Evaluation of the NHS England Innovation something only once, which are not taken into
Test Bed at Care City.’ Report prepared by Ann account in this number.
Raymond, Sachin Vadgama, Sonya Crowe, Steve 46. Vinnova Analysis (2015).
Morris and Martin Utley at UCL.

85
58 Victoria Embankment
London EC4Y 0DS
+44 (0)20 7438 2500
information@nesta.org.uk
@nesta_uk
www.facebook.com/nesta.uk
www.nesta.org.uk

Nesta is a registered charity in England and Wales with company number 7706036 and charity number 1144091.
Registered as a charity in Scotland number SCO42833. Registered office: 58 Victoria Embankment, London, EC4Y 0DS.