Digital

implementation
investment guide (DIIG):
integrating digital interventions
into health programmes
b
Digital
implementation
investment guide (DIIG):
integrating digital interventions
into health programmes
Digital implementation investment guide (DIIG): integrating digital interventions into health programmes
ISBN 978-92-4-001056-7 (electronic version)
ISBN 978-92-4-001057-4 (print version)
© World Health Organization 2020.
Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 IGO licence
(CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo/).
Under the terms of this licence, you may copy, redistribute and adapt the work for noncommercial purposes provided the work
is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific
organizations, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your
work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following
disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not
responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition.”
Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World
Intellectual Property Organization. (http://www.wipo.int/amc/en/mediation/rules/).
Suggested citation. Digital implementation investment guide: integrating digital interventions into health programmes. Geneva:
World Health Organization; 2020. Licence: CC BY-NC-SA 3.0 IGO.
Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris/.
Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders/. To submit requests for commercial use
and queries on rights and licensing, see http://www.who.int/about/licensing/.
Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it
is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The
risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user.
General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression
of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or
concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for
which there may not yet be full agreement.
The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended
by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary
products are distinguished by initial capital letters.
All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published
material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use
of the material lies with the reader. In no event shall WHO be liable for damages arising from its use.
Design and layout: RRD Design LLC
Contents
List of figures, tables and boxes v
Foreword vii
Acknowledgements viii
List of abbreviations ix
CHAPTER 1: INTRODUCTION 1
1.1 The Guide’s role in planning and implementing a digital health enterprise. . . . . . . . . . . . . . . . . . . . . . . 4
1.2 How to use this Guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 Key terms for using this Guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4 When to use this Guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
CHAPTER 2: FORM THE TEAM AND ESTABLISH GOALS 15
2.1 Determine roles and responsibilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Develop a common understanding of the health programme’s needs and goals. . . . . . . . . . . . . . . . . 22
2.3 Understand programme operations across levels of the health system. . . . . . . . . . . . . . . . . . . . . . . . . 23
CHAPTER 3: IDENTIFY HEALTH SYSTEM CHALLENGES AND NEEDS 29
3.1 Map the current state of programme activities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.1.1 Determine the health programme processes to target. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.1.2 Map the workflows for targeted processes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.1.3 Identify and confirm bottlenecks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.2 Conduct a root cause analysis of bottlenecks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3 Prioritize bottlenecks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4 Map programme-specific bottlenecks to generic health system challenges . . . . . . . . . . . . . . . . . . . . 38
CHAPTER 4: DETERMINE APPROPRIATE DIGITAL HEALTH INTERVENTIONS 43
4.1 Determine and select digital health interventions for the prioritized health system challenges . 46
4.2 Determine whether the enabling environment can support the
identified digital health interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.3 Determine what the interventions will need to do. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.3.1 Identify functional requirements and user stories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.3.2 Understand and manage expectations from end-users and stakeholders. . . . . . . . . . . . . . . . . . . 55
4.3.3 Map future-state workflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.4 Determine if existing digital health applications, platforms and enterprises can
achieve the requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.5 Progress check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
CHAPTER 5: PLAN THE IMPLEMENTATION 63
5.1 Infrastructure considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.2 Legislation, policy and compliance considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.2.1 Data management, privacy and security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.2.2 Regulation of new digital health technologies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.3 Leadership and governance considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.3.1 Governance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.3.2 External partnerships. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.4 Workforce and training considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.5 Services and applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.6 Standards and interoperability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.7 National digital health strategy and investment plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

iii
 CHAPTER 6: LINK THE DIGITAL HEALTH IMPLEMENTATION TO THE ENTERPRISE ARCHITECTURE 81
6.1 Assess the digital health enterprise architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.2 Identify common and enabling components and shared services (digital health platform). . . . . . 86
6.3 Link your digital health investments to the enterprise architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
CHAPTER 7: DEVELOP A BUDGET 91
7.1 Phases of implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
7.2 Cost drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
7.3 Budget matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
CHAPTER 8: MONITOR THE IMPLEMENTATION AND USE DATA EFFECTIVELY 101
8.1 Establish a logic model for your implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
8.2 Plan how you will conduct the monitoring and evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
8.3 Establish a culture of data use. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
8.4 Adaptive management: Use data to optimize interventions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
8.5 Progress Check. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
CHAPTER 9: VALUE PROPOSITION AND NEXT STEPS 119
References 122
Annexes 126
Annex 1.1 Glossary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Annex 1.2 Additional resources for further reading for planning and
implementing a digital health enterprise. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
Annex 2.1 Planning and implementation charter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Annex 2.2 Persona worksheet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Annex 3.1 Process matrix worksheet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
Annex 3.2 Worksheet for mapping bottlenecks to health system challenges. . . . . . . . . . . . . . . . . . . . . . . . 135
Annex 5.1 Questions for software developers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
Annex 5.2 Implementation considerations summary template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Annex 5.3 Implementation considerations for specific digital health interventions . . . . . . . . . . . . . . . . 138
Annex 5.4 Illustrative considerations to mitigate data management risks. . . . . . . . . . . . . . . . . . . . . . . . . 158
Annex 6.1 Linking digital health implementations to a national
digital health enterprise architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
Annex 7.1 Budget template. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Annex 8.1 Adaptive management checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Annex 8.2 Logic model template. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

iv
Figures, tables and boxes
Fig. 1.1.1. Planning and implementing a digital health enterprise: phases, steps and resources.. . . . . . . . . . . . . . . . 5
Table 1.1.2. Resources for planning and implementing a digital health enterprise.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Box 1.1.3. Resources detailing content for specific health programme areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Fig. 1.1.4. Essential processes of national digital health implementations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Fig. 1.2.1. Overview of the chapters in this Digital Implementation Investment Guide. . . . . . . . . . . . . . . . . . . . . . . . 9
Fig. 1.2.2. Summary of outputs within a costed implementation plan.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Fig. 1.3.1. Different digital health system architectures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Table 2.1.1. Key roles and their descriptions.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Box 2.1.2. Resources for establishing governance mechanisms.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Fig. 2.2.1. Example of Nigeria’s national health ICT vision extracted from the national eHealth strategy. . . . . . . . 22
Fig. 2.3.1. Illustrative example of a health system organogram and relationships.. . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Table 2.3.2. Examples of personas developed when planning an immunization programme.. . . . . . . . . . . . . . . . . . . 26

Fig. 3.1. Adaptation of CRDM approach for defining health system challenges.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Fig. 3.1.1.1. Process matrix illustrating three example processes within a typical vaccination programme.. . . . . . . 32
Fig. 3.1.2.1. Example workflow diagram for a service-delivery process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Box 3.1.2.2. Conventions that are generally used when mapping workflows.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Fig. 3.2.1. Using the 5 Whys model to identify the root cause of bottlenecks.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Table 3.3.1. Formula for scoring and ranking bottlenecks, with examples.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Fig. 3.4.1. WHO classification of health system challenges.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Box 3.4.2. Linking health system challenges to universal health coverage.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Fig. 3.4.3. Health system needs for universal health coverage.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Fig. 4.1. Redesigning health programme processes.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Fig. 4.1.1. Classifications of digital health interventions.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Fig. 4.1.2. Linkage between health system challenges and digital health interventions. . . . . . . . . . . . . . . . . . . . . . 48
Fig. 4.1.3. Digital health intervention selection matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Table 4.1.4. Bottlenecks discussed in a multistakeholder meeting.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Fig. 4.2.1. Roles and contributions of component “building blocks” to the
enabling environment for digital health enterprises.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Box 4.2.2. Useful resources for assessing the readiness of the enabling environment. . . . . . . . . . . . . . . . . . . . . . . . . 53
Fig. 4.3.1. Stakeholder relationships with a digital health intervention. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Table 4.3.1.1. An example scenario for a pregnancy-support digital health intervention.. . . . . . . . . . . . . . . . . . . . . . . . . 55
Fig. 4.3.2.1. Examples of different personas interacting with digital health interventions.. . . . . . . . . . . . . . . . . . . . . . 56
Fig. 4.3.3.1. Bottleneck and future-state workflow diagrams. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Box 4.4.1. The Digital Health Atlas and other repositories of digital health implementations. . . . . . . . . . . . . . . . . . 59
Fig. 4.5.1. Considerations for designing a digital health enterprise to deploy
selected digital health interventions.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

v
Fig. 5.1. Essential components of a digital health implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Table 5.2. Illustrative implementation considerations for digital health.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Box 5.3. Description of digital health interventions reviewed in the WHO guideline. . . . . . . . . . . . . . . . . . . . . . . 68
Box 5.1.1. Resource on hardware management.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Box 5.2.1.1. Examples of policies for data protection and regulation.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Box 5.2.2.1. Resources on regulation of medical device technologies.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Fig. 5.4.1. Kotter’s eight-step change model.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Fig. 5.4.2. Linking Kotter’s change model to BID’s touch strategy.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Fig. 5.5.1. Benefits and risks of different software models.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
Box 5.6.1. General considerations for standards and interoperability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Fig. 5.6.2. How mHero integrates digital health interventions using standards.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

Fig. 6.1.1. Digital health architectural approach.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

Box 6.2.1. Illustrative list of common components.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Fig. 6.2.2. Digital health investments within the digital health enterprise architecture.. . . . . . . . . . . . . . . . . . . . . . 87
Box 6.3.1. Key resources for building towards a digital health enterprise architecture.. . . . . . . . . . . . . . . . . . . . . . . 89
Fig. 6.3.2. OpenHIE architectural diagram illustrating mechanisms for
interoperability across different applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Fig. 6.3.3. Myanmar example of a national digital health enterprise architecture blueprint.. . . . . . . . . . . . . . . . . . 90

Fig. 7.1.1. Phases of implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

Table 7.2.1. Illustrative costs throughout the implementation life cycle.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Table 7.2.2. BID Initiative cost categories.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
Table 7.3.1. Summary budget matrix.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

Fig. 8.1. M&E and adaptive management as continual considerations for digital health implementations.. . 103
Fig. 8.1.1. Illustrative logic model of MomConnect digital health investment.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Fig. 8.2.1. Intervention maturity over time.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Fig. 8.2.2. Implementation maturity continuum.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Table 8.2.4. Examples of metrics from the Learning for Impact approach.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Fig. 8.3.1. Examples of data use. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Box 8.3.2. Immunization Data: Evidence for Action. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
Fig. 8.3.3. Data-driven accountability cycle.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Table 8.3.4. Cultural change factors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Fig. 8.3.5. The data journey, with and without data-use interventions.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Table 8.4.1. Traditional versus adaptive management.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Fig. 8.4.2. Adaptive management cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

Fig. 9.1. Summary of outputs towards a costed investment plan.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

Fig. 9.2. Critical sustainability elements.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

vi
Foreword
The transformative nature of digital technologies for health is undeniable. Today, over
half of the world’s population have access to a mobile phone. Increasing access to
mobile technologies has radically changed the way in which people may manage their
own health, as well as the way in which health services are delivered. Health systems
recognize that digital health technologies are critical for accelerating progress towards the
achievement of the Sustainable Development Goals. Greater investment, however, will be
needed to elevate the role of digital health in health systems, so that the positive impact
on health of individuals and populations can be fully realized.

The large-scale digital transformation of a health system is neither a quick nor a simple task. Rapid advancements in
digital technologies have made it easier to build individual technologies than to invest in and implement them such
that they function in a harmonized and complementary manner. Long-term systemic changes are needed, including a
change in the culture of using data. Investment must be carefully and thoughtfully coordinated for equitable access to
meet the full spectrum of health needs.

At the Seventy-First World Health Assembly, WHO’s Member States requested WHO not only to develop a global
strategy on digital health, but also to provide guidance for scaling up the implementation of digital health in line with
WHO’s Thirteenth General Programme of Work. In 2019, WHO launched the first WHO Guideline Recommendations on
Digital Interventions for Health System Strengthening to ensure that Member States use of digital health is informed by
the evidence for “what works”. This Digital implementation investment guide (DIIG): integrating digital interventions into
health programmes provides guidance on how to practically invest in and implement the recommendations outlined in
the WHO guideline according to national contexts, health sector needs and state of digital maturity. Investment must
be both effective and sustainable, with clearly anticipated health benefits for all.

Digital technologies are improving rapidly. Building a digital foundation that can be responsive to the diversity of
programme needs, but which also anticipates innovation is key. Under the framework of the emerging Global Strategy
on Digital Health, WHO, HRP, UNICEF, UNFPA, and PATH have developed the DIIG document to provide guidance on
investing in programmatic implementations for digital transformations of health systems in a systematic manner. Such
strategic investments allow for long-term sustainability on a national scale. Informed by experience across multiple
regions and agencies, the DIIG provides practical methods for approaching needs assessment, planning, investment
and implementation of digital health systems.

The lure of exciting new technologies and gadgets is ever present, but ultimately these technologies should be
promoting health, keeping the world safe, and serving the vulnerable. If implemented in a strategically harmonized
manner, leveraging the key principles and messages presented in the DIIG, these digital health systems are powerful
tools that will help us achieve the ultimate goal of health and well-being for all.

Dr Soumya Swaminathan
Chief Scientist
WHO

vii
Acknowledgements
The World Health Organization (WHO) and PATH thank the contributions of many individuals across different
organizations. This Guide was authored by Garrett Mehl, Maeghan Orton, Natschja Ratanaprayul and Tigest Tamrat of
the WHO Department of Sexual and Reproductive Health and Research; Hallie Goertz, Celina Kareiva, Carl Leitner and
Brian Taliesin of PATH; and Smisha Agarwal and Alain Labrique of the Johns Hopkins Bloomberg School of Public Health.

The following individuals reviewed and provided feedback for this Guide (in alphabetical order): Onyema Ajubor
(WHO), Peter Benjamin (Health Enabled), Ashley Bennett (PATH), Christina Bernadotte (PATH), Paul Biondich
(Regenstrief Institute), Sean Blaschke (UNICEF), Laura Craw (Gavi, the Vaccine Alliance), Marcelo D’ Agostino (WHO),
Carolina Danovaro (WHO), Hani Eskendar (International Telecommunication Union), Jun Gao (WHO), Jennie Greaney
(UNFPA – United Nations Population Fund), Karin Kallander (UNICEF), Manish Kumar (MEASURE Evaluation),
Mark Landry (WHO), Bernardo Mariano Jr. (WHO), Donna Medieros (Asian Development Bank), Alex Muhereza
(UNICEF), Rosemary Muliokela (WHO), Derrick Muneene (WHO), Maria Muniz (UNICEF), Henry Mwanyika (PATH),
Mohammed Nour (WHO), Steve Ollis (John Snow, Inc.), Olasupo Oyedepo (African Alliance of Digital Health Networks),
Jonathan Payne (Digital Impact Alliance), Liz Peloso (Independent Consultant), Caroline Perrin (Hôpitaux Universitaires
de Genève), Tina Purnat (WHO), Chilunga Puta (PATH), Steven Ramsden (The Global Fund to Fight AIDS, Tuberculosis
and Malaria), Daniel Rosen (Centers for Disease Control and Prevention), Lale Say (WHO), Merrick Schaefer (US Agency
for International Development), Chris Seebregts (Jembi Health Systems), Tamsyn Seimon (Independent Consultant),
Dykki Settle (PATH), Andreas Tamberg (The Global Fund to Fight AIDS, Tuberculosis and Malaria), Lori Thorell (UNICEF),
Jai Ganesh Udayasankaran (Sri Sathya Sai Central Trust, India), Martha Velandia (WHO), Steven Wanyee (Kenya Health
Informatics Association), Adele Waugaman (US Agency for International Development), Laurie Werner (PATH) and
Sylvia Wong (UNFPA – United Nations Population Fund).

The following individuals informed the development of this Guide through workshops (in alphabetical order):
Bem Aga (National Democratic Institute), Monica Amponsah (The Grameen Foundation), Dominic Atweam (Ministry
of Health Ghana), Abul Kalam Azad (Ministry of Health and Family Welfare Bangladesh), Mohini Bhavsar (Dimagi),
Clara Blauvelt (VillageReach), James BonTempo (Independent Consultant), Vajira H. W. Dissanayake (University of
Colombo, Sri Lanka), Sulaiman Etamesor (Federal Ministry of Health Nigeria), Mike Frost (Norwegian Institute of Public
Health), Erick Gaju (Ministry of Health Rwanda), Matt Hulse (USAID), Wogba Kamara (Ministry of Health and Sanitation
Sierra Leone), Onesmus Kamau (Ministry of Health Kenya), Charles Kwening (Literacy Bridge), Erin Larsen-Cooper
(VillageReach), Erica Layer (D-tree International), Portia Manangazira (Ministry of Health and Child Care Zimbabwe),
Yussif Ahmed Abdul Rahman (PATH) and Santigie Sesay (Ministry of Health and Sanitation Sierra Leone).

This work was funded by the Bill & Melinda Gates Foundation; Digital Impact Alliance; UNFPA – United Nations
Population Fund; US Agency for International Development; and The UNDP/UNFPA/UNICEF/WHO/World Bank Special
Programme of Research, Development and Research Training in Human Reproduction (HRP), a cosponsored programme
executed by WHO.

viii
List of abbreviations
ADB Asian Development Bank ITU International Telecommunication Union
AEHIN Asia eHealth Information Network LMIS logistics management information system
BID Better Immunization Data M&E monitoring and evaluation
CRDM Collaborative Requirements Development MNO mobile network operator
Methodology
MOH Ministry of Health
CRVS civil registration and vital statistics
MOHS Ministry of Health and Sanitation
DHA Digital Health Atlas
NGO nongovernmental organization
DHI digital health intervention
OPENHIE Open Health Information Exchange
DIIG Digital Implementation Investment Guide
RE-AIM Reach, Effectiveness, Adoption,
DPPI Directorate of Planning, Policy and Implementation and Maintenance
Information
RFP request for proposals
EIR electronic immunization application
TOGAF The Open Group Architecture Framework
EPI Expanded Programme on Immunization
UAG user advisory group
FHIR Fast Healthcare Interoperability Resource
UHC universal health coverage
HIS health information system
WHO World Health Organization
HL7 Health Level 7
HMIS health management information system
HSC health system challenge
ICD International Classification of Diseases
ICT information and communications
technology

ix
x
 CHAPTER

01 INTRODUCTION

This Digital Implementation Investment Guide (DIIG) aims to help governments

and technical partners plan a digital health implementation that focuses on one
or more health programmes to support national health system goals.

The Guide is designed to walk users of the document step-by-step through

planning, costing and implementing digital health interventions within a digital
health enterprise. This consists of selecting digital health interventions that are
aligned with identified health needs, appropriate to a specific country context
and integrated with existing technologies and the broader digital architecture.
Users of the Guide will learn from diverse experiences deploying digital health
technologies over the past decade and will be guided through a systematic
approach to designing, costing and implementing meaningful digital health
interventions that are part of a digital health enterprise.

A digital health enterprise comprises the business processes, data, systems and technologies used to support
the operations of the health system, including the point-of-service software applications, devices and
hardware and the underlying information infrastructure (such as the digital health platform) that deliver health
services accelerated and amplified by digital and data technologies. Digital health enterprise architectures
have varying degrees of maturity and institutionalization within the broader ecosystem. This document
makes a distinction between siloed digital health system architectures and exchanged digital health system
architectures that contribute to a national digital health enterprise architecture. Siloed digital health system
architectures are disconnected applications that aim to fulfil a project goal. These siloed digital health
implementations are implemented in the context of a time-bound, stand-alone digital health project, usually
to demonstrate proof of concept, findings from which may eventually contribute towards a government-
sponsored digital health implementation.
Exchanged digital health system architectures, on the other hand, consist of multiple applications leveraging
standards and connected through a health information exchange to address needs across various health
programmes, operating in a coordinated manner within a national digital health enterprise architecture.
Additionally, this Guide acknowledges the existence of siloed, integrated and ball-of-mud architectures
and steers the user towards planning and investing in cumulative and modular enterprise digital health
implementations that result in collective benefit across the health system. This Guide focuses on the
implementation of exchanged digital health system architectures that are modular in nature and support one
or more programmes across the health sector and, potentially, even beyond the health sector.

1
This Guide serves as a companion to the WHO guideline: WHO guideline
recommendations The WHO Guideline: +
recommendations on digital interventions for health on digital
interventions
for health system recommendations on digital
system strengthening (1) and provides a pragmatic strengthening
interventions for health system
process for reviewing WHO guideline–recommended strengthening (1) provides
digital health interventions and incorporating them evidence-based recommendations
into harmonized plans grounded in national systems on how specific interventions
and policy goals. Additionally, this document builds can address identified gaps in the
on Planning an information systems project: a toolkit for health system. This resource represents the first
public health managers (2), commissioned by Optimize: official guidelines from WHO exclusively on digital
Immunization Systems and Technologies for Tomorrow, health and enables policy-makers, managers and
a previous collaboration between the World Health other stakeholders to understand the implications
Organization (WHO) and PATH. Since the publication of of prioritized digital health interventions. This Guide
Planning an information systems project in 2013, countries provides a facilitated process for incorporating
have considerably increased their use of technologies for the recommended interventions within a health
health, reflecting a growing expectation across the globe programme area, aligned to identified health system
that digital health interventions be part of established challenges (see Annex 5.3).
health programmes to address persistent gaps in the Classification of
Digital Health
The WHO Classification of digital +
performance of health systems. This growth introduces Interventions v 1.0
health interventions (4) groups the
A shared language to describe the uses of digital technology for health

What is it? The classification of digital health interventions (DHIs) categorizes the different

new challenges in ascertaining the right functionality

ways in which digital and mobile technologies are being used to support health

various ways technologies support

system needs. Targeted primarily at public health audiences, this Classification
framework aims to promote an accessible and bridging language for health program
planners to articulate functionalities of digital health implementations. Also referred
to as a taxonomy, this Classification scheme is anchored on the unit of a “digital health
intervention,” which represents a discrete functionality of the digital technology to
achieve health sector objectives.

1.0 2.0 3.0 4.0

relative to identified needs, comparing the relevance of

Clients Healthcare Providers Health System Managers Data Services

health system needs into four

1.1 Targeted client
communication 1.3 Client to client
communication 1.6
On-demand
information 2.1
Client
identification and 2.5
Healthcare
provider 2.8 Healthcare
provider training 3.1 Human resource
management 3.3 Public health
event notification 3.6 Equipment and
asset management 4.1
Data collection,
management, 4.2 Data
coding 4.3 Location
mapping
services to clients registration communication and use
Transmit health event 1.3.1 Peer group for clients Provide training content to List health workforce Notification of public Monitor status of Parse unstructured data Map location of health
1.1.1 alerts to specific 2.8.1 healthcare provider(s) 3.1.1 cadres and related 3.3.1 health events from 3.6.1 health equipment 4.2.1 into structured data 4.3.1 facilities/structures
Client look-up of health Verify client Communication from Non-routine data
population group(s) 1.6.1 information 2.1.1 unique identity 2.5.1 healthcare provider(s) to identification information point of diagnosis 4.1.1 collection and
Assess capacity of Track regulation and Merge, de-duplicate, and Map location of
Transmit targeted health 1.4 Personal health
tracking 2.1.2 Enrol client for health
supervisor 2.8.2 healthcare provider(s)
3.1.2 Monitor performance of 3.6.2 licensing of medical
management
4.2.2 curate coded datasets or 4.3.2 health events
1.1.2 information to client(s)
based on health status or 1.7 Client financial
transactions
services/clinical care plan Communication and
2.5.2 performance feedback to
healthcare provider(s)
3.4 Civil Registration
and Vital Statistic
equipment
4.1.2 Data storage and
aggregation
terminologies
Map location of
4.3.3 clients
demographics Access by client to own healthcare provider(s) Prescription Manage certification/ Classify disease codes or and households
Transmit targeted alerts
1.4.1 medical records
Transmit or manage out 2.2 Client health
records Transmit routine news and
2.9 and medication 3.1.3 registration of healthcare
provider(s) 3.4.1 Notify birth event 3.7 Facility 4.1.3 Data synthesis and
4.2.3 cause of mortality
Map location of
1.1.3 and reminders to client(s) Self monitoring of health 1.7.1 of pocket payments by management management visualization 4.3.4 healthcare providers
1.4.2 or 2.5.3 workflow notifications to Record training credentials
diagnostic data by client client(s) healthcare provider(s) 3.1.4 3.4.2 Register birth event Automated analysis of
Transmit diagnostics Longitudinal tracking Transmit or track of healthcare provider(s) List health facilities and data to generate new
2.2.1 2.9.1 3.7.1 4.1.4
1.1.4 result, or availability of
result, to client(s)
1.4.3 Active data capture/
documentation by client 1.7.2
Transmit or manage
vouchers to client(s) for
of clients’ health status
and services
Transmit non-routine
2.5.4 health event alerts to
prescription orders
3.4.3 Certify birth event
related information information or predictions
on future events
4.4 Data exchange and
interoperability
Track client's medication 3.7.2 Assess health facilities
health services
Transmit or manage
Manage client’s structured
2.2.2 clinical records
healthcare provider(s) 2.9.2 consumption 3.2 Supply chain
management 3.4.4 Notify death event
Data exchange
1.2 Untargeted client
communication 1.5 Citizen based
reporting 1.7.3 incentives to client(s) for Manage client’s
Peer group for healthcare
2.5.5 providers 2.9.3 Report adverse drug events
Manage inventory and
3.4.5 Register death event
4.4.1 across systems
health services
2.2.3 unstructured 3.2.1 distribution of health
Transmit untargeted Reporting of health system clinical records Laboratory and 3.4.6 Certify death event
1.2.1 health information to an 1.5.1 feedback by clients
Routine health indicator
2.6 Referral
coordination 2.10 Diagnostics
commodities
Notify stock levels of
undefined population
2.2.4 data collection and Imaging 3.2.2 health
Transmit untargeted 1.5.2 Reporting of public health
events by clients management
2.6.1 Coordinate emergency Manangement commodities
3.5 Health
financing
1.2.2 health event alerts to response and transport Monitor cold-chain
Transmit diagnostic result 3.2.3 sensitive commodities
undefined group 2.10.1 to
Healthcare Manage referrals between healthcare provider 3.5.1 Register and verify client
2.3 provider decision 2.6.2 points of service within
Transmit and track
Register licensed drugs
3.2.4 and health commodities
insurance membership
support health sector 2.10.2 diagnostic orders Track insurance billing and
Manage procurement 3.5.2 claims submission
Manage referrals between 3.2.5 of
Provide prompts and 2.6.3 health and other sectors Capture diagnostic results commodities
2.3.1 alerts based according 2.10.3 from digital devices Track and manage
Report counterfeit or 3.5.3 insurance reimbursement
to protocol
Health worker 2.10.4 Track biological specimens 3.2.6 substandard drugs
Provide checklist
2.3.2 according to protocol 2.7 activity planning by clients Transmit routine payroll
3.5.4 payment to healthcare
and scheduling provider(s)
Screen clients by risk or
2.3.3 other health status Identify client(s) in need Transmit or manage
2.7.1 of services 3.5.5 incentives to healthcare
provider(s)
2.4 Telemedicine 2.7.2 Schedule healthcare
provider's activities 3.5.6 Manage budget and
expenditures
Consultations between
2.4.1 remote client and
healthcare provider
Remote monitoring of
2.4.2 client health or diagnostic
data by provider
Transmission of medical
2.4.3 data to healthcare provider
Consultations for case
2.4.4 management between
healthcare provider(s)

competing technologies and ensuring that investments categories: clients, health workers,
How to use it? The digital health interventions are organized into the following overarching groupings
based on the targeted primary user:
Interventions for clients: Clients are members of the public who are potential
or current users of health services, including health promotion activities.
Caregivers of clients receiving health services are also included in this group.
Interventions for healthcare providers: Healthcare providers are members
of the health workforce who deliver health services.

have the desired sustainable impact – all challenges that health system managers and
Interventions for health system or resource managers: Health system
and resource managers are involved in the administration and oversight of
public health systems. Interventions within this category reflect managerial
functions related to supply chain management, health financing, human
resource management.
Interventions for data services: This consists of crosscutting functionality to
support a wide range of activities related to data collection, management, use,
and exchange.

this updated publication aims to address. data services. This Guide uses the
World He alth O rgani zati on Classi f i c ati on of Di gi tal He alth I nte rve nti ons page 1

classifications to categorize various health system

Several key WHO sources have also informed the content challenges and digital health intervention solutions
and processes of this Guide. (see Chapters 3 and 4).
+
The WHO and International
National eHealth Strategy Toolkit + The World Health Assembly WHO Resolution
Telecommunication Union (ITU) on Digital Health represents global recognition
National eHealth strategy toolkit from WHO Member States on the potential of
(3), which provides for government digital technologies to advance the Sustainable
agencies a framework and Development Goals and support health systems. The
method for developing a national resolution includes a request that countries “consider,
page I

eHealth vision, an action plan as appropriate, how digital technologies could be

and a monitoring framework. A national eHealth or integrated into existing health systems infrastructure
digital health strategy specifies high-level goals and and regulation, to reinforce national and global
outcomes for the health system that drive a country’s health priorities by optimizing existing platforms and
eHealth priorities. This Guide can help translate that services” (5).
national-level strategy into specific digital health
implementations that can support the broader digital
+ The WHO Global strategy on digital health (6) serves
as a global call to action and operational plan for
health vision (see section 2.2).
WHO and Member States to effectively implement
and benefit from digital health for health systems to
achieve global and national goals.

2 Digital implementation investment guide

The process outlined in the Guide is also informed Be Data Driven: When an initiative is data driven,
by the Principles for Digital Development, which quality information is available to the right people
help stakeholders effectively and appropriately apply when they need it, and they are using those data
digital technologies in their health programmes to take action.
(digitalprinciples.org/principles). Core tenets of
these nine principles will be referenced throughout Use Open Standards, Open Data, Open Source,
this Guide. and Open Innovation: An open approach to digital
development can help to increase collaboration
Design with the User: User-centered design in the digital development community and avoid
starts with getting to know the people you are duplicating work that has already been done.
designing for through conversation, observation
and co-creation. Reuse and Improve: Reusing and improving is
about taking the work of the global development
Understand the Existing Ecosystem: Well- community further than any organization or
designed initiatives and digital tools consider the program can do alone.
particular structures and needs that exist in each
country, region and community. Address Privacy & Security: Addressing privacy
and security in digital development involves careful
Design for Scale: Achieving scale requires consideration of which data are collected and how
adoption beyond an initiative’s pilot population data are acquired, used, stored and shared.
and often necessitates securing funding
or partners that take the initiative to new Be Collaborative: Being collaborative means
communities or regions. sharing information, insights, strategies and
resources across projects, organizations and
Build for Sustainability: Building sustainable sectors, leading to increased efficiency and impact.1
programs, platforms and digital tools is essential
to maintain user and stakeholder support, as well
as to maximize long-term impact.

1 Text reproduced from Principles for Digital Development (7).

Introduction 3
1.1 The Guide’s role in planning and
implementing a digital health enterprise
A fully realized digital health enterprise delivers health in the digital age. The implementation of a digital
health enterprise includes the people who design, build, deploy and maintain the systems, accompanied by a
governance framework, an enabling policy environment and an operational plan. The process of planning and
implementing an appropriate digital health enterprise within the broader ecosystem includes several phases:

1. assessing the current state and enabling environment

2. establishing a shared understanding and strategic planning
3. defining the future state
4. planning enterprise architecture
5. determining health content requirements
6. monitoring and evaluation (M&E) and fostering data use
7. implementing, maintaining and scaling.

Fig. 1.1.1 outlines these steps as well as supporting resources (referenced in Table 1.1.2) that will help you successfully
navigate the process of planning and implementing an appropriate digital health enterprise. This Guide focuses
primarily on defining the future state and highlights appropriate resources associated with the other phases. (See
Box 1.1.3 for resources applied to specific use cases.)

4 Digital implementation investment guide

Introduction

Fig. 1.1.1. Planning and implementing a digital health enterprise: phases, steps and resources.

STEPS
PHASE ASSESSING THE Global Digital
CURRENT STATE + Conduct an inventory of existing or previously WHO Digital Digital Health HIS Stages of

01 AND ENABLING
ENVIRONMENT
used software applications, ICT systems
and other tools to better understand the
requirements for reuse and interoperability.
Health Atlas Health
Index
Investment
Review Tool
Continuous
Improvement

STEPS
ESTABLISHING + Develop a national digital health strategy WHO/ITU WHO WHO
PHASE A SHARED outlining overarching needs, desired activities National Guideline: Classification Principles Principles for

02
Recommendations
UNDERSTANDING and outcomes. eHealth on Digital of digital for Donor Digital
AND STRATEGIC + Define a vision for how the health system will Strategy Interventions for health Alignment Development
PLANNING be strengthened through the use of digital Toolkit Health System
Strengthening
interventions
technology.
STEPS
+ Formulate a digital health investment roadmap Digital Unicef
PHASE to support the national digital health strategy.
DEFINING THE Square Human WHO digital

03 FUTURE STATE + Plan and identify appropriate digital

interventions, alongside the health and data
content, to improve health system processes and ITU SDG
Global
Goods
Guidebook
Centered
design
toolkit
clearing-
house

address programmatic needs. Digital Digital ITU Digital

implementation investment health
investment framework Platform
STEPS guide handbook
+ Review the current state and develop an (this document) (eGov)
PHASE Open The Open Guidance World Bank
PLANNING THE architecture blueprint for the design of the digital Health Group for digital
04 ENTERPRISE
ARCHITECTURE
health implementations.
+ Identify validated open standards to ensure
data exchange, systems integration and future
Information
Exchange
(Open HIE)
Architecture
Framework
(TOGAF)
investing
in digital
health
identity
toolkit
proofing of digital health implementations.

STEPS
WHO Be He@lthy,
PHASE + Identify validated health content appropriate for handbook WHO Smart Guidelines
DETERMINING the implementation context. for WHO Core Be Mobile

05 HEALTH CONTENT Machine- indicator handbooks

+ Ensure use of content aligned with identified digitalizing Digital readable for non-
REQUIREMENTS primary accelerator sets communicable
standards for the future state. kits recommen- diseases
health care dations

STEPS WHO
M&E OF + Monitor your implementation to ensure digital PATH MEASURE
PHASE DIGITAL HEALTH Monitoring Defining data
implementations are functioning as intended and
06 IMPLEMENTATIONS
AND FOSTERING
DATA USE
and having the desired effect.
+ Foster data-driven adaptive change management
within the overall health system.
evaluating
digital health
interventions
and building
a data use
culture
demand
and use
resources

WHO
STEPS MAPS
PHASE IMPLEMENTING, toolkit: ADB Total
+ Maintain and sustain digital health mHealth Cost of
07 MAINTAINING AND
SCALING
implementations.
+ Identify risks and appropriate mitigations.
Assessment
and
Planning for
Ownership
Tool PDF for Print
Online environment
Scale
5
Table 1.1.2. Resources for planning and implementing a digital health enterprise.2

Phase Resources
PHASE 1 » WHO/ITU National eHealth strategy toolkit (3)
Assessing the » WHO Digital Health Atlas (8)
current state » Global Digital Health Index (9)
and enabling » Digital health investment review tool (10)
environment » HIS stages of continuous improvement toolkit (11)

PHASE 2
» Digital implementation investment guide (this document)
Establishing
» WHO Guideline: recommendations on digital interventions for health system strengthening (1)
a shared » WHO Classification of digital health interventions (4)
understanding » Principles of Donor Alignment for Digital Health (12)
and strategic » Principles for Digital Development (7)
planning

» ITU/DIAL SDG digital investment framework (13)

PHASE 3 » ITU Digital health platform handbook (14)
Defining the » Digital Square Global goods guidebook (15)
future state  » UNICEF Human-centered design toolkit (16)
» WHO Digital Clearinghouse (https://clearinghouse.who.int)

PHASE 4 » World Bank Digital identity toolkit (17)

Planning the » Open Health Information Exchange (OpenHIE) (18) 
enterprise » The Open Group Architecture Framework (TOGAF) Standard (19)
architecture » Guidance for investing in digital health (20)

» WHO Handbook for digitizing primary health care (21)

» WHO Smart Guidelines:
PHASE 5 + WHO Digital accelerator kits (22)
Determining + WHO Machine-readable recommendations
health content + WHO Fast Healthcare Interoperability Resource (FHIR) implementation guides (23)
requirements » Be He@lthy, Be Mobile handbooks for noncommunicable diseases (24)
» WHO 2018 global reference list of 100 core health indicators (25)
» Content for specific health domains (see Box 1.1.3)

PHASE 6
M&E of
» WHO Monitoring and evaluating digital health interventions (26) 
digital health » Defining and building a data use culture (27)
implementations » MEASURE data demand and use resources (28)
and fostering data
use

PHASE 7
Implementing, » The WHO MAPS toolkit: mHealth assessment and planning for scale (29)
maintaining and » Asian Development Bank (ADB) Digital health investment costing tool (30)
scaling

2 Additional resources for further reading included in Annex 1.2.

6 Digital implementation investment guide

 Box 1.1.3. Resources detailing content for specific health programme areas.
+ Civil registration and vital statistics (CRVS) digitisation guidebook (31)
+ Common requirements for maternal health information systems (32)
+ Electronic immunization registry: practical considerations for planning, development, implementation and
evaluation (33)
+ Electronic recording and reporting for tuberculosis care and control (34)
+ Mobile solutions for malaria elimination surveillance systems: a roadmap (35)

The development of a digital health enterprise is a different digital health interventions. Within a broader
dynamic process (Fig. 1.1.4). Depending on your country’s ecosystem of valuable documents, tools and processes
context, needs and constraints, you may reorder the important to planning and implementing digital
steps taken during each phase or even the overall health, this document focuses on several important
process. Or you may need to revisit earlier phases as phases, providing specific steps and outputs, which are
your national digital health ecosystem and health needs harmonized with the concepts, frameworks and terms of
change over time, requiring new strategic plans and these other prominent digital health resources.

Fig. 1.1.4. Essential processes of national digital health implementations.

National Health
Health National Inventory of Digital Assets Programme
Priorities Specific Needs
Digital
Implementation

DIIG
Investment Guide (DIIG)
Integrating Digital Interventions
into Health Programmes

APPLICATION
WITH
INTERVENTIONS

DIGITAL HEALTH INVESTMENT ENTERPRISE APPLICATION SCALE,

STRATEGY ROADMAP ARCHITECTURE WITH INSTITUTIONALIZE
INTERVENTIONS AND SUSTAIN

APPLICATION
WITH
INTERVENTIONS

Digital Maturity Governance Mechanisms Data Use Culture

of Country

NATIONAL eHEALTH STRATEGY BUILDING BLOCKS

Leadership & Strategy & Legislation, Standards & Services &

Governance Investment Policy, & Workforce Infrastructure Interoperability Applications
Compliance

Introduction 7
1.2 How to use this Guide
This Guide covers the different phases outlined in Fig. 1.1.1 by leveraging supporting resources, focusing the most
detail on phases 2–5. This document is designed to be used in two ways:
+ to support a facilitated planning process, resulting in a costed implementation plan suitable for a funder, whether
a government finance ministry or an agency (such as the World Bank, Gavi, The Global Fund or Bill & Melinda Gates
Foundation).
+ to provide stand-alone topical information and outputs, organized by chapter, for establishing and costing a digital
health implementation.

This Guide has been developed to accommodate BID INITIATIVE

different approaches for planning digital health BID Initiative CASE STUDY
implementations. It facilitates the stepwise case study:
or piecemeal approach to planning, as well as Step-by-step guide
comprehensive approaches to how countries develop
Throughout this Guide, you will find illustrations
digital investments that can be leveraged across more
from the BID Initiative (bidinitiative.org) of how
than one health programme. CHAPTERS 2–3 are useful
government stakeholders and technical partners
for reviewing the performance of the health system
navigated these steps in designing, costing and
and determining the challenges and needs within your
implementing a digital health intervention.
programme area, regardless of whether or not you seek
Launched in 2013, the BID Initiative was led
digital health interventions as part of the mechanism to
by PATH and the governments of Tanzania and
address the identified needs. CHAPTER 4 helps identify
Zambia. The BID Initiative was grounded in the
appropriate digital health interventions in line with
belief that better data plus better decisions
health system challenges and the maturity of the digital
would lead to better health outcomes. Although
ecosystem. CHAPTERS 5–7 can help ensure that your
BID initially focused on addressing critical
plans to implement digital health are viable and that
routine delivery of immunization services, the
digital applications are integrated and sustained within
approach was designed to be applicable to other
financial and other constraints.
health areas, such as nutrition or maternal and
Completing the Guide in its entirety will yield newborn health.
several outputs that can be combined into a costed
implementation plan or considered individually:
» summarized high-level financial plan and costing
» team roles and responsibilities and charter stating
shared vision and goals » proposal for governance mechanisms and risk
mitigation
» documented health programme processes and
end-user and data workflows » logic-model plan for M&E and adaptive
management.
» defined problem statement detailing specific
bottlenecks, challenges and needs in the health » Each chapter includes the following information:
system » suggested inputs and expected outputs
» identification of appropriate, prioritized digital » a step-by-step process to guide work
health interventions to address current challenges
and needs » illustrative examples from real programmes
» detailed assessment of the enabling environment » templates or worksheets to complete
and current constraints » additional resources for more information
» implementation plan that considers existing » links to relevant Principles for Digital Development.
resources, potentially reusable components,
environmental limitations and needs for standards The structure and key outputs of this Guide are outlined
and interoperability in Fig. 1.2.1.

8 Digital implementation investment guide

Fig. 1.2.1. Overview of the chapters in this Digital Implementation Investment Guide.

By the end of this Guide, you will be able to complete essential activities (see Fig. 1.2.1) that produce outputs that
comprise a costed implementation plan (see Fig. 1.2.2).

CHAPTER 01

INTRODUCTION

CHAPTER 03
CHAPTER 02
+ Historical and/or baseline data
+ Named team and list of stakeholders, + National digital health strategy
INPUTS including personas + Health programme objectives, progress
+ Shared vision of health programme goals INPUTS
and any evaluations
+ Organigram describing ministry of health
and relevant government bodies
IDENTIFY HEALTH SYSTEM
CHALLENGES AND NEEDS FORM THE TEAM AND
+ Analysis of the health programme ESTABLISH GOALS
processes
+ Current state task flow diagrams, user + Named team and list of stakeholders
journeys, programme processes + Shared vision of programme goals linked
OUTPUTS to digital health strategy
+ Prioritized pain points and health system OUTPUTS
challenges + Personas within the health programme

CHAPTER 05
CHAPTER 04
+ Personas and future state user task flow
+ Targeted health programme processes diagrams
+ Current state task flows, user journeys, + Enabling environment assessment,
and programme processes INPUTS digital landscape results
INPUTS + Functional and nonfunctional
+ Prioritized pain points and health system
challenges requirements

DETERMINE APPROPRIATE PLAN THE IMPLEMENTATION

DIGITAL HEALTH INTERVENTIONS
+ Prioritized digital health intervention(s) + Detailed implementation plan for each
+ Future state user journey / task flow intervention, specifying:
diagrams » G overnance, workforce and training, links
+ Enabling environment readiness, digital to strategy and investment plans
OUTPUTS
OUTPUTS landscape results & systems to leverage » Relevant health and data content,
+ Expectations, Functional and Infrastructure, existing digital systems,
nonfunctional requirements standards and interoperability needs

CHAPTER 06

+ Future state workflow with digital health

CHAPTER 07 interventions and requirements
INPUTS
+ National digital health architecture
+ Historical budgets and costs
INPUTS + Implementation plan LINK THE DIGITAL HEALTH INVESTMENT
TO THE BROADER ARCHITECTURE
DEVELOP A BUDGET
+ Current state and future state digital
health enterprise architecture
+ Programme budget defined by phases + Core and enabling components of
and cost drivers planned digital investment
OUTPUTS OUTPUTS + Common components for reuse or
+ Expected benefits over time
leveraged across programmes and
sectors

CHAPTER 09

INPUTS
+ Outputs from previous chapters
CHAPTER 08
VALUE PROPOSITION
+ Historical and/or baseline data and AND NEXT STEPS
analysis
INPUTS + Historical M&E and adaptive + Team, stakeholders and programme
management plans priorities
+ Defined health system challenges and
MONITOR THE IMPLEMENTATION pain points
AND USE DATA EFFECTIVELY + Appropriate digital health intervention,
requirements Costed
+ Logic model for digital health + Enabling environment assessment; Implementation
implementation OUTPUTS current state of digital architecture Plan
+ Plan for monitoring and evaluation, + Implementation plan including proposed
OUTPUTS maturity assessment digital investments
+ Plans for establishing culture of data use; + Phased budget, and coordinated
adaptive management for optimization investments
+ M&E and adaptive management plans

Introduction 9
10

Fig. 1.2.2. Summary of outputs within a costed implementation plan.

01 CHAPTER 02 02 CHAPTER 02 03 CHAPTER 03

Actors Priorities Programme Context

3.1 Personas & organigram

Health 3.2 Pain points

Goals 1.1 Team
2.1 Within health 3.3 Health System
Programme(s) & System Challenges and needs
1.2 Stakeholders
2.2 Within Digital 3.4 Current-state
1.3 Beneficiaries Health Strategy Workflow Diagrams

04 CHAPTER 04
06 CHAPTER 06
Common
Digital Context Architecture Services
Across
4.1 Prioritized Digital 6.1 Current 6.2 Future
Sectors
Digital Interventions Architecture Architecture
Context
4.2 Maturity and 4.3 Functional 6.3 Standards &
Readiness Requirements Interoperability Costed
4.4 Future-
Implementation
4.5 Digital 6.4 Point of Service
state
Workflow Inventory Applications 6.5 Shared Services Plan
Digital implementation investment guide

05 CHAPTER 05 07 CHAPTER 07 08 CHAPTER 08 09 CHAPTER 09

Implementation Budget Monitoring Value
Proposition
Financial and
8.1 Logic Model
Operational
Considerations 5.1 Implementation 7.1 Phases and 8.2 Monitoring &
Plan Costs evaluation

5.2 Enabling 7.2 Financing 7.2 Adaptive 9.1 Value

Environment Management Proposition
1.3 Key terms for using this Guide
A full glossary is included as Annex 1.1, but several key terms are useful to understand at the outset.

This Guide assists readers to develop a costed DIGITAL HEALTH PLATFORM: A shared digital health
implementation plan to support a digital health information infrastructure (infostructure) on which
enterprise comprising appropriate digital health digital health applications are built to support consistent
interventions targeting health system challenges and and efficient healthcare delivery. The infostructure
deployed within digital applications to strengthen the comprises an integrated set of common and reusable
performance of one or more health programmes and components that support a diverse set of
realize digital health outcomes within national digital digital health applications. The components consist of
health strategies. software and shared information resources to support
integration, data definitions and exchange standards for
BOTTLENECKS: Specific gaps or problems in the
interoperability and to enable the use of point-of-service
workflow of delivering health services specific to a
applications across health programme areas and use
health programme area, persona or process (for example,
cases (14).
“health workers have difficulty keeping track of when
pregnant women are due for an antenatal care visit”), DIGITAL HEALTH ENTERPRISE: The organizational unit,
in contrast to a health system challenge, which is a organization or collection of organizations that shares a
general representation of the problem across any health set of health goals and collaborates to provide specific
programme area (for example, “loss to follow-up”). health products and/or services to clients, along with the
business processes, data, systems and technologies used
COSTED IMPLEMENTATION PLAN: A document that
to support the operations of the health system, including
describes, in sequence, an identified set of challenges,
the point-of-service software applications, devices
accompanied by a contextually appropriate, financially
and hardware, governance and underlying information
justified plan for deployment and monitoring of
infrastructure (such as the digital health platform)
resources. The responsible party will use this plan to
functioning in a purposeful and unified manner.
obtain financial support to implement the proposed
activities for the digital health implementation within a This Guide distinguishes between four different types
specific timeline. The purpose of this Guide is to develop of digital health enterprise system architectures (see
a costed implementation plan for digital application(s) Fig. 1.3.1) along a continuum of maturity (37).
within an exchanged digital health system architecture
» SILOED: A digital health enterprise system
to address needs of health programme(s). architecture composed of stand-alone
application(s). A digital health project is a time-
DIGITAL HEALTH: Digital health is the systematic
bound implementation of a siloed digital health
application of information and communications enterprise, usually to demonstrate proof of
technologies, computer science, and data to support concept.
informed decision-making by individuals, the health
» MUD (Monolithic Unarchitected Software
workforce, and health systems, to strengthen resilience Distributions): Haphazardly structured, sprawling
to disease and improve health and wellness. (36). MUD systems are characterized by an evolving
agglomeration of functions, originating without
DIGITAL HEALTH APPLICATION: The software, a predetermined scope or design pattern, which
information and communications technology (ICT) accumulate technical debt.
systems or communication channels that deliver or
» INTEGRATED: A digital health enterprise system
execute the digital health intervention and health architecture in which two or more applications are
content (1, 14). directly connected to one another (that is, without
an intermediary data exchange), intended to
DIGITAL HEALTH ECOSYSTEM: The combined set of address one or more health system challenges and
digital health components representing the enabling fulfil health programme goals.
environment, foundational architecture and ICT
» EXCHANGED: A digital health enterprise system
capabilities available in a given context or country (14). architecture consisting of multiple applications

Introduction 11
 using standards to connect through a health programmes are government led and persist across
information exchange to address collective budget cycles as long as the underlying need persists.
needs across the health system, operating in Family planning and malaria control programmes are
a coordinated manner within a digital health
some examples of health programmes.
architecture.
HEALTH SYSTEM CHALLENGE: A generic (not health
DIGITAL HEALTH IMPLEMENTATION: The development domain specific) need or gap that reduces the optimal
and deployment of digital health application(s) and/ implementation of health services. Health system
or platform(s) to support and strengthen a health challenges represent a standardized way of describing
enterprise within a given context, accompanied by bottlenecks. For example, “loss to follow-up” is a health
a governance framework, operational plan, human system challenge used to generally describe specific
resources and related activities for its successful bottlenecks that may be articulated as “the person did
execution. not come back for their appointment” or “the person has
DIGITAL HEALTH INTERVENTION: A discrete not received a follow-up vaccination”.
technology functionality – or capability – designed INTEROPERABILITY: Interoperability is the ability of
to achieve a specific objective addressing a health different applications to access, exchange, integrate and
system challenge. Examples of digital health use data in a coordinated manner through the use of
interventions include decision support, targeted client shared application interfaces and standards, within and
communications, and stock notifications; see WHO across organizational, regional and national boundaries,
Classification of digital health interventions for a full to provide timely and seamless portability of information
list (4). and optimize health outcomes. (1, 14).
DIGITAL HEALTH INVESTMENT: Financial and other STAKEHOLDERS: All persons affected by or interested in
resources, including human resources, that are directed the consequences of a digital health implementation.
towards digital health implementations.
» The PLANNING TEAM includes stakeholders
DIGITAL HEALTH OUTCOME: The desired change in who are responsible for guiding the development
the health system or services by using digital health of the digital health implementation. This team
interventions. May also be known as an eHealth includes representatives from government and
outcome (3). implementing partners, where appropriate.
» END-USERS are individuals, typically health
DIGITAL HEALTH STRATEGY: An overarching plan that workers, who interact directly with the digital
describes high-level actions required to achieve national health intervention once implemented. End-users
health system goals. These actions may describe how may include health system managers who interact
new digital health components will be delivered or how with the data generated by the digital health
existing components will be repurposed or extended. intervention. Clients, or patients, could also be
end-users if they engage directly with the digital
May also be known as an eHealth strategy.
health intervention.
ENABLING ENVIRONMENT: Attitudes, actions, policies » BENEFICIARIES are individuals or members
and practices that stimulate and support effective, of the community who may benefit from the
efficient functioning of organizations, individuals and digital health intervention when used by another
programmes. The enabling environment includes end-user, such as a pregnant woman receiving
care from a health worker using a digital health
legal, regulatory and policy frameworks and political,
intervention like decision support to coordinate
sociocultural, institutional and economic factors (16). referrals.
These factors can include infrastructure, workforce,
» FUNDERS are organizations that provide resources
governance mechanisms and legislation and policies in
to design, develop and implement digital health
the country. implementations. They may be associated
with government agencies, nongovernmental
HEALTH PROGRAMME: Operational unit within a
organizations (NGOs), bilateral or multilateral
government ministry supporting formal activities agencies, private foundations or private-sector
institutionalized at a national or subnational level organizations.
to address clear priority health objectives. Health

12 Digital implementation investment guide

Introduction

Fig. 1.3.1. Digital health enterprise system architectures.

SILOED INTEGRATED EXCHANGED

A digital health enterprise system architecture A digital health enterprise system architecture in which A digital health enterprise system architecture consisting of multiple applications using
composed of standalone application(s). A digital two or more digital applications are directly connected to standards to connect through a health information exchange to address collective needs across
health project is a time-bound implementation each other (i.e. without an intermediary data exchange) the health sector, operating in a coordinated manner within a digital health architecture.
of a siloed digital health enterprise, usually to intended to address one or more health system
demonstrate proof of concept. challenges and fulfil health programme goals.
SHARED SHARED SHARED SHARED SHARED
SERVICE SERVICE SERVICE SERVICE SERVICE

HEALTH USE CASE HEALTH PROGRAMME

1 1 STANDARDS AND INTEROPERABILITY

APPLICATION APPLICATION

1 2 1 2 1 2
HEALTH PROGRAMME HEALTH PROGRAMME
OR USE CASE OR USE CASE

2 1 2 3
APPLICATION APPLICATION APPLICATION
APPLICATION
1 3 1 2 1 3 1 2
PROJECT GOALS

ineffective

ineffective HEALTH
PROGRAMME GOALS

MUD ineffective HEALTH SECTOR GOALS

Monolithic Un-architected
software Distribution. Applications
characterized by an evolving agglomeration of
functions, originating without a predetermined
scope or design pattern, which accumulate
technical debt.

TECHNICAL ECONOMIES OF SCALE

DEBT
ALIGNMENT WITH DIGITAL HEALTH STRATEGY

Digital Health Intervention Health System Challenge reusable functionalities

13
1.4 When to use this Guide
There are several situations where you might find this guide useful.

When identifying how best to apply the WHO guideline: The following resources provide examples of national
Recommendations on digital interventions for health digital health strategies:
system strengthening (1). For example, the guideline
» Every African country’s national eHealth strategy or
recommends the use of digital decision-support tools digital health policy (38)
for health workers within a health programme area;
» WHO Global Observatory for eHealth directory of
you would like to implement this recommendation but
eHealth policies (39).
need to better understand whether it is the appropriate
digital health intervention to address your health system When transitioning from a siloed digital health system
challenges, as well as understand how it fits within the architecture to an exchanged digital health system
context of the health programme, health system and architecture (see Fig. 1.3.1). This includes ensuring that the
digital health landscape. implementation of exchanged digital health enterprise
systems is costed and has thoroughly considered
When presented with a health system challenge and you
foundational aspects, such as governance, standards and
want to determine if and how to incorporate digital
interoperability.
health intervention(s) to address the identified challenge
within a health programme area. For example, there is When validating the need for a digital health investment
low use of health services, and you are thinking about in response to a funding request. This is a common
leveraging digital health to address this issue and achieve scenario; it is recommended that you follow the processes
an outcome of increased demand for services. in this Guide to adequately assess the health system
challenges and design a contextualized and impactful
When selecting specific digital health interventions to
digital health implementation that can support evolving
meet the objectives of a national digital health strategy.
and collective needs within a common digital health
For example, the digital health strategy may have listed
architecture.
“strengthening the workforce through digital technology”
as a digital health outcome. This Guide can help you select
appropriate interventions to meet this outcome and
understand how to design these interventions to work
well within your local context.

14 Digital implementation investment guide

 CHAPTER

02 FORM THE TEAM AND

ESTABLISH GOALS

Once you are ready to start using this Guide, the first step is to form the team
and establish goals for your investments in the digital health enterprise. In this
chapter, you will determine team roles and responsibilities, develop a common
understanding of the health programme’s goals, and begin to understand how the
health programme functions across all levels of the health system.

+ Common key roles and responsibilities (Table 2.1.1)

+ Planning and implementation charter (Annex 2.1)
TOOLS + Persona worksheet (Annex 2.2)

+ Identify team and stakeholder roles.

+ Establish programme needs, goals and objectives.
OBJECTIVES + Document structure of organizational units.

+ National digital health/eHealth strategy (if one exists)

+ Documents including health programme objectives, progress and any evaluations
INPUTS + Organograms describing directorates/departments in the Ministry of Health (MOH) and
relevant government bodies, such as Ministry of ICT, civil registrars and so on

+ Identification of team (Output 1.1)

+ List of stakeholders to engage (Outputs 1.1, 1.2)
+ Shared vision of priorities within the health programme and digital health strategy
OUTPUTS (Outputs 2.1, 2.2)
+ Organogram and personas within the health programme (Output 3.1)

Form the team and establish goals 15

 PRINCIPLES FOR
DIGITAL DEVELOPMENT

+ Understand how your work fits into the global development landscape. Identify
others working on the same problem in other geographies and determine if there
is a community of practice that relates to your work. Find the technical leaders
through virtual networks or communities of practice, such as the Global Digital
Health Network, the Asia eHealth Information Network (AeHIN), Global Digital
Health Partnership, Digital Health and Interoperability Working Group, Health Data
Collaborative, African Alliance for Digital Health and/or Implementing Best Practices
Initiative, who can help you disseminate your work to other teams, regions and
countries.

BE COLLABORATIVE 3
+ Engage diverse experts across disciplines, countries and industries throughout the
project life cycle. Create an engagement plan to apply this expertise at all phases
and incorporate insights through feedback loops. Look for tools and approaches from
other sectors and publish your findings so that they are available to other groups and
countries.
+ Plan to collaborate from the beginning. Build collaborative activities into proposals,
work plans, budgets and job descriptions. Identify indicators for measuring
collaboration in your M&E plan.

3 Text adapted from Principles for Digital Development: Be collaborative: Core tenets (7).

16 Digital implementation investment guide

2.1 Determine roles and responsibilities
Before embarking on the full planning process, you should form your team. Consider what skills, functions
and knowledge may help when crafting a comprehensive costed implementation plan for a funder. Seek out
individuals with adequate dedicated time who are qualified, motivated and knowledgeable over three areas:
governance, management and operations.

GOVERNANCE: Consult a national digital health Consider including these people on the management
governance committee or other government technical team:
oversight mechanism (see Fig. 1.1.4), if one exists, to
» project/process manager, who takes responsibility
ensure that the planned investment aligns with other for the delivery of this process
government investments and national priorities in the
» procurement manager
current or upcoming health-sector plan, or consider
forming such a mechanism if one does not exist. This » M&E analyst
committee can be responsible for providing overarching » digital health specialist/enterprise architect
direction and guidance. » health programme manager
» human-centred design advisers
If forming a governance committee, discuss the
governance principles and how responsibilities would » change management advisers
be shared, even if no formal governance policies exist » policy advisers
yet. Identify a senior ministry sponsor to chair the » implementing-partner representatives.
committee who can mobilize resources, align interests
and resolve potential conflicts, as well as a digital health OPERATIONS: The operations team works under the
lead who is tasked with oversight of deployment of the guidance of the management team, providing necessary
digital health enterprise. This process should lead to the technical expertise. Consider the skills needed to
development of a corresponding Terms of Reference. successfully implement the proposed work when
selecting operations staff, as they will provide important
Consider including these people on the governance perspectives on how to build out a viable plan. Team
committee: members may have multiple competencies, or you may
need more than one person for any one of these areas.
» senior ministry sponsor
» ministry lead for digital health, which may be Consider including these people as operations support:
Health Information Systems (HIS), M&E or
combined with the Ministry of ICT » business analysts (to develop requirements for the
» representatives of relevant additional ministries digital enterprise, consisting of applications and
(such as ICT, civil registrars, network regulators and platform)
local government) » software developers
» technical team leaders » system maintenance, optimization and end-user
» representatives of funding and technical agencies. support or help desk staff
» training or technical staff
MANAGEMENT: The management team will be
» database managers
responsible for completing the process to develop a
costed implementation plan as outlined in this Guide. » end-user representatives.
This team should expect to devote a substantial
Mapping stakeholders with the roles and responsibilities
level of their daily work time to these tasks. Look for
of individuals can help achieve this goal (see Annex
team members who possess a significant amount of
2.1). You will draw on various people at different points,
technical capacity; prior experience in managing large
and identifying expertise at the outset will speed the
implementations is an important asset.

Form the team and establish goals 17

planning process. Table 2.1.1 describes each of these roles require every role, these are perspectives that should
in detail to guide recruitment of the required skill sets. be considered either internally or through external
Although your digital health implementation may not mechanisms, such as proposals for technical assistance.

Table 2.1.1. Key roles and their descriptions.

Functional Role Description

area
GOVERNANCE Digital health May include a sponsor from a senior ministry and representatives from other relevant
governance ministries and funders, as well as technical team leaders. In some contexts, digital
mechanisms health governance committees are called Technical Working Groups.
These governing stakeholders oversee the digital health investment’s progress,
engage the right stakeholders, ensure financing and provide sufficient oversight to
the Management and Operations teams. They set the overall direction, make key
decisions and select important team members.
Digital health Accountable for successfully deploying the digital health intervention within
lead the context of the broader health system and may be tasked with developing or
refining the national digital health strategy. The digital health lead may also be
responsible for identifying needed policies and gaps, such as in data security,
standards and the national digital health enterprise architecture.
MANAGEMENT Project/process Takes on the responsibility for day-to-day direction of the implementation,
manager communicates with the Governance team and ensures that the system is deployed
on time and on budget. Ideally, an influential and skilled person within the MOH
will fill this role, but technical partners can support it. The project manager should
possess excellent managerial, technical and negotiation skills. This person also
facilitates and promotes adoption of the change-management process.
Procurement Ensures that implementation partners follow organizational guidelines for
manager contracting and licensing.
M&E analyst Reviews the planned digital health intervention to determine the key indicators
to use for monitoring progress and ensuring that national needs for reporting
indicators are met. In some contexts, this role may be called the data manager or
health management information system (HMIS) lead.
Digital health Ensures that the planned digital health implementation can operate within the
specialist/ current ICT infrastructure, recommends new infrastructure investments and
Enterprise reviews ICT human-resource capacity to ensure that the entire digital health
architect enterprise can be deployed sustainably. This person should have expertise in
informatics or HIS architecture.
Health Provide technical feedback on the programmatic needs and clinical workflows for
programme the specific health domain affected by the digital health implementation.
managers
Human-centred Engages the end users directly for co-design and co-creation of the planned
design expert digital health intervention. Trained in human-centered design, this person would
provide technical support in determining the most appropriate methodologies and
mechanisms for engaging the end-users from the beginning.
Change Explores the intended and unintended impact of the digital health interventions
management on current workflows, identifies potential risks (including human, technical and
expert system risks) and works with the team to develop mitigation plans for possible
failure points in the proposed implementation. This person also reflects on the
needs for change management and ensures that interventions take into account
end-users’ motivations to support adoption of the digital health interventions.
Implementing Represent organizations responsible for implementing the digital health
partner interventions within applications and the enterprise. Provide technical support,
representatives strategic direction and guidance and share practices from other contexts.

18 Digital implementation investment guide

 Functional Role Description
area
OPERATIONS Business analyst Analyses and documents workflow of the clinical care and health programme
processes and recommends digital health interventions relative to the prioritized
business requirements.
Software Transform requirements into specifications for information systems and
developers executable code.
Quality assurance Validate that the requirements have been met by the software developers and
testers oversee testing of the applications for bugs and functionality issues before
deployment.
Security and Manages and advises on security risks, including cloud hosting.
data-hosting
adviser
Systems Ensure that the digital health enterprise technologies are well maintained and
maintenance, operating at peak performance. Also provide different methods of support via
optimization and email, voice and in person to end-users and administrators of the digital health
end-user support enterprise.
or help desk staff
Training or Train and provide capacity-building support to end-users and administrators of the
technical staff digital health enterprise.
Database Maintain database performance and ensure that end-users can access and use
managers high-quality data.
End-user This important stakeholder group can give feedback on design choices, increasing
representatives ownership and agreement during implementation.

Box 2.1.2. Resources for establishing governance mechanisms.

For detailed case studies on establishing governance mechanisms, see the Broadband Commission Report Digital
health: a call for government leadership and cooperation between ICT and health (42), which describes processes
conducted at country level, bolstered by specific examples from Canada, Estonia, Malaysia, Mali, Nigeria, Norway,
the Philippines and Rwanda.

The Digital health convergence meeting toolkit (43) provides useful considerations for establishing governance
mechanisms through convergence meetings that bring together different stakeholders, including departments
in MOHs, development partners, international NGOs, experts in related fields and end-users. The convergence-
meeting approach offers a framework for bringing together government officials from different ministries,
digital health professionals in countries and technical experts committed to successfully implementing and
strengthening the HIS and digital health.

Form the team and establish goals 19

 CASE STUDY

Convening stakeholders and determining governance

structure in Sierra Leone
In 2016 following the Ebola viral disease outbreak, WHO facilitated a planning process with the Ministry
of Health and Sanitation (MOHS) for informing prioritization of digital systems addressing national health
priorities, in line with directives from the President’s Recovery Plan (40). This process sought to align the
use of digital tools with defined health system challenges in response to the immediate national priorities
of restoring health services, particularly related to malaria, acute malnutrition and maternal and newborn
deaths due to home deliveries (40). Furthermore, the fragmented approach to digital health investments
during the Ebola outbreak underscored the importance of governance structures and strategic planning.

During this process, WHO and the MOHS convened a diverse set of stakeholders representing programme
directorates leading work across various health domains, information systems and M&E of aggregate health
indicators, the Ministry of ICT, donors, NGOs and implementation partners (41). Over the course of a three-
day workshop, the convening organizations facilitated discussions on the specific health needs that would
be the focus of digital health investments, as well as the governance structure for developing a national
digital health enterprise architecture and administering digital health investments.

This process resulted in defining

the roles of different directorates
We, the stakeholders here gathered, under the firm
within the MOHS and assigning
leadership of the MOHS through the Directorate of
responsibilities for maintaining
Planning, Policy and Information (DPPI), hereby declare:
and harmonizing different
digital systems. The outcome of » Sierra Leone will develop a unified national
this meeting resulted in a joint architecture for our health information systems;
declaration across all programme » That will improve the availability, appropriate access
directorates within the MOHS and use of quality health information across all levels
known as the Bintumani of the health system;
Declaration (based on the » We will increase access to and use of health
name of the venue in which the information technology to improve service delivery
meeting and declaration was and demand for services to improve health outcomes;
held). The following is an excerpt » This process will be led, championed and sustained by
from the Bintumani Declaration: the DPPI for the benefit of all;
» We will strengthen our existing governance structure
to improve its effectiveness and participation by our
partners;
» We pledge to seek the commitment of government
and partners to provide technical and/or financial
resources to realize this vision.

These declarations – dubbed the Bintumani Declarations

– made on this 3rd day of August the year 2016 have been
endorsed by the Ministry of Health and Sanitation and its
partners. (41)

20 Digital implementation investment guide

 BID INITIATIVE
CASE STUDY

BID Initiative: User advisory groups and goals

The BID Initiative was designed to be country led and country owned, with partnership and
collaboration elevated as core values. In Tanzania, the Immunization and Vaccine Development
program (a department of the Ministry of Health, Community Development, Gender, Elderly and
Children) oversaw the overall governance of the BID Initiative, and a user advisory group (UAG) was
established to play a critical role in the prioritization, design and development of proposed approaches
and interventions (44). PATH and other technical partners provided management and operations
support. The UAG met in person monthly, and each member served as a champion for BID, sharing
information and gathering feedback in their local communities, as well as supporting dissemination
and implementation of the completed interventions.

The UAG provided an opportunity for end-users » poor data visibility into supplies at the facility
to get deeply involved in BID’s strategic planning and district levels
and decision-making from the outset. It included » complex data-collection forms and tools
15 representatives from all levels of the health » insufficient management of supply chains
system in the BID Initiative’s Arusha testing region, and logistics
including the Regional Immunization and Vaccine » inadequate capacity for data management
Officer. Members provided input on topics like and use at all levels of the health system.
supply chain, data collection, service delivery and
community involvement. End-user involvement is These problems were identified through desk
critical in understanding what end-users do, what reviews, consultation meetings, scoping visits,
they need and how to create a sense of ownership an in-depth analysis of demographic and
that is crucial to the success of any initiative. A immunization data and a landscape of the
similar UAG was also set up in the Livingstone countries’ digital health infrastructure at the outset
District of Zambia (45). of the BID Initiative.

These stakeholders helped validate and refine The following tools are available from the BID
a list of the most critical routine problems with Initiative to help with forming UAGs.
immunization service delivery that they would
» The Stakeholder analysis tool (46) helps
work to address through the initiative: identify and map individuals and
» incomplete or untimely data organizations to include on your team.

» lack of unique identifiers for infants » The UAG terms of reference (47) can
be adapted to reflect the goals and
» inaccurate or uncertain target population for responsibilities of your unique UAG.
calculating immunization rates
» difficulty identifying infants who do not start
immunization or who drop out (defaulter
tracing)

Form the team and establish goals 21

2.2 Develop a common understanding of the
health programme’s needs and goals
Once you have formed the planning team, convene them and any necessary additional stakeholders to
articulate a common understanding of the main goals of the health programme and how they align with
your country’s national digital health strategy (if there is one). The team should also review core programme
documents, data and assessment reports that describe the programme’s goals and objectives, including how it
has performed to date.

In this review of health programme documents, aim to health strategy might include overarching
clearly identify the following: goals like “improved access to health
services”. This may be followed by “digital
1. Short- and long-term goals and objectives of health outcomes” (possibly stated as “eHealth
the health programme outcomes” or “health ICT outcomes”, as in
the example of Nigeria in Fig. 2.2.1), such as
a. Assess how the program aligns with priorities “effective use of telemedicine and ICT for
under the national strategic health plan or health worker training and support” (48).
other government strategies for investing c. Assess how well these goals align with the
in health. Ensure that all stakeholders needs of the population that the health
have a shared understanding of the health programme targets. Should the health
programme’s goals. programme focus on particular populations
b. Assess how the health programme aligns or groups to improve equity and coverage?
with the national digital health strategy (if
one exists). For example, the national digital

Fig. 2.2.1. Example of Nigeria’s national health ICT vision extracted from the national
eHealth strategy.

NIGERIA
NATIONAL By 2020, Health ICT will help deliver and enable universal health coverage — whereby
HEALTH Nigerians will have access to the services they need without incurring financial risk.
ICT VISION

Increased
Improved Increased Increased Increased equity in,
UHC access to coverage of uptake of Improved financial access to,
OUTCOMES health health health quality of care coverage for and quality of
health services,
services services services health care information,
and financing.
Effective use
Effective use of ICTs for
Effective of CRVS, HRIS, Effective use of delivering
use of NHMIS & LMIS Effective use Effective use of ICT for health appropriate
of mobile ICT for decision
HEALTH ICT telemedicine for tracking messaging & support & insurance health services
and use of demand & other for those who
OUTCOMES ICT for health and supply cash transfers within the health-related need them
worker training of health for demand continuum financial most based on
and support services and creation of care transactions epidemiology
commodities and ability to
pay
Nationally scaled integrated Health ICT services and applications supported by
LONG-TERM Nigerian Health Information Exchange implemented with appropriate funding,
ICT OUTPUTS
infrastructure & equipment, training & policies.

Source: Government of Nigeria National health ICT strategic framework, 2016 (48).

22 Digital implementation investment guide

 2. Health programme indicators, or measures of b. Assess whether data to understand
success, and evaluations to help assess whether programme performance are adequate and
health programme goals are being met, which routinely collected or whether additional
will help establish deficiencies within the data are required to improve programme
assessment; articulate these as “gaps”.
health programme and begin to identify where
to target potential digital health investments c. Use this discussion to identify any
other objectives that are valuable to the
a. Team members focused on M&E should programme stakeholders but have not been
identify and summarize appropriate health clearly articulated in the documented goals
programme performance reports prior to the and objectives reviewed in (1) above.
stakeholder meeting.

2.3 Understand programme operations

across levels of the health system
All stakeholders need to understand how the current health programme operates in practice, including the
workflows and information flows across all levels of the health system (a focus of Chapter 3). You may find it
helpful to document the structure of the health system and the types of workers and their roles at each level of
the system. Resources like organizational diagrams (organograms) and health workforce operational guidance
may exist that describe the health programme’s management, human-resource structure and expected roles
and responsibilities.

During this step, try to describe the following: You may find it helpful to create a diagram using
lines and arrows to illustrate the levels of the health
» the different tiers of the health system, including
the community, district and provincial levels system, facility types and cadres, their relationships to
one another and their collective responsibilities (see
» the types of health provider and management
Fig. 2.3.1).
workforce cadres associated with the programme
area and their relationships with the levels of the
health system
» linkages across different health programme areas,
such as immunizations and antenatal care
» the names of the health facility types and the
health workforce cadres associated with providing
care within the health programme area.

Form the team and establish goals 23

24

Fig. 2.3.1. Illustrative example of a health system organogram and relationships.

Ministry of Health & Family Welfare (MoHFW)

Immunization and Vaccine Director General of Family

Development (IVD) Planning (DGFP)

Assistant Director, Deputy Director, Assistant Director,

Monthly Report National Manager Family Planning Family Planning Clinical Contraception
for EPI
MIS Form-4

Proof of
Delivery
Form Upazilla Family
National Regional Planning Officer Medical Officer, Medical Officer,
Logistics Monthly Immunization and Clinical Contraception Clinic
Manager Report Vaccine Officer DVDMT
and Supply Monthly
Requisition (RIVO) MIS Form-2
Form Report
and Supply
Requisition Medical Officer,
Form ClMCH-Family Planning
Proof of Delivery Form
Assistant
MIS Form -3
Upazilla Family
Health Care Center Reporting
Planning Officer Form-1
UH&FWC Maternal
healthcare Progress Report
Digital implementation investment guide

District Senior
IVD/EPI Immunization Tally Sheets Family Welfare
Village Monthly Report and Vaccine (F201 & F202) Family Visitor
Community
Executive Officer Report Officer (DIVO) Planning
Inspector Sub Assistant Family Welfare
MIS Form-1 Community Visitor (FWV)
Medical Officer

Facility
Community Register Rural Urban
Health Worker Clinic Nurse Clinic Nurse
Family Welfare Assistant Pharmacist Medical Officer,
Health & Family
Welfare Center

Immunization Family planning National level District level Facility level Community level
programme programme
Once you have mapped the general health system, » challenges routinely faced that negatively affect
identify the range of end-users and beneficiaries of the the persona’s responsibilities and health outcomes,
specific health programme. These individuals may be or what affects job satisfaction
the same ones identified in the organogram – health » vision of success from the persona’s perspective, or
workers, community-level supervisors, clinical staff what would make it easier to perform the job well.
and district-level managers – and they may include
For clients, include the same characteristics as in
individuals from NGOs and other sectors of government.
provider personas, plus the following:
To help understand specific end-users and beneficiaries
» barriers to accessing appropriate services and
of the health programme, you should create personas, completing recommended treatments (including
which are generic aggregate descriptions of the different financial, information, geographic and cultural
types of people involved in or benefiting from the obstacles that may prevent use)
health programme. Personas help stakeholders view » vision of successful care from the client’s
the objectives and programmatic challenges from the perspective, or what would improve satisfaction
vantage point of the people who deliver or receive health with health services that the client has received.
services. Personas also help align stakeholders around
shared definitions and perceptions. Finally, they provide As an example, Table 2.3.2 describes key personas that
a common point of reference on who delivers health commonly engage with or benefit from childhood
services, monitors or supervises services and, ultimately, immunization programmes (for more information,
receives services. see Chapter 4). When you design the digital health
implementation, you will directly involve the identified
For the purposes of this Guide, consider creating personas and incorporate their feedback into the design,
personas for managers, health workers and clients in the particularly if they are intended end-users of digital
health programme. For managers and health workers, health interventions. Once you have identified and
list personal and professional aspirations and challenges described each persona, detail their user stories and
for each persona, as well as familiarity with and use of routine interactions within each health programme and
technology. Include the following characteristics: between personas. Chapter 4 describes the process of
» responsibilities within the health programme area developing user stories in greater detail.
» any dependencies (actions or individuals) required See Annex 2.2 for a worksheet to use when developing
to trigger essential activities personas.

Form the team and establish goals 25

Table 2.3.2. Examples of personas developed when planning an immunization programme.

MANAGER PERSONA: Dr Regina Flowers, Regional Immunization Officer

Reports and communicates district-level lessons to policy-makers at the national level.
RESPONSIBILITIES
Oversees funding for and supply of vaccines.
Age: 50+
DEMOGRAPHICS Education: MD
Fluent in English. Uses computers and smartphones regularly.
Vaccine use and supply data are frequently missing or erroneous. Data reports are delayed and
CHALLENGES
difficult to synthesize. Field staff are not well trained.
WHAT WOULD
Adequate funding. High-quality and regular data. Engaged and responsive policy-makers.
SUCCESS LOOK LIKE

MANAGER PERSONA: Mr Paul Dacosta, District Immunization Officer

RESPONSIBILITIES Provides vaccines and supervision to clinics. Supervises 30 clinics. Records vital statistics
Age: 50+
DEMOGRAPHICS Education: MPH
Fluent in English. Owns a smartphone. Can use computers.
Unable to supervise all the clinics due to time constraints. Data not reported regularly by the
CHALLENGES
clinics.
WHAT WOULD Regular contact with the clinics with time to troubleshoot. High-quality and regular reports
SUCCESS LOOK LIKE from the clinics.

PROVIDER PERSONA: Ms Cissy Dialo, Clinic Nurse

RESPONSIBILITIES Administers vaccines in the clinic or during home visits.
Age: 25+
DEMOGRAPHICS Educated
Owns a basic personal phone.
CHALLENGES Clients often miss or delay immunization. Vaccines frequently out of stock.
WHAT WOULD
A system to manage all the clients to ensure that they receive vaccines on time.
SUCCESS LOOK LIKE

CLIENT PERSONA: Mrs Marisa Mukumba and Jenny, Client Mother and Child
RESPONSIBILITIES Mother takes the child for immunizations.
Child’s father has a mobile phone.
DEMOGRAPHICS
Mother is literate.
CHALLENGES Forgets when the next immunization is due. Local clinic does not regularly have supplies.
WHAT WOULD Good access, on time, to quality care for the child. Going to the facility when health workers are
SUCCESS LOOK LIKE there and needed vaccines are in stock.

Source: Adapted from Product vision for the BID Initiative, 2014 (49).

26 Digital implementation investment guide

 ˆ Members of your team (Output 1.1)
ˆ Stakeholders to engage throughout the planning and implementation
You should process (Output 1.2)
now have an ˆ Key outcomes from the national digital health strategy (if one exists) that
understanding you intend to build on
of the following: ˆ Priority goals (Outputs 2.1, 2.2), organogram (Output 3.1) and personas
(Outputs 3.1, 1.3) within the health programme(s) you are addressing.

This information can be collected in the worksheet in Annex 2.1.

Form the team and establish goals 27

28 Digital implementation investment guide
 CHAPTER

03 IDENTIFY HEALTH SYSTEM

CHALLENGES AND NEEDS

In the previous chapter, you aligned a team around the primary goals of the
health programme and identified key national strategies (health programme and
digital health) to guide the planning process. In this chapter, you will pinpoint
specific health programme processes and articulate the bottlenecks that you
seek to improve, which will set the stage for selecting appropriate digital health
interventions.

+ Process matrix (Annex 3.1)

+ Root causes of common bottlenecks (Fig. 3.2.1)
TOOLS + Bottleneck-mapping worksheet (Annex 3.2)

+ Examine current processes and workflows for the health programme area.
+ Identify and prioritize bottlenecks, also known as pain points, within the health
programme area.
OBJECTIVES + Link bottlenecks to standardized health system challenges to determine actionable
areas for improvement.

+ Named team and list of stakeholders engaged throughout the planning and
implementation process
INPUTS + Shared vision of health programme goals
+ Personas within the health programme

+ Current-state (“status quo”) workflow diagrams illustrating the user journey of selected
health programme processes (Output 3.4)
OUTPUTS + Prioritized bottlenecks (Output 3.2) mapped to list of health system challenges to be
addressed (Output 3.3)

29
 PRINCIPLES FOR
DIGITAL DEVELOPMENT

+ Engage with your target end-users and consult existing research to develop an
understanding of the people, networks, cultures, politics, data needs, infrastructure
and markets that make up your ecosystem before designing your initiative or tool.
+ Coordinate with other implementing organizations, civil society and the
UNDERSTAND THE government early on to learn from successful and unsuccessful initiatives in the
EXISTING ECOSYSTEM 4 ecosystem, to avoid duplicating efforts and to integrate with existing technical
systems more easily.

+ Incorporate multiple user types and stakeholders in each phase of the project
life cycle to direct feature needs and revise the design. Here, users are people who
will interact directly with the tool or system, and stakeholders are people who will
be affected by or have an interest in the tool or system, such as people whose data
are being collected, government officials or researchers who may study the data
collected.
+ Design tools that improve users’ current processes, saving time, using fewer
resources and improving quality.
+ Develop a context-appropriate digital implementation informed by end-users’
priorities and needs, considering the ecosystem and accepting that some digital
DESIGN WITH approaches will not be appropriate.
THE USER 5 + Develop the digital enterprise in an incremental and iterative manner, with clear
objectives and purpose in mind.
+ Ensure that the design is sensitive to and considers the needs of the historically
underserved.
+ Embrace an iterative process that allows for incorporating feedback and adapting
your implementation after initial testing and launch.
+ Be open about setting expectations and let people opt out of participating in the
design process.

4 Text adapted from Principles of Digital Development: Understand the existing ecosystem: Core tenets (7).
5 Text adapted from Principles of Digital Development: Design with the user: Core tenets (7).

30 Digital implementation investment guide

In this chapter, you will pinpoint specific health programme processes and articulate the bottlenecks that you
seek to improve, which will set the stage for selecting appropriate digital health interventions (see Fig. 3.1).
This process builds on the Public Health Informatics Institute’s Collaborative Requirements Development
Methodology (CRDM), a commonly used approach for defining the problem, identifying how it could be
improved, and describing how the improved process would need to function (50).

Fig. 3.1. Adaptation of CRDM approach for defining health system challenges.

The CRDM approach is broken down into three areas of concentration:

BUSINESS PROCESS BUSINESS PROCESS REQUIREMENTS

ANALYSIS REDESIGN DEFINITION

Think Rethink Describe

How do we do How should we do How can an information system
our work now? our work now? support our work?

+ Define goals and + Examine tasks and + Define specific tasks to be

objectives. workflow. performed for optimized
+ Determine the + Identify business processes.
health programme inefficiencies. + Describe the
processes to target. + Restructure tasks and implementation of business
+ Map the user workflow. rules.
journey/workflow + Select one or more + Determine whether the
within each of these digital health enabling environment can
processes. interventions support the selected digital
+ Identify pain to address the health interventions.
points within each prioritized health + Define what the digital
workflow. system challenges. health interventions should
do based on end-user
needs and stakeholder
expectations.
+ Determine if existing ICT
systems and digital health
projects can be leveraged to
support your digital health
interventions.

3.1 Map the current state of programme activities

First, you will need to understand how the health programme currently functions, or the workflow of the
programme. This requires thinking through the tasks that are performed to meet the goals and priorities of the
health programme articulated in Chapter 2. In other words, you will analyse the various processes within the
health programme and identify the bottlenecks that prevent optimal delivery of those services, resulting in a
diagram of the current state.

The process for mapping the current state has three steps.

1. Determine the health programme processes to target.

2. Map the user journey (illustrated through a workflow) within each of these processes.
3. Identify bottlenecks within each workflow.

Identify health system challenges and needs 31

 3.1.1 DETERMINE THE HEALTH PROGRAMME PROCESSES TO TARGET

Health programme processes, also called business process and the expected outcomes. Describe tasks in
processes, consist of a set of activities or tasks as much detail as possible, and include every step of the
performed together to achieve the objectives of the process.
health programme area or health system (51). Processes
The process matrix in Fig. 3.1.1.1 illustrates health
involve different personas and may cross multiple levels
programme processes for an immunization information
of the health system. For example, within the area of
system. In this example, the Expanded Programme on
antenatal care, health programme processes can include
Immunization (EPI) constitutes the health programme,
activities associated with identifying pregnant women,
with the goal of providing universal immunization for
generating demand for services, monitoring supplies and
all children (52). Within the EPI health programme,
commodities, managing and following up with clients,
there may be a variety of processes, such as stock
and reporting (see Fig. 3.1.1.1).
management and patient management for tracking
Start by identifying all of the priority processes for your vaccination history. These processes have a set of tasks
health programme. For each of the identified processes, that one of the personas needs to carry out, such as
list the objectives, the inputs needed, the expected registering clients, ordering stock and recording stock
outputs, the specific sets of tasks that make up the levels. See Annex 3.1 for a process matrix worksheet.

Fig. 3.1.1.1. Process matrix illustrating three example processes within a typical
vaccination programme.

# PROCESS OBJECTIVE TASK SET OUTCOMES

Patient Maintain a database of » Register patient More complete registration
A management all newborns, with their » Search for existing record of newborns
vaccination history. » Maintain patient database

Vaccination Ensure that all infants are » Define national schedule More timely vaccination,
management vaccinated with all vaccine » Plan vaccinations higher vaccination coverage
doses in the national » Send reminders
B
schedule. » Register vaccinations
» Monitor vaccination
coverage

Stock Ensure that vaccines and » Order stock Higher availability and
management other stock are always » Receive lower wastage of vaccine
available when needed, » Store and other stock
C while minimizing wastage » Count stock
and excess stock. » Monitor balances,
expiry dates, wastage
and usage

Service delivery Ensure that the provider is » Provide counselling Provider is following correct
providing quality services » Diagnose protocol and vaccination
(providing vaccinations) » Dispense (provide schedule, more timely
to clients by having the vaccination) vaccinations, increased
necessary vaccination history » Refer coverage, improved quality of
D
and list of which vaccinations » Update record with which care, greater documentation
are needed. vaccinations have been of vaccinations provided
given
» Schedule and inform
client of next visit

Source: Adapted from WHO/PATH Planning an information systems project, 2013 (2).

32 Digital implementation investment guide

 3.1. 2 MAP THE WORKFLOWS FOR TARGETED PROCESSES

Now that you have identified and described important Develop workflows for the different processes through
processes in the health programme, you can further discussions with the people who provide services within
examine the user journey to understand where to make the health programme. Try to get multiple perspectives of
improvements. Describing these processes as they occur, how work is actually performed, rather than how health
with as much input from programme implementers with system managers may think (or hope) the work is done.
on-the-ground experience as possible, is vital to capture This should also be complemented by mapping the range
events as they typically happen, rather than how they are of processes and interventions delivered during a given
officially supposed to happen or imagined to happen in interaction. By doing so, you can ensure that you avoid
theory. Detailing tasks, or the specific activities within a designing around a single health need but missing other
health programme process, will uncover opportunities to interactions that the health provider may have at the
improve the overall process. same time with the same patient.

Workflows, or task flow diagrams, are one way of Ultimately, any separate workflows that are part of
illustrating the user journey. Workflow diagrams are a process should tie together when the analysis is
visual representations of the progression of activities complete. As stakeholders review activities within the
(tasks, events and interactions) performed within a different processes, they can reflect on the challenges
health programme process. These diagrams help visualize that prevent achieving the outlined activities. When
specific activities within the process and illustrate the designing an intervention, you could use the workflow
interactions between the personas who perform those diagrams to explain how the health programme works,
activities (see Fig. 3.1.2.1). The result of one task generally the interrelations between people and places and the
triggers another task, until the final process objective issues to address in order to improve performance. You
is reached. All tasks associated with the process being may also want to review common workflow diagrams
mapped should appear at least once on the workflow documented in WHO digital accelerator kits (22) relevant
diagram. These diagrams also map how information to the programme area(s), which can offer a starting point
moves through the system and can be used to identify for discussion and comparison with your own workflow
and illustrate where bottlenecks occur. (See Box 3.1.2.2 systems.
for a description of symbols generally used in workflow
diagrams.)

Identify health system challenges and needs 33

Fig. 3.1.2.1. Example workflow diagram for a service-delivery process.
SERVICE DELIVERY POINT

Patient

1.
Start Encounter

2. 3. 4. 6. 7. 8.
YES
Health worker

Provide Diagnose Treatment Dispense Update Record Inform Next Visit End
Counselling Available?
NO

5.

Refer

Legend

Task Sub-tasks Decision Start End

Source: Adapted from WHO/PATH Planning an information systems project, 2013 (2)
.

Box 3.1.2.2. Conventions that are generally used when mapping workflows.
» Tasks are represented as boxes.
» Diamonds represent decision points.
» Boxes with double lines represent bundles of numerous subtasks.
» Circles represent start and end points of the workflow.

For more information on standard notation for workflow diagrams, see the CRDM website (50)
and WHO’s Optimizing person-centric record systems: a handbook for digitalizing primary
health care (53).

34 Digital implementation investment guide

 BID INITIATIVE
CASE STUDY
BID Initiative: Business processes
It was hard for Oliver Mlemeta, the sister in charge at the Usa River Health Facility in Tanzania, to know how
many children to expect on immunization day. To figure out which children were due, she would spend hours
sifting through dense immunization registries and tallying numbers. Then she had to make sure she had
adequate supplies on hand. If her vaccine stock was low, she would spend more time calling other clinics and
retrieving what she needed by motorbike. Sometimes, she would have to turn mothers and children away
because there were not enough supplies.

When her facility’s immunization day arrived, Oliver and her team would vaccinate hundreds of children.
Afterwards, more days of reporting awaited the team; they often worked nights and weekends to record
metrics into paper ledgers by hand. Once the data were sent to the district, it was just as difficult for district
staff to provide feedback that could help Oliver improve services. She rarely knew how she and her facility
were performing because the flow of data was often unidirectional.

The BID Initiative was designed to

make Oliver’s work easier, to help her
do her job better and to reach more
• Vaccine Information Management System
children with lifesaving vaccines. Its • RITA Birth Registration
• National ID
vision was to empower countries to • DHIS2
• Health Facility Registry
enhance immunization and overall
health service delivery by improving
data collection, quality and use. In
doing so, the BID Initiative worked
closely with the governments of
Tanzania and Zambia to identify
the most critical challenges to
immunization service delivery,
defining 16 business processes and
associated tasks related to operating
a national immunization information
system in Africa, including client
registration, vaccine administration
and report generation. These processes
were developed through ongoing
consultation with key country
stakeholders and can be found in the
Product vision for the BID Initiative (49).

Identify health system challenges and needs 35

 3.1.3 IDENTIFY AND CONFIRM BOTTLENECKS

As you create the workflows, challenges – or bottlenecks stakeholders who are involved in performing the work,
– should emerge. These are areas where failures in as well as with clients attempting to access the health
service delivery occur, where health workers experience system. This validation should take place with the health
frustrations or even where patients may be lost to workers and clients who know what happens on a daily
follow-up. Bottlenecks are the specific gaps that basis and who can share rich insights into how these
prevent personas from reaching their goals of success activities work in practice, rather than with directors and
and achieving positive health outcomes. Bottlenecks supervisors. You could organize discussions with those
contribute to the suboptimal implementation of health at the frontlines who can additionally articulate their
programmes and are often causes of the failure to meet challenges in delivering health services, highlighting the
the programme’s goals. bottlenecks. Reviewing the workflow diagram with them
and explaining your goals and what you will do with
For example, in the workflow shown in Fig. 3.1.2.1, you
the information will improve accuracy in representing
may find that clients routinely do not show up for their
the workflows. This process of engaging with personas
expected first encounters (Task 1). Further discussion
will help document gaps between the current state and
may reveal that clients experience difficulties (such as
the desired future state of the health programme by
when articulating their needs) that prevent them from
identifying the following:
benefiting from a health worker’s consultation and
diagnosis (Task 2). Issues like inaccurate diagnoses or » inefficiencies or gaps
adherence to clinical protocols during consultations may » efficiencies that can be gained with repeatable
emerge as additional bottlenecks associated with the tasks
health worker, also occurring at Task 2. » redundancy in tasks, such as information collected
more than once
You could validate that the workflow diagrams are
accurate through observations and interviews with » blocks to the optimal flow from one task to the
next.

3.2 Conduct a root cause analysis of bottlenecks

A root cause analysis helps identify the factors underlying the bottlenecks occurring in health programme
processes. This analysis narrows the list of actionable bottlenecks and determines which issues can be
addressed with available resources.

Three types of root causes may emerge through this may not be because the vaccines are out of stock, but
process. because a supervisor has discouraged the vaccinator
from opening a multidose vial for a single child, resulting
1. PHYSICAL: tangible, material items failed in
in wastage. In this case, other types of nondigital
some way.
interventions to mitigate the recurring problem would
2. HUMAN: people did something wrong or did be more appropriate.
not do something required.
The 5 Whys method is an intuitive way to identify the
3. ORGANIZATIONAL: a system, process or policy
root cause of a bottleneck (see Fig. 3.2.1). This method
that people use to make decisions is faulty.
involves asking “Why does this problem exist?” five
The root cause analysis may also reveal situations where times, or until you get to the foundational roots of
a digital health intervention may not be warranted or the problem. It is important to include those who are
ideal. For example, the reason vaccines are not given experiencing the problem directly to be involved in
to every child who comes to an immunization camp determining the “Whys”.

36 Digital implementation investment guide

Fig. 3.2.1. Using the 5 Whys model to identify the root cause of bottlenecks.

PROBLEM: C LIENT DOES NOT RECEIVE THE NECESSARY MEDICATION AT THE FACILIT Y

WHY? The clinic ran out of medicine

WHY? Clinic logistics manager did not order enough medicine

WHY? Did not know how much demand there would be for the medicine

WHY?
Did not have accurate historical data on the amount of medicines
needed monthly

WHY? Aggregating the data from multiple paper forms takes time

Source: iSixSigma: Determine the root cause: 5 whys (54).

Find more examples and worksheets to support the 5 Whys process in the iSixSigma online library (54).

3.3 Prioritize bottlenecks

Now that you have a better understanding of their root causes, you can rank the bottlenecks, identifying those
that contribute the most to preventing success of the health programme. Although you may ideally like to
address every bottleneck, limits on resources often require focusing on the most important challenges or the
ones that you feel you have the ability to influence. Stakeholders should discuss and review the bottlenecks and
their root causes, comparing them until they agree on a ranking from most to least critical.

Factors that influence bottleneck rankings include Table 3.3.1 offers some considerations for ranking
whether the issue can actually be resolved, the capacity bottlenecks, which you may find useful in guiding
of stakeholders to drive this change and the potential discussion and reaching consensus among stakeholders.
impact of resolving the issue. Also take the perspectives The table describes three hypothetical examples of
of different personas into account when prioritizing bottlenecks, which receive scores of Low (1), Medium
bottlenecks. For example, in Task 2 of Fig. 3.1.2.1 (“Provide (2) or High (3) based on responses to three questions.
Counselling”) and Task 3 of Fig. 3.1.2.1 (“Diagnose”), the Adding the scores together gives the prioritized ranking.
most important bottleneck for the client may be not Alternatively, you may choose to list and prioritize the
having appropriate information on the type of services bottlenecks by debate.
she should be receiving, whereas the health worker
may feel most frustrated by not having the clinical tools
necessary to screen patients.

Identify health system challenges and needs 37

Table 3.3.1. Formula for scoring and ranking bottlenecks, with examples.

1.
How much 2.
impact What is the 3.
does this likelihood of Is this Prioritized
Bottleneck bottleneck overcoming important to a Score Ranking
have on the this wide range of
process? bottleneck? stakeholders?
(1–3) (1–3) (1–3)
Example
Aggregating data
from paper forms is Low (1) High potential (3) Yes (3) 1+3+3=7 HIGHEST
burdensome and rarely
done correctly.
Example
Mother’s belief that
Medium potential
newborns should not be Medium (2) Some (2) 2+2+2=6 MEDIUM
(2)
immunized if they seem
“small” in size.
Example
Budget is not available
to provide blood
High (3) Low potential (1) No (1) 3+1+1=5 LOWEST
pressure cuffs to all
clinics for hypertension
screening.

3.4 Map programme-specific bottlenecks to

generic health system challenges
At this point, focus on describing the highest priority bottlenecks using a common vocabulary, so you can link
them to possible interventions. WHO developed a classification for health system challenges that standardizes
the categories of common bottlenecks experienced at various levels of the health system (see Fig. 3.4.1).
This classification provides a consistent method for grouping the diverse ways that various participants,
from patients to health workers to decision-makers, have for expressing very granular, programme-specific
bottlenecks and their root causes. It also ensures that all stakeholders have a common understanding of the
challenges and a consistent language for articulating the need, as well as for mobilizing support and funding.

Note that digital health may not be the most suitable will ultimately design an approach that makes the most
approach for addressing the specific challenge. Health sense within your environment and that meets the
system challenges can and should be addressed in needs of your stakeholders and end-users. Chapter 4
different ways. Nondigital approaches, such as training, guides you through the steps of identifying digital health
supervision or paper tools, are often more appropriate interventions for the specific health system challenges
solutions, and the optimal solution for your context that you have prioritized.
may require a combination of approaches. Your team

38 Digital implementation investment guide

Fig. 3.4.1. WHO classification of health system challenges.

Health System Challenges

1 Information 3 Quality 6 Efficiency

1.1 Lack of population 3.1 Poor patient experience

6.1 Inadequate workflow
denominator management
Insufficient health
Delayed reporting 3.2 worker competence Lack of or inappropriate
1.2 of events 6.2 referrals
Low quality health
Lack of quality/ 3.3 commodities Poor planning and
1.3 reliable data 6.3 coordination
Low health worker
Communication 3.4 motivation 6.4 Delayed provision of care
1.4 roadblocks
Insufficient continuity Inadequate access to
Lack of access to 3.5 of care 6.5 transportation
1.5 information or data
Inadequate supportive
Insufficient utilization of 3.6 supervision
1.6 data and information
Poor adherence to
7 Cost
1.7 Lack of unique identifier 3.7 guidelines
High cost of manual
7.1 processes
2 Availability 4 Acceptability 7.2 Lack of effective resource
allocation
Insufficient supply Lack of alignment with 7.3 Client-side expenses
2.1 of commodities 4.1 local norms
Lack of coordinated payer
Insufficient supply Programs which do not 7.4 mechanism
2.2 of services 4.2 address individual beliefs
and practices
Insufficient supply
2.3 of equipment 8 Accountability
2.4 Insufficient supply of
qualified health workers
5 Utilization
Insufficient patient
8.1 engagement
5.1 Low demand for services
Unaware of service
5.2 Geographic inaccessibility 8.2 entitlement
Low adherence to Absence of community
5.3 treatments 8.3 feedback mechanisms
5.4 Loss to follow up Lack of transparency in
8.4 commodity transactions
Poor accountability
8.5 between the levels of
the health sector
Inadequate understanding
8.6 of beneficiary populations

Identify health system challenges and needs 39

 Box 3.4.2. Linking health system challenges to universal health coverage.
Although you have gone through an extensive process to identify the key bottlenecks and health system
challenges, you may have overlooked some areas. You may want to take the further step of mapping your
bottlenecks to the different contributors to achieving universal health coverage (UHC). The goals of UHC aim to
ensure access to all individuals who need services and that these services are delivered with the intended quality
and do not cause financial hardship (55).

The Tanahashi framework is a common approach to articulating the set of health system challenges that need to
be overcome in efforts to achieve UHC for specific health interventions within target populations (see Fig. 3.4.3).
Reviewing the different sections of the framework, such as supply and demand, can also highlight the range
of different health system challenges that should be addressed. For example, it may be difficult to overcome
the challenge of loss to follow-up of clients (continuous coverage) if there is low demand for services (contact
coverage) and clients are not coming to facilities in the first place. The updated framework used here includes the
following layers (56):
» ACCOUNTABILITY COVERAGE: the proportion of those in the target population known and registered in
the health system
» ACCESSIBILITY AND AVAILABILITY OF SERVICES: includes ensuring availability of commodities,
equipment and human resources and accessibility to health facilities
» CONTACT COVERAGE: proportion of clients who have contact with relevant facilities, health workers and
services among the target population
» CONTINUOUS COVERAGE: the extent to which clients receive the full course of intervention required to
be effective
» EFFECTIVE COVERAGE: the proportion of individuals receiving satisfactory health services among the
target population
» FINANCIAL COVERAGE: the proportion of patients protected from impoverishment due to health-related
costs.

Annex 3.2 provides a template for mapping bottlenecks to health system challenges and examples of bottlenecks
linked to health-system-challenge classifications by UHC layer.

40 Digital implementation investment guide

 Fig. 3.4.3. Health system needs for universal health coverage.

Health system challenges

limiting UHC for target population

Financial coverage
affordability The proportion of patients protected
from impoverishment due to health-
related costs

Potential
Effective coverage for
Quality The proportion of individuals receiving digital
satisfactory health services among the health
target population interventions

Continuous coverage
The extent to which clients receive the full
course of intervention required to be effective

Health System Goal

Demand
Contact coverage
Proportion of clients who have contact with
relevant facilities, providers and services among
the target population

Availability of commodities and equipment

Ensuring availability of commodities and equipment

Supply
Availability of human resources
Ensuring availability of human resources

Accessibility of health facilities

Ensuring access to health facilities

Accountability coverage
Accountability The proportion of those in the target population registered into
the health system

Target Population

Source: Mehl & Labrique, 2014 (56).

ˆ Analysed the health programme processes to be improved

At this point in the (Annex 3.1)
process, you have ˆ Mapped the current state (“status quo”) of the user journey and
accomplished the workflows within the health program processes (Output 3.4)
following tasks: ˆ Prioritized bottlenecks (Output 3.2) and health system challenges
(Output 3.3) to be addressed, detailed in template (Annex 3.2).

Identify health system challenges and needs 41

42 Digital implementation investment guide
 CHAPTER

04 DETERMINE APPROPRIATE
DIGITAL HEALTH INTERVENTIONS

In the previous chapter, you examined different health programme processes and
identified key health system challenges. In this chapter, you will rethink the way
these tasks are performed and reflect on interventions to address the identified
bottlenecks.

+ Bottleneck-mapping worksheet (Annex 3.2)

+ WHO Classification of digital health interventions (4)
+ Digital health intervention selection matrix (Fig. 4.1.3)
TOOLS + Digital Health Atlas (8) or digital landscape review
+ Global Digital Health Index (9) or other digital maturity assessment

+ Identify digital health interventions aligned to the health system challenges prioritized
in Chapter 3.
+ Define the requirements for each proposed digital health intervention.
+ Determine whether the capability for each proposed digital health intervention exists
OBJECTIVES and whether it can be used within the existing digital health enterprise or if new
investment is required to achieve this capability.
+ Understand the state of maturity for digital health in the country and the readiness of
the enabling environment.

+ Analysis of the targeted health programme processes

+ Current-state (“status quo”) workflow diagrams of the user journey with the health
INPUTS programme processes
+ Prioritized bottlenecks and health system challenges to be addressed

+ List of prioritized digital health interventions (Output 4.1)

+ Future-state user journey/workflow diagrams (Output 4.4)
+ Enabling-environment assessment (Output 4.2)
OUTPUTS + Functional and nonfunctional requirements (Output 4.3)
+ Landscape analysis to determine whether or how the existing digital health enterprise
can be leveraged (Output 4.5)

43
 PRINCIPLES FOR
DIGITAL DEVELOPMENT

+ Define and communicate what being open means for your initiative.
+ Adopt and expand on existing open standards, such as Health Level 7 Fast
Healthcare Interoperability Resource (HL7 FHIR): specifications developed by,
agreed to, adopted by and maintained by a community that enable sharing of data
across digital applications and the digital health platform.
+ Share nonsensitive data after ensuring that data privacy needs are addressed; to
encourage open innovation by any group or sector, do not place restrictions on data
USE OPEN
use.
STANDARDS,
OPEN DATA,
+ Use existing open source and open standards–based software where appropriate
OPEN SOURCE, AND
to help automate data sharing, connect your tool or system with others and add
OPEN INNOVATION 6
flexibility to adapt to future needs.
+ Develop any new software code to be open source, which anyone can view, copy,
modify and share, and distribute the code in public repositories.
+ Enable innovation by sharing freely without restrictions, collaborating widely and
co-creating tools when it makes sense in your context.

+ Identify the existing technology tools (local and global), data and frameworks
being used by your target population, in your geography or in your sector. Evaluate
how these could be reused, modified or extended for use in your program.
+ Develop modular, interoperable approaches instead of those that stand alone or are
attempting to be all-encompassing in their features. Interoperability will ensure that
you can adopt and build on components from others and that others can adopt and
build on your tool in the future; and swap out systems when improved – standards-
REUSE AND
based –solutions become available.
IMPROVE 7
+ Collaborate with other digital development practitioners through technical working
groups, communities of practice and other knowledge-sharing events to become
aware of existing tools and to build relationships that could lead to the future reuse
and improvement of your tool.

7 Text adapted from Principles for Digital Development: Use open standards, open data, open source, and open innovation: Core tenets (7).

44 Digital implementation investment guide

In the previous chapter, you examined different health programme processes and identified key health system
challenges. In this chapter, you will rethink the way these tasks are performed and reflect on interventions to
address the identified bottlenecks (see Fig. 4.1). This is called business process redesign. It is at this stage where
you will critically assess whether and how digital health applications may be used to alleviate the identified
bottlenecks. Then, based on your decision, you can begin to describe what digital health applications must do to
effectively support the optimized health programme processes.

Fig. 4.1. Redesigning health programme processes.

The CRDM approach is broken down into three areas of concentration:

Chapter 4

BUSINESS PROCESS BUSINESS PROCESS REQUIREMENTS

ANALYSIS REDESIGN DEFINITION

Think Rethink Describe

How do we do How should we do How can an information system
our work now? our work now? support our work?

+ Define goals and + Examine tasks and + Define specific tasks to be

objectives. workflow. performed for optimized
+ Determine the + Identify business processes.
health programme inefficiencies. + Describe the implementation
processes to target. + Restructure tasks and of business rules.
+ Map the user workflow. + Determine whether the
journey/workflow + Select one or more enabling environment can
within each of these digital health support the selected digital
processes. interventions health interventions.
+ Identify pain to address the
prioritized health
+ Define what the digital
points within each health interventions should
workflow. system challenges. do based on end-user
needs and stakeholder
expectations.
+ Determine if existing ICT
systems and digital health
projects can be leveraged to
support your digital health
interventions.

A digital health intervention encompasses the When identifying potential digital health interventions,
functionality needed to alleviate the bottleneck and consider the ability of the health system to absorb and
improve functioning of the health system. The digital sustain the planned interventions. Additionally, assess
health intervention may be implemented using a whether existing applications that offer the functionality
set of software applications and hardware, which needed for the digital health interventions may already
when combined are referred to as the digital health exist in your country that could be reused or adapted
application. The interventions within a digital health for your implementation. Digital health applications are
application can be as simple as a weekly mobile phone “the software and ICT systems that deliver or execute
call to bridge the long distances separating patients from the digital health intervention and health content” (14).
health workers, or as complex as helping health workers Digital health enterprises include one or more digital
manage clinical records that are connected to laboratory, health applications, but also comprise the hardware,
logistics, reporting and human resource management standards, people, processes, policies, governance and
systems. underlying information infrastructure that support the
operations of a health programme or health system (14).

Determine appropriate digital health interventions 45

Identifying and selecting digital health interventions incorporating the interventions.
includes the following steps. 4. Determine if existing digital health applications
1. Select one or more digital health interventions within the enterprise can be leveraged to
to address the prioritized health system support your digital health interventions. This
challenges (if digital is appropriate). If digital will help you understand how your proposed
is appropriate, select interventions that have implementation can integrate with or use
demonstrated effectiveness, and determine the functionality of existing digital health
how these interventions can address your applications and shared services within the
health system challenges. enterprise.

2. Determine whether the enabling environment Table 4.1.4 provides a worksheet to help you think
can support the selected digital health through how to address the identified bottlenecks from
interventions. This includes understanding the perspectives of different personas (developed in
the ecosystem and absorptive capacity of the Chapter 2) as you work through these steps. For each
environment in which the interventions will be combination of persona and bottleneck, describe
implemented to ensure their feasibility. the specific information or functionality needed and
3. Define what the digital health interventions possible measures of success, once the intervention is
should do based on end-user needs and used to overcome the bottleneck. Then you can identify
stakeholder expectations. This includes appropriate digital health interventions, together with
determining what the future state of the the enabling infrastructure and capacity needed to
workflow or user journey will be after implement them.

4.1 Determine and select digital health interventions

for the prioritized health system challenges
Over the past two decades, a variety of digital health approaches have been tested as ways of alleviating health
system challenges that have not otherwise been adequately addressed. From health-promotion messages sent
to clients to applications that track stock levels, digital health interventions have been used individually (siloed)
or combined with shared services using data exchange standards to form robust and extensible digital health
enterprises.

You could reflect on the root causes of each health been documented to support health system needs and
system challenge to understand how a particular address challenges (see Fig. 4.1.2). Each digital health
intervention may overcome or mitigate it (see Chapter 3). intervention included in the Classification represents a
Involve potential end-users (clients, health workers, discrete unit of technology functionality to address a
supervisors and so on) at this stage as they may be health programme need or overcome a health system
instrumental in understanding whether and how digital challenge. Furthermore, the Classification provides a
health interventions can help address the identified standardized vocabulary that public health practitioners
issues. You may also find it appropriate to address a and software vendors can understand when expressing
health system challenge by combining digital and how the digital health intervention should function.
nondigital approaches. Your review of this document should facilitate your
understanding of the opportunities that may exist when
To begin selecting digital health interventions,
building a digital health enterprise and how the digital
first review the WHO Classification of digital health
health interventions will address identified health
interventions shown in Fig. 4.1.1 (4). This classification
system challenges.
system presents the diverse ways that technology has

46 Digital implementation investment guide

Fig. 4.1.1. Classifications of digital health interventions.

1.0 2.0
Clients Healthcare Providers
1.1 Targeted client 1.3 Client to client On-demand Client Healthcare 2.8 Healthcare
communication communication 1.6 information 2.1 identification and 2.5 provider provider training
services to clients registration communication
Transmit health event 1.3.1 Peer group for clients Provide training content to
1.1.1 alerts to specific 2.8.1 healthcare provider(s)
Client look-up of health Verify client Communication from
population group(s) 1.6.1 information 2.1.1 unique identity 2.5.1 healthcare provider(s) to
Personal health 2.8.2 Assess capacity of
Transmit targeted health 1.4 tracking Client financial 2.1.2 Enrol client for health
supervisor
healthcare provider(s)
1.1.2 information to client(s)
based on health status or 1.7 transactions
services/clinical care plan
2.5.2
Communication and
performance feedback to
Prescription
demographics Access by client to own
Transmit targeted alerts
1.4.1 medical records
Transmit or manage out 2.2 Client health
records
healthcare provider(s)
Transmit routine news and
2.9 and medication
1.1.3 and reminders to client(s) Self monitoring of health 1.7.1 of pocket payments by management
1.4.2 2.5.3 workflow notifications to
or diagnostic data by client client(s) Longitudinal tracking healthcare provider(s)
Transmit diagnostics Transmit or track
Transmit or manage 2.2.1 of clients’ health status 2.9.1 prescription orders
1.1.4 result, or availability of 1.4.3 Active data capture/ Transmit non-routine
result, to client(s) documentation by client 1.7.2 vouchers to client(s) for and services 2.5.4 health event alerts to
health services healthcare provider(s) 2.9.2 Track client's medication
2.2.2 Manage client’s structured
clinical records
consumption
Untargeted client Citizen based Transmit or manage
1.2 communication 1.5 reporting 1.7.3 incentives to client(s) for Manage client’s
2.5.5 Peer group for healthcare
providers
2.9.3 Report adverse drug events
health services
2.2.3 unstructured
Transmit untargeted Reporting of health system clinical records Referral Laboratory and
1.2.1 health information to an 1.5.1 feedback by clients
Routine health indicator
2.6 coordination 2.10 Diagnostics
undefined population
Reporting of public health 2.2.4 data collection and Imaging
Transmit untargeted 1.5.2 events by clients management Coordinate emergency Manangement
2.6.1 response and transport
1.2.2 health event alerts to
undefined group
Healthcare Manage referrals between 2.10.1 Transmit diagnostic result
to healthcare provider
2.3 provider decision 2.6.2 points of service within
support health sector 2.10.2 Transmit and track
diagnostic orders
Provide prompts and 2.6.3 Manage referrals between
2.3.1 alerts based according
health and other sectors 2.10.3 Capture diagnostic results
from digital devices
to protocol
Health worker 2.10.4 Track biological specimens
2.3.2 Provide checklist
according to protocol 2.7 activity planning
and scheduling
Screen clients by risk or
2.3.3 other health status Identify client(s) in need
2.7.1 of services
2.4 Telemedicine 2.7.2 Schedule healthcare
provider's activities
Consultations between
2.4.1 remote client and
healthcare provider
Remote monitoring of
2.4.2 client health or diagnostic
data by provider

2.4.3 Transmission of medical

data to healthcare provider
Consultations for case
2.4.4 management between
healthcare provider(s)

3.0 4.0
Health System Managers Data Services
3.1 Human resource 3.3 Public health 3.6 Equipment and Data collection, 4.2 Data 4.3 Location
management event notification asset management 4.1 management, coding mapping
and use
List health workforce Notification of public Monitor status of Parse unstructured data Map location of health
3.1.1 cadres and related 3.3.1 health events from 3.6.1 health equipment 4.2.1 into structured data 4.3.1 facilities/structures
Non-routine data
identification information point of diagnosis 4.1.1 collection and
Track regulation and Merge, de-duplicate, and
Monitor performance of 3.6.2 licensing of medical
management
4.2.2 curate coded datasets or 4.3.2 Map location of
health events
3.1.2 healthcare provider(s)
3.4 Civil Registration equipment
4.1.2 Data storage and terminologies
Manage certification/ and Vital Statistic aggregation 4.3.3 Map location of

3.1.3 4.2.3 Classify disease codes or clients and households

registration of healthcare
3.7 Facility 4.1.3 Data synthesis and cause of mortality
provider(s) 3.4.1 Notify birth event management visualization 4.3.4 Map location of
healthcare providers
Record training credentials 3.4.2 Register birth event Automated analysis of
3.1.4 of healthcare provider(s) List health facilities and
3.7.1 data to generate new
4.1.4 Data exchange and
3.4.3 Certify birth event
related information information or predictions
on future events
4.4 interoperability
Supply chain 3.7.2 Assess health facilities
3.2 management 3.4.4 Notify death event
Data exchange
4.4.1 across systems
3.4.5 Register death event
Manage inventory and
3.2.1 distribution of health 3.4.6 Certify death event
commodities

3.2.2 Notify stock levels of

Health
health commodities
3.5 financing
3.2.3 Monitor cold-chain
sensitive commodities
Register and verify client
3.5.1 insurance membership
3.2.4 Register licensed drugs
and health commodities
3.5.2 Track insurance billing and
claims submission
3.2.5 Manage procurement
of commodities
Track and manage
Report counterfeit or 3.5.3 insurance reimbursement
3.2.6 substandard drugs
by clients Transmit routine payroll
3.5.4 payment to healthcare
provider(s)
Transmit or manage
3.5.5 incentives to healthcare
provider(s)

3.5.6 Manage budget and

expenditures

Source: WHO Classification of digital health interventions, 2018 (4).

Determine appropriate digital health interventions 47

Fig. 4.1.2. Linkage between health system challenges and digital health interventions.

HEALTH SYSTEM DIGITAL HEALTH INTERVENTION (DHI)

CHALLENGE (HSC) Digital functionality for addressing
Need or problem to be addressed the Health System Challenge

Manage inventory and distribution

Insufficient 3.2.1 of health commodities
supply of
commodities 3.2.2 Notify stock levels of health commodities

Provide prompts and alerts based

2.3.1 according to protocol
Healthcare provider’s 2.3.2 Provide checklist according to protocol
poor adherence to
clinical guidelines 2.5.2 Communication and performance
feedback to healthcare provider(s)

2.7.2 Schedule healthcare provider’s activities

Lack of access 2.2.4 Routine health indicator data collection

and management
to information
or data 4.1.2 Data storage and aggregation

4.1.3 Data synthesis and visualizations

Loss to Transmit targeted alerts and reminders

1.1.3 to client(s)
follow-up
of clients 2.2.1 Longitudinal tracking of clients’ health
status and services

When determining which digital health interventions Fig. 4.1.3 illustrates the links between programmatic
are appropriate, consider reviewing the WHO Guideline: bottlenecks and health system challenges and suggests
Recommendations on Digital Interventions for Health appropriate digital health interventions to address the
System Strengthening (1), which lists evidence-based challenges, based on recommendations from the WHO
digital health interventions and considerations for guideline. Recommended digital health interventions
implementation. Should WHO recommendations on are based on extensive reviews of scientific evidence
digital health interventions not yet exist for the health demonstrating the interventions’ value in addressing
system challenges you have identified, you could draw specific health system challenges. Annex 5.3 provides
from research evidence and regional experiences to a list of common health system challenges associated
determine whether to use alternative or existing digital with digital health interventions featured in the WHO
health interventions in your context or whether to guideline.
consider a nondigital approach more carefully (see
Box 1.1.3). Keep in mind that WHO does not recommend
investments in digital health interventions that do not
have a robust evidence base.

48 Digital implementation investment guide

Fig. 4.1.3. Digital health intervention selection matrix.

ILLUSTRATIVE Bottleneck described as RELEVANT DIGITAL HEALTH CONTRIBUTION

BOTTLENECKS HEALTH SYSTEM INTERVENTIONS (DHI) TO UNIVERSAL
articulated in CHALLENGE (HSC) DHI included DHI that are not included HEALTH COVERAGE
programme area with an evidence in WHO guideline but (UHC)
base from WHO provide appropriate
guideline functionality
Decision Communication
support
2.3 for health 2.5.2 and performance
feedback to health
workers workers
“Health workers do
not counsel clients Digital
appropriately.”
5.3 Poor adherence to
guidelines
provision
of training
“Health workers are 2.8 content
unsure of which/when to health
services are due.” workers/
mLearning

Effective
coverage
Decision Coordination
2.3 support 2.6 of emergency
for health transportation
workers
Lack of referrals/ Transmission or
“Clients are not referred Provider- management
on time.” 6.2 inappropriate to-provider 1.7.2 of vouchers to
referrals telemedicine client(s) for health
2.4 consultations services
for case
management

“Clients do not
continue taking their
medication.” Targeted client Personal health
“Clients do not 5.1 Low adherence to 1.1 communication,
such as 1.4.1 tracking/access
by client to own
Continuous
complete the treatments coverage
reminders health record
recommended number
of visits/treatments.”

“Services are too far for

clients.” Client-to- On-demand
2.4.1 provider information
“Roads are inaccessible 5.2 Geographic telemedicine 1.6.1 services: client Contact
during rainy season [or inaccessibility lookup of health coverage
other period].” information

Source: Adapted from the WHO Classification of digital health interventions, 2018 (4).

Consider the range of digital health interventions that evidence-based recommendations and any country
may be appropriate for your identified health system experiences will help you identify a set of digital health
challenges. Cross-checking these against the WHO interventions suitable for the expressed needs.

Determine appropriate digital health interventions 49

 CASE STUDY

Determining appropriateness of digital health interventions

for health system challenges
As part of the joint process with the Ministry of Health and Sanitation in Sierra Leone for identifying
health system challenges and assessing the appropriateness of digital health interventions, WHO
convened stakeholders to review challenges faced within the health system and across different
programmatic areas. This was a consultative process to identify areas of improvement and assess
whether digital health interventions may play a role in addressing some of the issues that were raised.
Table 4.1.4 gives some examples of bottlenecks raised during this multistakeholder assembly, as well as
discussions about the role of digital health interventions in addressing the root causes of these health
system challenges.

Table 4.1.4. Bottlenecks discussed in a multistakeholder meeting.

BOTTLENECK
BOTTLENECK DISCUSSIONS ON THE ROLE OF
ROOT CAUSES EXPRESSED AS A
ARTICULATED DIGITAL HEALTH INTERVENTIONS
LINKED TO STANDARDIZED
THROUGH IN ADDRESSING ROOT CAUSES OF
BOTTLENECK HEALTH SYSTEM
DISCUSSIONS HEALTH SYSTEM CHALLENGES
CHALLENGE
Inequitable High attrition rate of HSC 2.4. Digital health interventions such as
distribution of trained midwives from Insufficient supply listing health workforce cadres could be
trained midwives hard-to-reach areas of qualified health used to track the distribution of deployed
Overall lack of trained workers health workers (DHI 3.1.1).
midwives nationally Provider-to-provider telemedicine could
assist with enabling less-skilled health
workers to consult with midwives and
other relevant health professionals
(DHI 2.4.4).
However, a digital health intervention
may not be well suited for addressing
the root cause of overall lack of trained
midwives nationally.
Poor functioning Inadequate HSC 5.2. Digital health interventions are not well
or lack of medical maintenance of Geographic suited to address the root cause of this
infrastructure, equipment inaccessibility bottleneck.
including power, Infrastructural
blood services and constraints
water
Frequent stockouts Limited prediction of HSC 2.1. Although digital health interventions
of drugs and stockouts for ordering Insufficient supply may not alleviate the constraints
supplies required drugs in a of commodities associated with inadequate vehicles or
timely manner, both availability of commodities at the central
from the central to level, a digital health intervention such
the district level and as notification of stock levels could be
from the district to the used to communicate when there are
primary healthcare unit stockouts and request replenishments in
Poor monitoring of drug a timely manner (DHI 3.2.2).
availability
Inadequate logistics
(such as vehicles or
fuel) for delivering
commodities

50 Digital implementation investment guide

4.2 Determine whether the enabling
environment can support the identified
digital health interventions
Digital health interventions are delivered through digital health applications, which ideally are linked to a
supportive digital health platform comprising shared services and enabling components (in cases where
there are multiple applications); together with the people, processes and policies that support and use them
to deliver health services to clients, these applications and platform make up the digital health enterprise.
Successful deployment of digital health applications requires a thorough knowledge of the ecosystem where
the interventions will be deployed and whether they can be supported in that environment. Understanding this
context can inform the feasibility of implementing the digital health enterprise, as well as demonstrate where
system integrations will be required.

For example, in settings with limited infrastructure and A comprehensive digital health strategy establishes the
governance structures, it may be prudent to opt for less vision for how digital health approaches will support
complex digital health implementations until these a national health system and provides the operational
building-block enabling factors evolve to a more mature details necessary to achieve this vision. The digital
state. Regardless, each subsequent investment in health investment roadmap provides an overview of the
digital health should contribute cumulative value to the national vision and financial implications for stepwise
functioning of the digital health enterprise, addressing investment in foundational and health programme–
health needs within the health programme and across specific digital health applications and shared services.
the health system. Investments in ball-of-mud health The results of one or more digital-maturity-assessment
software characterized by an evolving agglomeration of outputs provide a practical assessment of the progress
functions, originating without a predetermined scope of establishing different critical enabling components
or design pattern, which are monolithic contribute to an of governance and environment represented in the
accumulation of technical debt and are not advised (see digital health strategy, the HIS, interoperability of the
Fig. 1.3.1). digital health enterprise or the digital health application
implementation scale (see Box 4.2.2 for resources).
The national digital health strategy, investment
A national inventory of digital health assets, such as
roadmap, country assessment of digital maturity,
the Digital Health Atlas (8), provides an overview of
national inventory of digital assets and enterprise
the knowledge, experience, targeted health focus,
architecture documentation, if available, should serve
digital characteristics and maturity of specific digital
as a starting point for understanding the priorities and
applications and shared services within the country
state of the national digital health ecosystem, and hence
(see Box 4.4.1 for more information). The national vision
the feasibility of selected digital health interventions
for an enterprise architecture provides documentation
and the context for integrating the prioritized
of logical organizational and business processes of
interventions into the national system (see Fig. 1.1.4).
the national health system and its supporting data,
The digital health strategy outlines a country’s vision
applications, shared services and digital infrastructure,
as it relates to the enabling environment (see Fig. 4.2.1),
with clearly defined goals and objectives for achieving
such as legal, regulatory and policy frameworks and
future health goals.
ICT workforce needs, as well as the ICT environment,
including infrastructure and foundational architecture.

Determine appropriate digital health interventions 51

Fig. 4.2.1. Roles and contributions of component “building blocks” to the enabling environment for
digital health enterprises.

+ Direct and coordinate eHealth at the national level; ensure

alignment with health goals and political support; promote
LEADERSHIP, awareness and engage stakeholders.
GOVERNANCE AND + Use mechanisms, expertise, coordination and partnerships to
MULTISECTOR develop or adopt eHealth components (e.g. standards).
ENGAGEMENT + Support and empower required change, implementation
of recommendations and monitoring results for delivery of
expected benefits.

+ Ensure a responsive strategy and plan for the national

eHealth environment. Lead planning, with involvement of
STRATEGY AND
major stakeholders and sectors.
INVESTMENT
+ Align financing with priorities; donor, government and
private-sector funding identified for medium term.

ENABLING
+ Adopt national policies and legislation in priority areas;
review sectoral policies for alignment and comprehensiveness;
ENVIRONMENT LEGISLATION, establish regular policy reviews.
POLICY AND
COMPLIANCE + Create a legal and enforcement environment to establish trust
and protection for consumers and industry in eHealth practice
and systems.

+ Make eHealth knowledge and skills available through internal

expertise, technical cooperation or the private sector.
WORKFORCE
+ Build national, regional and specialized networks for eHealth
implementation.
+ Establish eHealth education and training programmes for
health workforce capacity-building.

STANDARDS AND
+ Introduce standards that enable consistent and accurate
collection and exchange of health information across health
INTEROPERABILITY
systems and services.

+ Form the foundations for electronic information exchange

across geographical and health-sector boundaries. This
INFRASTRUCTURE includes the physical infrastructure (e.g. networks), core
services and applications that underpin a national eHealth
environment.

ICT + Provide tangible means for enabling services and systems

ENVIRONMENT that deliver health content and; access to, and exchange and
SERVICES AND
management of information and content. Users include the
APPLICATIONS
general public, patients, providers, insurance, and others. The
means may be supplied by government or commercially.

Source: WHO/ITU National eHealth strategy toolkit, 2012 (3).

52 Digital implementation investment guide

Depending on your timeline, you may want to develop include ability to work offline, multiple language
short-term plans that take into account the current settings, and password protection (57). However, many
enabling environment, while contemplating medium- of these factors usually fall outside the health sector,
and long-term plans that benefit from your deliberate making it difficult to address gaps within the proposed
new investments to catalyse some components of intervention’s life span or budget. Other components
the environment to mature further. A comprehensive of the enabling environment may require lengthy
analysis of the enabling environment will also help processes, such as developing a skilled workforce or
you document the nonfunctional requirements, adopting national information standards, and therefore
which describe general attributes and features to also fall outside the time frame of the proposed
ensure usability and overcome technical and physical intervention (3). (See Box 4.2.2 for a list of resources to
constraints. Examples of nonfunctional requirement help assess the enabling environment.)

Box 4.2.2. Useful resources for assessing the readiness of the enabling environment.
» National eHealth strategy toolkit (3) is an expert, practical guide that provides governments, their ministries
and stakeholders with a solid foundation and method for developing and implementing a national eHealth
vision, action plan and monitoring framework.
» Global Digital Health Index (9) is a resource for quantitatively assessing the maturity of the enabling
environment; using indicators based on the National eHealth strategy toolkit (3), countries can track their
progress towards fulfilling the building blocks of an enabling environment.
» HIS stages of continuous improvement toolkit (11) supports assessment, planning and prioritizing
interventions and investments to strengthen an HIS, measuring current and desired HIS status across five
core domains and mapping a path towards improvement.
» HIS interoperability maturity toolkit (58) supports identification of the key domains for interoperability and
the required levels of maturity to achieve HIS interoperability goals.
» Assessing the enabling environment for establishing a contextualized national digital health strategy (59)
provides a systematic, structured approach to assessing the enabling environment for digital health based
on the development of Nigeria’s National Health ICT Strategic Framework.
» Information and communication technologies for women’s and children’s health: a planning workbook (60)
outlines a systematic approach to determining countries’ absorptive capacity for digital health interventions
and offers considerations for a variety of factors related to the enabling environment, spanning from policy
and infrastructure to sociocultural issues, and ways of understanding and mitigating potential risks.

Additionally, seek the experiences of global communities of practice to learn about how other MOHs and
implementers are developing infrastructure, policies and strategies and using digital health interventions to
address different health system challenges.

Determine appropriate digital health interventions 53

4.3 Determine what the interventions will need to do
Once you have selected which types of digital health interventions will appropriately address your health
system challenges and their suitability for your context, you can start defining the requirements of each
intervention and the digital health application that will be used to execute the intervention.

The requirements should not be just digitizing the Answer these three questions to determine the
processes identified in the current-state workflows requirements (see Fig. 4.3.1).
developed in Chapter 3; also think through ways that
1. What do members of the planning team
introducing efficiencies can optimize the performance of
(stakeholders) expect the digital health
the health system and programme area.
interventions to do for them?
2. What do end-users expect the digital health
interventions to do for them?
3. What do beneficiaries expect the digital health
interventions to do for them?

Fig. 4.3.1. Stakeholder relationships with a digital health intervention.

Planning The planning team includes stakeholders who are

Team responsible for guiding the development of the
digital health system

End-users End-users are health workers who interact

directly with the digital health implementation
once implemented. Clients, or patients, could also
be users if they engage directly with the digital
health system

Beneficiaries Beneficiaries are clients or members of the

community who may benefit from the use
of the digital health system

4.3.1 IDENTIFY FUNCTIONAL REQUIREMENTS AND USER STORIES

Functional requirements describe what the digital health The functional requirements consist of simple
application needs to do to address the health system statements that summarize what the end-user needs
challenges identified in Chapter 3. These requirements the digital health intervention to do (see Table 4.3.1.1
answer the question, “What does the intervention need for an example). Start with a brainstorming session
to do to help overcome a health system challenge?” to generate all possible scenarios that your team
Software developers use the functional requirements can imagine in a logical sequence along the process
as a reference to ensure that the intervention meets the workflow developed in Chapter 3. Consider all end-users
needs of the targeted end-users. who will access the intervention directly or who will
need to access information provided by the intervention
to make decisions.

54 Digital implementation investment guide

Table 4.3.1.1. An example scenario for a pregnancy-support digital health intervention.

END-USER REQUIRED FUNCTIONALITY REASON

As a I need a list of my clients whose So that I can plan my day accordingly, including adequate
community nurse antenatal visits are due today supplies to conduct antenatal care checkups.
As a I need to look up a client’s past So I can evaluate whether the client is in a high-risk
community nurse health record category and needs additional advice.
As a I need to be able to send my client So that she can remember to come to the clinic for a
community nurse an SMS reminder follow-up ultrasound in the third trimester.

You may find it helpful at this time to prioritize the most versus nice to have may also depend on the connectivity
important functions (“must have”) and identify others requirements, such as how urgently the information
that may be optional to develop, if time and budget is needed (for example, the information needs to be
permit (“nice to have”). This distinction of must have available in real time).

4.3. 2 UNDERSTAND AND MANAGE EXPECTATIONS FROM

END-USERS AND STAKEHOLDERS

It is also essential to understand how the digital health » other stakeholders, those with a keen interest in
implementation will work for everyone involved the success of the digital health implementation
and the health programme, such as members of
and how to improve the usability and value of the
the planning team.
applications for end-users. Some functions may not
be part of a end-user’s workflow or direct experience Understanding the perceptions, roles and
but are relevant to the stakeholders. A senior policy- responsibilities, as well as the motivations, of the people
maker may need to see aggregate performance data who will interact with or be affected by the digital health
characterized by geographic region, for example, intervention ensures that the intervention responds
but a health worker working in close contact with to these human needs. Make the lists of these actors
communities may not have immediate use for this as broad or as close to the health programme as your
macrolevel information. Refer to the current-state team feels is necessary. Briefly describe each kind of
workflow diagrams, which you created in Chapter 3, to stakeholder or end-user, and assess their potential
determine who performs what tasks within a health responses to identified digital health interventions
system and their roles as beneficiaries, end-users or (see Fig. 4.3.2.1 for an example). Once this is done, the
stakeholders. design team may choose to modify the requirements
Consider three distinct groups when describing what the to mitigate potential risks and ensure that the digital
intervention needs to do (the functional requirements): health implementation meets broader needs.

» end-users, the people who will actually use the

intervention
» beneficiaries, individuals or groups whose health
the intervention is intended to improve

Determine appropriate digital health interventions 55

Fig. 4.3.2.1. Examples of different personas interacting with digital health interventions.

POTENTIAL RESPONSE TO DIGITAL HEALTH INTERVENTIONS

GROUP DESCRIPTION
(POSITIVE AND NEGATIVE)

Stakeholder Community May be excited to have an innovative project in the community. Will
leader want to understand the information given to end-users. Will want to be
involved in or leading all planning meetings.

Stakeholder District May need information from the digital health intervention at an aggregate
programme geographic level to manage programme resources.
manager

End-user Clinic nurse May be apprehensive about switching to a new way of working,
especially if older in age and accustomed to the traditional ways that
service is provided. Will need more information on how the digital health
intervention will impact their work, including job security and relationship
with supervisor.

End-user Pregnant May be excited to be involved in this new initiative but may also worry
woman about what her husband will think. She knows she will also have to ask
her older children or neighbour’s kids to help her figure out what she is
expected to do with the phone.

4.3.3 MAP FUTURE-STATE WORKFLOW

Once you have gathered the different requirements diagrams because you may need to plan for additional
and assessed their implications on end-users and locations where data will be managed or used for
stakeholders, you can begin to conceptualize the decision-making.
future-state workflow. In Chapter 3, you diagrammed
To create the future-state workflow diagrams, follow
the current state of the health programme. Now
these steps.
you could develop new versions of these diagrams,
workflow diagrams of the future state, to illustrate the 1. Review the current-state workflow diagrams
processes in the desired system where the digital health and the end-user profiles. Considering these
intervention overcomes the bottlenecks (see Fig. 4.3.3.1). factors will ensure that the intervention is
To do this, reimagine the health programme optimized realistic.
with a digital health implementation in place. Indicate
2. Redraw the workflow diagrams from Chapter 3
the digital health moments, those points in the process to include proposed digital and nondigital
where digital health interventions address bottlenecks changes that together optimize to address
to improve the current state. The number of future-state health system challenges.
diagrams may differ from the number of current-state

56 Digital implementation investment guide

Fig. 4.3.3.1. Bottleneck and future-state workflow diagrams.

CURRENT STATE PATIENT MANAGEMENT

! Mother forgets No unique identification

Client

when an system - Clerk takes hours

1. immunization to find the client’s records
is due using the paper system
HEALTH FACILITY

Start Arrive at
facility

! ! ! Bottleneck where digital health

intervention would add value
2. 3. 4. 6.
YES
Gather client Search Match Check in End
Clerk

details for client found? client

NO
No unique identification
system - Clerk is required
to gather several identifying 5.
factors e.g. First name, family
name, birthdate, etc. Create client
record

FUTURE STATE PATIENT MANAGEMENT

Mother is given a
Client

reminder of child’s
1. 2. immunization due date
1.1.3
three days ahead of time.
HEALTH FACILITY

Start Client Arrive at

receives SMS facility
reminder DIGITAL HEALTH
MOMENTS
3. 4. 5. 7. Transmit targeted
YES 1.1.3 alerts and reminders to
client(s)
Gather client Search Match Check in End
Clerk

details for client found? client

2.1.1
NO 2.1.1 Verify client unique
identity
SEARCH USING UNIQUE ID

Clerk now only requires a 6.

few minutes to find the
client’s records.
Create client
record

4.4 Determine if existing digital health

applications, platforms and enterprises
can achieve the requirements
A digital health intervention operates through a digital health application, which when connected to other
shared technologies (such as data-exchange enabling components and registries in the platform), together
with the supported business processes, policies that govern them and people that use them, make up a digital
health enterprise. The best scenario is that your digital health investment leverages or extends the digital health
enterprise that is already in place in your context. Building on existing investments, such as by expanding the
functionality of existing applications or the connected platforms (like shared services) or incorporating new
health content into the tools that health workers already use, can limit the fragmentation of the digital health
enterprise and support the sustainability of your intervention.

Determine appropriate digital health interventions 57

 BID INITIATIVE
CASE STUDY

BID Initiative: Data quality and use interventions

Together with end-users from all levels of the health system, the BID Initiative collaboratively developed
a comprehensive set of tools and approaches to improve immunization data collection, quality and use
spanning data-management policies and practices, information system products and training on the tools,
as well as methods for better using data to make critical decisions about immunization services. These
improvements included the following:
» an electronic immunization application (EIR) linked to a client registry (Tanzania only) and supply chain
information
» automated, simplified report generation
» data-use campaigns
» microtraining videos
» peer-support networks
» barcodes or QR codes on child health cards and vaccine supplies
» targeted supportive supervision for health workers
» data visualizations and dashboards to monitor facility and neighbouring facility performance.

The BID team applied a “top-down” approach with a “bottom-up” view when drafting these interventions.
They started by considering Tanzania’s and Zambia’s national strategies, incorporating the current context
of the end-users (such as the functional architecture) before considering the facility applications (such as
the technical architecture) that were in use. With key stakeholders, they also assessed existing information
systems and how they matched the defined requirements, along with the projects and pilots that had come
before, so as not to duplicate efforts or invest in technologies and strategies that do not yield a high impact.
Adapted from The BID Initiative Story: Implementing solutions (61).

To increase your awareness of this ecosystem and » digital health interventions included in the digital
identify opportunities for improving interoperability and health implementation, system infrastructure and
contributing to a shared and sustainable digital health other environmental requirements
enterprise, conduct a landscape analysis and inventory of » key data supplied by each individual digital health
existing digital health applications, enabling components, implementation
shared services and enterprises used in your country. » availability of end-user and technical
The team conducting the landscape analysis should documentation
gather the following information for each digital health » extent that the application(s) can exchange data
implementation: within the broader digital health enterprise
» number of end-users, categorized by cadre » standards being used to define the data structure,
exchange and storage.
» group or individual responsible for maintenance
» levels of the health system or departments in a You may wish to leverage the Digital Health Atlas (8) to
health facility affected by the intervention review existing digital health inventories or conduct one
» software programs in use, including version in your region or country (see Box 4.4.1).
numbers, licences, recurrent costs and operating
systems

58 Digital implementation investment guide

 Box 4.4.1. The Digital Health Atlas and other repositories of digital health
implementations.

Identifying existing digital health interventions to use for your planned work can be challenging. WHO’s
Digital Health Atlas (DHA) can help you conduct a digital health inventory (8). The DHA is a web-based tool for
cataloguing and tracking digital health implementations. Searching the DHA for implementations filtered by
health domain, health system challenge area, software applications or context may reveal opportunities for reuse
and collaboration.

You may also consider tapping into existing communities of practices, such as the Global Digital Health Network
(62) or a regional network like the AeHIN (63). In some cases, a recent landscape report highlighting other digital
health interventions and ICT systems in the country may be available. These repositories may include current
as well as historical deployments that have ended for reasons other than funding cessation; insights from such
deployments may help you smartly plan for and avoid pitfalls that others have faced. Examples of such landscape
analysis reports include the following:
» Accessing the enabling environment for ICTs for health in Nigeria: a landscape and inventory (64)
» Bangladesh eHealth inventory report (65)
» mHealth in Malawi: landscape analysis (66).

In addition to assessing the landscape of digital that is (frequently) Free and Open Source Software
health applications in your context, you may also (FOSS), is supported by a strong community, has a clear
consider leveraging software applications identified governance structure, is funded by multiple sources,
in The Global Goods Guidebook (15). This guidebook has been deployed at significant scale, is used across
compiles mature digital health applications that use multiple countries, has demonstrated effectiveness,
open standards, are supported by a robust developer is designed to be interoperable and is an emergent
community, have demonstrated effectiveness and can standard application” (67). For more information on the
be adapted to different countries and use cases (15). A criteria used to determine and assess the maturity of
mature digital health software global good is “software software global goods, visit the Digital Square wiki (67).

Determine appropriate digital health interventions 59

Your investment in digital health implementations may deployment timelines. Additionally, for interventions
offer opportunities to contribute new functionality to focused on health workers, you could determine which
software global goods that may offer benefits beyond existing ICT tools (hardware and applications) various
your programme area, and similarly, your implementation cadres have experience with to maximize appropriateness
may benefit greatly from linking to or using existing of hardware specifications prior to implementation of the
functionalities, which can reduce costs and accelerate digital health intervention(s).

4.5 Progress check

You have now determined the digital health interventions and identified digital health applications to
support your planned implementation.

ˆ Digital health interventions that are appropriate to address the

bottlenecks and health system challenges identified in Chapter 3
(Output 4.1)
ˆ Digital health interventions that can be supported in your local
At this point in the context (Output 4.2)
process, you have ˆ Expectations of what the digital health intervention should do, or the
an understanding of functional requirements (Output 4.3)
the following: ˆ Future-state workflow based on the selected interventions and
functional requirements (Output 4.4)
ˆ Assessment of available digital health applications and enterprises
that your chosen digital health interventions can integrate with or
leverage (Output 4.5).

60 Digital implementation investment guide

Fig. 4.5.1 highlights the major considerations discussed so interventions you have selected will work with currently
far when selecting digital health interventions and the deployed applications and fit within the larger digital
digital health applications and platform in which they health ecosystem. Chapter 5 will go into greater detail
will be implemented. As you review these factors, check on other considerations when implementing a digital
that the digital health interventions you have selected health enterprise that delivers the selected digital health
are still the most appropriate. Also consider how the interventions and related functional requirements.

Fig. 4.5.1. Considerations for designing a digital health enterprise to deploy selected
digital health interventions.

ˆ What are the health system challenges identified by the

Identified pain stakeholder group?
points and health ˆ Will digital health interventions be appropriate or are there non-
system challenges digital approaches that should also be considered?

ˆ Has the digital health intervention been shown to address your

Evidence-base and health system challenges?
experience with ˆ Will more than one be needed to address the health system
the implementing challenges?
the digital health ˆ Have the digital health interventions been tested or implemented
intervention(s) at the scale necessary?

ˆ Will the enabling and ICT environment be able to support the

Enabling selected digital health intervention?
Environment ˆ What precautions will you need to take in planning the digital
health implementations (also elaborated on in Chapter 5)?

ˆ How well does the enterprise meet the user needs and stakeholder
expectations?
Requirements ˆ Does the enterprise fit well within the existing culture, language and
workflow processes?

ˆ Are there other digital health interventions and ICT enterprises being
Linkages to other implemented in the targeted context?
ICT Enterprises ˆ Will you be able to leverage the additional enterprises?

Determine appropriate digital health interventions 61

62 Digital implementation investment guide
 CHAPTER

05 PLAN THE IMPLEMENTATION

The previous chapters helped you identify which digital health interventions
to implement and why. This chapter examines more closely how to plan for
implementing your prioritized interventions within a digital health enterprise,
recognizing the iterative nature of the implementation process.

+ Questions to ask technology vendors (Annex 5.1)

TOOLS + Key considerations for implementing a digital health intervention (Annex 5.3)

+ Review critical factors to consider when planning to implement a digital health

intervention.
OBJECTIVES + Develop a realistic implementation plan that is responsive to the enabling environment.

+ Future-state user journey/workflow diagram (Chapter 4)

+ Enabling-environment assessment (Chapter 4)
+ Functional and nonfunctional requirements (Chapter 4)
INPUTS + Landscape analysis results (Chapter 4)
+ Personas (Chapter 2)

+ Detailed implementation plan for selected digital health interventions (Output 5.1)
OUTPUTS + Documentation in the Implementation Summary Template (Annex 5.2)

63
 PRINCIPLES FOR
DIGITAL DEVELOPMENT

+ Plan for sustainability from the start.

+ Develop a definition of sustainability for your initiative.
+ Identify and implement a sustainable business model.
+ Use and invest in local ICT service providers.
BUILD FOR + Engage local governments and integrate national strategies into programming.
SUSTAINABILITY 8 + Collaborate instead of competing, and partner to identify the best approach with
the greatest impact.
+ Build a programme that can be adapted as end-user needs and the context change.

8 Text adapted from Principles for Digital Development: Build for sustainability: Core tenets (7).

64 Digital implementation investment guide

The previous chapters helped you identify which digital health interventions to implement and why. This
chapter examines more closely how to plan for implementing your prioritized interventions within a digital
health enterprise, recognizing the iterative nature of the implementation process.

Digital health implementations are broadly based on the Depending on the complexity of the prioritized digital
following critical components (see Fig. 5.1): health interventions and the maturity of the digital
health enterprise, the requirements for ensuring
1. appropriate and accurate health content and
effective implementations can vary considerably.
information, defined based on the health
However, for any digital health application to scale and
programme guidelines and related evidence-
become institutionalized within a health programme,
based practices, and data needs for the
it must align with the infrastructure, legislation and
programme or use case
policies, and country or implementation leadership
2. the digital health intervention itself, and governance. These components, or foundational
consisting of the discrete digital functionality building blocks of a digital health strategy, contextualize
being applied to achieve the health objectives the implementation of the digital health application
(the digital health interventions selected in and are critical for its viability and sustainability.
Chapter 4)
While most digital health implementations tend to
3. digital health applications, which represent focus extensively on the technological aspects like the
the software and communication channels software and hardware, effective partnerships and a
that facilitate the delivery of digital health well-trained workforce underpin the success of any
interventions combined with health content, implementation. Additionally, maximizing opportunities
and which may be supported by shared services for interoperability and linkages to broader systems
such as registries and an interoperability layer based on data standards will reduce fragmented siloed
4. foundational ICT and enabling environment digital health implementations and enhance the overall
(such as governance, infrastructure, legislation digital health ecosystem in the country. (Financial
and policies, workforce, and enterprise investments are also one of the building blocks and will
architecture, including services, applications, be discussed in more detail in Chapter 7.)
standards and interoperability) in which the
implementation is situated.

Plan the implementation 65

Fig. 5.1. Essential components of a digital health implementation.

Digital Health Digital Applications

Health Content Interventions Software and communication
Information that is aligned with
recommended health practices
or validated health content
+ A discrete function of digital
technology to achieve health
sector objectives
channels that facilitate delivery
of the digital interventions and
health content, supported by
shared services

LEADERSHIP & GOVERNANCE

LEGISLATION, WORKFORCE
STRATEGY & SERVICES & POLICY, &
INVESTMENT APPLICATIONS COMPLIANCE

STANDARDS &
INTEROPERABILITY

INFRASTRUCTURE

Foundational Layer: ICT and Enabling Environment

In this chapter, you will explore the different influence the success of your implementation. After
considerations and key questions that can help guide completing this chapter, you may want to reassess
your digital health implementation (see Table 5.2). These whether the digital health interventions you selected are
considerations are based on the seven foundational still feasible for your context based on your evaluation of
building blocks, as well as additional factors that may these considerations.

Table 5.2. Illustrative implementation considerations for digital health.

Factor Illustrative considerations

STRATEGY AND » Is there a digital health strategy or investment roadmap in place? How do your identified digital
INVESTMENT health interventions align with the national strategy and current and proposed digital health
investments in the country?
» What new investments are required to make it possible to ensure that your digital health
interventions are integrated into an existing or future digital health enterprise architecture?
INFRASTRUCTURE » What are the electricity conditions at the deployment sites?
» What kinds of connectivity and bandwidth are available at the deployment sites?
» If the planned intervention will use mobile technology, what types of devices are end-users
familiar with?
LEGISLATION, » Are there mechanisms for ensuring the privacy and security of information during
POLICY AND transmission?
COMPLIANCE » Are there relevant policies for unique IDs and identity management for digital health
implementations that involve clients/patients?
» Are there procedures for redundant data storage in case of primary data loss?
» Are there other policies, such as HIS policies, to which digital health implementations will have
to adhere?

66 Digital implementation investment guide

 Factor Illustrative considerations
LEADERSHIP AND » Is there a national digital health governance framework or action plan, such as technical
GOVERNANCE working groups?
» Are partnership terms and formal collaborations documented in a memorandum of
understanding?
» Do separate departments or divisions oversee ICT and digital health?
» Is there a group or committee that combines ICT staff and staff from public health vertical
programmes (such as malaria, tuberculosis, HIV/AIDS or maternal and child health) that you
should include?
WORKFORCE » Are training procedures in place for health workers to build capacity with digital health
applications?
» If health workers will use a digital application concurrently with paper-based systems, how will
the double work be managed?
» How are change management and transitioning to digital approaches being supported?
» Are there considerations and actionable policies to support health workers who may find their
roles redundant following the introduction of a particular digital implementation?
» Are plans for health workers’ privacy and safety in place regarding the use or exchange of data
with chosen digital health interventions?
SERVICES AND » Are there existing digital health applications that can support the selected digital health
APPLICATIONS interventions?
» Are there mechanisms for procuring hardware locally or leveraging existing hardware?
» Are there processes in place for maintaining the hardware and software? How will updates to
the software be pushed out to the end-users?
» What are the regulations and procedures for hosting and storing data?
» Does your implementation reference or link to shared services in the digital health platform,
such as identification registries, a national health workforce registry or the master facility list?
STANDARDS AND » Is a digital health enterprise architecture or blueprint in place? Does your digital health
INTEROPERABILITY intervention use data standards that are compatible with other systems in the country?
(See also Chapter 6.) » Are there reusable components, such as terminology services and data dictionaries, that you
could incorporate?
» Is there an interoperability framework to help guide how systems support one another?
» Are there maintained data exchange standards (such as HL7 FHIR) that will need to be
considered?
» Is there a national interoperability mediator (for data exchange) that applications will need to
leverage?
HEALTH CONTENT » What are the evidence-based clinical and/or public health guidelines from which your health
content will be derived?
» What are the processes that will be followed to ensure that algorithms, decision support,
checklists, messages, schedules and other operational components of the health content are in
line with evidence-based recommendations and best practices?
» What are the data and indicators that are critical to the functioning of the digital health
intervention and the derivative outputs, such as recommended indicators and performance
metrics?
» How can you leverage and contribute to an existing data dictionary or terminology shared
service?

Sources: Adapted from WHO/ITU National eHealth strategy toolkit, 2012 (3); The MAPS toolkit, 2015 (29).

Plan the implementation 67

 Box 5.3. Description of digital health interventions reviewed in the WHO guideline.
Annex 5.3 expands on additional implementation considerations for specific digital health interventions prioritized in
the WHO guideline (1). If you are considering one of these interventions, review Annex 5.3 for a more nuanced list of
factors that could affect your implementation.
Digital health Definition Synonyms and
intervention other descriptors
Birth notification Digital approaches to support the notification of births, Ⱥ Birth event alerts
to trigger the subsequent steps of birth registration and Ⱥ Enabling health workers and community to
certification, and to compile vital statistics  transmit alerts/notifications when a birth has
occurred
Death notification Digital approaches to support the notification of deaths, Ⱥ Death surveillance
to trigger the subsequent steps of death registration and Ⱥ Death event alert
certification, and to compile vital statistics, including cause- Ⱥ Enabling health workers and communities to
of-death information  transmit alerts/notifications when a death
has occurred
Stock notification Digital approaches for monitoring and reporting stock levels, Ⱥ Stock-out prevention and monitoring
and commodity and consumption and distribution of medical commodities. Ⱥ Alerts and notifications of stock levels
management This can include the use of communication systems (e.g. SMS) Ⱥ Restocking coordination
and data dashboards to manage and report on supply levels of
Ⱥ Logistics management and coordination
medical commodities 
Client-to-provider Provision of health services at a distance; delivery of health Ⱥ Consultations between remote client/patient
telemedicine services where clients/patients and health workers are and health worker
separated by distance  Ⱥ Clients/patients transmit medical data (e.g.
images, notes and videos) to health worker
Provider-to-provider Provision of health- services at a distance; delivery of health Ⱥ Consultations for case management between
telemedicine services where two or more health workers are separated by health workers
distance  Ⱥ Consulting with other health workers,
particularly specialists, for patient case
management and second opinion
Targeted client Transmission of customized health information for different Ⱥ Notifications and reminders for
communication audience segments (often based on health status or appointments, medication adherence, or
(targeted demographic categories). Targeted client communication may follow-up services
communication to include: Ⱥ Health education, behaviour change
individuals) communication, health promotion
i. transmission of health-event alerts to a specified
population group; communication based on a known client’s
ii. transmission of health information based on health status health status or clinical history
or demographics; Ⱥ Alerts for preventive services and wellness
iii. alerts and reminders to clients; Ⱥ Notification of health events to specific
iv. transmission of diagnostic results (or of the availability of populations based on demographic
results).  characteristics

Health worker Digitized job aids that combine an individual’s health Ⱥ Clinical decision support systems (CDSS)
decision support information with the health worker’s knowledge and clinical Ⱥ Job aid and assessment tools to support
protocols to assist health workers in making diagnosis and service delivery, may or may not be linked to
treatment decisions  a digital health record
Ⱥ Algorithms to support service delivery
according to care plans and protocol
Digital tracking of Digitized record used by health workers to capture and store Ⱥ Digital versions of paper-based registers for
patients’/ clients’ health information on clients/patients in order to follow-up specific health domains
health status and on their health status and services received. This may include Ⱥ Digitized registers for longitudinal health
services within digital service records, digital forms of paper-based registers programmes, including tracking of migrant
a health record populations’ benefits and health status
(digital tracking) for longitudinal health programmes and case management
logs within specific target populations, including migrant Ⱥ Case management logs within specific target
populations. populations, including migrant population
Provision of The management and provision of education and training Ⱥ mLearning, eLearning, virtual learning
Educational and
training content content in electronic form for health professionals. In contrast Ⱥ Educational videos, multimedia learning and
to health workers to decision support, health worker training does not need to access to clinical and non-clinical guidance
(mobile learning/ be used at the point of care.  for training reinforcement
mLearning)

68 Digital implementation investment guide

5.1 Infrastructure considerations
Many digital health applications and platforms have seen limited adoption and success because they
were piloted in areas that had inadequate access to electricity and mobile networks or were otherwise
inappropriately designed for the context. Understanding the available infrastructure is foundational to defining
the scope and feasibility of the digital health application. Several practical approaches help alleviate constraints
posed by infrastructural limitations. For example, erratic power supply and Internet connectivity may cause
challenges when uploading data to a central computer in real time and may even result in data loss. However, if
you know about connectivity issues during the planning stages, you could design the digital health application
to facilitate offline data collection, supported by manual transfer of data to a central computer where the data
are stored. Similarly, if irregular access to electricity keeps end-users from charging digital devices, you could
consider alternatives during planning, such as battery packs or solar chargers.

In the early stages of planning, consider the following DEVICES: What types of devices can be sourced locally?
questions to ensure that the infrastructure can support If the planned intervention will use mobile technology,
the implementation or to determine whether you need what types of devices do end-users currently have? How
to establish contingency plans. comfortable are they with using different functions on
their phones? Do they make voice calls only, or do they
ELECTRICITY: What are the electricity conditions at the
also use text messaging or interactive applications?
deployment sites? Is there a ready and stable supply of
electricity? Does this electricity supply vary by season DIGITAL LITERACY AND LANGUAGE: What is the
or during inclement weather? What alternatives exist to level of digital literacy (proficiency in operating digital
grid power, such as solar, wind or water turbines? Also devices) of the target population? If you plan to deploy
consider training end-users on power management of the intervention across regions, how will you account for
devices: turning devices off, shutting off Bluetooth and variations in digital literacy in accessing information over
so on. digital devices, as well as the range of languages that the
content would need to include?
CONNECTIVITY: What kind of connectivity is available
at the implementation sites? Do end-users have reliable
voice and text-messaging coverage? Do they have
stable low- or high-speed Internet coverage? If possible,
speak directly with end-users or mobile network Box 5.1.1. Resource on hardware
regulators about coverage and connectivity. Coverage management.
reports from mobile network operators (MNOs) often » BID Initiative Equipment support strategy (68)
overstate connectivity or present only a snapshot of the
coverage. If you plan to scale to regions with unreliable
connectivity, consider interventions and applications
that can work offline, and adequately plan for the
amount of data that may be needed to be stored offline
to ensure that the application still performs well with
large amounts of data waiting to be sent to server.

Plan the implementation 69

5.2 Legislation, policy and compliance considerations
It is important to understand the national policies and regulations that may apply and explore relevant global
best practices when national policies are lacking. These could include regulations for hosting data and using
personally identifiable information, processes for informed consent, relevant standards and linkages with other
systems. A successful digital health implementation plan will assess the current policy environment, adapt the
design to that environment and ensure that policies are sufficiently implemented. Evaluate these considerations
in conjunction with leadership and governance considerations (see section 5.3), as sound policy relies on leaders
and governance structures to ensure its effectiveness and accountability (69, 70, 71).

5. 2.1 DATA MANAGEMENT, PRIVACY AND SECURITY

Clear guidance and documentation for access to and » privacy protection for patients, caregivers and
sharing of data strengthen the security of the systems health workers
that process and store data (such as data warehouses). » security of protected information during
This can also help clarify who has access to what data, transmission
when and for what purpose and head off potential risks » devices that can access each server where
associated with inappropriate data use. information is held

Guidance facilitates necessary privacy protection for » access and use of data that donors and
implementing partners collect and store, as well as
sensitive data and mitigates against breaches that place
for research purposes
the health system and beneficiaries at risk. See Box 5.2.1.1
» guidance on data flows from one place to another
for examples of policies for protecting data, as well as
Annex 5.4 for an illustrative checklist to mitigate data » use of cloud-based services
management risks. » data ownership
Look for the following guidance or policies when » procedures for redundant data storage in case of
planning for data access and security needs: primary data loss
» other policies, such as HIS policies, to which
interventions will have to adhere.

70 Digital implementation investment guide

 Box 5.2.1.1. Examples of policies for data protection and regulation.
When defining the guidelines, consider principles of assessment, such as the UN Global Pulse’s Risk, harms and
benefits assessment tool (72). This resource serves as an internal tool to better understand data privacy, ethics
and data-protection compliance mechanisms associated with use of big data in development and humanitarian
contexts.

The General Data Protection Regulation developed by the European Union provides useful considerations for
managing data protection, privacy and security. These include client rights to1:
» be informed on how their data will be used
» access their data
» correct or rectify the collected data
» have their data deleted, or “be forgotten”, if the data were collected unlawfully or deemed no longer necessary
» restrict data processing or completely object to the processing of personal data for the purposes of
advertising or direct marketing
» transfer the data to another party without interference (73).

Guidance from CDC2 and UNAIDs3 on data security, privacy and confidentiality; and relevant standards curated by
ISO TC 215 on health informatics4 will provide additional operational and technical requirements.

The African Union Convention on Cyber Security and Personal Data Protection provides similar articles, including
these examples:
» Article 18 The right to object: Any person has the right to object, on legitimate grounds, to the processing
of the data relating to him/her. He/She shall have the right to be informed before personal data relating to
him/her are disclosed for the first to third parties or used on their behalf for the purposes of marketing, and
to be expressly offered the right to object, free of charge, to such disclosures or uses.
» Article 21 Security obligations: The data controller must take all appropriate precautions, according to the
nature of the data, and in particular to prevent such data from being altered or destroyed, or accessed by
unauthorized third parties. (74)
While establishing such policies in countries is critical, raising awareness of and enforcing these policies are
equally important.

1 https://ec.europa.eu/info/priorities/justice-and-fundamental-rights/data-protection/2018-reform-eu-data-protection-rules/eu-data-
protection-rules_en
2 https://www.cdc.gov/nchhstp/programintegration/docs/pcsidatasecurityguidelines.pdf
3 https://www.unaids.org/en/resources/documents/2019/confidenTality_security_tool_user_manual
4 https://www.iso.org/committee/54960/x/catalogue/p/1/u/0/w/0/d/0

5. 2.2 REGULATION OF NEW DIGITAL HEALTH TECHNOLOGIES

Many countries are developing national regulatory technologies for their member countries. Although
systems for health-related technology. Regional still emerging, international regulatory bodies are also
groups, such as the Digital Regional East African establishing measures for quality assurance and safety of
Community Health Initiative and the African Union’s “software as a medical device” (75), such as software that
New Partnership for Africa’s Development, encourage monitors blood pressure or diagnoses infections. These
regional-level regulation or qualification of new efforts at different levels can result in multiple layers of

Plan the implementation 71

regulation that you will need to examine and determine » How can you ensure that your implementations
how to balance. (For more information about regulations are aligned with guidelines governing the
of medical technologies, see Box 5.2.2.1.) connectivity of digital medical innovations, such as
connected diagnostics?
As technology continues to evolve, the procedures and » Is the current regulatory system fully financially
policy considerations for introducing new technologies resourced and staffed?
will also likely need to accelerate. The implementation
leadership teams may want to consider how new
technologies are integrated into the health system
and what the policy foundation is for regulating the Box 5.2.2.1. Resources on regulation
technologies. Consider the following questions when of medical device
technologies.
planning to meet regulatory requirements.
» International Medical Device Regulators
» How are new digital technologies approved for use,
forum (76)
procurement and marketing within the public and
private sectors of the health system? (Public and » WHO Medical devices: regulations (77)
private systems often differ in regulatory rigor.)

5.3 Leadership and governance considerations

Having a clear understanding of the governance and leadership structure will help you craft an inclusive
planning process and foster a sense of ownership of the implementation. Governance may include a steering
committee or a decision-making board composed of a range of stakeholders (see Chapter 2). Engaging the right
stakeholders from the beginning increases the possibility that they will incorporate future investments in the
implementation into their annual budgets and workplans (29, 69, 78, 79).

5.3.1 GOVERNANCE

Answering the following questions will help you committee that combines ICT staff and staff
facilitate active engagement with existing governance from public health vertical programmes (such as
malaria, tuberculosis, HIV/AIDS and maternal and
mechanisms.
child health) that you should include?
» Are meetings scheduled at regular intervals to » Does the Ministry of Education cover aspects of
coordinate across leadership and build consensus training health workers? How is the Ministry of ICT
on project directions and proposed changes? involved in supporting the programme? Should you
» Are partnership terms and formal collaborations include a different ministry or department that
documented in a memorandum of understanding? manages the governance of administrative units
(such as districts, provinces or regions), like the
» Do separate departments or divisions oversee
Ministry of Local Administration?
ICT, M&E and digital health? Is there a group or

72 Digital implementation investment guide

 5.3. 2 EXTERNAL PARTNERSHIPS

Implementing digital health interventions often opportunities to access a larger number of end-users and
requires working with a range of partners in addition reduce the cost of the intervention by bundling it with
to governmental organizations that are traditionally other MNO services. In turn, the MNO may value the
considered the public health sector. These partners enhancement of brand awareness and status that such a
can include NGOs, academic institutions, civil society partnership can provide.
organizations, donor organizations, and technology
Also determine how you may need to work with a
and software companies. Also consider local MNOs,
technology vendor and, if so, who that vendor should be.
telecommunications groups, the consumer products
The technology vendor as a partner may provide training
industry and pharmaceutical companies. Identify
and installation support during initial deployment of
the resources, competencies and capacity needed
the digital health implementation but have different
to complement implementation needs and improve
procedures, pricing and expectations in other phases.
the potential for success. You may also want to
The first step in hiring a technology vendor is to develop
consider aligning implementation of the digital health
a requirements document (based on the requirements
intervention with other large-scale health initiatives,
identified in Chapter 4), which serves as the basis for a
such as a national maternal health effort, in which
request for proposals (RFP) for soliciting competitive
technical inputs, training and supportive supervision can
bids. You circulate the RFP to potential vendors, who
be combined. Be sure to include technical partners and
then respond with associated costs and details of the
selected contractors as collaborators when developing
products and services they can provide to meet the
the implementation plan.
requirements (2). The RFP should also state the criteria
Consider the strategic advantages that each partner you will use to evaluate submitted proposals.
brings. For example, partnering with MNOs may provide

Plan the implementation 73

5.4 Workforce and training considerations
Chapter 2 listed the key governance, management and operations personnel needed for a digital health
implementation. At this point, you will focus on the workforce directly involved in the digital health enterprise,
such as community and facility health workers and healthcare managers. Although some health workers will
show great enthusiasm for new technologies, introducing and institutionalizing the intervention with all of the
workforce or client end-users can be challenging.

Establish training programmes for each cadre of services. Start simple and minimize the number of
health worker and for healthcare managers who are required fields, if possible. Maintain technical support for
involved with the digital implementation. Intensive end-users by levels. In the subsequent phases, you may
introductory training on how to use new applications, start working more intensively on change management
followed by regular refresher training, is vital. Explore and processes to accept a paperless future state.
models like training of trainers and engaging local NGOs
Involve end-users. Incorporate end-users’ cultural
when scaling up training in a systematic way. Be sure to
preferences, and plan to adapt the design for new
appropriately contextualize the model you choose for
contexts. Also plan for changing the intervention’s
regional variations. Several studies have also highlighted
content and interfaces over time. Design information
how important it is to train not just the health workers
structures, images and icons so they can be changed
who will directly interface with the devices, but
easily if you scale to a different context. Involve end-
also the managers who will provide oversight and
users to make sure that the data you collect will have
ensure accountability. Additionally, you may consider
a purpose, which helps ensure that collected data are
embedding metadata (system-generated data on how
actually used.
the digital health intervention is being used) to monitor
performance of health workers and determine where Consider demonstrating how the data health workers
you may need to intervene or provide additional end- collect will be used in the health programmes they
user support. implement. Recognize that many health workers and
their managers may not intuitively understand digital
Build in sufficient time to learn the new digital
health and the accompanying technology vocabulary.
health intervention, recover from errors and increase
Ensure that health workers understand how data are
comfort and speed in using the application within the
stored and used, including the value of the data to them
enterprise. Also assess if the digital health intervention
within their responsibilities. Consider making a regular
will introduce a significant change that may require
plan to share the data back with them, showing how the
more intense training of health workers (and therefore
data was used for decision-making at higher levels and
additional funding and training-of-trainers courses).
how the data can be used to improve their own work.
Start identifying champions at different levels to help
inspire and motivate others. Transitioning from paper- Communicate expectations and best practices for
based to digital systems is typically a phased process. managing passwords and personal health information,
In the initial phases, health workers will likely have to including why these are considered optimal behaviours.
enter data twice, once on paper and once digitally, which
can annoy workers and disrupt their ability to provide

74 Digital implementation investment guide

 BID INITIATIVE
CASE STUDY

BID Initiative: Workforce implementation

The BID Initiative developed a workforce implementation strategy (80) informed by John Kotter’s eight-step
model for change management (see Fig. 5.4.1). “Touches”, or visits, were used to introduce and provide training
on new tools and approaches, allowing health workers time to internalize the practice of data analysis and
use (see Fig. 5.4.2). Successful implementation requires buy-in from local health workers, so BID centred its
approach on health workers, both as trainers and trainees. Instead of having BID Initiative staff lead touches,
district immunization officers led activities within facilities and provided direct support to health workers.

Fig. 5.4.1. Kotter’s eight-step change model.

Implementing and
sustaining for change

Engaging and
enabling the 8
step

organization ANCHOR THE NEW APPROACHES IN THE CULTURE

step 7
CONSOLIDATE GAINS AND PRODUCE MORE CHANGE – DON’T LET UP

6
step
CREATE SHORT-TERM WINS

Creating
5
step
the climate
REMOVE OBSTACLES
for change

4
step
COMMUNICATE THE CHANGE VISION

3
step
CREATE A VISION FOR CHANGE

step2
BUILD A GUIDING TEAM

1
step
CREATE URGENCY

Plan the implementation 75

 BID INITIATIVE
CASE STUDY

Fig. 5.4.2. Linking Kotter’s change model to BID’s touch strategy.

ROUTINE BID KOTTER’S OBJECTIVES CHANGE MANAGEMENT

ACTIVITIES STEPS ACTIVITIES

Touch 1 1 and 2 Introduce the intervention and prepare Select a guiding team involving
for Touch 2. Urgency is defined and health facility in-charges with
advocated for. the district officials and regional/
provincial teams for support.

Touch 2 3 and 4 Introduce the electronic immunization Communicate the vision for data use
registry to health providers and and data quality interventions and how
train them on how to use it to collect that vision contributes to addressing
immunization data and use the data for challenges faced by the facility,
decision-making. through posters, messaging, etc.

Touch 3 5 and 6 Provide immediate follow-up with the Create short-term wins by sharing
health providers. During this touch, health success stories early in the process,
workers are also encouraged to use the such as the ability to use the
system and to build their confidence. systems to easily collate information
or prepare reports.

Touch 4 7 and 8 Emphasize the decision-making and data Health providers are supervised and
use culture. Ensure new ways behavior is continue to use the new system.
recognized and rewarded to embed the Health workers are also trained in
change in the organizational culture. several different data use scenarios
using the built-in decision-making
Institutionalize data use and data quality process. Future supervision is
interventions and embed them into handed over to the district.
existing public processes using, for
instance, supportive supervision.

Source: BID Initiative briefs: recommendations and lessons learned: change management (80).

The following are additional BID Initiative tools and resources for workforce implementation:
» Rollout strategy (81)
» Facility and district visit strategy (82)
» Spotting and addressing resistance to change (83)
» Change readiness assessment tool (84)
» Coaching/supportive supervision job aid (85).

76 Digital implementation investment guide

5.5 Services and applications
Services and applications are the devices and tools, including software, used to collect, transmit, access and
maintain health information, including the ICT systems that your digital health interventions will integrate
with and leverage (3). Services include digital registries for health workforce, master patient indexes and health
facilities, and applications include a variety of software and ICT systems that may be operating in your country,
such as HMIS, indicator-reporting dashboards, and terminology and identity management services, as well
as any enabling components like the interoperability layer. Although specific requirements for services and
applications will differ based on the digital health interventions that you have selected, consider the following
to improve the uptake of your intervention.

Leverage existing hardware or reduce procurement for products and services and cannot switch to
of new hardware. Many digital implementations now another vendor without incurring substantial costs
(87)? Are multiple vendors or a large community
rely on end-users’ phones or other devices or on locally
with experience providing support available for
procured devices for deploying interventions. As much this specific digital health application or shared
as possible, avoid investing in duplicative hardware that service?
only a single unit or department will use. Also consider
deployment of mobile-device management tools to Test the implementation for functionality and
ensure that purchased hardware is being used for its stability. Technology vendors typically work in iterations
intended purpose and to reduce wastage (86). to develop applications and organize tests to assess the
application’s usability and stability. Note that this testing
Determine an appropriate software and licensing plan
should be planned as part of a deployment strategy with
to execute your digital health interventions. Based
continual end-user experience tests until the application
on the landscape analysis conducted in Chapter 4, you
is ready to scale. A formal sign-off with the developers is
should be aware of which digital health interventions,
needed at each of the stages (2).
applications and shared services are present in your
country. Consider the advantages and disadvantages » End-user experience tests, also called usability
of different strategies before deciding on a specific testing, are conducted to ensure that end-
users can navigate through the system and
software model. Fig. 5.5.1 summarizes the predominant
perform the tasks as intended. This testing
software models available for use, along with their should be conducted continually throughout the
benefits and risks. Annex 5.1 provides more detailed development process in order to incorporate end-
questions that you may want to discuss with potential user feedback into the process as early as possible.
providers of software applications. » An end-user acceptance test is conducted in a
» To what extent is the application configurable controlled setting with test data to assess whether
by internal teams or requires customization by the application performs as described in the
external teams? For example, if facility names or requirements document prior to signing off on the
hierarchies change or new indicators are added, functionality.
can they be reconfigured within the platform, or do » A functionality test is conducted in a real-life
you need the software vendor to make changes? setting with a limited number of end-users
What is the expected frequency of changes? entering real data and using the application in the
» How will changes or reconfigurations to the way that it was intended.
application be pushed to the end-users using » Once the stability of the application has been
outdated versions, or how will you bring in health tested in controlled and real-life settings with
workers for reinstallation? a limited number of end-users, a load test (or
» Are national vendors available to provide support? stress/volume test) is done to assess whether the
What are the risks of vendor lock-in, in which the application continues to function as intended with
customer becomes dependent on the vendor a larger number of simultaneous end-users.

Plan the implementation 77

 » Throughout this process, put in place mechanisms » Health content embedded in the digital health
to collect and respond to end-user issues, application – such as counselling content or
particularly in the early stages of piloting and decision-support algorithms – reflects evidence-
deployment. This mechanism will need to describe based practices derived from clinical or public
how end-users are expected to report problems, health recommendations, as well as national
who investigates and responds, and how quickly guidelines.
they should respond.
» The digital health application will be able to
capture relevant and appropriate data and make
Ensure that the health content is in line with evidence- possible calculations of prioritized indicators for
based recommendations and best practices. Based on key decision-making and monitoring performance
the health programme area where your digital health of the end-users, as well as inputs, outputs and
intervention is focused, consult relevant resources and impact.
country experts to ensure that the following is true for » Content – where relevant – reflects appropriate
your intervention. Leverage resources such as WHO theories of change and is culturally appropriate,
Digital Accelerator Kits and Computable Guidelines locally validated and in line with national
Standards for up to date health and data content aligned guidelines.
with WHO Guideline and data recommendations.

Fig. 5.5.1. Benefits and risks of different software models.

MODEL BENEFIT S RISKS

CUSTOM-DEVELOPED Examples: • You have control over technology, • Custom development tends to be difficult
SOFTWARE Project Optimize demonstration functionality and design. to manage within time and budget.
Build a software system from projects in Albania, Guatemala, • The development experience creates • Control over design does not guarantee
scratch. Senegal and Vietnam. ownership and improves sustainability. satisfaction with the end product, as that
• It is possible to engage the local IT industry. depends on the capabilities of the technical
team.
• Long-term support depends on the
continued availability of individuals.

COMMERCIAL Examples: • The lead time from selection to • Often expensive and sold with unclear and
OFF-THE-SHELF Sage Enterprise Resource implementation is normally shorter. complex fee structures, for example, a fee-
SOFTWARE Planning, which is in use in many • You can evaluate it before buying. per-server processor.
Buy a commercially available countries in Francophone Africa • The product is maintained and upgraded • Commercial off-the-shelf software is not
product. for essential medicines. (at a cost). often designed for implementation in low-
• It has normally been tested and refined in resource settings.
other implementations.

FREE PACKAGED Examples: • Shorter lead time. • There is often no contract, so service
SOFTWARE USAID/John Snow, Inc.: • Possibility to evaluate. and warranty for bug-fixing depends on
Software developed by a donor • PipeLine • No upfront cost (but maintaining or goodwill of one or two individuals and
organization or technical agency. • Supply Chain Manager customizing it may require investment). there is no institutional support.
Alternatively, a system developed • Many implementation and running costs
by a neighboring country. World Health Organization: are hidden.
• Vaccination Supplies Stock
Management tool
• District Vaccine Data
Management tool

OPEN SOURCE Examples: • You have the right to make changes to the • Can end up with a poorly supported
SOFTWARE OpenLMIS software. product.
The source code as well as OpenMRS • You can engage the local IT industry. • A loosely knit community might not be
the software product is freely DHIS2 • Benefit from communities and able to provide the business relationship
available. Often, a community has OpenSRP share development costs with other you need.
been formed to support the open organizations. • Some of the implementation and running
source software. costs are hidden.

SOFTWARE AS A Examples: • Highly feasible to implement and maintain. • Data hosted on remote servers: not always
SERVICE (SaaS) Logistimo • Clarity about the cost to implement and in agreement with national policy.
Database and application hosted Magpi run a SaaS application. • Ministries of health are not often well
on remote servers, and software is CommCare • Investment in improved software can easily positioned to pay a regular service fee.
sold (or offered freely) as a service Ona be shared among customers.
that can be contracted per user
and per month or year.

Source: WHO/PATH Planning an information systems project, 2013 (2).

78 Digital implementation investment guide

5.6 Standards and interoperability
Standards and interoperability are also critical factors for ensuring the success of digital health
implementations. For different systems to be interoperable, or to meaningfully exchange information with one
another, procedures and data standards must be established that ensure a common language and facilitate data
exchange. Interoperability is not only critical for driving cost efficiencies and reducing fragmentation across
different digital systems, but is also necessary to support the continuity of care as patients engage at different
points of service in the health system, and the digital health enterprise will need to ensure the golden thread of
continuity is maintained across each provider and facility.

Chapter 6 will elaborate on the concepts of standards broader digital health enterprise architecture. Skipping
and interoperability and describe how the proposed this review step may result in a fragmented and siloed
digital health implementation can further link to a digital health implementation.

Box 5.6.1. General considerations for standards and interoperability.

Where appropriate, link to identification registries, such as a national health workforce registry, client
registry or master facility list. For example, some countries have a facility registry with a master facility list that
includes unique identifiers of health facilities and related information on the services they provide. The facility
registry provides an identification number that can link different applications, such as electronic medical records
and a telemedicine system. A similar example is a health worker registry, which provides unique identifiers for
each health worker. These identifiers can facilitate exchanging data across the different applications in which
digital health interventions are housed.

Use data standards for exchanging health information. Global bodies such as Health Level 7 (HL7), International
Classification of Diseases (ICD) and International Health Exchange have established standards, rules that
allow information to be shared and processed in a uniform, consistent manner (88, 89). These data standards
allow stakeholders to align on common data models, which then facilitates exchanging information across
components of the digital health enterprise. For example, data terms used in an HMIS need to align with data
terms used in an electronic medical record to allow comparison of indicators and analysis of health information.
Developing a dictionary of health terms can be a gradual process that uses already established data standards
while also curating local data.

5.7 National digital health strategy and

investment plan
Determine whether you have a national digital health or eHealth strategy, and review the priorities, policies and
operational aspects to establish areas of alignment and differences. The digital health strategy may detail the
current policies and different enabling factors, which are critical to understanding the readiness of the national
environment to support additional digital health applications. Also determine whether a national investment
planning process has been conducted detailing the status, functionality, maturity and inventory of the national
digital health enterprise and any plans for future digital health investments that may align with your needs.

Plan the implementation 79

 CASE STUDY

mHero – using data standards to effectively integrate

different digital health interventions
mHero combines three technologies – IntraHealth’s iHRIS software, UNICEF’s RapidPro and Jembi’s OpenHIM
– into a powerful communication tool for ministry and health workers (90).

iHRIS is open source health workforce information system software used to capture and maintain
information for planning, managing, regulating and training the health workforce. RapidPro is an open source
platform for building an interactive messaging system. OpenHIM is an interoperability layer for standards-
compliant health information exchange, such as OpenHIE.

The combined tool, mHero, allows the MOH to instantly send information to health workers’ mobile phones
and enables health workers to send time-sensitive information back to ministry officials. This was done
through rigorous adoption of open international standards, such as HL7 FHIR, for health data exchange, based
on the OpenHIE framework. By using these standards, information about the health workers could be derived
from the health workforce registry (iHRIS) and then used by the RapidPro messaging system to seamlessly
communicate with health workers around the country (see Fig. 5.6.2). For example, information on the health
facility associated with the health worker could be used as a way to target the messaging to health workers.
Furthermore, the open source and open standards approach means that the mHero platform does not depend
on any specific piece of software, which allows MOHs to readily integrate mHero into their HIS.

Fig. 5.6.2. How mHero integrates digital health interventions using standards.

CURRENT STATE

SHARED SERVICES DATA SOURCES DATA SERVICES

Digital Health Platform

Shared Services Institution- Analytics,

Based HIS & Data Dashboards, Aids,
Business Domain Registry Services Sources Repositories
Services
FACILIT Y
REGISTRY

Facility data HW data

Population- EXTERNAL SYSTEMS
Based HIS & Data
Sources
INTER-LINKED REGISTRY
ENTERPRISE

INTEROPERABILITY LAYER
enabling components

DIGITAL HEALTH APPLICATIONS WITH DIGITAL INTERVENTION FOR:

Health programmes / Use cases
A Health Workforce
B Use Case
C Health Programme
D Use Case
Communication
Point of Service Applications

Source: IntraHealth International, 2017 (90).

80 Digital implementation investment guide

 CHAPTER LINK THE DIGITAL HEALTH

06 IMPLEMENTATION TO THE
ENTERPRISE ARCHITECTURE

Over the course of the previous chapters, you developed a plan for a digital health
implementation focused on addressing identified health system challenges, with
a clear understanding of the digital health interventions and other functionalities
required. To advance this from being a siloed digital health implementation to
an exchanged digital health system architecture, it is also important to consider
the costs and implementation requirements that would enable you to cohesively
benefit from and support the broader digital health enterprise, avoiding the
missteps of investing in fragmented digital systems or ball-of-mud software
applications trying to do everything.

+ OpenHIE diagram (Fig. 6.3.2)

+ TOGAF architecture building blocks (Annex 6.1)
TOOLS + Illustrative list of common components

+ Link the digital health implementation to the broader digital health enterprise
architecture.
OBJECTIVES + Ensure that the costed implementation plan is reflected in the digital health enterprise
architecture.

+ Defined future-state workflow and functional requirements (Chapter 4)

INPUTS + National digital health enterprise architecture (if available in country)

+ Core functional requirements for the planned digital health investment within the
enterprise architecture (Outputs 4.3, 6.2)
+ Shared services and enabling components that can be reused or leveraged by other
OUTPUTS health programme areas or sectors (Outputs 6.3, 6.5)
+ Identification of which applications and shared services already exist and which will
require further investment (Outputs 6.1, 6.4)

81
 PRINCIPLES FOR
DIGITAL DEVELOPMENT

+ Identify the existing technology tools (local and global), data and frameworks being
used by your target population, in your geography or in your sector. Evaluate how
these could be reused, modified or extended for use in your program.
+ Develop modular, interoperable approaches instead of those that stand alone or are
attempting to be all-encompassing in their features. Interoperability will ensure that
you can adopt and build on components from others and that others can adopt and
build on your tool in the future; and swap out systems when improved – standards-
REUSE AND
based –solutions become available.
IMPROVE 1
+ Collaborate with other digital development practitioners through technical working
groups, communities of practice and other knowledge-sharing events to become
aware of existing tools and to build relationships that could lead to the future reuse
and improvement of your tool.

1 Text adapted from Principles for Digital Development: Reuse and improve: Core tenets (7).

82 Digital implementation investment guide

Over the course of the previous chapters, you developed a plan for a digital health implementation focused on
addressing identified health system challenges, with a clear understanding of the digital health interventions
and other functionalities required. To advance this from being a siloed digital health implementation to an
exchanged digital health system architecture (see Fig. 1.3.1), it is also important to consider the costs and
implementation requirements that would enable you to cohesively benefit from and support the broader digital
health enterprise, avoiding the missteps of investing in fragmented digital systems or ball-of-mud software
applications trying to do everything. When investments in digital health implementations are aligned with
a nationally governed digital health enterprise architecture, in which systems contribute to and derive value
from one another, the potential for your implementation to be sustained, scaled and institutionalized is greatly
improved. In contexts where there may not be a defined digital health enterprise architecture, ITU’s Digital
health platform handbook (14) and the OpenHIE architecture specification (91) and community of practice (18)
offer useful resources for planning a digital health enterprise architecture.

6.1 Assess the digital health enterprise architecture

A digital health enterprise architecture, if available in the country, outlines current and planned business
processes, data, systems and technologies and provides an overview of the standards, information exchange
and interoperability profiles that can be optimized across the enterprise. A clearly established enterprise
architecture describing how different processes, data, systems and technologies function together is crucial
for guiding interoperability to support data exchange between digital health applications, as well as collective
functioning goals. An enterprise architecture blueprint lays the foundation for scaling up and sustaining digital
health applications that are standardized and interoperable and that will facilitate access to higher quality, more
complete information, in turn resulting in better decisions and more improved health outcomes across multiple
areas. The architectural approach provides a view of all the necessary building blocks and a rational method of
understanding, defining and manageably implementing digital health interventions.

Typically, there are four layers, or viewpoints, that functional architecture. The technical architecture
describe various aspects of the health enterprise includes the required applications and technology that
architecture (see Fig. 6.1.1). The health programme or should be integrated and standardized to facilitate
business processes, such as the ones you developed the delivery of identified digital health interventions
in Chapters 3 and 4, depend on data augmented by and address the health sector goals. Adherence to
appropriate health content for optimal functioning, appropriate health data and ICT standards becomes
including decision-making and effective action. These the critical link, or “glue”, towards achieving greater
aspects are based on the health programme needs, interoperability.
as established in Chapters 3 and 4, and comprise the

Link the digital health implementation to the enterprise architecture 83

Fig. 6.1.1. Digital health architectural approach.

BUSINESS
processes and
activities use…

DATA
that must be collected,
organized, safeguarded and
distributed using…

APPLICATIONS
such as open source or custom
information systems and digital health
solutions that run on…

TECHNOLOGY
such as the e-Government Integrated Data
Centre (eGIDC) and cellular phone networks.

Although this type of architectural vision may not sure that the architecture is sustainable as the needs
exist or be fully developed for your context, you could for digital evolve within and across health programmes.
consult the TOGAF and OpenHIE framework in Fig. 6.3.2 Given the rapid changes in technology, as well as the
to understand the different components typically changes in health programme needs, the architecture
found in developing a shared digital health enterprise should be built in a way to support additions of
architecture. Understanding the current architecture new applications, upgrades to legacy systems and
and its constituent functional and technical component adaptations to new demands. Within this planning
parts will facilitate understanding the gap between what process, the implementation team should consider
currently exists and can be leveraged and what may be costs for future upgrades, maintenance and remodelling
missing, requiring further targeted investment through of the architecture as additional needs emerge, such
your costed implementation plan. as including new shared services or data exchange
standards.
Lastly, depending on the complexity of the architecture,
it may be helpful to consider a systems audit and seek
the assistance of an expert in data exchange to make

84 Digital implementation investment guide

 BID INITIATIVE
CASE STUDY

BID Initiative: Enterprise architecture approach

Multiple disconnected health data systems can stunt the efforts and potential of new technologies.
Successful digital health enterprises take stock of the existing landscape and build upon it, with the goal of
sustaining efforts well beyond a project life span. The BID Initiative worked closely with the governments
of Tanzania and Zambia to integrate interventions into the broader health system and overarching health
strategies.

The BID Initiative defined its enterprise architecture in terms of whole-system behaviours rather than
specific technologies. It was not intended to be a definitive description of any single country’s health
enterprise architecture, but rather intended as a starting point that could adapt to a specific country’s needs
and reflect its unique context.

A number of core principles drove the architectural choices reflected in the BID design.
» Data collection is integrated into the workflow. This principle reflects the fact that for data quality
and timeliness to improve, the use of data must be woven into the fabric of each workflow participant’s
business processes. All data will be captured electronically, as soon as possible and as close as possible
to the step in the workflow where the data are generated.
» Data will be shared to support multiple workflows. As an example of this principle, child
immunization transactions can be leveraged to track inventory transactions.
» Users have access to the data necessary to perform their duties. One of BID’s objectives was to
improve data quality and use; therefore, the workflow participants must have access to actionable,
readily understandable data.
» Interoperability and openness: The preference was to adopt existing standards wherever possible
and to adapt them where necessary. There was no expectation that new standards would need to be
developed to support the BID Initiative.
» Sustainability: This principle means that simple, stable, readily adoptable solutions would be favoured
over technologically “sophisticated” ones that would be difficult to deploy nationally. Communications
infrastructure would be leveraged and centralized ICT solutions preferred over ones that are
decentralized.
Adapted from Product vision for the BID Initiative, Chapter 2 (49).

Link the digital health implementation to the enterprise architecture 85

6.2 Identify common and enabling components and
shared services (digital health platform)
When linking your digital health implementation to the broader digital health enterprise architecture, consider
the core functionalities, or components, that are unique to your use case or health programme, as well as
the components that can be generalized and reused in other health programme areas or even beyond the
health sector. These common components (also called reusable components) represent opportunities for
joint investment, allowing you to stitch together and harmonize different digital health interventions across
programme areas and even sectors, while also facilitating the establishment of a common architecture from
which all digital health implementations can benefit (see Fig. 6.2.2). Think of these common components as
like roads and water pipes – “build once, use multiple times” – that you are likely to need for your digital health
implementation and that others will also need and can collectively contribute towards and benefit from in
subsequent adaptations and deployments. These reusable components may also be used outside the health
sector, such as in education and social protection. The collective of common components is known as the digital
health platform and can be subdivided into shared services and enabling components.

You could align your digital health implementation

to link to and share these common components as
Box 6.2.1. Illustrative list of common
they become available in the digital health enterprise components.
architecture (14). Standards and APIs (application
programming interfaces), which are codes and software The following are examples of common
that allow two software programs to communicate components that can be extended across other
with each other, can help link the digital health implementation areas:
implementation to common components. » authentication services to determine access
and control privileges
If these components do not yet exist in your setting, you
could direct some of your digital health investment to » identity management services, such as
support establishing a common component like a shared unique IDs for clients, health workers and
facilities
service that other health programme areas can later
reuse. Not all of the necessary common components » terminology services and reference data
represented in an architectural diagram will be supporting metadata needs, including for
data elements and indicators
available at the same time; investments are often made
sequentially as finances become available and needs » geolocation services
arise. Accordingly, your digital health implementation » payment services to facilitate financial
will evolve over time by contributing investments or transactions
incorporating new components as the digital health » analytics engines supporting dashboards
ecosystem and architecture matures (14). and similar tools
» scheduling and decision-logic engines
» data warehousing to support storage and
archiving using common standard formats
» enterprise service bus to support a data
exchange “interoperability layer”.

86 Digital implementation investment guide

Fig. 6.2.2. Digital health investments within the digital health enterprise architecture.

ENTERPRISE

SHARED SERVICES DATA SOURCES DATA SERVICES

Shared Services Institution-Based Analytics, LEGEND

HIS & Data Sources Dashboards, Aids,
Repositories Funded
Business Registry Services Investment for software under
Domain Services Digital and non- new software development
Terminology digital datasets Data tools Investment for Funded fully
Services
Health Mgmt
Client
sourced from that are strengthening
software
functional
software
info system
Registry processes that used to
Logistics Mgmt Facility systematically facilitate Investment for
scaling functional Software for
info system Registry record information decision software decommissioning

within health making

Shared Product Registry institutions and Reused POS Unique POS
Health Record Health Worker (e.g., GIS) application application
facilities (e.g.
Digital Health Platform

Registry
Service Availability
and Readiness
Provides Provides Assessment
software software (SARA))
capabilities, capabilities, EXTERNAL SYSTEMS
or “services”, or “services”,
specific to that are Population-Based
healthcare canonical/ HIS & Data Sources
that may be master “lists,” Non-health
leveraged which are Digital and non-digital digital
by other enforced by datasets sourced from applications,
applications specified processes that systematically services,
across the governance record information about interoperability
digital health mechanisms members of a population enabling
enterprise (e.g. Census, Demographic components,
and Health Surveys (DHS)) and data sources
with capabilities
that may be
leveraged by the
digital health
enterprise (e.g.
Provides software capabilities to support national unique
INTER- managed data exchange and interoperability
OPERABILITY identification
LAYER between applications and services across system, national
Enabling the digital health enterprise procurement
Components
systems, national
weather service
data sources, etc.)
APPLICATIONS WITH INTERVENTIONS FOR:

A Health Programme B Use Case

C Health Programme
D Use Case
Point of Service Applications

Applications that are reused within more than one use case or
programme area

Applications that are uniquely used for only one use case or health
programme area

Link the digital health implementation to the enterprise architecture 87

6.3 Link your digital health investments to the
enterprise architecture
This Guide provides only an introductory overview for linking digital health investments within health
programmes to the broader national digital health enterprise architecture. Executing this process will require
more detailed consultations with personnel who have expertise in developing national digital health enterprise
architectures. Box 6.3.1 highlights key resources that can facilitate this process.

It is recommended that you identify or develop a » What are the core components for this digital
diagram reflecting the current state of the national health implementation?
digital health enterprise architecture and include in » Of those core components, which existing
your costed implementation plan a diagram detailing a common components can be reused or leveraged
future state that shows proposed investments in digital from other health programme areas? For example,
an analytics engine may be a required component
services and applications (see Fig. 6.3.3 for an example).
that another programme area has already
The current state depicts how different systems are implemented and can be reused and shared (for
currently implemented, which may be as disparate example, as a shared service).
applications that are siloed or at best paired directly
» What common components are new requirements
with other applications. In the future-state diagram, that the digital health investment can support and
highlight planned new and emerging digital components contribute to the national digital health enterprise
that others are implementing, as well as the common architecture?
and programme-specific functionalities that your
digital health investment will focus on, specifying the Illustrative current- and future-state diagrams of
applications, common services and interoperability Myanmar’s national digital health enterprise architecture
requirements that your system will leverage or are provided as guidance (see Fig. 6.3.3), which you could
contribute towards. Annex 6.1 provides a template use as a template to diagram existing (green, yellow) and
for thinking through how the proposed digital health planned (red, grey) common components of your costed
implementation can link to the broader architectural implementation plan. In your diagram, note the use of
requirements. specific digital health applications encompassing digital
health interventions at the point of service, shared
As you begin linking your digital health investment to services and enabling components for each of your
the broader architecture and planning for the costs that identified digital health interventions.
would entail, consider the following questions.

88 Digital implementation investment guide

 Box 6.3.1. Key resources for building towards a digital health enterprise architecture.
» Digital health platform handbook: building a digital information infrastructure (infostructure) for health
focuses on combining digital health interventions into an interoperable whole, identifying the standards
and interoperability requirements for enabling a cohesive digital health enterprise architecture (14).
» OpenHIE describes a reusable architectural framework that leverages health information standards,
enables flexible implementation by country partners and supports exchange of individual components
(see Fig. 6.3.2). OpenHIE also serves as a global community of practice to support countries towards “open
and collaborative development and support of country-driven, large-scale health information sharing
architectures” (18).
» TOGAF is an industry-standard enterprise architecture methodology providing detailed guidance to
support the establishment of a flexible, integrated hierarchy of business, data, applications and technology
architectures to optimize digital health interventions (92).

Fig. 6.3.2. OpenHIE architectural diagram illustrating mechanisms for

interoperability across different applications.

1 2 3

DIGITAL
4 5 6 7 8
HEALTH
PLATFORM

10

Legend 6. [Health] Facility Registry (HFR) serves as a central authority

to uniquely identify all places where health services are
1. Logistics Management Information System (LMIS) stores and
administered within the country.
aggregates routine supply chain data and facilitates its analysis for
improving supply chain management. 7. Health Worker Registry (HWR) is the central authority for
maintaining the unique identities of health providers within the
2. Shared Health Record (SHR) is a repository containing the
country.
normalized version of content created within the community
after being validated against each of the previous registries. It is a 8. Product Registry (PR) is the central authority for defining
collection of person-centric records for patients with information products, such as health and medical commodities, and their
in the exchange. categorization.
3. Health Management Information System (HMIS) stores 9. Interoperability Services Layer receives all communications
routinely collected aggregate health data and facilitates their from external services within a health geography and orchestrates
analysis with the goal of improving the quality of health services. data-exchange processing among the external systems and the
OpenHIE component layer.
4. Terminology Service (TS) serves as a central authority to uniquely
identify the clinical activities that occur within the care-delivery 10. Point of Service [applications], such as the OpenMRS electronic
process by maintaining a terminology set mapped to international medical records system and the RapidPro Communications
standards, such as ICO10, LOINC, SNOMED and others. Engine, are used by clinicians and community health workers
to access and update a patient’s person-centric shared health
5. Client Registry (CR) manages the unique identity of citizens
information and record healthcare transactions, and to
receiving health services within the country.
interactively communicate, respectively.

Source: Figure and legend adapted from OpenHIE architecture v2, 2019 (93).

Link the digital health implementation to the enterprise architecture 89

Fig. 6.3.3. Myanmar example of a national digital health enterprise architecture blueprint.

CURRENT STATE

SHARED SERVICES
Shared Services Institution-Based Population-Based Analytics,
HIS & Data Sources HIS & Data Sources Dashboards, &
Digital Aids
Facility Surveys
(SARA)
Health Facility MoHS Website
Registry Dashboards
eLMIS Census
(mSupply, Logistimo)
Health Workforce
Registry GIS EXTERNAL SYSTEMS
eHMIS (DHIS2)

RCSC Database

DITT Data Centre

POINT OF SERVICE APPLICATIONS

Tablets
(Health Worker Apps)
Electronic Medical
Records (EMR)
TB (OpenMRS, GeneX
RCDC Surveillance Alert, QuanTB)
ePIS (EMR and HIS) LEGEND
RMNCAH, NCDs,
RTA and Injuries Investment for Funded
new software software under
development

Investment for Funded fully

strengthening functional
software software

FUTURE STATE Investment for

scaling functional Software for
software decommissioning

SHARED SERVICES DATA SOURCES DATA SERVICES Reused POS Unique POS
application application
Digital Health Platform

Shared Services Institution-Based Population-Based Analytics,

HIS & Data Sources HIS & Data Sources Dashboards, Aids,
Repositories
Health Facility Facility Surveys
Registry (SARA)
Census MoHS Website
Dashboards
Health Workforce
Registry eHMIS (DHIS2)
Civil registration &
Terminology Services vital statistics GIS EXTERNAL SYSTEMS
(HDD) eLMIS
(mSupply, Logistimo)
Household Surveys Data Warehouse
(DHS, STEPS, others)
Shared Health
Record IDSR

RCSC Database
ENTERPRISE

INTER- DITT Data Centre

OPERABILITY Authentication Encryption Transformation
LAYER
Enabling Queuing Validation Translation
E-Governance
Components
Agriculture
DIGITAL HEALTH APPLICATIONS WITH DIGITAL INTERVENTION FOR:
Health programmes / Use cases
A Malaria B HIV/AIDS C TB D Patient Portal
Point of Service Applications

DHIS2 DHIS2
DHIS2
OpenMRS OpenMRS

GIS PWID GeneX

Tracking
Patient
Portal
MS Access FUCHIA QuanTB

In the current state, there is a limited use of data components are used to facilitate interoperability across
exchange standards, and digital systems use a direct different digital health implementations and shared
(integrated) connection to shared services. In the services.
future state, data exchange standards and enabling

90 Digital implementation investment guide

 CHAPTER

07 DEVELOP A BUDGET

This chapter will help you develop a budget for implementing and sustainably
operating your digital health intervention within your specific digital health
ecosystem. You will identify cost drivers for each phase of the digital health
implementation, including budget considerations related to interoperability, and
you will develop a budget for the life span of the investment. Cost considerations
for specific digital health interventions are further detailed in Annex 5.3.

+ Budget template (Annex 7.1)

TOOLS

+ Understand cost drivers by implementation phase.

+ Accurately calculate funding needs.
OBJECTIVES + Identify co-financing opportunities across other health programmes and sectors.
+ Historical budgets and costs
INPUTS + Implementation considerations (Chapter 5)

+ Programme budget (Outputs 7.1, 7.2)

OUTPUTS

91
 PRINCIPLES FOR
DIGITAL DEVELOPMENT

+ Plan and design for scale from the start.

+ Develop a definition of scale for your initiative.
+ Keep your design simple, flexible and modular to make it easy to change your
content and adapt to other contexts.
+ As you make technology choices, think about whether those choices will make it
easier or harder to scale.
+ Identify partners early who can help scale your tool or approach.
DESIGN FOR
SCALE 1
+ Consider your funding model, including the cost per end-user, options for
generating revenue, social business models and other financial paths to sustaining
the initiative.
+ Gather evidence and demonstrate impact before attempting to scale.
+ Don’t attempt to scale without fully validating that your initiative is appropriate in
a new context and addresses a priority need.

1 Text adapted from Principles for Digital Development: Reuse and improve: Core tenets (7).

92 Digital implementation investment guide

This chapter will help you develop a budget for implementing and sustainably operating your digital health
intervention within your specific digital health ecosystem. You will identify cost drivers for each phase of the
digital health implementation, including budget considerations related to interoperability, and you will develop
a budget for the life span of the investment. Cost considerations for specific digital health interventions are
further detailed in Annex 5.3.

7.1 Costs by phase of implementation

Fig. 7.1.1. Phases of implementation.

DEVELOPMENT
& SETUP DEPLOYMENT INTEGRATION & SCALE SUSTAINED
INTEROPERABILITY OPERATIONS

ONGOING/ALL PHASES

The full costs of implementations of digital health ONGOING/ALL PHASES: This is not a distinct phase but
interventions are frequently underestimated because instead refers to elements that affect the budget across
budgets often focus on the costs related to the initial the implementation life cycle, such as human resources
demonstration or deployment and do not take into and governance.
account the resources needed for long-term operation
DEVELOPMENT AND SETUP: During this phase, you
and maintenance. Inaccurate budget estimates can
design and prepare for implementation. You will incur
thwart the sustainability of interventions, especially
many of the costs during this phase, including workflow
when demonstrations or pilot projects transition to
mapping and defining the future state. You will also
programmes operating at scale.
begin working with technology vendors and purchasing
Understanding the total cost of ownership, or the equipment needed to support the deployment.
the resources required to support a digital health Within this phase, you should begin to think through
intervention throughout its life cycle, will help you requirements for interoperability and exchange with
make more informed purchasing decisions and better other systems, adopting appropriate standards and
communicate funding requirements to donors, partners ensuring that the intervention leverages any relevant
and other stakeholders. When considering the total cost existing components or ICT systems, such as data
of ownership and developing a budget, it is important exchanges, HMIS and registries.
to consider costs associated with different phases of
implementation (see Fig. 7.1.1).

Develop a budget 93
DEPLOYMENT: During this phase, the digital health user to deploy your intervention on a larger scale. During
implementation goes live, often in a pilot setting. It is this phase, you may need to invest a significant portion
important to budget resources to support end-user of expenses in long-term assets, such as purchasing
testing and iteration for refinement during this phase. equipment or improving facilities, including network and
electrical infrastructure, as well as the human resources
INTEGRATION AND INTEROPERABILITY: Although
needed to maintain the quality of the deployment.
these elements should be addressed during design
and deployment, they are so important to long-term SUSTAINED OPERATIONS: After your digital health
sustainability that they have been called out in a implementation scales, you will enter into sustained
separate phase, as they may need to be reviewed and operations. Consider recurring costs during this
updated continuously. As your deployment expands phase, as well as continued M&E, ongoing data-use
and the digital health enterprise architecture evolves, activities and ways to share learnings with the larger
reflect on the additional needs for your digital health community. The annual sustained operations costs
implementation to integrate and exchange data with will be key in determining feasibility of sustaining the
existing systems. digital implementation in the long run. The estimated
annual sustained operations costs can be used to inform
SCALE: You will begin to expand the reach of your digital
government budgetary allocations in future years.
health implementation during this phase, so consider
the number of future end-users and the cost per end-

7.2 Cost drivers

For each of the phases listed in Section 7.1, identify the specific cost categories and their related cost drivers
that will affect your budget. Table 7.2.1 lists several examples of cost categories and the factors, or cost drivers,
that influence (increasing or decreasing) those cost categories that are associated with each phase of the
implementation. Table 7.2.1 also describes important considerations to take into account when estimating
costs. For additional guidance, consider consulting other tools, such as ADB’s digital health investment
costing tool (30).

94 Digital implementation investment guide

Table 7.2.1. Illustrative costs throughout the implementation life cycle.3

Cost Up-front
Categories Cost Drivers Considerations versus
recurring

ONGOING/ALL PHASES

Management » Complexity of intervention » What is the level of effort for programme management staff Recurring
and staffing » Full-time Equivalents associated with training, vendor relationship management and
(FTEs) needed other meetings?
» Staff training needed » Does staff capacity already exist on your team? Will you need to
» Turnover shift tasks or hire and train new resources?
» Overhead » Is there an opportunity to build capacity with existing staff at a
lower cost than hiring new staff?
Governance » Number of stakeholders » What is the estimated time for development, approval and uptake Recurring
needed for co-ordination of supportive policies?
» Full-time Equivalents » Whose approval is needed to begin or finalize this work (for
(FTEs) needed example, parliamentary approval or executive approval by the
» Time needed for approvals MOH)?
» Amount of travel and » How many agencies or approval processes are needed to make
meetings required for changes to the health system?
buy-in, co-ordination, and » Do you have technical staff available who are skilled in policy
approvals analysis and governance or advocacy?
» Translation required » Do you need to work with external consultants to analyse, create
» Overhead and institutionalize new governance structures?
» How often will policies and regulations be renewed or revisited?
» How frequently will the governance body meet? Will travel be
required?

DEVELOPMENT AND SETUP

Software » Scale of implementation » What is the licensing model? For example, is it open source or Up-front
licensing (i.e. number of end users, proprietary? What are the licensing costs, and how will these
cost per number of devices, etc.) change with scale?
environment » Is there a flat fee per number of end-users or an individual fee per
end-user or device?
and per end-
» Is there a platform fee or cost to add end-users?
user
Software » Complexity of features and » If you are working with a software vendor, what are the costs to Up-front
customization, functionality required add features now or in the future?
including » Staff training needed » If the software is open source, is there a responsive, established
adding » Turnover end-user community that will provide ongoing support and help
» Full-time Equivalents add features at no cost?
additional
(FTEs) needed » Do you have skilled, available technical staff who can customize
languages the software? What is the level of effort required? Is the software
» Translation required
well documented? Are there multiple vendors or a large
community with experience providing support for this specific
digital health tool?
» What are the costs to contract with a consultant who is skilled
and familiar with the software code to do the customizations?
» What are the costs to translate terms and develop the software in
additional languages, if needed?
Application » File size of the software » What is the level of effort for staff to install and configure the Up-front
installation application application? If you are replacing an existing application, consider
and » Sophistication of hardware the time needed to uninstall the previous application and transfer
data to the new system.
configuration » Speed of internet » What are the requirements for accessing sensitive information?
connectivity
Are proper protocols in place? Has the proper delegation of duties,
such as a data processor/data handler, been established?

3 Note that this table does not include additional overhead costs, such as utilities and office supplies, that would be required irrespective of the digital
health implementation.

Develop a budget 95
Cost Up-front
Categories Cost Drivers Considerations versus
recurring
Inter- » Use of standards or lack » What is the cost to interoperate with existing systems? Recurring
operability thereof » What efforts will be needed to ensure that the system complies
with other » Maturity of interoperability with relevant standards, including open standards?
systems standards used

Hardware » Existence of “Bring your » Where will data be stored (for example, in the cloud, on local Recurring
own device” policies/ servers or on backup servers)?
number of devices needed » Do end-users need devices?
» Sophistication of device(s)
needed

DEPLOYMENT

End-user » Amount of travel and » How will you collect end-user feedback? Recurring
testing meetings required » How frequently will you modify or iterate on the functionalities
» Translation required within the digital health implementation?
Cost and » Internet costs in country » What is the cost for the Internet bandwidth or mobile data Recurring
availability » Reliability of electricity in needed for the system to operate properly?
of data country » Will you need to equip your office with generators to ensure that
connectivity the system remains available during power outages?
and power » Do you need solar chargers, car chargers or spare batteries for
reliable device charging?
Training » Existing capacity gaps » Is there a fee for initial training? Recurring
» Amount of travel required » Are there travel and other logistical costs associated with training?
» Gaps in existing training » Do you need to create new training and capacity-building
materials and curriculum materials?
» Scale and frequency of » What training methodology will be employed (for example, on-
training needed the-job, classroom or mixed-use training)?
» How long are the trainings?
» How many people need to be trained?
» How frequently will you offer training to new end-users as the tool
scales?
Roll-out » Number of end users » Are there per diems, lodging, fuel and transport costs associated Up-front
» Size of targeted geographic with transporting the needed hardware to the sites?
area » What are the costs associated with communicating? Are
there marketing materials that need to be developed for
communication purposes?

INTEGRATION AND INTEROPERABILITY

Data » Existing data sharing » Has an architecture for data sharing been established? Recurring
collection policies » Are relevant standards for data exchange and coding available, or
and use » Existing data/terminology will they need to be developed?
standards » Are there fees associated with using the coding standards?
» Licensing fees associated » Do the tools already support the identified standards?
with standard use » Are there additional interoperability considerations?
» Use of standards or lack
thereof in the current
digital health enterprise

SCALE

Any category » Number of end users » Are additional staff needed? What additional support structures Recurring
that will be » Complexity of intervention are needed?
affected by » Size of data collected and » Will additional people need to be trained?
expanding stored » Is additional hardware or storage needed?
reach » Are additional software licences needed?
» Are additional voice or data services needed?

96 Digital implementation investment guide

 Cost Up-front
Categories Cost Drivers Considerations versus
recurring

SUSTAINED OPERATIONS

Voice and data » Number of end users » How many text messages and voice minutes will be used? Recurring
services » Size of data collected and » How much mobile data will be needed for each end-user?
(mobile data stored » Can you negotiate a below-market rate with an MNO?
» Amount of information
plan, Internet, needed to disseminate
number of text
messages)
Hardware » Number of end users and » How often will you replace hardware? (The typical replacement Recurring
maintenance, devices managed rate is approximately 20–25% per year.)
ongoing ad- » Sophistication of the » What are typical maintenance costs?
device » How many staff members are needed for ongoing administration
ministration » Amount of travel needed
and replace- of hardware? What are their costs related to travel to
for on-site maintenance deployment sites?
ment rate support
Subscriptions » Number of end users » Is there a subscription fee? Recurring
» Per user subscription fee » Are there costs to receive software updates or to access specific
features?
» How will upgrades be verified?
Software » Number of end users » Will you need to pay new licence fees when you update to new Recurring
maintenance » Per user licensing fee software versions?
(fixing bugs, » Amount of bug fixes » Will volunteers from the open source community be able to do
needed maintenance, or will you have to hire a developer? Consider that
adding features, » Anticipated updates
some updates may require additional development and testing.
maintaining released per year » Will you get support from a vendor or from programme staff?
customizations) Consider the budget implications of operations support for
system crashes or to address issues with software performance.
Transfer of » Full-time Equivalents » How much staff time will be needed to transition ownership to Recurring
ownership (FTEs) needed the government or another entity? What capacity-building will
» Staff training needed this require?
» Turnover » Will licensing costs change due to an increase in the number of
» Number of end users end-users?
» Will the new owner need to procure new hardware?
Refresher » Complexity of training » What is the staff attrition rate? Recurring
training and curriculum » How frequently will you provide refresher training?
additional » Turnover » What other training activities and materials will you offer?
» On the job training
training vs. formal training » How will ongoing support and supervision be used?
activities mechanism
M&E and » Full-time Equivalents » Who will monitor the use of the new tools and the quality of the Recurring
data-use (FTEs) needed data in the systems, and what are the associated costs to do so?
activities » Complexity and scope of » Will you conduct periodic evaluations of the introduction of the
intervention new tools, uptake of the systems or even the impact? If so, what
are the associated costs?
» What processes need to be strengthened or developed to build
a culture of data use (such as consistent data review meetings,
nonfinancial incentives to data champions and so on)?
Collective » Complexity and scope of » How will you share learnings and findings with the larger Recurring
benefit, such intervention community? What costs are associated with the sharing?
as sharing » Number of stakeholders
needed for co-ordination
learnings
Source: Adapted from Principles for Digital Development: How to calculate total lifetime costs of enterprise software solutions (94).

Develop a budget 97
 BID INITIATIVE
CASE STUDY

BID Initiative: How much does an EIR cost?

How much does it cost to implement BID Initiative interventions in the test sites in Tanzania and Zambia?
What are the up-front and recurring costs of operating BID interventions? What are the resources required
to implement interventions in each district? How do the costs of providing immunization services and
reporting compare before and after BID interventions are implemented?

These are some of the key costing questions the BID team asked as it planned to implement interventions in
health facilities in Tanzania and Zambia. The answers were used to make decisions about implementation,
scaling and adaptation.

PATH health economists led an economic evaluation assessing both the financial and economic costs of
implementing BID interventions in Tanzania and Zambia. A key component of this evaluation involved
estimating the total cost of ownership of BID interventions, including the financial costs incurred to develop,
deploy, integrate, scale and sustain the interventions (see Table 7.2.2 for more detailed cost drivers). The data
to estimate the total cost of ownership were gathered from project records and through tracking resources
used by implementation teams.

Table 7.2.2. BID Initiative cost categories.

Cost category Cost description

ONGOING/ALL PHASES

Meetings with government officials to get their buy-in and plan for implementation
Meetings in their regions or provinces, including meetings associated with developing the
rollout strategy for the region or province
Printing
Printing of guidelines
Tanzania only

DEVELOPMENT AND SETUP

System design and

development Design and development of the EIR in each country; includes testing the EIRs
EIR that is being used
Design and development of first versions of EIRs in each country that were later
Learning costs
shelved
Peer learning
A visit to Zambia for peer-learning exchange
Tanzania only
Costs to enter previous immunization records of each child from the paper
Back-entry costs
immunization registers into the EIR
Hardware Tablets and covers, chargers and QR code/barcode scanners for health facilities

98 Digital implementation investment guide

 Cost category Cost description
DEPLOYMENT

Training Training of staff responsible for rolling out the interventions to the health facilities
Per diems, lodging and transport associated with deployment of the EIR to health
facilities and district immunization offices; transport includes vehicles purchased
Roll-out
(one for each country) and expenditures for fuel and maintenance of those vehicles,
as well as hiring other vehicles used for the deployment

SUSTAINED OPERATIONS

Internet connectivity Access to Internet for uploading data and transferring data to higher levels
Data hosting Server for EIR data
Supportive Per diems and transport costs for BID Initiative or MOH staff to provide supportive
supervision supervision to health facilities after deployment of interventions
Printing Printing of barcodes used on immunization cards
Adapted from Mvundera et al., 2019 (95).

The economic evaluation showed the following results.

» Costs of developing, deploying and maintaining EIRs are less than 10 US dollars per child under 1 year of
age, but can be less than 5 US dollars per child in a country like Tanzania that has a large birth cohort.
» Hardware and deployment of the EIRs are the cost categories that account for a large share of these
costs (95).

BID believes that subsequent EIR development and deployment costs may be even less because of the
ability of other low- and middle-income countries to leverage the EIR systems that were developed for
Tanzania and Zambia as well as through leveraging learnings generated from these deployments.
Adapted from Di Giorgio & Mvundera, 2016 (96).

Develop a budget 99
7.3 Budget matrix
Once you have identified your cost categories by investment phase and the related drivers of those cost
categories, use a budget matrix (see Table 7.3.1) to create a detailed budget for your implementation across the
expected life span before the intervention will require significant updates. This time frame is typically about five
years but could be longer or shorter depending on the selected intervention. Budget matrices are also useful
for comparing costs across digital health interventions. You could use financial data in historical procurement
records and from past implementations, along with RFPs from developers and implementers, to complete the
budget matrix.

In creating your budget, be sure to indicate components that would be funded through the existing programme to
show country and partner co-investment. A detailed budget template is included in Annex 7.1.

Table 7.3.1. Summary budget matrix.

Budgeting category Year 1 Year 2 Year 3 Year 4 Year 5 Five-year

pilot total
Ongoing/all phases

Development and setup

Deployment

Integration and
interoperability

Scale

Sustained operations

TOTAL

Source: Adapted from Principles for Digital Development:

How to calculate total lifetime costs of enterprise software solutions (94).

While preparing your costed implementation plan, you should also be able to demonstrate how this investment will
improve the status quo in terms of projected impact. This may include proving the comparative value of this digital
health investment over other types of activities, including nondigital investments. Modelling methodologies, such as
the Lives Saved Tool (LiST), can help project the overall health impact of the investment. More details on how to apply
this tool to digital health investments and examples of impact projections for different digital health interventions can
be found in the Asian Development Bank’s Digital health impact framework (97).

100 Digital implementation investment guide

 CHAPTER

08 MONITOR THE IMPLEMENTATION

AND USE DATA EFFECTIVELY

Monitoring and evaluation and continuous improvement by responding to the

changes induced by the digital health implementation, also known as adaptive
management, are necessary components of ensuring the viability and ultimate
impact of your efforts.

+ Monitoring and evaluating digital health interventions: a practical guide to conducting

research and assessment (26)
+ Adaptive management checklist (Annex 8.1)
TOOLS + Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework
(98)
+ Logic model template (Annex 8.2)

+ Develop and execute a plan to monitor fidelity and quality of the implementation.
+ Develop and execute a plan to assess the impact of the implementation on expected
process and outcome indicators.
OBJECTIVES + Determine activities required to build and promote a strong culture of data use.
+ Understand how adaptive management approaches may improve efficiencies and
impact.

+ Historical and/or baseline data and analysis

INPUTS + Historical M&E and adaptive management plans
+ Logic model for digital health implementation (Output 8.1)
+ A plan to guide the M&E and adaptive management of the digital health
OUTPUTS implementation (Outputs 8.2, 8.3)

101
 PRINCIPLES FOR
DIGITAL DEVELOPMENT

+ Design programmes so that impact can be measured continually and

incrementally, focusing on outcomes, not just outputs.
+ Make use of existing data, including open data sets and data from interoperable
systems.
+ Use rigorous data collection methods. Consider and address potential biases
and gaps in the data collected, perform data quality checks, and maintain strong
documentation behind collected data.
+ Close knowledge gaps by contributing data to the development community and
using data and interoperability standards.
+ Use quality real-time or timely data to support rapid decision-making, improve
programming for end-users and inform strategy.
BE DATA
DRIVEN 1
+ Present data in formats that are easy to interpret and act on, such as data
visualizations.
+ Create a culture of data use by prioritizing capacity-building and data-use efforts
across all stakeholder groups, including the groups whose data are being collected.
+ Be holistic about data collection and analysis. Collect data from multiple
sources, and use a mix of data collection and analysis methods. Analyse your data
collaboratively with stakeholders.
+ Identify and use open data and interoperability standards.
+ Collect and use data responsibly according to international norms and standards.

1 Text adapted from Principles for Digital Development: Reuse and improve: Core tenets (7).

102 Digital implementation investment guide

After going through the steps to plan the digital health implementation but prior to deploying the
implementation, you can embed mechanisms to monitor the implementation, such as collecting baseline data,
and use the insights from the emerging data to increase your implementation’s impact and efficiency. Continual
monitoring of activities at different stages of the implementation is critical for ensuring its long-term success, as
is responding to external changes and new learnings.

Fig. 8.1. M&E and adaptive management as continual considerations for digital
health implementations.

Monitoring and evaluation

LEADERSHIP & GOVERNANCE

LEGISLATION, WORKFORCE
STRATEGY & SERVICES & POLICY, &
INVESTMENT APPLICATIONS COMPLIANCE

STANDARDS &
INTEROPERABILITY

INFRASTRUCTURE

Adaptive Management
Source: Batavia & Mechael, 2016 (59).

This chapter explores three important and interrelated can help implementations achieve their objectives in
concepts for continual assessment of your a more effective and efficient way. Although M&E has
implementation: historically been used as an accountability and reporting
tool, it can also drive growth and improvement.
1. monitoring and evaluation
Monitoring during deployment can ensure that the
2. establishing a culture of data use entire operation functions as intended, from the digital
3. adaptive management. health intervention’s performance to the way in which
end-users interact with the intervention to the kind of
Together, these processes support flexible, responsive data that the intervention generates (99).
project design that will enable you to refine your
Ultimately, you will need to demonstrate the
digital health implementation as circumstances
contribution of the digital health implementation to
change and priorities shift. The backbone of this
health system performance and, where possible, to
success, and of strong HIS in general, is rigorous, rapid
improved health outcomes.
and continual M&E, which enables a steady cycle of
learning, iterating and adapting. Robust M&E efforts

Monitor the implementation and use data effectively 103

8.1 Establish a logic model for your implementation
Defining a framework in the form of a logic model will processes, outputs, outcomes and impacts (4). They
help you refine and understand your programme’s goals provide a graphical representation, which may serve as
and objectives and conceptualize the relationship a catalyst for engaging and communicating with key
between them, the underpinning programme activities stakeholders, including implementers, in an iterative
required to achieve them and the anticipated outcomes. process, often in the wake of changes in design or
programmatic implementation. Fig. 8.1.1 is an illustrative
Logic models aim to clarify programme objectives and
logic model for MomConnect in South Africa (26).
aid in identifying expected causal links from inputs,

Fig. 8.1.1. Illustrative logic model of MomConnect digital health investment.

INPUTS PROCESSES OUTPUTS OUTCOMES IMPACT

Partnerships Utilization of digital Utilization of
health system health services
§ With implementing § Meetings and § Improved registration § Improved use of
agencies, service contract agreement of pregnant women antenatal care services
providers, local with all relevant (registered women (e.g. completion of
health authorities stakeholders that are < 20 weeks ANC visits 1–4)
§ Policy-level gestation) § Improved delivery care
support at local and § Community-based (e.g. facility-based
national level identification/ deliveries, delivery by
subscription of skilled birth attendant
Health promotion messaging pregnant women (SBA))
§ Programme § Development § Improved postpartum
promotion of appropriate care § Reduction in
§ Development promotional Strengthening human number of
materials resources § Improvements in
of standardized child health (e.g. stillbirths
health promotion § Development of § Health workers’
reported use of mobile early attendance of
messages with marketing and postnatal care (PNC) § Reduction in
guidelines and promotional tools for data collection neonatal/infant/
(facility-based for newborns)
milestones channels child mortality
providers that report § Improvements in
§ Consensus of use of mobile tools) disease-specific care/
expert panel on § Reduction
Improved efficiency management (e.g. HIV in maternal
health messages early identification/
§ Provider time spent on mortality
services treatment for
Health-care facility and community inputs newborns)
§ Adequacy and § Provider training/ Improved § Improved continuity
availability of orientation to technology use of MNCH services (e.g.
human resources digital health § Functional messaging coverage of ANC, SBA,
(HR) system service (e.g. messages and PNC)
sent per end-user
§ Linkages with § Recruitment of during pregnancy Improved knowledge
existing monitoring staff to address HR period) § Increased provider
systems gaps knowledge (e.g. target
§ Technical performance
§ Ongoing supportive of the service (e.g. providers can correctly
supervision average time to recall at least two
of technology complete subscription maternal danger signs)
implementation on digital health § Increased knowledge of
system) subscribers enrolled to
Technology receive messages
§ Network coverage § Testing and
and power updating digital Supply side Supply side
health system § “Help desk” usage and § Utilization of help
§ Testing and desk services (e.g. ANC
adaptation of based on user response
response clients that utilize help
mobile application § Satisfaction with desk services)
§ Provider mobile § Regular checking of services received at the
provider phones to health-care facility Funding
equipment § Subscriber willingness
check for broken/ § Satisfaction with help
lost/not working to pay
desk services
Funding § Health facility
aggregate outputs
§ Adequate timeline, budget and sources
Funding
§ NGO/implementing
partner costs
§ End-user costs
§ Incremental costs to
health system

104 Digital implementation investment guide

Logic models link inputs (programme resources) with OUTPUTS are the products of process activities. From
processes (activities undertaken in the delivery of a technological perspective, technology inputs (such
services), outputs (products of processes), outcomes as hardware and software devices), coupled with the
(intermediate changes) and impact. capacity-building to ensure their appropriate and
sustained use, correspond to changes in program
In the context of digital health implementations,
outputs, including improvements in performance and
INPUTS encompass all resources that go into the
adoption by end-users. Ultimately, these technological
programme. In this model, technology inputs are
outputs are anticipated to correspond to improved
differentiated from programmatic inputs aimed at
functioning of health systems (such as governance,
providing health services. Programmatic inputs (human
human resources and commodity management) and
resources, training and other materials development)
service delivery (such as increased number of health-
are also distinguished from policy inputs, which aim to
worker visits and increased proficiency of health workers
improve linkages with treatment and care, as well as to
in service delivery). Improvements in service delivery
consider factors related to affordability, including user
include increased outreach and follow-up, improved
fees. Technology inputs include not only the hardware
availability and quality of services, improved service
and software, but also the feedback loop to ensure that
integration and, among health workers, increased
technology is responsive to stakeholder needs, including
proficiency and accountability.
clients and health workers. Inputs may also include the
advocacy needed to secure necessary funding and policy OUTCOMES refer to the intermediate changes that
changes. emerge as a result of inputs and processes. Outcomes
may be considered according to three levels: health
PROCESSES are the activities undertaken in the
system, health worker and client. At the health-
delivery of digital health interventions. Processes fall
system level, outcomes encompass domains of
into three distinct areas: technology (capacity-building
efficiency (technical and productive), increased service
and platform enhancements), developing a national
responsiveness to meet client needs and increased
implementation strategy and health capacity-building.
coverage of targeted health services. At the health-
From the programmatic side, digital health worker level, increased knowledge, productive efficiency
interventions require two critical processes: a national (time allocation) and quality of care are anticipated.
implementation strategy and capacity-building. It is Finally, at the client level, digital health interventions
recommended that digital health implementations are anticipated to correspond to changes in knowledge,
that fall within the latter stages of development and efficiency (technical and productive), service
evaluation (such as effectiveness to implementation responsiveness, adherence to treatment protocol and,
science) undergo a series of activities that help ultimately, demand for services.
formulate a larger national implementation strategy,
IMPACT may be considered according to domains within
including identifying priority areas for rollout, linking
your focus programme areas, such as performance of
with routine health services delivery, targeting cadres of
health systems (increased time health workers spend on
health workers required to provide services and fostering
clinical care), population health (reductions in morbidity
public–private partnerships. With regard to capacity-
and mortality) and other population benefits, including
building, introducing digital health interventions may
reductions of household out-of-pocket payments
require short- and long-term human-resource inputs,
for care corresponding to catastrophic costs for care
including initial and refresher training of health workers,
seeking.
as well as ongoing supportive supervision. Finally, and
perhaps most critically, ensuring information security,
including confidentiality of client records, and managing
data are key required processes.

Monitor the implementation and use data effectively 105

8.2 Plan how you will conduct the
monitoring and evaluation
Planning to deploy a digital health intervention includes EVALUATION is the systematic assessment of an
determining what to monitor to ensure that the ongoing intervention to determine whether it is
intervention is working as planned and what to evaluate fulfilling its objectives and to demonstrate an effect
to ensure that it is having the effects you expected. on health outcomes (26). A formal evaluation allows
These efforts may include tracking performance, you to attribute a range of outputs, outcomes or
changes in processes, health outcomes, end-user economic values to the intervention, which can show
satisfaction with health systems, cost-effectiveness or evidence of benefit. If you are planning an intervention
shifts in knowledge and attitudes. that will require evidence to receive political or
financial support for scale, generating this evidence
MONITORING helps answer the question: Is the
in an early deployment will help you create a strong
intervention working as intended? Monitoring digital
case. Evaluation is also an increasingly important
health interventions, often using routinely collected
consideration for the digital health field as it works
data, can measure changes in performance over time
to harmonize and learn from various deployments,
and allow for adaptive management or course correction
shifting from small-scale pilots to the broader
based on the results (26). Monitoring, alongside
institutionalization of digital health. In addition,
processes for taking action, creates tight feedback loops
evaluation can help you understand if your intervention
that stimulate ongoing planning and learning, which is
is having the intended programme impacts, such as on
critical to creating a culture of data use and fostering
client access and use of services, efficiency of health
adaptive management. Effectively monitoring your
workers and quality of care. However, evaluation can
intervention enables you to identify issues in software
be resource intensive and requires staff with a strong
code, recognize when end-users are facing challenges
background in research design and evaluation to support
and make sure that the intervention is achieving the
the work.
targets you have set. Plan for and support these essential
monitoring activities early in the design process, Evaluation needs will also evolve throughout the
but also be sure to build them into all stages of the implementation life cycle. Initially, the evaluation
implementation’s life cycle. may focus on determining whether the digital health
intervention has an effect on health practices and
As the implementation matures, monitoring activities
outcomes, such as improving health workers’ adherence
may focus on the digital health intervention’s fidelity and
to protocols or increasing timeliness of services. The
quality: Do the realities of field implementation alter the
focus of evaluations will gradually shift to economic
functionality and stability of the intervention? Are the
assessments and implementation research questions
content and delivery of the intervention of high enough
that explore issues surrounding scale, sustainability
quality to yield intended outcomes? Finally, as the
and changes in policy and practices. Health economic
intervention scales, monitoring may increasingly focus
assessments demonstrating a return on investment will
on its integration with the broader health system and
be especially critical to justify the use of digital health
the policy environment surrounding, for instance, data
interventions over other types of potential interventions.
privacy, management and use, as well as ensure that the
appropriate levels of training and end-user support are
in place to maintain fidelity of impact, within budget
constraints.

106 Digital implementation investment guide

Ideally, M&E occur in close balance with each other to better meet the needs of end-users and the
and are structured to answer questions that are most organization. Fig. 8.2.1 illustrates the evolution of M&E
relevant at each stage of the implementation. At the needs during the development and deployment stages
early stages, you could use M&E results to iteratively of implementation.
redesign and test the digital health intervention

Fig. 8.2.1. Intervention maturity over time.

INTERVENTION MATURITY OVER TIME

NATIONAL IMPLEMENTATION
FUNCTIONALITY FIDELITY

STABILITY QUALITY
PROTOTYPE

MONITORING
EVALUATION

USABILITY EFFICACY EFFECTIVENESS IMPLEMENTATION RESEARCH

FEASIBILITY ECONOMIC / FINANCIAL EVALUATION

Maturity models can serve as measuring sticks or implementation is meeting the intended objectives or
indicators of progress by helping identify opportunities addressing the targeted health system challenges. If the
for improvement and allowing teams to critically implementation is not proving to be usable or stable,
assess the resources, budget and time required of your team could consider improving or even potentially
the intervention (see Fig. 8.2.2). Determining an halting it, especially if it cannot pass thresholds for
implementation’s maturity stage can also inform feasibility or demonstrate an added value over existing
what must be monitored and to what degree the practices.

Monitor the implementation and use data effectively 107

Fig. 8.2.2. Implementation maturity continuum.

Implementation Maturity Continuum

Component When Potential Measures & Guiding Questions

• End-user entry of phone number

is correct
• Rate of agreement in data
recording between training rounds
(i.e. end-user accuracy)
• Quality control reports on How well and consistently are
end-users end-end-users able to operate
Pre-launch the system?
QUALITY & during • Feedback from end-users on Are the content and use of the
implementation content system adequate for yielding
• Incorrect schedules or content intended outcomes?
updates
• Timestamps on form submissions
• Number of form submissions/
worker
• Data patterns similar across
workers/geographic areas

• Stability reports
• Functionality reports
Do the realities of field
• Phone loss or damage implementation alter the
During • Poor network connectivity functionality and stability of the
FIDELITY system?
implementation • Power outages
Is the system being used
• End-user forgets password appropriately or as designed?
• Incorrect intervention delivery by
end-user

• Server downtime Does the system consistently

operate as intended?
• SMS failure rate
Is the system responsive during
STABILITY Pre-launch peak conditions or high volume
• Network connectivity of data transmission?
• Server operation capacity What is the failure rate from the
server side?

• SMS content
• SMS schedules
• SMS timing Does the system meet the
requirements for addressing
• Form content the identified health system
FUNCTIONALITY Pre-launch challenge?
• Form schedules
• Application functions Does the system operate as
intended?
• Comparison of requested system
vs delivered system
• QA test case adherence

Source: Adapted from WHO Monitoring and evaluating digital health interventions, 2016 (26).

108 Digital implementation investment guide

 CASE STUDY

Myanmar “Learning for Impact” model of continuous monitoring

UNFPA Myanmar developed the Love Question, Life Answer (LQLA) mobile application to promote
accessibility of evidence-based information about life skills and sexual and reproductive health and rights
to adolescents and young people in and out of school in Myanmar. Through active promotion on different
channels, the number of downloads reached more than 43 000 end-users by the end of 2018. As the app was
being deployed, UNFPA determined that it would not be sufficient to measure only the end-users.

To improve monitoring and uptake of the app, UNFPA designed a suite of routine practical tools known
as Learning for Impact. The Learning for Impact monitoring approach looks at the categories of system
performance, usage, engagement, health outputs and health outcomes, each linked to objective metrics.
Through this continuous monitoring, implementation teams can make corrective actions to refine different
aspects of the deployment.

Table 8.2.4. Examples of metrics from the Learning for Impact approach.

Definition in the
Category Metric context of the Potential corrective actions
implementation
STABILITY System Percentage of time for a Poor system uptime requires the original
uptime given period for which a software development team to optimize server
system is operational versus performance.
nonoperational
FIDELITY Monthly Number of end-users in a Consistently poor monthly active end-user
active end- given month numbers may be an indication of a need to:
users » improve system performance
» increase advocacy or marketing efforts
» improve engagement through upgrades.
QUALITY Net promoter User satisfaction with A low net promoter score indicates that end-
score the intervention, as users may not be satisfied with the tool. This
measured by willingness to may prompt additional investigation to see what
recommend it to others end-users are unsatisfied with (such as features,
stability or utility), which can then lead to
targeted improvements.
HEALTH Improvement Percentage of women/ If the intervention is not increasing sexual and
OUTPUTS of sexual and men 15–24 years old reproductive health knowledge, this indicates
reproductive who correctly identify content areas that may need more focus, either
health both ways of preventing in education efforts outside the use of the
knowledge the sexual transmission intervention or in the intervention itself.
of HIV and reject major
misconceptions about HIV
transmission
HEALTH Uptake of Number of adolescents/ If service uptake has not improved during the
OUTCOMES sexual and young people who have period of using the intervention, additional
reproductive used integrated sexual and investigation should be done as to why (such as
health reproductive health services health system constraints, difficult-to-use clinic
services (disaggregated by services, finder or delays to care).
age and gender)

Monitor the implementation and use data effectively 109

The RE-AIM framework may also be considered as a Adoption: the absolute number and proportion of
comprehensive way to think through your M&E needs as organizational units, individuals or settings that adopt a
you scale up the implementation. The following are the given intervention
components of the RE-AIM framework (98):
Implementation: the fidelity to the various elements
Reach: the level of penetration of an intervention in of an intervention’s protocol, including consistency
terms of the proportion of eligible participants who of delivery as intended and the time and cost of the
receive the intervention intervention; at the individual level, implementation
refers to clients’ use of the intervention strategies
Effectiveness: the impact on targeted outcomes,
including potential negative effects Maintenance: the extent to which a program or
policy becomes institutionalized or part of routine
organizational practices and policies.

8.3 Establish a culture of data use

To sustain a digital health implementation, there needs Examples of data use include a national manager of a
to be a culture that values the collection of high-quality health area adjusting allocations of health workforce
data, as well as the actions taken as a result of that personnel to meet changing burdens on the local,
information. M&E that produces believable data can district and regional levels or a minister of health using
help foster a culture of data use in which: data visualizations to predict how much financial
support a certain health programme will need over the
» people demand and seek out high-quality data to
inform their decision-making next five years. See Fig. 8.3.1 for additional examples of
how data can be used to build and support a robust
» people are motivated and empowered to act on
culture of data use.
the data
» managers support those who are collecting the
data and reinforce transparent use of data
» leadership implements data policies that stress
the value and systematic collection of data while
modelling the use of data (27).

110 Digital implementation investment guide

Fig. 8.3.1. Examples of data use.

BENEFIT E X AMPLES HOW TO EVALUATE SUCCESS

• Targeted advocacy efforts help to address higher-than- • Did the system produce credible data for these
BETTER INDICATORS
average vaccination dropout rates in specific population indicators?
FOR STRATEGIC
groups.
PLANNING • Were managers able to act on this information?
• Credible estimates of vaccine wastage rates for each • Did the information change decisions and how did
health centre lead to tailored vaccination strategies to that benefit the programme?
reduce wastage.
• Was there any measurable impact on outcome
• High failure rates in certain types of cold chain equipment indicators, such as vaccination coverage?
lead to the discontinuation of that equipment.

• A district officer validates a vaccine request based on • Did the system lead to operations that are more
BETTER DAY-TO-DAY
the available stock, target population and average efficient? For example, was there a reduction in
DECISION-MAKING
consumption in the health centre that sent the request. buffer stocks or wastage?
• A nurse uses the immunization register of her clinic to • Did the system lead to better availability of stock?
find the children who are falling behind their vaccination • Did it change the way people work and did that
schedule. improve health outcomes (for example, higher
• A warehouse manager analyses average demand and coverage, lower dropouts)?
makes sure that stock is kept between minimum and
maximum levels.

• In Senegal, some health programmes have outsourced • Do the system data accurately reflect reality?
BETTER CONTROL
the distribution of their commodities to the national
AND OVERSIGHT • Did the system highlight poor performance?
pharmacy. With access to stock and delivery information,
they can regularly monitor the arrangement.
• In Turkey, pharmacists scan barcodes when they
dispatch drugs to make sure that the insurance system
is not overcharged.
• Through a last-mile stock management system, managers
can monitor whether some health centres or districts are
regularly overstocked or experiencing stockouts.

• Health workers enter monthly reports directly into a • Comparing how people spent their time before
REDUCED
computer or mobile device and transmit them implementation of the system change and how
ADMINISTRATIVE
electronically. they spend it since the change, what are the
BURDEN
differences?
• Aggregate coverage reports are generated automatically
by the system.

Source: WHO/PATH Planning an information systems project, 2013 (2).

Box 8.3.2. Immunization Data: Evidence for Action.

The Immunization Data: Evidence for Action (IDEA) review is a global synthesis of existing evidence aimed at
increasing the use of high-quality data to improve immunization coverage (100). It provides a clearer picture
of what works to improve immunization data use, why it works and where knowledge gaps still exist. The
systematic realist review considered nearly 550 documents, including published literature, working papers,
project evaluations and reports. The following are some key findings from the review: interconnected and
comprehensive strategies provide stronger results; data use leads to increase of quality of data; and efforts to
systematize data use lead to long-term intervention success. Additional information and resources can be found
at Technet-21 (101).

Monitor the implementation and use data effectively 111

Although the high quality of data is important, quality reflect the needs of different end-users, individuals are
does not guarantee data use at an individual, facility, more likely to use data for decision-making.
community or organizational level. Additionally, while
Building a culture of data use requires careful
a trained cadre of data end-users is critical, data must
planning, steady application and the decision-making
meet the requirements of multiple end-users and
infrastructure to allow for change. Data, in order to be
end-user scenarios, informing both national policy and
actionable, must be translated from complex charts and
service delivery among health workers. Furthermore,
figures into digestible information and a clear series of
the right people (at all levels) must be able to get the
messages and directives (see Fig. 8.3.3). As the digital
information they need, when they need it, for their
health intervention demonstrates success, an activity
purposes.
during scale may include changes in how the data
Data must be translated into information at the right outputs inform policy decision-making. As individuals
level of detail to inform national-level resource planning increasingly use data in their day-to-day decisions,
and state-, provincial- and local-level programme they will gradually become more invested in the quality
management, depending on the needs of the decision- of that data, even working to improve it. Data use is
maker. At the facility level, for example, an EIR may therefore a cyclical process. As the quality improves,
enable nurses to search for individual-level data about end-users’ confidence in that data will also increase,
patients who have recently defaulted on a vaccine. The and they will be more likely to use the data to make
same registry, to be truly functional, must also convey decisions. Teams should also make a point of measuring
immunization coverage data at the district, regional and progress along the way. A deliberate, systematic
provincial levels to meet the needs of higher-level health approach will bring about enduring improvements in
workers, who must track which facilities are meeting data use, data maturity and sustainability-enabling
targets and which are underperforming. When presented factors.
with actionable information that has been aggregated to

Fig. 8.3.3. Data-driven accountability cycle.

QUALIT Y
DATA

Complex numerical data, in the

form of tables and graphs, are
generated.

DATA UNDERSTANDABLE
DEMAND INFORMATION

Targeted plans are developed, An interpretation of

and programmes are DATA USE BY the data facilitates
improved GOVERNMENTS understanding of the
main takeaways.

ACTION ACTIONABLE
(POLICY OR PROGRAM) MESSAGE

The information is used to help The information is used to

focus on what needs to help focus on what needs to
be done. be done.

Source: Moore et al., 2018 (102).

112 Digital implementation investment guide

Four other factors can help complete cultural change, as described in Table 8.3.4.

Table 8.3.4. Cultural change factors.

Earlier, you identified your health system challenges, along with the information needed to
Awareness of address these challenges. Implementing an awareness campaign that builds a case for action
need appropriate across different levels of the health system can increase support for the digital
health intervention.
Motivation can come from both external drivers (such as job performance indicators or financial
Motivation to incentives) and internal drivers (such as care for the community and country). Recognition by
act peer networks and data-use champions may also stimulate motivation.
Empowerment Empowerment often entails changes to formal policies and job descriptions that support
to act individuals’ ability to identify and act on information.
Individuals must feel reasonably confident in their ability to identify and review the relevant
Skills to use and data, interpret the information and then develop conclusions and corresponding action items.
improve quality Beyond initial training, there should be a feedback loop to monitor performance, as described
earlier in this chapter, and ongoing performance support to continuously improve.

BID INITIATIVE
CASE STUDY

BID Initiative: Building a culture of data use

What does a culture of data use look like on the data use, including data-use guides, readiness
ground? For Oliver Mlemeta, a nurse at the Usa assessments and guidelines on supportive
River Health Facility in Tanzania, it means she will supervision that complemented the use of the EIR.
be trained on how to use data in a meaningful way
so that she can better serve her patients. Simplified,
automatically generated reports will reduce the Fig. 8.3.5. The data journey, with and
chance of error and allow her more time to do without data-use interventions.
what she loves most: care for patients. A monthly
Without With data use
dashboard tells her how her facility is performing data use interventions interventions

compared to neighbouring health centres and allows

Oliver to adjust services accordingly. If she has health facility health facility
questions, she can reach out to her peers using the level level

WhatsApp platform, a communication forum where

health workers provide peer support as they adopt
Community mobilization
Supportive supervision

new tools and practices. district

Data use campaigns

district
level level

A primary goal of the BID Initiative was to improve

data use at all levels of the health system (see
regional
Fig. 8.3.5). Building a culture of data use requires level
regional
level

products that ease data collection and visibility,

policies to support the culture and people who
can enforce the policies by establishing effective national national/global
level level
practices. To this end, BID introduced a number
of different tools and approaches to strengthen Source: BID Initiative briefs: recommendations and
lessons learned: data use (103).

Monitor the implementation and use data effectively 113

 BID INITIATIVE
CASE STUDY

The following are key learnings from implementing these data-use interventions in Tanzania and Zambia.
» Teach data-analysis skills with the facility’s existing data to help nurses identify challenges
currently affecting service delivery and pinpoint ways to address those issues. This foundation in
data analysis better prepares nurses to adopt new tools and to adapt their data-analysis skill sets to
different service areas, such as malaria.
» Electronic tools, as well as revised paper forms, must go through an iterative process with
feedback from facility, district, regional and provincial members of UAGs. This allows software
developers to understand how health workers will use data and information and ensures the creation
of intuitive tools that enable access to data for planning and service delivery.
» Use targeted, supportive supervision and tools, such as job aids and dashboards, for data
visualization to identify low-performing facilities. These tools should also present a methodology
to walk through the challenges associated with the facility’s performance, as well as an approach to
identify steps to improve performance.
» Create peer-support networks to connect health workers with other facilities in their district.
These networks provide an opportunity for nurses to ask questions of one another and to receive
support in real time using messenger platforms like WhatsApp. For instance, health workers may pose
questions about how to calculate indicators. Regional leads may also use the network to communicate
with nurses and facility in-charges by sharing immunization updates.
» Engage regional-, provincial- and national-level stakeholders. Although nurses at the facility and
district level are the critical data end-users and will benefit from greater data visibility, stakeholders
at all levels should be involved to foster a culture of data use across the health system. Readiness
assessment tools and data dashboards for decision-making enable management of that change.

The following are additional tools developed by the BID Initiative for building a culture of data use:
» Spot check form (104)
» Data-use culture job aid (105)
Adapted from BID Initiative briefs: recommendations and lessons learned: data use (103).

114 Digital implementation investment guide

8.4 Adaptive management:
Use data to optimize interventions
Adaptive management requires realizing that change stakeholder engagement. It uses cycles of structured
happens and building in the ability to respond to the decision-making and, increasingly, real-time data to
change (106). Emerging from the interdisciplinary make strategic and operational decisions throughout
need and understanding that complex development the implementation’s life cycle. Adaptive management
issues require nimble solutions, adaptive management is only possible to the extent that data on performance
calls for incremental, steady iteration (107). Adaptive are available.
management ensures that M&E plan outputs are
Real-time data empowered by digital health
continuously used to improve the digital health
interventions can facilitate adaptive management
intervention or larger digital health investment.
by enabling rapid, timely feedback in the form of
Adaptive management may include adjusting behavioural changes, performance metrics and M&E
interventions, trying out new workflows, retiring indicators, allowing for prompt course correction.
unsuccessful processes or scaling approaches that Adaptive management fosters changes to traditional
have demonstrated value. It is a continuous process management approaches in several ways (see Table 8.4.1).
of learning by doing, steady feedback and ongoing

Table 8.4.1. Traditional versus adaptive management.

Traditional management Adaptive management

» Relies on fixed best practices and » Reinforces participatory approaches, iteration and flexibility
standardization determined at the start of an throughout the implementation life cycle
implementation » Change is contextual and informed by end-users and other key
» Change is top-down and driven by the stakeholders
organization and donors » Requires the capacity for constant change and strategic course
» Requires management planning and repetition correction

An ongoing cycle of decision-making, monitoring, When developing your adaptive management plan
assessment and feedback leads to a better to optimize and sustain interventions, consider the
understanding of development issues and an improved following questions.
management strategy based on what is learned.
» Are your programming and interventions based on
Be aware that transforming how data are used can evidence or following a logical theory of change?
generate resistance throughout all levels of the » How does your organization identify and mitigate
health system because of changes in accountability, uncertainties and risks?
collaboration, communication, decision-making, job » Who is involved in decision-making at an
descriptions and other operational practices. Although implementation, programme or organizational
the digital health intervention may be functional, stable, level?
usable and effective, this resistance can affect the overall » What mechanisms does your team or organization
efficacy of the intervention. Combining an effective have to periodically pause and reflect?
monitoring approach with improved data-culture » How does your team or organization discuss and
practices can help mitigate this risk. learn from missteps or failures? What mechanisms
for knowledge management does your team or
organization have to capture and share lessons
learned?

Monitor the implementation and use data effectively 115

 » Does your team have mechanisms for translating implementation, but they are critical during times
learnings into change? If so, how do you integrate of uncertainty or when key milestones are delivered.
change at an implementation, programme or Contextual factors, M&E data, learnings and other data
organizational level?
recorded throughout the process can all inform these
» What tools and mechanisms does your team intentional pause-and-reflect moments.
use to determine the appropriate approach to
interventions (such as theories of change, benefit Engage stakeholders to consider the implications of
analyses, stakeholder analyses and so on)? how work is progressing and if the data indicate that
» How do you evaluate progress, taking into account a change is needed; if so, allow stakeholders to inform
uncertainties and repeated cycles of learning and the possible alternatives. Analyse all of these inputs to
change? make an evidence-based decision to stay the course or
» How do you analyse and communicate results? redirect. Throughout the process, document learnings,
alternatives that were considered, decisions and actions,
A full checklist of considerations can be found in as these may be used to inform future pause-and-reflect
Annex 8.1. Fig. 8.4.2 illustrates the steps you could moments. If action has been taken to redirect, schedule
take to apply adaptive management during your time to reassess the new plan and identify new areas
implementation. When planning your digital health where uncertainty may call for additional reflection.
intervention, schedule regular moments to pause and
reflect. Ideally, these reviews occur regularly throughout

Fig. 8.4.2. Adaptive management cycle.

ADAPT REVIEW

Review current
Update context and
implementation analyse real-time
plan, if needed, M&E data to assess
and iterate. implementation.

ACT

Engage
stakeholders
Stay the course to consider
or adapt. implications
and alternatives.
Pause and
reflect

Document Make evidence-

learning, based decisions
decisions and to stay the course
actions. or adapt.

116 Digital implementation investment guide

8.5 Progress check

ˆ Problem statement detailing specific challenges and needs in

the health system
ˆ Identified digital health interventions to address the current
challenges
At this point in the ˆ Enabling-environment assessment defining possible constraints
process, you should ˆ Implementation plan that is appropriate for the environmental
have the following limitations
outputs: ˆ Linkages of this specific investment to the broader set of digital
health activities and enterprise architecture
ˆ High-level financial plan and costing
ˆ M&E plan, including for adaptive management and data use.

Monitor the implementation and use data effectively 117

118
 CHAPTER

09 VALUE PROPOSITION AND

NEXT STEPS

Fig. 9.1 provides an overview of the key components to complete as you finalize
your costed implementation plan. You may use this costed implementation
plan to obtain the necessary digital health investment for your proposed
implementation. Beyond resource mobilization, following this process should
give you more confidence that the selected digital health interventions that you
plan to implement within a larger digital health enterprise architecture:
» address identified bottlenecks and health programme needs
» align with the existing national digital health strategy
» fit within your local context and ecosystem
» promote an exchanged digital health enterprise system architecture that can
contribute to broader health sector goals.

While this process takes time, it should result in long- Lastly, as you embark on the digital health
term cost savings by reducing resources wasted on implementation, continue to consider evolution of the
misaligned, ineffective or siloed digital health enterprise larger ecosystem. How can your investment continue
system architectures, while increasing the likelihood for to contribute to the broader digital health enterprise
health impact by addressing identified health system architecture? How can you use the data effectively to
challenges. Additionally, the selected interventions continually improve your investment and its health
should fit within the existing national digital health impact? Remember that building sustainable digital
strategy, enterprise architecture and context, ensuring health enterprises is a dynamic process, and as the local
long-term sustainability of the investment. context changes over time, you may need to consider
new or additional digital health interventions or refine
your thinking on the health system challenges to be
addressed.

119
120

Fig. 9.1. Summary of outputs towards a costed investment plan.

Health Goals
01 CHAPTER 02 02 CHAPTER 02 03 CHAPTER 03
» Form the team and establish goals
» Establish personas and needs in health
Actors Priorities Programme Context
structure
3.1 Personas & organigram
» Assess and document user and data
workflows 3.2 Pain points
» Identify pain points and health system
1.1 Team
challenges 2.1 Within health 3.3 Health System
Programme(s) & System Challenges and needs
» Articulate expected outcomes, benefits, 1.2 Stakeholders
impact
2.2 Within Digital 3.4 Current-state
1.3 Beneficiaries Health Strategy Workflow Diagrams

Digital Context 04 CHAPTER 04

06 CHAPTER 06
Common
» Review digital maturity and readiness Digital Context Architecture Services
» Detail priority digital interventions in Across
optimized flows
4.1 Prioritized Digital 6.1 Current 6.2 Future
Sectors
» Inventory digital assets and gaps, ability to Interventions Architecture Architecture
reuse
» Articulate required functions and systems 4.2 Maturity and 4.3 Functional 6.3 Standards &
Readiness Requirements Interoperability Costed
» Detail standards and interoperability needs
Implementation
» Diagram current state and proposed future 4.4 Future- 4.5 Digital 6.4 Point of Service
state architecture
state
Workflow Inventory Applications 6.5 Shared Services Plan
Digital implementation investment guide

Financial and Operational 05 CHAPTER 05 07 CHAPTER 07 08 CHAPTER 08 09 CHAPTER 09

Considerations Implementation Budget Monitoring Value
» Describe policy and enabling environment Proposition
» Explain investment phases and budget
8.1 Logic Model
» Highlight governance mechanisms
and risk mitigation strategies 5.1 Implementation 7.1 Phases and 8.2 Monitoring &
» Present detailed actions including scale- Plan Costs evaluation
up, operation and maintenance, training,
monitoring and change management 5.2 Enabling 7.2 Financing 7.2 Adaptive 9.1 Value
Environment Management Proposition
» Outline logic model, and value proposition
of the investment on expected outcomes
 BID INITIATIVE
CASE STUDY

BID Initiative: Sustainability

An important component of sustaining a digital health enterprise is sharing lessons learned with the larger
community. Here are a few from the BID Initiative that consider policy, technical, institutional and financial
elements (see Fig. 9.2).

Fig. 9.2. Critical sustainability elements.

POLICY
Understand the policy TECHNICAL
environment present in Map government capacity to manage
the country, including solutions over the long-term; develop
gaps in existing policies a capacity-building plan to ensure
and strategies. the human skills and necessary
infrastructure are in place.

INSTITUTIONAL
Identify the level of
engagement and FINANCIAL
participation across the Ensure that the cost of
health system in the maintaining and replacing
design and adaptation solutions over time is both
of solutions. feasible for the government
and built into the appropriate
funding mechanism.

Work from the beginning with a core group of stakeholders across the government and other key
organizations. This will ensure a complete understanding of the challenges to be addressed and that the
solutions address those challenges and meet end-user needs.

Build key champions within the government and key stakeholder groups. These champions are essential to
advocate for adopting solutions and long-term funding.

Balance the need for a “proof of concept” (seeing it to believe it works) with the need to begin sustainability
planning. The key issues of technical capacity, policy environment and financing need to be considered from
the beginning.

Create a realistic, shared vision among partners and the government from the start. This vision will cover
what needs to be in place for sustainability (infrastructure, policy, capacity and financing) and determine
how to implement process or system changes.

Secure costing data as quickly as possible (including cost estimates if necessary). This will build
understanding of both the level of financing required and the savings possible in other budget areas because
of greater efficiencies and smoother processes.
Adapted from BID Initiative briefs: recommendations and lessons learned: sustainability (108).

Value proposition and next steps 121

 References
1. WHO guideline: recommendations on digital interventions 16. Human-centered design toolkit. UNICEF; in press.
for health system strengthening. Geneva: World Health
Organization; 2019. Licence: CC BY-NC-SA 3.0 IGO. 17. Digital identity toolkit: a guide for stakeholders in Africa.
Washington (DC): World Bank Group; 2014 (https://
2. World Health Organization, PATH. Planning an information openknowledge.worldbank.org/handle/10986/20752/,
systems project: a toolkit for public health managers. accessed 17 February 2020). Licence: CC BY 3.0 IGO.
Seattle: PATH; 2013 (https://path.org/resources/planning-
an-information-systems-project-a-toolkit-for-public- 18. OpenHIE [website] (https://ohie.org/, accessed 18 February
health-managers/, accessed 18 February 2020). 2020).

3. World Health Organization, International 19. The TOGAF Standard: version 9.2. The Open Group; 2018
Telecommunication Union. National eHealth strategy (https://www.opengroup.org/togaf/, accessed 18 February
toolkit. Geneva: International Telecommunication Union; 2020).
2012 (https://www.itu.int/dms_pub/itu-d/opb/str/D- 20. Drury P, Roth S, Jones T, Stahl M, Medeiros D. Guidance
STR-E_HEALTH.05-2012-PDF-E.pdf, accessed 17 February for investing in digital health. Manila: Asian Development
2020). Bank; 2018 (ADB Sustainable Development Working Paper
4. World Health Organization. Classification of digital health Series, No. 52; https://www.adb.org/sites/default/files/
interventions: a shared language to describe the uses publication/424311/sdwp-052-guidance-investing-digital-
of digital technology for health. Geneva: World Health health.pdf, accessed 18 February 2020).
Organization; 2018 (WHO/RR/18.06; https://www.who.int/ 21. Handbook for digitalizing primary health care: optimizing
reproductivehealth/publications/mhealth/classification- person-centered tracking and decision-support systems
digital-health-interventions/en/, accessed 18 February across care pathways. World Health Organization; in press
2020). (https://www.who.int/reproductivehealth/publications/
5. Resolution WHA71.7. Digital health. In: Seventy-first World handbook-digitalizing-primary-health-care/en/).
Health Assembly, 26 May 2018. Geneva: World Health 22. Digital accelerator kits. World Health Organization; in press
Organization; 2018 (https://apps.who.int/gb/ebwha/pdf_ (https://www.who.int/reproductivehealth/publications/
files/WHA71/A71_R7-en.pdf, accessed 18 February 2020). digital-accelerator-kits/en/).
6. Global strategy on digital health 2020–2024. World 23. Implementation guide. In: FHIR Release 4 [website]. Ann
Health Organization; draft (https://www.who.int/docs/ Arbor (MI): HL7 International; 2019 (https://www.hl7.org/
default-source/documents/gs4dh.pdf?sfvrsn=cd577e23_2, fhir/implementationguide.html, accessed 17 February
accessed 18 February 2020). 2020).
7. Principles for Digital Development [website] (https:// 24. World Health Organization, International
digitalprinciples.org/principles/, accessed 18 February Telecommunication Union. Be he@lthy, be mobile
2020). Licence: CC BY-SA 4.0. handbooks. In: Noncommunicable diseases and their risk
8. Digital Health Atlas [website] (https://digitalhealthatlas. factors [website]. Geneva: World Health Organization; 2020
org/en/-/, accessed 18 February 2020). (https://www.who.int/ncds/prevention/be-healthy-be-
mobile/handbooks/en/, accessed 17 February 2020).
9. Global Digital Health Index [website] (https://www.
digitalhealthindex.org/, accessed 18 February 2020). 25. 2018 global reference list of 100 core health indicators (plus
health-related SDGs). Geneva: World Health Organization;
10. Digital health investment review tool. Maternal and Child 2018 (https://www.who.int/healthinfo/indicators/2018/en/,
Survival Program; 2018 (https://www.mcsprogram.org/ accessed 17 February 2020). Licence: CC BY-NC-SA 3.0 IGO.
resource/digital-health-investment-review-tool/, accessed
17 February 2020). 26. Monitoring and evaluating digital health interventions: a
practical guide to conducting research and assessment.
11. HIS stages of continuous improvement toolkit. Chapel Geneva: World Health Organization; 2016 (https://www.
Hill (NC): MEASURE Evaluation; 2019 (https://www. who.int/reproductivehealth/publications/mhealth/digital-
measureevaluation.org/his-strengthening-resource- health-interventions/en/, accessed 18 February 2020).
center/his-stages-of-continuous-improvement-toolkit/,
accessed 17 February 2020). 27. Arenth B, Bennett A, Bernadotte C, Carnahan E, Dube M,
Thompson J et al. Defining and building a data use culture.
12. Principles of Donor Alignment for Digital Health [website] Seattle: PATH; 2017 (https://www.path.org/publications/
(https://digitalinvestmentprinciples.org/, accessed 17 files/DHS_Data_Use_Culture_wp.pdf, accessed 18
February 2020). February 2020).
13. International Telecommunication Union, Digital Impact 28. Data demand and use. In: MEASURE Evaluation [website].
Alliance. SDG digital investment framework and call to Chapel Hill: Carolina Population Center, University of North
action. Geneva: International Telecommunication Union; Carolina at Chapel Hill (https://www.measureevaluation.
2018 (https://digitalimpactalliance.org/research/sdg- org/our-work/data-demand-and-use/, accessed 18
digital-investment-framework/, accessed 17 February February 2020).
2020).
29. WHO Department of Reproductive Health and Research,
14. Digital health platform handbook: building a digital United Nations Foundation, Johns Hopkins University
information infrastructure (infostructure) for health. Global mHealth Initiative. The MAPS toolkit: mHealth
Geneva: International Telecommunication Union; in press. assessment and planning for scale. Geneva: World
Health Organization; 2015 (https://www.who.int/
15. Digital Square. The global goods guidebook. Seattle: PATH; reproductivehealth/topics/mhealth/maps-toolkit/en/,
2019 (https://digitalsquare.org/global-goods-guidebook/, accessed 18 February 2020).
accessed 18 February 2020).

122 Digital implementation investment guide

30. Standards and Interoperability Lab – Asia. Digital health 44. Mwanyika H. Capturing the end user’s perspective: BID’s
investment: costing tool. Manila: Asian Development Bank; user advisory group. In: BID’s Latest [blog]. BID Initiative;
2020 (http://sil-asia.org/costing-tool/, accessed 17 February 2014 (https://bidinitiative.org/blog/capturing-the-end-
2020). user-perspective-bids-user-advisory-group/, accessed 18
February 2020).
31. CRVS digitisation guidebook: a step-by-step guide to
digitising civil registration and vital statistics processes 45. Njobvu FS. User advisory group established in Zambia. In:
in low resource settings. African Programme for the BID’s Latest [blog]. BID Initiative; 2014 (https://bidinitiative.
Accelerated Improvement of CRVS (http://www.crvs-dgb. org/blog/user-advisory-group-established-in-zambia/,
org/en/, accessed 18 February 2020). accessed 18 February 2020).
32. Common requirements for maternal health information 46. BID Initiative. Stakeholder analysis tool [document].
systems: produced with the Collaborative Requirements Seattle: PATH; 2018 (http://bidinitiative.org/wp-content/
Development Methodology. Seattle: PATH; 2012 (https:// uploads/1.-TOOL_StakeholderAnalysis_FINAL.docx,
path.azureedge.net/media/documents/MCHN_mhis_ accessed 17 February 2020).
crdm.pdf, accessed 18 February 2020).
47. BID Initiative. User advisory group terms of reference
33. Pan American Health Organization. Electronic [document]. Seattle: PATH (http://bidinitiative.org/wp-
immunization registry: practical considerations for content/uploads/13.-TOOL_UserAdvisoryGroup_TOR_
planning, development, implementation and evaluation. FINAL.docx, accessed 17 February 2020).
Washington (DC): PAHO; 2017 (http://iris.paho.org/xmlui/
handle/123456789/34865, accessed 18 February 2020). 48. National health ICT strategic framework 2015–2020.
Government of Nigeria; 2016 (https://www.who.int/goe/
34. World Health Organization, Management Sciences for policies/Nigeria_health.pdf?ua=1; accessed 18 February
Health, KNCV Tuberculosis Foundation. Electronic recording 2020).
and reporting for tuberculosis care and control. Geneva:
World Health Organization; 2012 (WHO/HTM/TB/2011.22; 49. BID Initiative. Product vision for the BID Initiative. Seattle:
https://www.who.int/tb/publications/electronic_ PATH; 2014 (https://www.path.org/publications/files/
recording_reporting/en/, accessed 18 February 2020). VAD_bid_product_vision.pdf, accessed 18 February 2020).

35. Mobile solutions for malaria elimination surveillance 50. Collaborative Requirements Development Methodology
systems: a roadmap. Vital Wave; 2017 (https://vitalwave. (CRDM) [website]. Decatur (GA): Public Health Informatics
com/wp-content/uploads/2017/08/VITALWAVE-BMGF- Institute, The Task Force for Global Health; 2016 (https://
Mobile-Tools-for-Malaria-Surveillance-Roadmap.pdf, www.phii.org/crdm/, accessed 18 February 2020).
accessed 18 February 2020). 51. Business process. In: Business Dictionary [website].
36. Digital Health and Interoperability Working Group. Annual WebFinance Inc.; 2020 (http://www.businessdictionary.
meeting, Washington (DC), 11 December 2019. com/definition/business-process.html, accessed 17
February 2020).
37. IHE quality research and public health (QRPH) white paper:
extracting indicators from patient-level data. Integrating 52. The expanded programme on immunization. Geneva:
the Healthcare Enterprise; in press. World Health Organization; 2013 (http://www.who.int/
immunization/programmes_systems/supply_chain/
38. Vota W. Every African country’s national eHealth strategy or benefits_of_immunization/en/; accessed 18 February
digital health strategy. ICTworks. 4 December 2019 (https:// 2020).
www.ictworks.org/african-national-ehealth.strategy-
policy/#.Xkqhg5NKiu5, accessed 17 February 2020). 53. Optimizing person-centric record systems: a handbook for
digitalizing primary health care. World Health Organization;
39. Directory of eHealth policies. In: Global Observatory for in press.
eHealth [website]. Geneva: World Health Organization;
2020 (https://www.who.int/goe/policies/countries/en/, 54. Determine the root cause: 5 whys. In: iSixSigma [website]
accessed 17 February 2020). (https://www.isixsigma.com/tools-templates/cause-
effect/determine-root-cause-5-whys/, accessed 18
40. National Ebola recovery strategy for Sierra Leone February 2020).
2015–2017. Government of Sierra Leone; 2015 (https://
ebolaresponse.un.org/sites/default/files/sierra_leone_ 55. Universal health coverage (UHC) fact sheet. Geneva: World
recovery_strategy_en.pdf, accessed 18 February 2020). Health Organization; 2018 (https://www.who.int/news-
room/fact-sheets/detail/universal-health-coverage-(uhc)),
41. Detailed meeting report. In: Sierra Leone Health accessed 18 February 2020).
Information Systems Interoperability Workshop, Freetown
(Sierra Leone), 2–4 August 2016:18 (https://www. 56. Mehl G, Labrique A. Prioritizing integrated mHealth
healthdatacollaborative.org/fileadmin/uploads/hdc/ strategies for universal health coverage. Science.
Documents/SL_HIS_Interoperability_Meeting_Report_ 2014;345(6202):1284–7. doi:10.1126/science.1258926.
Final__2_.pdf, accessed 18 February 2020). 57. Nonfunctional requirements. In: SAFe Framework 5.0
42. Digital health: a call for government leadership and [website]. Boulder (CO): Scaled Agile Inc.; 2020 (http://
cooperation between ICT and health. Broadband www.scaledagileframework.com/nonfunctional-
Commission for Sustainable Development; 2017 requirements/, accessed 17 February 2020).
(https://www.broadbandcommission.org/Documents/ 58. Health information systems interoperability maturity
publications/WorkingGroupHealthReport-2017.pdf; toolkit. Chapel Hill (NC): MEASURE Evaluation; 2019
accessed 18 February 2020). (https://www.measureevaluation.org/resources/tools/
43. Digital health convergence meeting toolkit. Asian health-information-systems-interoperability-toolkit/,
Development Bank; 2018 (https://www.adb.org/sites/ accessed 17 February 2020).
default/files/publication/468216/digital-health-converge-
meeting-tool-kit.pdf; accessed 18 February 2020).

References 123
59. Batavia H, Mechael P. Toolkit: assessing the enabling 74. African Union Convention on cyber security and personal
environment for establishing a contextualized national data protection. Addis Ababa: African Union; 2014 (https://
digital health strategy. Washington (DC): United Nations au.int/en/treaties/african-union-convention-cyber-
Foundation; 2016 (http://ict4somlnigeria.info/wp-content/ security-and-personal-data-protection/, accessed 18
uploads/2016/03/Toolkit-assessing-enabling-environment_ February 2020).
FINAL.pdf, accessed 18 February 2020).
75. Software as a medical device (SaMD): key definitions.
60. Information and communication technologies for women’s In: International Medical Device Regulators Forum, 9
and children’s health: a planning workbook. Geneva: December 2013 (IMDRF/SaMD WG/N10FINAL:2013; http://
World Health Organization (https://www.who.int/pmnch/ www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-
knowledge/publications/ict_mhealth.pdf, accessed 18 131209-samd-key-definitions-140901.pdf, accessed 18
February 2020). February 2020).
61. BID Initiative. Implementing solutions. In: The BID Initiative 76. International Medical Device Regulators forum [website]
Story [website]. Seattle: PATH (https://bidinitiative.org/ (http://www.imdrf.org/, accessed 18 February 2020).
story/implementing-solutions/, accessed 17 February 2020).
77. Medical devices: regulations. In: World Health Organization
62. Global Digital Health Network [website]. Baltimore: The [website] (https://www.who.int/medical_devices/safety/
Johns Hopkins University Center for Communication en/, accessed 18 February 2020).
Programs; 2019 (https://www.globaldigitalhealthnetwork.
org/, accessed 18 February 2020). 78. The journey to scale: moving together past digital health
pilots. Seattle: PATH; 2014 (https://www.path.org/
63. Asia eHealth Information Network (AeHIN) [website]. resources/the-journey-to-scale-moving-together-past-
Manila: AeHIN; 2016 (http://www.aehin.org/, accessed 18 digital-health-pilots/, accessed 18 February 2020).
February 2020).
79. Beyond scale: how to make your digital development
64. Accessing the enabling environment for ICTs for health in program sustainable [e-book]. Digital Impact Alliance;
Nigeria: a landscape and inventory. Washington, DC: United 2017 (https://digitalimpactalliance.org/research/beyond-
Nations Foundation; 2014 (http://ict4somlnigeria.info/wp- scale-how-to-make-your-digital-development-program-
content/uploads/2016/03/nigeria-landscape-report.pdf, sustainable/, accessed 18 February 2020).
accessed 18 February 2020).
80. BID Initiative briefs: recommendations and lessons
65. Bangladesh eHealth inventory report. Bangladesh learned: change management. Seattle: PATH; 2017
Knowledge Management Initiative; 2014. (http://bidinitiative.org/wp-content/uploads/VAD_BID_
LessonsLearned_ChangeMngmt_v1_rev03.pdf, accessed 17
66. mHealth in Malawi: landscape analysis. Malawi Ministry of February 2020).
Health and Population; 2018 (https://static1.squarespace.
com/static/548487dce4b08bf981fe60d5/t/5b18fb5f6d2a 81. BID Initiative briefs: recommendations and lessons
73891c7a2a1f/1528363963131/FINAL_malawi_mhealth_ learned: rollout strategy. Seattle: PATH; 2017 (http://
landscape_analysis_May_2018.pdf, accessed 18 February bidinitiative.org/wp-content/uploads/VAD_BID_
2020). LessonsLearned_Rollout_v1_rev04.pdf, accessed 17
February 2020).
67. What are global goods. In: Digital Square Wiki [website].
Washington (DC): Digital Square; 2018 (https://wiki. 82. BID Initiative. Facility and district visit strategy for
digitalsquare.io/index.php/What_are_Global_Goods, introducing data quality and data use interventions
accessed 18 February 2020). [document]. Seattle: PATH; 2018 (http://bidinitiative.org/
wp-content/uploads/2.-TOOL_FacilityVisitStrategy_FINAL.
68. BID Initiative. Equipment support strategy [document]. docx, accessed 17 February 2020).
Seattle: PATH (http://bidinitiative.org/wp-content/
uploads/12.-TOOL_Equipment_Support_Strategy_FINAL. 83. BID Initiative. Spotting and addressing resistance to change
docx, accessed 17 February 2020). [document]. Seattle: PATH (http://bidinitiative.org/wp-
content/uploads/6.-TOOL_Addressing-Resistance_FINAL.
69. Global Digital Health Index Indicator Guide. In: Global doc, accessed 17 February 2020).
Digital Health Index [website] (http://gdhi-showcase-
lb-602552207.us-east-1.elb.amazonaws.com/indicators_ 84. BID Initiative. Change readiness assessment tool for health
info, accessed 18 February 2020). workers [document]. Seattle: PATH (http://bidinitiative.
org/wp-content/uploads/7.-TOOL_Change-Readiness-
70. Marcelo A, Medeiros D, Ramesh K, Roth S, Wyatt P. Assessment-Facilities_FINAL.docx, accessed 17 February
Transforming health systems through good digital health 2020).
governance. Manila: Asian Development Bank; 2018 (ADB
Sustainable Development Working Paper Series, No. 51; 85. BID Initiative. Coaching/supportive supervision job aid
https://www.adb.org/publications/transforming-health- [document]. Seattle: PATH (http://bidinitiative.org/wp-
systems-good-digital-health-governance, accessed 18 content/uploads/11.-TOOL_Coaching_Supervision_Guide_
February 2020). FINAL-1.docx, accessed 17 February 2020).
71. Empel S. Way forward: AHIMA develops information 86. Mobile device management. In: Wikipedia [website]
governance principles to lead healthcare toward better (https://en.wikipedia.org/wiki/Mobile_device_
data management. Journal of AHIMA. 2014;85(10):30–32 management, accessed 18 February 2020).
(https://library.ahima.org/doc?oid=107468 - .XkvrXpNKiu6,
accessed 18 February 2020). 87. Vendor lock-in. In: Wikipedia [website] (https://
en.wikipedia.org/wiki/Vendor_lock-in, accessed 18
72. Risks, harms and benefits assessment tool. In: UN Global February 2020).
Pulse [website] (https://www.unglobalpulse.org/policy/
risk-assessment/, accessed 17 February 2020). 88. Health Level 7. In: Wikipedia [website] (https://
en.wikipedia.org/wiki/Health_Level_7, accessed 18
73. Right to erasure (‘right to be forgotten’). In: General Data February 2020).
Protection Regulation, Chapter 3, Art. 17 (https://gdpr-info.
eu/art-17-gdpr/, accessed 18 February 2020).

124 Digital implementation investment guide

89. What is interoperability in healthcare? In: HIMSS Resource 103. BID Initiative briefs: recommendations and lessons
Library [website]. Chicago: Healthcare Information and learned: data use. Seattle: PATH; 2017 (http://bidinitiative.
Management Systems Society; 2019 (https://www.himss. org/wp-content/uploads/VAD_BID_LessonsLearned_
org/what-interoperability, accessed 18 February 2020). DATAUSE_v1_rev06.pdf, accessed 17 February 2020).
90. mHero [website]. Chapel Hill (NC): IntraHealth 104. BID Initiative. Spot check form [document]. Seattle: PATH
International; 2017 (https://www.mhero.org/, accessed 18 (http://bidinitiative.org/wp-content/uploads/3.-TOOL_
February 2020). SpotCheck_FINAL.xlsx, accessed 17 February 2020).
91. OpenHIE architecture specification, version 2.0. OpenHIE; 105. BID Initiative. Data use culture job aid [document]. Seattle:
2019 (https://ohie.org/architecture-specification/, accessed PATH (http://bidinitiative.org/wp-content/uploads/8.-
17 February 2020). TOOL_Data_Use-Culture_FINAL.docx, accessed 17
February 2020).
92. TOGAF content metamodel [presentation]. In: TOGAF
standard courseware V9 edition, module 7. The Open 106. Kareiva C, Bernadotte C, Gilbert S, Goertz H. Compasses
Group; 2009 (http://www.togaf.info/togaf9/togafSlides9/ when there are no maps: the growing importance of
TOGAF-V9-M7-Metamodel.pdf, accessed 18 February adaptive management to the development sector and the
2020). role of real-time data. Seattle: PATH; in press.
93. OpenHIE architecture. OpenHIE; 2019 (https://ohie. 107. What is adaptive management? In: Learning lab [website].
org/#arch, accessed 17 February 2020). Licence: CC BY 4.0. Washington (DC): United States Agency for International
Development, 2018 (https://usaidlearninglab.org/lab-
94. How to calculate total lifetime costs of enterprise software notes/what-adaptive-management-0, accessed 18
solutions. In: Principles for Digital Development [website] February 2020).
(https://digitalprinciples.org/resource/howto-calculate-
total-cost-enterprise-software/, accessed 18 February 108. BID Initiative briefs: recommendations and lessons
2020). learned: sustainability. Seattle: PATH; 2017 (http://
bidinitiative.org/wp-content/uploads/VAD_BID_
95. Mvundera M, Di Giorgio L, Lymo D, Mwansa FD, Ngwegwe LessonsLearned_Sustainability_v1_rev03.pdf, accessed 17
B, Werner L. The costs of developing, deploying and February 2020).
maintaining electronic immunisation registries in Tanzania
and Zambia. BMJ Global Health 2019;4:e001904. doi:10.1136/
bmjgh-2019-001904.
96. Di Giorgio L, Mvundera M. What are the costs and benefits
of implementation? In: BID’s Latest [blog]. BID Initiative;
2016 (http://bidinitiative.org/blog/what-are-the-costs-
and-benefits-of-implementation/, accessed 17 February
2020).
97. Jones T, Drury P, Zuniga P, Roth S. Digital health impact
framework user manual. Manila: Asian Development
Bank; 2018 (ADB Sustainable Development Working Paper
Series, No. 57; https://www.adb.org/sites/default/files/
publication/465611/sdwp-057-digital-health-impact-
framework-manual.pdf, accessed 18 February 2020).
98. What is RE-AIM. In: RE-AIM [website] (http://www.re-aim.
org/about/what-is-re-aim/, accessed 18 February 2020).
99. Agarwal S, Lefevre A, Labrique A, Mehl G, Cargo M.
Evidence for scale: the development of a monitoring and
evaluation framework for mHealth – MomConnect case
study. London: GSMA; 2014 (https://www.gsma.com/
mobilefordevelopment/programme/mhealth/evidence-
for-scale-the-development-of-a-monitoring-and-
evaluation-framework-for-mhealth-momconnect-case-
study/, accessed 18 February 2020).
100. Immunization data: evidence for action. A realist review
of what works to improve data use for immunization,
evidence from low- and middle-income countries [précis].
Seattle: PATH, Pan American Health Organization; 2019
(https://path.azureedge.net/media/documents/PATH_
IDEA_Precis_R1.pdf, accessed 18 February 2020).
101. Immunization Data: Evidence for Action (IDEA). In:
Technet-21 [website] (https://technet-21.org/en/topics/
idea/, accessed 18 February 2020).
102. Moore M, Faherty LJ, Fischer SH, Bouskill KE, DaVanzo J,
Setodji CM et al. Data and family planning: an evaluation
of two programs of the Bill & Melinda Gates Foundation.
Santa Monica: Rand Corporation; 2018 (https://www.
rand.org/pubs/research_briefs/RB9984.html, accessed 18
February 2020).

References 125
 ANNEXES
Annex 1.1 Glossary
ADAPTIVE MANAGEMENT. The process of building COSTED IMPLEMENTATION PLAN. A document that
in the ability to respond to change using incremental, describes, in sequence, an identified set of challenges,
steady iteration to continually improve a digital health accompanied by a contextually appropriate and
implementation. financially justified mitigation strategy. A costed
implementation plan, or proposal, can be used to obtain
API. Stands for application programming interface.
financial support to implement the proposed activities
A code that allows two software programs to
of a government-driven digital health investment.
communicate with each other. The API defines the
correct way for a developer to write a program that CURRENT STATE. The flow of events that a client
requests services from an operating system or other experiences when seeking or receiving a particular
application.1 health service as they currently occur.

BENEFICIARY. Clients or members of the community DIGITAL HEALTH. Digital health is the systematic
who may benefit from the digital health implementation application of information and communications
when used by another end-user. technologies, computer science, and data to support
informed decision-making by individuals, the health
BOTTLENECK. A specific problem or gap in the delivery
workforce, and health systems, to strengthen resilience
of health services that reduces optimal implementation
to disease and improve health and wellness. 2
of the health programme; may also be referred to as pain
point. A generic or nonprogramme-specific bottleneck is DIGITAL HEALTH APPLICATION. The software, ICT
a health system challenge. system or communication channel that delivers or
executes the digital health intervention and health
CLIENT. An individual who is a potential or current user
content. 3
of health services; may also be referred to as patient or
beneficiary. DIGITAL HEALTH ECOSYSTEM. The combined set of
digital health components representing the enabling
COMMON COMPONENTS. Core functionalities of
environment, foundational architecture and ICT
applications that can be generalized and reused for other
capabilities available in a given context or country.
health programme areas or beyond the health sector;
also called reusable components.

1 Adapted from Digital health terminology guide. AeHIN; 2018 (https://aehin.hingx.org/Share/Details/3819/, accessed 24 January 2019).
2 Consensus definition of digital health, Digital Health and Interoperability Working Group Key terms and theory of change small working group;
[Presentation] 2019 (https://docs.google.com/presentation/d/1TnTFaunk-1WLlG4sKJQ_aSfjmfmivvcENil4mY4XxJs, accessed 18 February 2020).
3 Adapted from Digital health platform handbook: building a digital information infrastructure (infostructure) for health. Geneva: International
Telecommunication Union (in press).

126 Digital implementation investment guide

DIGITAL HEALTH ENTERPRISE. The business processes, DIGITAL HEALTH PLATFORM. A shared digital
data, systems and technologies used to support health information infrastructure (infostructure) on
the operations of the health system, including the which digital health applications are built to support
digital health applications, point-of-service software consistent and efficient healthcare delivery. The
applications, other software, devices, hardware, infostructure comprises an integrated set of common
standards, governance and underlying information and reusable components that support a diverse set of
infrastructure (such as the digital health platform) digital health applications. The components consist of
functioning in a purposeful and unified manner. This software and shared information resources to support
guide distinguishes between four different types of integration, data definitions and exchange standards for
digital health enterprise system architectures along a interoperability and to enable the use of point-of-service
continuum of maturity: siloed, ball of mud, integrated applications across health programme areas and use
and exchanged. cases.6

DIGITAL HEALTH IMPLEMENTATION. The development DIGITAL HEALTH PROJECT. A time-bound

and deployment of digital health application(s), implementation of a siloed digital health application,
platform(s) or enterprise within a given context, usually to demonstrate proof of concept.
accompanied by an operational plan, human resources,
DIGITAL HEALTH STRATEGY. An overarching plan
training and related activities for successful execution.4
that describes high-level actions required to achieve
DIGITAL HEALTH INTERVENTION. A discrete national health system goals. These actions may describe
technology function designed to achieve a specific how new digital health components will be delivered
objective addressing a health system challenge in or how existing components will be repurposed or
order to improve a health programme process and help extended. The foundational building blocks of a digital
strengthen the overall health system. health strategy include infrastructure, legislation,
policies, leadership and governance, standards and
DIGITAL HEALTH INVESTMENT. Financial and other
interoperability, and financing. May also be known as an
resources, including human resources, that are directed
eHealth strategy.
towards digital health implementations. This document
aims to guide the development of a costed digital DIGITAL HEALTH SYSTEM. A digital health system
health investment for an exchanged digital health comprises all of the digital technology used to support
implementation. the operations of the overall health system. Included
in this system are software applications and systems,
DIGITAL HEALTH MOMENT. A point in the process of
devices and hardware, technologies, and the underlying
providing health services where gaps and inefficiencies
information infrastructure.7
occur and at which a digital health intervention can be
applied, providing functionality that can improve the ENABLING ENVIRONMENT. Attitudes, actions, policies
health programme process. and practices that stimulate and support effective
and efficient functioning of organizations, individuals
DIGITAL HEALTH OUTCOME. What will be achieved or
and programmes. The enabling environment includes
changed in the health system or services by using digital
legal, regulatory and policy frameworks and political,
health interventions; may also be known as an eHealth
sociocultural, institutional and economic factors.
outcome. 5
END-USER. An individual, such as a health worker,
manager or client, who interacts directly with the
intervention once implemented.

4 Adapted from Classification of digital health interventions: a shared language to describe the uses of digital technology for health. Geneva: WHO;
2018 (https://www.who.int/reproductivehealth/publications/mhealth/classification-digital-health-interventions/en/).
5 Adapted from National eHealth strategy toolkit. Geneva: World Health Organization and International Telecommunication Unit; 2012
(https://www.itu.int/dms_pub/itu-d/opb/str/D-STR-E_HEALTH.05-2012-PDF-E.pdf, accessed 22 January 2019).
6 Adapted from Digital health platform handbook: building a digital information infrastructure (infostructure) for health. Geneva: International
Telecommunication Union (in press).
7 Adapted from Digital health platform handbook (in press).

Annexes 127
ENTERPRISE ARCHITECTURE. A blueprint of business HEALTH PROGRAMME. Operational unit within a
processes, data, systems and technologies used to help government ministry supporting formal activities
implementers design increasingly complex systems institutionalized at a national or subnational level
to support the workflow and roles of people in a large to address clear priority health objectives. Health
enterprise, such as a health system.8 programmes are government led and persist across
budget cycles as long as the underlying need persists. EPI
EVALUATION. The systematic assessment of an ongoing
and malaria control programmes are some examples of
or completed intervention to determine whether
health programmes.
the intervention is fulfilling its objectives and to
demonstrate an effect on health outcomes.9 HEALTH PROGRAMME PROCESS. A set of activities
involving different personas that is required to
EXCHANGED SYSTEM ARCHITECTURE. A system
achieve an objective or carry out a function of a health
architecture consisting of multiple applications
programme; also referred to as a business process.
connected through a health information exchange
to address collective needs across the health sector, HEALTH SYSTEM CHALLENGE. A generic (not health
operating in a coordinated manner within a digital domain specific) need or gap that reduces the optimal
health enterprise. implementation of health services. Health system
challenges represent a standardized way of describing
FUNCTIONAL REQUIREMENTS. Description of what
bottlenecks.
the digital system needs to do to support the tasks that
make up the health system process and address the INDICATOR. A quantitative or qualitative factor or
identified health system challenges. variable that provides a simple and reliable means to
measure achievement.12
FUNDER. Private foundation, NGO, bilateral or
multilateral agency, or private-sector organization that INTEGRATED SYSTEM ARCHITECTURE. A digital health
provides resources to design, develop and implement enterprise system architecture in which two or more
digital health investments or projects. digital health applications are directly connected to
one another (without an intermediary data exchange),
FUTURE STATE. The desired flow of events where
intended to address one or more health system
the digital health intervention has overcome the
challenges and fulfil health programme goals.
bottlenecks.
INTEGRATION. Integration is the inter-connectivity
HEALTH INFORMATION SYSTEM. A system that
requirements needed for two applications to securely
integrates data collection, processing, reporting, and
communicate data to and receive data from another.
use of the information necessary for improving health
service effectiveness and efficiency through better INTEROPERABILITY. Interoperability is the ability of
management at all levels of health services.10 different applications to access, exchange, integrate and
cooperatively use data in a coordinated manner through
HEALTH MANAGEMENT INFORMATION SYSTEM.
the use of shared application interfaces and standards,
An information system specially designed to assist in
within and across organizational, regional and national
the management and planning of health programmes,
boundaries, to provide timely and seamless portability of
as opposed to delivery of care.11
information and optimize health outcomes.

8 Adapted from Digital health platform handbook (in press).

9 Adapted from WHO evaluation practice handbook. Geneva: World Health Organization; 2013 (http://www.who.int/iris/handle/10665/96311,
accessed 11 September 2019).
10 World Health Organization. Developing health management information systems: a practical guide for developing countries. Manilla: World Health
Organization; 2004 (https://iris.wpro.who.int/bitstream/handle/10665.1/5498/9290611650_eng.pdf)
11 WHO. Developing health management information systems: a practical guide for developing countries
12 WHO evaluation practice handbook, 2013.

128 Digital implementation investment guide

LOGIC MODEL. A graphic depiction of the shared SCENARIO. A set of simple statements that summarizes
relationships between the resources, activities, outputs, what the end-user needs the digital health intervention
outcomes and impact of a health programme, showing to do.
the relationship between the programme’s activities and
SILOED APPLICATION. A stand-alone digital health
its anticipated effect.13
application consisting of one or more digital health
MATURITY MODEL. A framework used in ICTs and interventions to address one or more health system
digital health to help situate implementations based challenges and fulfil the project goals.
on technological, organizational or environmental
STAKEHOLDER. Any person who is affected by or
characteristics.
interested in the consequences of the digital health
MONITORING. The process of collecting and analysing implementation; stakeholders include the planning
routinely collected data to compare how well an team, end-users, beneficiaries and funders.
intervention is being implemented against expected
STANDARD. In software, a standard is a specification
results and measure changes in performance over time.14
used in digital application development that has been
MUD (MONOLITHIC UNARCHITECTED SOFTWARE established, approved and published by an authoritative
DISTRIBUTION). Software characterized by an evolving organization. These rules allow information to be
agglomeration of functions, originating without shared and processed in a uniform, consistent manner
a predetermined scope or design pattern, which independent of a particular application.
accumulate technical debt.15
TASK. A specific action in a health programme process.
NONFUNCTIONAL REQUIREMENTS. General attributes
TECHNICAL DEBT. Technical debt in software
and features of the digital system to ensure usability and
development and systems architecture describes the
overcome technical and physical constraints. Examples
risk of taking shortcuts and making short-term fixes
of nonfunctional requirements include ability to work
(which later require costly revisions and add-ons); rather
offline, multiple language settings, and password
than investing in carefully designed, robust solutions
protection.
(which cost more upfront, but have lower maintenance
PERSONA. A generic aggregate description of a person and feature development costs over time).17
involved in or benefiting from a health programme.
TOTAL COST OF OWNERSHIP. The resources required
PLANNING TEAM. The group of stakeholders to support a digital health intervention throughout its
responsible for guiding the development of the digital life cycle.
health implementation.
WORKFLOW. A visual representation of the progression
REQUEST FOR PROPOSALS (RFP). A document that of activities, events and decision points in a logical flow
solicits proposals, often made through a bidding process, illustrating the interactions within the business process;
by an agency or company interested in procuring a the diagram also maps how information moves through
commodity or service.16 the process and helps visualize where bottlenecks are
occurring.
ROOT CAUSE ANALYSIS. The process of identifying
the underlying factors causing a bottleneck in the
health programme.

13 Adapted from Logic models. In: CDC Approach to Evaluation [website] (https://www.cdc.gov/eval/logicmodels/index.htm).
14 Adapted from WHO evaluation practice handbook, 2013.
15 Adapted from Foote B & Yoder J. Big ball of mud. Pattern Languages of Programs Conference, Monticello (IL), September 1997.
16 Adapted from Request for proposal. In: Wikipedia [website] (https://en.wikipedia.org/wiki/Request_for_proposal, accessed 24 January 2019).
17 Adapted from Technical Debt. Wikipedia [website]. Available from: https://en.wikipedia.org/wiki/Technical_debt#cite_note-2
[Accessed 29 September 2020].

Annexes 129
Annex 1.2 Additional resources for planning and
implementing a digital health enterprise
Phase Steps Resources
» Health information systems interoperability maturity
PHASE 1 Conduct an inventory of existing toolkit: assessment tool. MEASURE Evaluation; 2017
or previously used software (https://www.measureevaluation.org/resources/
Assessing the applications, ICT systems and publications/tl-17-03b).
current state other tools to better understand » Information and communication technologies for
and enabling the requirements for reuse and women’s and children’s health: a planning workbook. WHO
environment interoperability. (https://www.who.int/pmnch/knowledge/publications/
ict_mhealth.pdf?ua=1).
Develop a national digital health
strategy outlining overarching
PHASE 2
needs, desired activities and
Establishing outcomes. » Data Use Partnership: the journey to better data for data
a shared health in Tanzania. PATH; 2016 (http://www.path.org/
understanding and Define a vision for how the health publications/detail.php?i=2734).
strategic planning system will be strengthened
through the use of digital
technology.
» Collaborative Requirements Development Methodology:
Formulate a digital health participant tools. PATH and Public Health Informatics
investment roadmap to support Institute; 2019 (https://www.path.org/resources/
the national digital health strategy. collaborative-requirements-development-methodology-
PHASE 3 participant-tools/).
Plan and identify appropriate » Keisling K. Introduction to mHealth: how to approach
Defining the future digital health interventions, mHealth. 2014 (http://healthenabled.org/wordpress/wp-
state  alongside the health and data content/uploads/2017/09/mhealth_approaches-1.pdf).
content, to improve health » Planning an information systems project: a toolkit for
system processes and address public health managers. WHO and PATH; 2013
programmatic needs. (https://path.azureedge.net/media/documents/TS_opt_
ict_toolkit.pdf).

Review the current state and

develop an architecture blueprint
PHASE 4 for the design of the digital health » Connecting health information systems for better health:
implementations. leveraging interoperability standards to link patient,
Planning the provider, payor and policymaker data. PATH; 2014 (http://
enterprise Identify validated open standards www.jointlearningnetwork.org/resources/connecting-
architecture to ensure data exchange, systems health-information-systems-for-better-health/).
integration and future proofing of
digital health implementations. 
Identify validated health
PHASE 5 content appropriate for the
Determining implementation context.
health content Ensure use of content aligned
requirements with identified standards for the
future state.

130 Digital implementation investment guide

Phase Steps Resources
» Bridging real-time data and adaptive management: ten
lessons for policy makers and practitioners. US Agency for
Monitor your implementation to International Development; 2018 (https://www.usaid.gov/
PHASE 6 digital-development/rtd4am/policy-design-lessons/).
ensure systems are functioning as
M&E of intended and having the desired » Defining and building a data use culture. PATH; 2017
digital health effect. (https://www.path.org/publications/files/DHS_Data_Use_
implementations Culture_wp.pdf).
and fostering Foster data-driven adaptive » Integrating big data into the monitoring and evaluation of
data use change management within the development programmes. UN Global Pulse; 2016 (https://
overall health system. www.slideshare.net/unglobalpulse/integrating-big-data-
into-the-monitoring-and-evaluation-of-development-
programmes).
» Beyond scale: how to make your digital development
program sustainable. DIAL; 2017 (https://
PHASE 7 Maintain and sustain digital health digitalimpactalliance.org/research/beyond-scale-how-to-
Implementing, implementations. make-your-digital-development-program-sustainable/).
maintaining and Identify risks and appropriate » The journey to scale: moving together past digital health
scaling mitigations. projects. PATH; 2014 (https://www.path.org/resources/
the-journey-to-scale-moving-together-past-digital-health-
pilots).

Annexes 131
Annex 2.1 Planning and implementation charter
As you determine roles and responsibilities and develop a common goal, you can you use this charter template to list
out the overall vision, scope, health programmes to be targeted and other key information related to your planning and
implementation efforts. See Chapter 2 for more details.

VISION/OBJECTIVES
A concise description of what outcomes
are expected from the planning and
implementation. Describe how the
organization will benefit at the end of the
project.
BACKGROUND
Current situation that requires a change;
inventory of existing tools and systems;
context diagram that visually represents
the project participants, problems and
opportunities.
FUNCTIONAL SCOPE

A brief description of the main functional

blocks or modules that will be included.

HEALTH AREA SCOPE

Which of the Ministry of Health departments
and programmes will eventually use this
intervention?
Will it include only a subset at first,
and then be expanded?
GEOGRAPHIC SCOPE
Where will the intervention be implemented
over time? Where will it be piloted? Who will
be using it? District people or also at the
health center level?

PARTICIPANTS

List of individuals whose input has been

gathered as part of the scope definition.

TIMING

By when do you expect the intervention to be

operational at the pilot level?
And at scale?

132 Digital implementation investment guide

Annex 2.2 Persona worksheet
This persona worksheet can be used to think through the different potential end-users and better understand their
needs and challenges as you design the digital health implementation. Use this persona worksheet as a starting point;
you may conduct more detailed persona descriptions as you engage further with the end-users. See Chapter 2 for more
details.18

Demographics Name, Photo and Type of Persona

» Gender » Name (can be real or illustrative)
» Age » Photo of persona to help with visualization and storytelling
» Community
» Language(s) used
Roles and Responsibilities

Context Description
» Does this end-user own a digital device? Is yes, what kind?
» Level of familiarity with digital tools?
» Rural or urban?
» Internet connectivity?
» Availability of electricity and water?
» Homogeneous or heterogeneous population?
» Distance to nearest health facility?
Challenges
» What are the routine challenges this end-user faces?
» Long distances travelled without reliable mode of transportation?
» Sufficient training and performance monitoring?
» Workload?
» It would be beneficial to include quotes given directly in interviews for the persona you are creating
What does success look like from the perspective of the persona?
» What are their motivations?

For example: When clients are happy with the services? Not having to wait a long time before seeing a health worker?

18 For more examples of persona-mapping templates, please see Demand for health services workbook: a human-centred approach. UNICEF
(http://hcd4i.org/wp-content/uploads/2018/10/unicef_digitalhealthinterventions_final2018-1.pdf).

Annexes 133
Annex 3.1 Process matrix worksheet
This process matrix worksheet will allow you to identify the different health programme processes and tasks, so you
can map the workflow and identify bottlenecks within these processes. See Chapter 3 for more details.

Health system
# Process Objective Task set Outcomes Bottlenecks challenge
A Antenatal To provide timely » Assess » Women receive » Health worker » Poor adherence
care and appropriate pregnant appropriate did not check to guidelines
referral referrals to a woman’s care and are for all the (HSC 3.7)
higher level facility danger signs. referred in a danger signs. » Insufficient
Illustrative example

or healthcare » Stabilize timely manner. » Health worker health worker

provider. pregnant was unaware competence
woman’s of whether or (HSC 3.2)
conditions. where to refer. » Geographic
» Arrange » Referral facility inaccessibility
emergency is too far away. (HSC 5.2)
transport.

134 Digital implementation investment guide

Annex 3.2 Worksheet for mapping bottlenecks to
health system challenges
This worksheet continues from Annex 3.1, where you identified key tasks within the targeted processes, as well as
possible bottlenecks. Use this worksheet to elaborate on the different bottlenecks within your workflows and begin
determining which ones you will address. See Chapter 3 for more detail.

UHC Layer Illustrative bottleneck Health system Rank

affected challenge by stakeholder team
» “Health workers do not show at their health 3.4 Low health worker
posts/facilities.” motivation
EFFECTIVE
» “Health workers turn away clients.”
COVERAGE/
QUALITY » “There is a high attrition of health workers.”
» “It is hard to retain health workers.”
» “Health workers do not counsel clients 3.7 Poor adherence to
appropriately.” guidelines
» “Health workers are not following their
counselling/treatment protocols.”
» “Health workers are not sure of what services to
provide.”
» “The guidelines are difficult to interpret for
health workers.”
» “Clients do not receive care on time.” 6.4 Delayed provision of
» “Clients have to wait a long time before receiving care
appropriate treatment.”
» “Clients do not continue taking their medication.” 5.3 Low adherence to
» “Clients do not complete the recommended treatments
CONTINUOUS
number of visits/treatments.”
COVERAGE
» “Clients do not return for [XX] services and 5.4 Loss to follow-up
appointments.”
» “Clients discontinue or drop out of services.”
» “Health workers are unable to trace clients.”
» “Clients move around a lot and do not come back
to the same place for services.”
» “Clients do not want to use [XX] service.” 5.1 Low demand for
» “Clients do not know where they can access [XX] services
CONTACT
service.”
COVERAGE
» “Services are too far for clients.” 5.2 Geographic
» “Roads are inaccessible during rainy season [or inaccessibility
other period].”
» “There is a lot of stigma with accessing [XX] 4.1 Lack of alignment
service.” with local norms
» “Clients feel afraid/ashamed to seek [XX] service.”

Annexes 135
Annex 5.1 Questions for software developers
As you select the specific applications that you will use in the digital health implementation, it is critical to have some
background information on the level of support that software developers will provide, as well as the capabilities of the
software application. This worksheet provides key questions that can be used to guide your discussion with a particular
software developer. This worksheet could also be used as a guide for creating a proposal scoring rubric.19

Question Reasoning
What is your largest implementation?
How many end-users are part of that Determine if the vendor has experience or evidence that they are able to
implementation? How many records are in support the size of your desired implementation.
that database?
If your end-users typically access the system and provide all of their reports
How many end-users can use the software
on Friday afternoons, you do not want the system to fail or have very poor
at the same time?
performance during those times.
What components of the proposed To follow a principle such as technology independence, knowing
intervention use proprietary software; the licensing requirements early is important. For system maintenance,
commercial, off-the-shelf software; open knowing the underlying technology and corresponding robustness of either
source software? the software provider or the open-source community can be important.
What service-level agreement for uptime What amount of time is tolerable for the system to be unavailable? A total
do you guarantee each month? How of 95 percent uptime translates to eight hours each week. Usually, the
many hours of maintenance is the system
vendor will apply security updates to the software on your behalf. Yet, you
unavailable each month and when are
those typically scheduled? would not want this to occur during key periods of system use.

How would you integrate with our health If an integrated system is a key principle, knowing that the application
information system? Can you provide has a demonstrated architecture for data exchange is necessary. If the
examples of how you have done this integration has never been done before, it may be considered an unsupported
before? customization that requires ongoing maintenance fees.
Data security is critical for health information systems. Information such
How do you safeguard the security and as patient records can be hard to replace and should not fall into the wrong
privacy of our data? What were the results
hands. Data can be lost due to disasters such as a flood or fire but also to
of your most recent external audit?
hackers or a malware infection.
How often would our data be backed up?
Can you provide your disaster recovery If data are an asset, knowing that the vendor has processes to store and
plans? When was the last exercise and restore your system in the event of an emergency is important.
what were the results?
Early clarification of roles and responsibilities for deploying the software is
What training and support services do you needed to understand the overall costs. Training end-users is often a large
provide? What times are support services part of the deployment budget. Sometimes, the vendor will provide training
available? for administrators and train your trainers. Do your normal hours of operation
coincide with the support hours provided?
For ease of use, the system user interface should be in the language of your
What languages does your software
end-users. If the language is not currently supported, ideally the vendor has
support?
capabilities that allow you to localize the various terms.
What are the annual maintenance and Sometimes hidden fees obscure the true costs of the system. Maintenance
licensing fees? How much are these fees fees of upwards of 20 percent of the software license may be required when
expected to increase annually? the contract is signed.
What interoperability standards does your
system support? Have you demonstrated It is valuable to understand the software vendors attention to data
conformity with specific standards? With exchange standards, and their ability to implement the software within a
which use cases and other systems have broader digital health enterprise environment using standards.
you achieved interoperability?

19 Source: Planning an information systems project: a toolkit for public health managers, Annex 7. PATH; 2013 (https://path.azureedge.net/media/
documents/TS_opt_ict_toolkit.pdf).

136 Digital implementation investment guide

Annex 5.2 Implementation considerations
summary template
Using the following worksheet, compile your implementation considerations for each digital health intervention,
taking into account the different factors within the enabling environment.

See Chapter 5 for more detail, as well as the Who Guideline: Recommendations on Digital Interventions for Health System
Strengthening for implementation considerations for selected interventions.

Enabling environment Implementation considerations Implementation considerations

factors for digital health intervention 1 for digital health intervention 2
1. STRATEGY AND
INVESTMENTS

2. INFRASTRUCTURE

3. LEGISLATION, POLICY AND

COMPLIANCE

4. LEADERSHIP AND
GOVERNANCE

5. WORKFORCE

6. SERVICES AND
APPLICATIONS

7. STANDARDS AND
INTEROPERABILITY

8. HEALTH CONTENT

Annexes 137
Annex 5.3 Implementation considerations for
specific digital health interventions
This annex describes specific implementation considerations for a selected set of digital health interventions. These
specific considerations are in addition to the factors discussed in Chapter 5 that relate to general implementation
considerations for all interventions.

The interventions listed in this appendix are based on the interventions that were prioritized in the WHO Guideline:
Recommendations on Digital Interventions for Health System Strengthening, which include:
» birth and death notification
» stock notification and commodity management
» client-to-provider telemedicine
» provider-to-provider telemedicine
» targeted client communication
» health worker decision support
» digital health record for tracking of patients’/clients’ health status and services
» provision of training and educational content to health workers via mobile devices (mLearning).

INTERVENTION 1: BIRTH AND DEATH NOTIFICATION

DESCRIPTION
OF THE DIGITAL Digital approaches to support the notification of births and deaths in order to trigger subsequent steps
HEALTH of birth registration and certification and compile vital statistics20
INTERVENTION
» Lack of access to information or data (lack of reporting of events)
COMMONLY
» Lack of population denominator
ASSOCIATED
» Lack of quality/reliable data
HEALTH SYSTEM
CHALLENGES » Inadequate understanding of beneficiary populations
» Lack of unique identifier
» Align to national policies and laws for legal identity, as well as issuance of unique IDs. This
intervention should strengthen the entire CRVS and avoid developing systems that do not link to
health services or CRVS systems. For example, is the health system allowed to notify about vital
CONSIDERATIONS events?
BEFORE
» Align to national policies and laws around electronic storage of data, data privacy, data
DEPLOYING
protection and so on.
» Explore sociocultural barriers associated with communicating about births/deaths and address
the way these dynamics will influence notifications via digital devices.

20 Adapted from Classification of digital health interventions: a shared language to describe the uses of digital technology for health. Geneva: WHO;
2018 (https://www.who.int/reproductivehealth/publications/mhealth/classification-digital-health-interventions/en/).

138 Digital implementation investment guide

 » Consider mechanisms to ensure the completeness of the data and whether demand-generation
activities are needed to incentivize reporting by explaining its benefits. Implementers should
be aware, however, of any reporting targets placed on health workers and ensure that birth and
death data are validated before being released to the civil registration system.
CONSIDERATIONS » Consider processes for deduplication/validation of the notification. Strengthen processes to
DURING ensure that all notifications lead to both legal registration and issuance of certificates.
DEPLOYMENT » Consider how best to ensure the quality and timeliness of birth and death data, such as by
checking on low performers identified through digital performance data or spot checks. Other
ways to help improve data quality include standardizing the definitions associated with reporting
birth and death events, such as for stillbirths, and making these definitions accessible to those
inputting the data.
OPPORTUNITIES
FOR INTEROPERA-
Consider linking birth notification to health services that have high coverage, such as immunization
BILITY AND LINK-
services or health facilities that offer very high rates of institutional delivery. It is important, however,
AGES TO OTHER
to consider whether the civil registration system can absorb an increase in notifications.
DIGITAL HEALTH
INTERVENTIONS
There is a risk in overinvesting in notifications without a robust civil registry system on the back end.
Without this, notifications will have no legal value, and the data are less likely to be properly stored
and managed.
Implementers should note that increases in notifications will in turn require that the health system
RISKS AND and civil registration services be prepared to absorb higher demand for registration. This is a potential
MITIGATION bottleneck in the registration and validation process and could deter populations from continuing
STRATEGIES notifications. Consider how to improve accessibility and shorten the connection between the health
workers or communities providing the notifications and the CRVS sector undertaking the registration.
Data on births/deaths may be particularly vulnerable for financial value (for example, information used
for advertising and marketing). Establish mechanisms to preserve data privacy, ownership, access,
integrity and protection of patient information.
CRVS digitisation guidebook: a step-by-step guide to digitising civil registration and vital statistics
ADDITIONAL
processes in low resource settings. African Programme for the Accelerated Improvement of CRVS
RESOURCE
(http://www.crvs-dgb.org/en/).

Considerations for cost categories for birth/death notification

Cost category Description

ONGOING/ALL PHASES

Governance Personnel for partnership building and coordination meetings to align with stakeholders (such as
Technical Working Group members, implementing partners and MNOs)
Meeting costs (e.g. transportation, personnel time)
Management and Personnel to oversee overall programme
staffing

DEVELOPMENT AND SETUP

Outreach and raising Dissemination to the community about the intervention and how to make notifications, which
awareness may be conducted by outreach through community health workers, pamphlets, billboards and/or
mobile message blasts
Personnel for system setup and end-user support

Annexes 139
 DEPLOYMENT

Equipment/ hardware Devices (such as mobile phones, tablets and computers) used by key informants for conducting
birth notifications
Setup of cloud hosting or physical server, which would require physical and virtual security and
authentication
Initial training Development/adaptation of training curricula and standard operating procedures, which can
include materials for train-the-trainer approaches
Training on standard operating procedures for the recipient of the birth/death notifications (health
workers and civil registrar personnel)

INTEGRATION AND INTEROPERABILITY

Content adaptation Development/adaptation of content and requirements for the registration system; for death
registration, this may require mapping to processes for death certificates, death surveillance and
ICD codes, 21 as well as requirements for insurance closure and social protection mechanisms
Design of technology architecture to link the notification with the birth registration system or with
health records; for death registration, this may require linkages to verbal autopsy systems
Review and incorporation of policies related to identity management and civil registration,
including for obtaining unique identifiers
Technology Software customization of the digital system for completing birth registration information,
adaptation including generation of unique identifiers
Embedding of security features, such as authorization for end-user access control and data
encryption to ensure protection of data
Definition of integration or interoperability requirements, including data definition and message
formats
Software linkage between birth registration application and the health record, ideally using a
unique identifier, such as a unique personal ID (e.g. a national ID number)
Human resources Additional personnel for increased coordination with partners to follow up on software integrations
and governance

SCALE

Training and adaptive Training for additional personnel interacting with the birth registration software system
management Additional training for supervisory personnel on continuous monitoring of the system at scale
Additional training for ICT support staff to provide end-user support, troubleshooting, backup and
recovery at scale
Periodic review meetings to discuss feedback on system performance and challenges

SUSTAINED OPERATIONS

Refresher training and Additional personnel to ensure ongoing maintenance of the integrated system and integration of
adaptive management data
Refresher training or continued community outreach to facilitate uptake of notification processes
Periodic review meetings to discuss feedback on system performance and challenges
Incentives for reporting birth and death notifications, particularly if relying on community
members and key informants for the notifications

21 See International classification of diseases. WHO (http://www.who.int/classifications/icd).

140 Digital implementation investment guide

 Communication/ SMS text message, USSD voice call and/or data transmission charges based on volume of
data exchanges communication content and communication channel; note that both SMS and USSD are unsecure
channels and that, unlike with USSD, with SMS an unencrypted record will remain on the sender’s
phone, creating an additional risk
Short code maintenance fee, which represents a simplified number for clients to use when
registering for the service
Aggregator maintenance fees, which enable communication across multiple network carriers
Technology Data hosting (such as server maintenance or cloud-hosting fees)
maintenance Software maintenance, licensing and upgrade fees
Hardware maintenance, including insurance and repair/replacement of hardware

INTERVENTION 2: STOCK NOTIFICATION AND COMMODITY MANAGEMENT

DESCRIPTION Digital approaches for monitoring stock levels and distribution of medical commodities, which can
OF THE DIGITAL include using communication systems (such as SMS) and data dashboards to manage and report on
HEALTH supply levels of medical commodities22
INTERVENTION
COMMONLY » Insufficient supply of commodities (which could be attributed to wastage of expired stocks due
ASSOCIATED to lack of good planning, forecasting and redistribution systems)
HEALTH SYSTEM » Geographic inaccessibility
CHALLENGES » Lack of effective resource allocation
» Lack of transparency in commodity transactions
» Delayed reporting of events
CONSIDERATIONS Consider the need for training at all levels of the health system, including training of health workers
BEFORE to send stock reports, of support staff (such as cold-chain technicians) to manage stock and of facility
DEPLOYING workers to assess stock levels.
Reinforce training by the basic processes of inventory management and stock distribution. Since
management staff at national and subnational levels make decisions according to the data on whether
or not to supply health facilities and health workers with stock replenishments, introducing the digital
system should also be accompanied by refresher training on the basic processes of supply chain
management.
CONSIDERATIONS » Ensure that the digital systems and ordering routines are flexible enough to respond to local
DURING needs. For instance, where systems deal with quarterly stock orders, ensure that they can also
DEPLOYMENT accommodate unexpected or seasonal needs.
OPPORTUNITIES Prioritize integrating notifications with existing data reporting systems, including national or
FOR INTEROPERA- subnational information management systems where available, such as supply chain, logistics and
BILITY AND LINK- warehouse management information systems.
AGES TO OTHER Consider integrating the stock notification system with a data dashboard that displays the
DIGITAL HEALTH notification, receipt of commodity at the station and action taken, among other data, to ensure
INTERVENTIONS transparency.
RISKS AND The digital reporting of stock levels will introduce a level of transparency in commodity transactions
MITIGATION that may be new to the health system. Ensure that there is no harm or reprisal to health workers for
STRATEGIES reporting stockouts or wastage; instead, emphasize explaining the benefits of reporting stockouts so
that they can be addressed. To motivate continued reporting, ensure that some action is possible when
stockouts are reported.
ADDITIONAL Critical success factors for deploying digital LMIS. John Snow, Inc. (https://www.jsi.com/JSIInternet/Inc/
RESOURCE Common/_download_pub.cfm?id=18286&lid=3).

22 Adapted from Classification of digital health interventions, 2018.

Annexes 141
Considerations for cost categories for stock notification

Cost category Description

ONGOING/ALL PHASES

Governance Personnel for partnership building and coordination meetings to align with stakeholders (such as
Technical Working Group members, implementing partners and MNOs)
Meeting costs (e.g. transportation, personnel time)
Management and Personnel to oversee overall programme
staffing Personnel for system setup and end-user support (such as monitoring stability of software and
troubleshooting system failures)
Personnel to monitor data generated by the system and provide feedback, corrective actions and so on.

DEVELOPMENT AND SETUP

Content adaptation Defining list of commodities to be monitored and mapping their identification codes to global
standards
Human-centred design process to define requirements within appropriate context, including
mapping business processes, understanding personas of intended end-users and documenting
functional and nonfunctional requirements
Development/adaptation of dashboards for monitoring data collected by the system
Technology Software customization to adapt the stock notification system to the commodities that need to
adaptation be tracked and thresholds for notifying stockouts (such as commodities for notification or logic of
when to trigger a notification)
Dashboards for monitoring the performance of the system and visualizing aggregated data
End-user testing among targeted populations to ensure optimal end-user experience
Refinement of the intervention in response to feedback from end-user testing to ensure that
requirements and context are taken into account

DEPLOYMENT

Equipment/ hardware Devices (such as, mobile phones and tablets) for operating the stock notification system and for
health workers to use to track commodity levels
Server/cloud for storing data generated by the system, which includes ensuring there is a locked,
air-conditioned physical space for a server; some contexts may store data in a cloud, in which case
a physical server may not be required
Computers at the district and/or national level for monitoring system performance and viewing
reporting dashboards
Initial training Development/adaptation of training curricula and standard operating procedures for using the
system
Initial training for health workers interacting with the system
Training for supervisory staff on standard operating procedures and continuous monitoring

INTEGRATION AND INTEROPERABILITY

Technology Design of technology architecture to link the notification with the broader LMIS
adaptation Software integration with broader LMIS
Human resources Additional personnel to define interoperability requirements and data exchanges
Additional personnel to ensure the ongoing maintenance of the integrated system
Additional personnel for increased coordination with partners to follow up on software integrations
and governance

142 Digital implementation investment guide

 SCALE

Training and adaptive Additional training for personnel interacting with the LMIS
management Additional training for supervisory personnel on continuous monitoring of the LMIS
Additional training for ICT support staff to provide end-user support, troubleshooting, backup and
recovery

SUSTAINED OPERATIONS

Refresher training and Refresher training for health workers interacting with the system
adaptive management Refresher training for supervisory staff on continuous monitoring and use of data emerging from
the system
Periodic review meetings to discuss feedback on system performance and challenges
Communication/ SMS, voice call and/or data transmission charges for submitting data on stock levels
data exchanges
Technology Software maintenance and licence fees
maintenance Hardware maintenance, including insurance and repair/replacement

INTERVENTION 3: CLIENT-TO-PROVIDER TELEMEDICINE

DESCRIPTION The delivery of healthcare services where clients/patients and health workers are separated by
OF THE DIGITAL distance23
HEALTH
INTERVENTION
COMMONLY » Geographic inaccessibility
ASSOCIATED » Insufficient (coverage) supply of qualified health workers
HEALTH SYSTEM » Delayed provision of care
CHALLENGES
» Inadequate access to transportation
» Client-side expenses
CONSIDERATIONS » Determine the mechanisms for outreach and raising awareness about this intervention, such as
BEFORE through mass media communication, community outreach and so on.
DEPLOYING » Clarify clinical protocols to explain what can and cannot be done in the remote consultation. For
example, determine what type of cases still warrant face-to-face contact. Consider whether it
is possible or desirable for clients to meet health workers in person before making connections
over digital services.
» Involve the relevant professional bodies as well as the health workers and clients in the planning,
design and implementation of the telemedicine programme to ensure that their needs and
concerns are met, such as to educate health workers on the legal frameworks governing
telemedical exchanges.
CONSIDERATIONS » Ensure that use of the technology does not negatively affect the relationship between the
DURING patient and health worker, particularly when end-users are learning about the technology and
DEPLOYMENT how to operate the devices.
» Pay special attention to the needs, preferences and circumstances of particularly disadvantaged
or hard-to-reach groups, including people with low literacy or few digital literacy skills and
people with limited control over or access to mobile devices.
» Consider how services can be made available to people with disabilities, such as sight or hearing
impairments, or with poor access to electricity or poor network coverage. Strategies to increase
access to telemedicine in these cases may include providing public kiosks, for example.

23 Adapted from Classification of digital health interventions, 2018.

Annexes 143
OPPORTUNITIES Integrate with provider-to-provider telemedicine in cases where referral to another health worker is
FOR INTEROPERA- required.
BILITY AND LINK- Link with targeted client communication to follow up on clients/patients following the consultation.
AGES TO OTHER
DIGITAL HEALTH
INTERVENTIONS
RISKS AND There may be risks with unaccredited or unlicensed health workers using a client-to-provider
MITIGATION telemedicine system. Establish a clear legal framework for implementing telemedicine, including
STRATEGIES licensing and regulation of health workers using it.
Ensure that there is capacity for and a plan to respond to calls requiring an emergency response.
It may be hard to predict adoption and growth. Prediction and usage modelling needs to be in place,
with plans and resources to scale, if required.
ADDITIONAL Telemedicine toolkit. Novartis Foundation (https://www.novartisfoundation.org/telemedicine-toolkit).
RESOURCES Framework for the Implementation of a Telemedicine Service. WHO PAHO
(https://iris.paho.org/handle/10665.2/28414)

Considerations for cost categories for client-to-provider telemedicine

Cost category Description

ONGOING/ALL PHASES

Governance Personnel for partnership building and coordination meetings to align with stakeholders (such as
Technical Working Group members, implementing partners and MNOs)
Meeting costs (transportation, personnel time)
Management and Personnel to oversee overall programme
staffing Clerical staff to answer and triage incoming calls (may not be necessary if clinical staff can also do
the call intake)
Clinical staff to provide consultations or refer to a specialist, if needed, which may be particularly
expensive if the service needs to be available 24-7
Access to referral specialists in cases requiring expertise not currently provided by available clinical
staff (such as dermatology or radiology)
Personnel for routine monitoring of system performance, including tracking of dropped calls and
use of the service
Personnel for system setup and end-user support

DEVELOPMENT AND SETUP

Outreach and raising Development of materials on how to access the intervention (such as pamphlets and billboards
awareness displaying the number to dial)
Dissemination to clients about the intervention (such as messages sent to a phone bank of
numbers to communicate availability of the telemedicine service)
Technology Software customization for communication and exchanging health content based on the
adaptation modalities/communication channels to be used, such as video conferencing, transmission of data/
images, voice calls and so on
Security features, such as end-user authentication schemes when recording callers’ demographic
and health information

144 Digital implementation investment guide

 DEPLOYMENT

Equipment/ hardware Computer with dedicated software system for audio and/or video connections for health workers
to conduct the consultation
Audio- or videoconferencing equipment, which may include headsets and trunk lines (central
lines that can direct voice calls, images and video to multiple lines and across different network
operators)
Initial training Development/adaptation of training protocols and standard operating procedures, including call
intake, consent and referral processes
Initial training to health workers on how to use the telemedicine system

INTEGRATION AND INTEROPERABILITY

Technology Design of technology architecture to link the telemedicine system with other interventions, such
adaptation as targeted client communication
Software customization to reflect integration
Human resources Additional personnel to define interoperability requirements and data exchanges
Additional personnel to ensure the ongoing maintenance of the integrated system
Additional personnel for increased coordination with partners to follow up on software integrations
and governance

SCALE

Training and adaptive Additional training for health workers conducting the telemedicine
management Additional training for supervisory personnel on continuous monitoring
Additional training for ICT support staff to provide end-user support, troubleshooting, backup and
recovery
Periodic review meetings to discuss feedback on system performance and challenges

SUSTAINED OPERATIONS

Refresher training and Refresher training and continuous support to health workers on how to use the telemedicine
adaptive management system
Periodic review meetings to discuss system performance and workflow integration
Communication/ Airtime and/or transmission of data files, depending on the volume and modality of the client-to-
data exchanges provider communication (modalities/communication channels may include videoconferencing,
transmission of data or images, web-based platforms, voice calls and interactive voice response;
the caller may incur these costs unless there are provisions for the service to be toll-free, enabling
costs to be absorbed by the organization/facility providing the remote consultation)
Support line for client experiences and feedback
Technology Software maintenance and licence fees
maintenance Hardware maintenance, including insurance and repair/replacement of hardware

Annexes 145
 INTERVENTION 4: PROVIDER-TO-PROVIDER TELEMEDICINE

DESCRIPTION The delivery of healthcare services where two or more health workers are separated by distance, often
OF THE DIGITAL a lower level health worker consulting with a specialist or more skilled health worker24
HEALTH
INTERVENTION
COMMONLY » Insufficient supply of qualified health workers
ASSOCIATED » Insufficient (coverage) supply of services
HEALTH SYSTEM » Geographic inaccessibility
CHALLENGES
» Insufficient health worker competence
» Lack of or inappropriate referrals
» Delayed provision of care
» Inadequate access to transportation
CONSIDERATIONS » Develop protocols to educate health workers on the use of the technology.
BEFORE » Explore whether changes to licensing and legislation are necessary to support any changes in
DEPLOYING health workers’ scopes of practice.
CONSIDERATIONS » Ensure that the distribution of roles and responsibilities among different health workers is clear,
DURING including through regulations and job descriptions.
DEPLOYMENT » Explore whether changes to salaries or incentives for health workers are needed to reflect any
changes in scopes of practice.
» Build trust between professionals who are considering establishing links between facilities
across institutions, such as through twinning programmes.
OPPORTUNITIES Use master facility lists/registries and health worker registries to facilitate information exchange
FOR INTEROPERA- across facilities and health workers, respectively.
BILITY AND LINK- Link with digital client records through unique identifiers in order to have the patient/client history
AGES TO OTHER during the consultation.
DIGITAL HEALTH
INTERVENTIONS
RISKS AND Clarify liability issues for health workers using telemedicine systems and determine what can and
MITIGATION cannot be done during remote consultations; the approach should not be a substitute for the adequate
STRATEGIES training of health workers.

Considerations for cost categories for provider-to-provider telemedicine

Cost category Description

ONGOING/ALL PHASES

Governance Personnel for partnership building and coordination meetings to align with stakeholders (such as
Technical Working Group members, implementing partners and MNOs)
Management and Personnel to oversee overall programme
staffing Health worker providing the assistance with clinical case, which may be particularly expensive if
the service needs to be available 24-7
Referral providers/specialists (such as dermatology or radiology) providing the consultations
Personnel for system maintenance and end-user support

DEVELOPMENT AND SETUP

Outreach and raising Dissemination to health workers about the telemedicine service
awareness

24 Adapted from Classification of digital health interventions, 2018.

146 Digital implementation investment guide

 Technology Software customization for communication and exchanging health content, which may be based
adaptation on the modalities/communication channels to be used for videoconferencing, transmission of data
or images and voice calls
Security features, such as end-user authentication schemes, when relaying clients’ health
information
End-user testing among health workers to ensure optimal end-user experience and alignment with
workflows
Refinement in response to feedback from end-user testing to ensure that requirements and
context are taken into account

DEPLOYMENT

Equipment/ hardware Computer with dedicated software for audio and/or video connections for health workers to
conduct the consultation
Audio- or videoconferencing equipment, which may include headsets and trunk lines (central
lines that can direct voice calls, images and video to multiple lines and across different network
operators)
Database to log all incoming calls, audio and images
Server/cloud for storage of recorded calls, audio and images, including a locked, air-conditioned
physical space for the server; some contexts may store data in the cloud, which would require
cloud-hosting fees
Initial training Development/adaptation of training protocols and standard operating procedures, including
referral processes
Initial training of health workers on how to use the telemedicine system

INTEGRATION AND INTEROPERABILITY

Technology Design of technology architecture to link the telemedicine system with other interventions
adaptation Software customization and incorporation of data-exchange mechanisms
Human resources Additional personnel to define interoperability requirements and data exchanges
Additional personnel to ensure the ongoing maintenance of the integrated system
Additional personnel for increased coordination with partners to follow up on software integrations
and governance

SCALE

Training and adaptive Additional training for health workers conducting the telemedicine
management Additional training for supervisory personnel on continuous monitoring
Additional training for ICT support staff to provide end-user support, troubleshooting, backup and
recovery
Periodic review meetings to discuss feedback on system performance and challenges

SUSTAINED OPERATIONS

Refresher training and Refresher training to health workers on how to use the telemedicine system
adaptive management Periodic review meetings to discuss system performance and workflow integration
Communication/ Airtime and/or transmission of data files, depending on the volume and modality of the provider-
data exchanges to-provider communication
Technology Software maintenance, updates and licence fees
maintenance Hardware maintenance, including insurance and repair/replacement of hardware

*See resources under Client-to-provider telemedicine for additional resources

Annexes 147
 INTERVENTION 5: TARGETED CLIENT COMMUNICATION

DESCRIPTION Transmission of customized health information for different audience segments (often based on health
OF THE DIGITAL status or demographic categories), which may include transmission of
HEALTH 1. health-event alerts to a specific population group;
INTERVENTION 2. health information based on health status or demographics;
3. alerts and reminders to clients; and
4. diagnostic results (or the availability of results)25
COMMONLY » Low demand for services
ASSOCIATED » Low adherence to treatments
HEALTH SYSTEM » Loss to follow-up
CHALLENGES
» Insufficient patient engagement
» Unaware of service entitlement
» Lack of access to information
» Lack of alignment with local norms (stigma)
CONSIDERATIONS » Determine the mechanisms to enrol the targeted population in the service, such as through
BEFORE health appointments, advertised short codes, community outreach and so on.
DEPLOYING » Ensure that clients are actively made aware of how to opt out of receiving the targeted client
communication. Attention needs to be paid to clearly communicating consent procedures to
clients. Inform clients on the intended uses of their data, including to enable subsequent further
contact with them and over what period of time, and their right to be forgotten/opt out.
» Ensure that the content of the communication accurately reflects the reality of the available
commodities and services. For example, encouraging women to seek family planning at their
nearest health facility is appropriate if a full range of contraception and advice is available there,
including the relevant commodities.
» Consider testing to ensure that the messages are understood as intended and that any necessary
colloquial translations are used. Consider the languages used for the content to reach the target
audiences, including whether they are in active spoken or written use. Also consider anti-spam
regulations and test that messages are not caught in spam filters.
» Consider whether to include two-way communication with clients to enable their interaction
and response to the health system.
CONSIDERATIONS Pay attention to the circumstances of people who have poor access to electricity or poor network
DURING coverage, people who cannot afford a mobile device or voice and data charges, and people who have
DEPLOYMENT limited autonomy, because their access to phones is controlled by another person, for example.
Give particular attention to the needs, preferences and circumstances of especially disadvantaged or
hard-to-reach groups, including people with low literacy or few digital literacy skills, people speaking
minority languages, migrant populations in new settings, people affected by emergency situations and
people with disabilities, such as sight or hearing impairment.
Ensure that any sensitive content or personal data transmitted and stored are held on a secure server
with protocols in place for destroying the data when appropriate.
OPPORTUNITIES Link with the digital health record as a mechanism to tailor messages and content delivered to clients.
FOR INTEROPERA- Link with personal health tracking interventions, such as “access by client to own medical record” and
BILITY AND LINK- “self-monitoring of health/diagnostic data client.”
AGES TO OTHER
DIGITAL HEALTH
INTERVENTIONS
RISKS AND There is a risk of disclosing sensitive health content, particularly in the context of shared phones
MITIGATION or when individuals do not have full access to their devices. Consider any demographic or health
STRATEGIES characteristics that could put the targeted population at greater risk and ensure that the way the
information is provided and accessed is sensitive to this. Procedures need to be in place to ensure that
individuals are not unduly pressured to provide personal information.

25 Adapted from Classification of digital health interventions, 2018.

148 Digital implementation investment guide

ADDITIONAL mHealth mobile messaging toolkit. PATH; 2014 (https://www.path.org/publications/files/TS_mhealth_
RESOURCES mobile_messaging_toolkit.pdf).
Making content meaningful: a guide to adapting existing global health content for different audiences.
Johns Hopkins Center for Communication Programs; 2014 (https://www.kmtraining.org/sites/default/
files/supplement-making-content-meaningful.pdf).

Considerations for cost categories for targeted client communication

Cost category Description

ONGOING/ALL PHASES

Governance Personnel for partnership building and coordination meetings to align with stakeholders (such as
Technical Working Group members, implementing partners and MNOs)
Meeting costs (transportation, personnel time)
Management and Personnel to oversee overall programme
staffing Personnel for partnership building and coordination meetings to align with stakeholders (such as
MOH counterparts, other implementing partners and MNOs)
Personnel for routine system performance and delivery of communication content (such as
monitoring read receipts and failures)
Personnel to review incoming messages/calls, if there is bidirectional communication

DEVELOPMENT AND SETUP

Content adaptation Development/adaptation of health content to be communicated with clients, which may be
developed by reviewing existing clinical guidelines to ensure that the health content is validated
and from a trusted source; the adaptation process may require translating the content to the
different health literacy levels and languages spoken among the targeted population, as well as
ensuring optimal format and mode of delivery
Adaption to the appropriate communication channel(s), which may include additional adaptations
to the different communication channels: text-based communication (SMS, WhatsApp); audio
communication, which can vary by dialect; or the use of visual aids (pictures, interactive features
and videos) for less literate populations
Technology Software customization for transmitting the communication content, which can include the
adaptation frequency and logic of when communication content should be transmitted
Short code setup, which represents a simplified number for clients to use when registering for the
service
Database to log incoming and outgoing communication exchanges
End-user testing among targeted populations to ensure optimal end-user experience
Refinement of the intervention in response to feedback from end-user testing to ensure that
requirements and context are taken into account

DEPLOYMENT

Outreach and raising Registration of clients to enrol in the service, which could be done through a number that clients
awareness can use to register/subscribe themselves to receive messages or through recruitment by health
workers or other staff
Dissemination to clients about the service and how to subscribe (such as pamphlets, billboards
and/or SMS blasts)
Equipment/ hardware Computers for monitoring system performance and uptake
Server/cloud for storage of recorded calls, audio and images
Mobile devices (often leveraging the devices that clients/individuals already own)

Annexes 149
 INTEGRATION AND INTEROPERABILITY

Content adaptation Content may be adapted to reflect direct linkages to medical records
Technology Integration of client identification: unique client identification, ideally by means of a unique
adaptation personal identifier, needs to be built into the system design and registration process to ensure the
fidelity of message delivery; in some cases, a proxy identifier, such as a mobile phone number, is
used where it can be ascertained that it is valid and consented
Integration and interoperability standards, profiles and APIs to enable data integration and
interoperability with other systems, such as client health records and call centres
Human resources Personnel to implement system and data integrations to enable interoperability between
communication systems and other national systems, such as medical records
Personnel to monitor system and data integration to ensure merging of data between systems
Personnel to ensure the ongoing maintenance of the integrated system and integration of data

SCALE

Training and adaptive Additional training for ICT support staff to provide end-user support, troubleshooting, backup and
management recovery
Periodic review meetings to discuss feedback on system performance and challenges
Human resources Personnel for increased coordination with partners to follow up on software integrations and
governance for unique identifiers
Personnel for monitoring intervention coverage, particularly for hard-to-reach populations

SUSTAINED OPERATIONS

Communication/ SMS, USSD, voice call and/or data transmission charges based on volume of communication
data exchanges content and communication channel
Technology Software maintenance, updates and licence fees
maintenance Short code maintenance fees

150 Digital implementation investment guide

 INTERVENTION 6: HEALTH WORKER DECISION SUPPORT

DESCRIPTION Digitized job aids that combine an individual’s health information with the health worker’s knowledge
OF THE DIGITAL and clinical protocols to assist health workers in making diagnosis and treatment decisions26
HEALTH
INTERVENTION
COMMONLY » Poor adherence to guidelines
ASSOCIATED » Inadequate supportive supervision
HEALTH SYSTEM » Lack of or inappropriate referrals
CHALLENGES
» Insufficient supply (coverage) of qualified health workers
» Insufficient health worker competence
CONSIDERATIONS » Check the relevance and quality of the decision-support content (such as algorithms) and that
BEFORE it aligns with evidence-based clinical guidance, such as WHO or national guidance. This type of
DEPLOYING validation can be done through mechanisms like an independent review and using mock cases
to test the intended output from the algorithms. Also consider built-in mechanisms to update
content remotely as algorithms evolve.
» Assess health workers’ skills and knowledge to ensure that they have adequate capacity to obtain
accurate data before input, to avoid erroneous outputs.
CONSIDERATIONS » Make sure that health workers understand during training that the support provided is based
DURING on existing guidelines and policy. While health workers may deviate from the recommendations,
DEPLOYMENT they should be clear about their rationale for doing so. Where possible, enable cases to be
documented in which health workers feel they need to deviate from the guidance proposed by
the decision-support system.
» Health workers should consider explaining the use of devices and seeking clients’ permission
before using them to improve the acceptability to patients of using digital decision-support
devices. Patients should also be made aware that the information from the counselling may
be saved and used at future visits to improve quality and continuity. Any concerns with
acceptability may be mitigated by, for example, health workers showing the patient the inputs
and results or listening to the messages or videos together so that the device does not become a
barrier in the consultation.
» Improve awareness among staff and supervisors about the value of portable devices, and develop
ground rules or codes of conduct for when and how devices should be used.
OPPORTUNITIES Consider integrating decision-support tools with patient health records, such as digital health records
FOR INTEROPERA- for tracking clients’ health status and services, to more easily incorporate the patient’s health history.
BILITY AND LINK- Consider integrating decision-support tools with digital tools for planning and scheduling health
AGES TO OTHER worker activity.
DIGITAL HEALTH
INTERVENTIONS
RISKS AND Issues with unvalidated or erroneous content/algorithms can result in poor quality of care. The
MITIGATION underlying content needs to undergo thorough rounds of validation and testing and be rooted in
STRATEGIES reliable sources, such as national clinical protocols and global guidelines.
Decision-support algorithms can be quite complex, so be sure to build in adequate time for testing
all the paths of the algorithm with any changes to the software. Consider using automated tools for
testing.
Consider that following the algorithm may mean that health workers spend more time with clients
(rather than skipping steps). This may result in frustration for health workers who have an increased
workload and clients who face longer waiting periods. Share results of improved adherence to
guidelines with health workers and explain the benefits of higher quality care to clients (to justify
waits); in some settings, many of the follow-up visits may be eliminated because correct care is given
the first time, thus ultimately saving the clients and health workers time.
Referral linkages may need to be strengthened to support possible increases in the number of patients
seeking care for previously undetected needs now being revealed by the decision-support system.

26 Adapted from Classification of digital health interventions, 2018.

Annexes 151
Considerations for cost categories for health worker decision support

Cost category Description

ONGOING/ALL PHASES

Governance Personnel for partnership building and coordination meetings to align with stakeholders (such as
Technical Working Group members, implementing partners and MNOs)
Meeting costs (transportation, personnel time)
Management and Personnel to oversee overall programme
staffing Personnel for system setup and end-user support (such as monitoring stability of software and
appropriate functioning of algorithms and troubleshooting system failures)

DEVELOPMENT AND SETUP

Content adaptation Development/adaptation of decision-support pathways/algorithms based on clinical guidelines;

national guidelines may not always be as clear as needed for programming, so an expert group may
need to be engaged for clarity and consensus
Validation of algorithms and decision-support logic to be embedded into the decision-support
system
Technology Software customization adapted to the validated decision-support logic
adaptation End-user testing among targeted populations to ensure optimal end-user experience
Refinement of the intervention in response to feedback from end-user testing to ensure that
requirements and context are taken into account

DEPLOYMENT

Equipment/ hardware Devices (such as mobile phones, tablets and so on) for operating the decision-support software
system used by the health workers
Computers for monitoring system performance and end-user management
Initial training Development/adaptation of training curricula and standard operating procedures for using the
decision-support system
Initial training for health workers interacting with the decision-support system

INTEGRATION AND INTEROPERABILITY

Technology Review and incorporation of policies related to identity management

adaptation
Human resources Software customization to enable interoperability with external systems, such as DHIS2 for
aggregate-level reporting; these integrations are most commonly done through an API, which
details the rules and protocols for communicating between different systems

SCALE

Training and adaptive Additional training for ICT support staff to provide end-user support, troubleshooting, backup and
management recovery
Periodic review meetings to discuss feedback on system performance and challenges
Human resources Additional personnel for increased coordination with partners to follow up on software integrations
and governance for unique identifiers

152 Digital implementation investment guide

 SUSTAINED OPERATIONS

Refresher training and Refresher training for health workers interacting with the decision-support system
adaptive management Periodic review meetings to discuss feedback on system performance and challenges
Technology Software maintenance and licence fees
maintenance Hardware maintenance, including insurance and repair/replacement of hardware
Ongoing adaptation and updating of decision-support logic as new clinical recommendations
emerge

INTERVENTION 7: DIGITAL TRACKING OF CLIENTS’ HEALTH STATUS

AND SERVICES

DESCRIPTION Digitized record used to capture, store, access and share health information on a client or grouping
OF THE DIGITAL of clients, which may include digital service records, digital forms of paper-based registers for
HEALTH longitudinal health programmes and case management logs within specific target populations,
INTERVENTION including migrant populations27
COMMONLY » Delayed reporting of events
ASSOCIATED » Lack of quality/reliable data
HEALTH SYSTEM » Insufficient continuity of care
CHALLENGES
» Delayed provision of care
» Poor planning and coordination
CONSIDERATIONS » Ensure that adequate policy and legal processes and protections, using either a card-based or
BEFORE biometric-based identifier, and telecommunications infrastructure are consistently available
DEPLOYING across facilities and programmes to provide accurate patient identification and facilitate the
digital tracking of health services.
» Consider whether the digital health records for tracking clients’ health status and services have
adequate infrastructural support to be maintained over time. Start-up costs and infrastructural
requirements for a digital tracking system tend to be higher than for paper-based interventions.
When used appropriately and effectively, the costs of digital health interventions are amortized,
and cost savings may be realized in the long run. However, in contexts where basic health
infrastructure is limited, including human resources like supervisors and managers, the digital
tracking system may be very resource intensive.
CONSIDERATIONS » Consider an incremental approach in transitioning from a paper-based data collection form to
DURING a digital form. Closely following the layout of the paper-based form in the digital format may
DEPLOYMENT reduce end-users’ learning curve. Additionally, instead of creating an application that captures all
disease or health areas simultaneously, consider a step-by-step approach, introducing end-users
to modules gradually before adding new ones.
» Improve awareness among staff and supervisors about the value of portable devices, and develop
ground rules or codes of conduct for when and how devices should be used.
OPPORTUNITIES Link to unique identifiers, such as a local or national ID system, to provide a foundational digital
FOR INTEROPERA- identity that can facilitate longitudinal follow-up and linkages to other systems and digital health
BILITY AND LINK- interventions; such unique IDs would help health workers search for clients and reduce the potential
AGES TO OTHER for duplicate registration of clients in community and facility systems.
DIGITAL HEALTH Link digital health records with decision support to enhance the delivery of care while maintaining the
INTERVENTIONS health record and tracking patient history.
Integrate with commodity-reporting systems/LMIS to record supplies used during visits (rapid
diagnostic tests, medicines, condoms distributed and so on).

27 Adapted from Classification of digital health interventions, 2018.

Annexes 153
RISKS AND Health workers may face the added work burden of operating a dual system when using both a
MITIGATION manual/paper-based system and the digital tool. Establish a plan or processes to replace manual/
STRATEGIES paper-based systems or account for the dual burden of managing these two systems.
Consider local policies on digital identities when designing a programme to ensure that the
programme does no harm. Digital tracking of individuals’ health status may be controversial in some
circumstances, such as among refugees or other groups who lack firm legal status in particular
settings. The extent to which such groups may trust tracking depends on who is doing the tracking and
how the information is likely to be used.
ADDITIONAL Electronic immunization registry: practical considerations for planning, development, implementation and
RESOURCE evaluation. Pan American Health Organization; 2018
(http://iris.paho.org/xmlui/handle/123456789/34865).
Handbook for digitizing primary health care: optimizing person-centred digital tracking and decision
support systems. World Health Organization; in print
(www.who.int/reproductivehealth/publications/handbook-digitalizing-primary-health-care/en/)

Considerations for cost categories for digital tracking of clients’ health status and services

Cost category Description

ONGOING/ALL PHASES

Governance Personnel for partnership building and coordination meetings to align with stakeholders (such as
MOH counterparts, other implementing partners and MNOs)
Meeting costs (transportation, personnel time)
Management and Personnel to oversee overall programme
staffing Personnel for system setup and end-user support (such as monitoring stability of software and
troubleshooting system failures)
Personnel to monitor data generated by the system and provide feedback, corrective actions and so on

DEVELOPMENT AND SETUP

Content adaptation Mapping of healthcare cadres’ workflows and responsibilities across the different levels of the
health system, used to determine the content to be included in the system
Development/adaptation of the data dictionary for the digital forms recording client health
information in the system, which may include aligning the data-collection form with global data-
coding standards, such as the ICD
Development/adaptation of algorithms from clinical guideline recommendations, if being
integrated with decision support
Technology Software customization to adapt to the data-collection and decision-support needs
adaptation Dashboards for monitoring the performance of the system and visualizing aggregated data
End-user testing to ensure optimal end-user experience
Refinement of the intervention in response to feedback from end-user testing to ensure that
requirements and context are taken into account

154 Digital implementation investment guide

 DEPLOYMENT

Equipment/ hardware Devices (such as mobile phones and tablets) for operating the decision-support system used by the
health workers
Security features, such as end-user authentication schemes, passwords and data encryption for
recording and sharing client health information
Computers for monitoring system performance and viewing reporting dashboards
Initial training Development/adaptation of training curricula and standard operating procedures for using the
system
Initial training for health workers interacting with the system
Training for supervisory staff on standard operating procedures and continuous monitoring

INTEGRATION AND INTEROPERABILITY

Content adaptation Review and incorporation of policies related to identity management

Technology Software customization to enable interoperability with external systems, such as DHIS2 for
adaptation aggregate-level reporting; these integrations are most commonly done through an API, which
details the rules and protocols for communicating between different systems

SCALE

Training and adaptive Additional training for ICT support staff to provide end-user support, troubleshooting, backup and
management recovery
Periodic review meetings to discuss feedback on system performance and challenges
Human resources Personnel for increased coordination with partners to follow up on software integrations and
governance for unique identifiers
Personnel for monitoring intervention coverage, particularly for hard-to-reach populations

SUSTAINED OPERATIONS

Refresher training Refresher training for health workers interacting with the system
Refresher training for supervisory staff on continuous monitoring and use of data emerging from
the system
Periodic review meetings to discuss feedback on system performance and challenges
Communication/ Data (such as 3G), SMS or wireless connection (or other forms of communication) for submitting
data exchanges data-collection forms
Content adaptation Ongoing adaptation and update of decision-support logic as new clinical recommendations emerge
Technology Server/cloud for storing data generated by the system, including a locked, air-conditioned physical
maintenance space for the server; some contexts may store data in the cloud, which requires cloud-hosting fees
Software maintenance and licence fees
Hardware maintenance, including insurance and repair/replacement of hardware

Annexes 155
 INTERVENTION 8: DIGITAL PROVISION OF TRAINING AND EDUCATIONAL
CONTENT TO HEALTH WORKERS

DESCRIPTION Management and provision of education and training content in electronic form for health
OF THE DIGITAL professionals; in contrast to decision support, health worker training does not need to be used at the
HEALTH point of care28
INTERVENTION
COMMONLY » Insufficient health worker competence
ASSOCIATED » Poor adherence to guidelines
HEALTH SYSTEM » Inadequate supportive supervision
CHALLENGES
» Lack of or inappropriate referrals
CONSIDERATIONS » Ensure that the information is from a source considered trustworthy and credible by health
BEFORE workers in your setting. For example, the information loaded on the mLearning system should be
DEPLOYING based on validated content or should align with national or WHO clinical guidance.
» Ensure that the programme is end-user tested among health workers, both those in practice and
those in training, to ensure that their needs and concerns are met.
» Consider network capacity and coverage, especially if mLearning materials may be videos, which
can be time-consuming to download in certain settings.
» Consider usage needs of the mLearning content, as to whether or not you need to report on
which resources are accessed more frequently than others, how many times and during what
times of day, and then ensure that systems/applications can support these needs.
CONSIDERATIONS » Improve awareness among staff and supervisors about the value of portable devices and develop
DURING ground rules or codes of conduct for when and how devices should be used to increase the
DEPLOYMENT acceptability of mLearning.
» Consider if health workers can earn credits for continuing education using these materials as a
way of increasing their uptake.
» Involve the relevant professional bodies, including national certification or institutional boards,
to ensure that the content of the mLearning programmes aligns with current scopes of practice
and national training curricula for health workers.
OPPORTUNITIES Embed mLearning content on devices used by health workers for other digital health interventions to
FOR INTEROPERA- help maximize resources and enable health workers to access content on a routine basis.
BILITY AND LINK- Link mLearning with human resource information systems to update certification of health workers.
AGES TO OTHER
DIGITAL HEALTH
INTERVENTIONS
RISKS AND Issues with unvalidated or erroneous educational and training content can result in poor quality of
MITIGATION care. The underlying content needs to undergo thorough rounds of validation and testing and be rooted
STRATEGIES in reliable sources, such as national clinical protocols and global guidelines.
ADDITIONAL Open Deliver [app]. mPowering Frontline Health Workers; 2018 (https://partnerships.usaid.gov/
RESOURCE partnership/mpowering-frontline-health-workers/).

28 Adapted from Classification of digital health interventions, 2018.

156 Digital implementation investment guide

Considerations for cost categories for mLearning

Cost category Description

ONGOING/ALL PHASES

Governance Personnel for partnership building and coordination meetings to align with stakeholders (such as
MOH counterparts, other implementing partners and MNOs)
Meeting costs (transportation, personnel time)
Management and Personnel for system setup and end-user support (such as monitoring stability of software and
staffing troubleshooting system failures)
Personnel to provide technical support related to exams and feedback on assignments

DEVELOPMENT AND SETUP

Content adaptation Development/adaptation of mLearning content in a digital format (videos and other forms of
multimedia, for example), which may include adapting existing digital training modules or creating
new modules based on validated health content or clinical guidelines and customization from
global repositories of digital training materials; the adaptation process may also require translating
the content to different languages or skill levels of targeted health workers
Technology Software customization to incorporate the adapted training content to be transmitted
adaptation End-user testing among health workers to ensure optimal end-user experience and alignment with
workflows
Refinement in response to feedback from end-user testing to ensure that requirements and
context are taken into account

DEPLOYMENT

Equipment/ hardware Devices (such as mobile phones and tablets) for use by the health workers (if they are not using
their own devices)
Computers at the district and/or national level for monitoring system performance
Initial training Initial training for health workers interacting with the system

INTEGRATION AND INTEROPERABILITY

Technology Software integration with accreditation databases held by healthcare professional councils or
adaptation registration bodies
Software integration with human resource information systems or registries

SCALE

Training and adaptive Additional training for ICT support staff to provide end-user support, troubleshooting, backup and
management recovery
Periodic review meetings to discuss feedback on system performance and challenges

SUSTAINED OPERATIONS

Refresher training Refresher training for health workers interacting with the system
Communication/ Data-transmission charges if the training content is not stored on the device or requires periodic
data exchanges updates
Technology Software maintenance and licence fees
maintenance Ongoing adaptation and updating of new training content
Hardware maintenance, including insurance and repair/replacement of hardware

Annexes 157
Annex 5.4 Illustrative considerations to mitigate
data management risks
The following is a list of illustrative considerations to think through when mitigating risks associated with data
management and protecting data privacy broken down by phases: before, during and after data collection. Note
that this is not an exhaustive list, and it will have to be reviewed alongside national policies, where available.
Please see Chapter 5 for more information.29,30

BEFORE DATA COLLECTION

1 ˆ Who is responsible and accountable for determining the purposes of data collection for this
implementation or determining what personal data to collect from whom? Please provide the
name(s) of the entity/entities (separated by semicolons), which may assist legal with updating any
agreements.
2 ˆ What is the purpose of collecting personal data and is it truly necessary?
3 ˆ Will the personal data collected during the implementation serve multiple purposes (such as
contact management and fundraising, assessment of eligibility for benefits and research and so
on)?
4 ˆ Who is the personal data about?
5 ˆ Who will supply the data?
6 ˆ What kind of personal data will you be collecting?
7 Will you be collecting any of these categories of special data?
a ˆ Association data (religious, political, trade association)
b ˆ Racial or ethnic data
c ˆ Biometric data
d ˆ Genetic data
e ˆ Criminal or disciplinary history
f ˆ Health data
g ˆ Sexual orientation, gender identity or sexual activity data
8 ˆ Will any of the personal data be about key populations (for example, commercial sex workers,
people who inject drugs or LGBTQIA) or other specific population groups, such as children and
adolescents?
9 ˆ Could the data realistically identify specific individuals, alone or in combination with other data
sources?
10 ˆ Would collection of these data put certain individuals or groups of individuals at risk of harm?

29 Adapted from Ali J, Labrique AB, Gionfriddo K, Pariyo G, Gibson DG, Pratt B. et al. Ethics considerations in global mobile phone–based surveys of
noncommunicable diseases: a conceptual exploration. Journal of Med Internet Research. 2017;19(5),e110. doi:10.2196/jmir.7326.
30 Adapted from UN High-Level Committee on Management. Personal data protection and privacy principles. United Nations; 2018
(https://www.unsystem.org/personal-data-protection-and-privacy-principles).

158 Digital implementation investment guide

 11 Does your implementation involve any of the following technologies or any other technologies that
appear to present a high risk to the rights of data subjects?
a ˆ Innovative technology like artificial intelligence
b ˆ Automated processing of benefits
c ˆ Social media networks or other online services for children
d ˆ Large-scale profiling of data subjects
e ˆ Biometric data
f ˆ Genetic data
g ˆ Data matching from multiple data sources
h ˆ Invisible processing (processing significant amounts of data not obtained from the data
subjects or their representatives)
i ˆ Tracking data (IP or geolocation)
12 ˆ What technological measures will be taken to ensure the data is secure (such as encryption)?

DURING DATA COLLECTION

1 ˆ How will you obtain and store the personal data?

2 ˆ Where will personal data be stored?
3 ˆ What measures will be taken to obtain informed consent?
4 ˆ Who will own the data? Will the data subjects have access to their own data?

AFTER DATA COLLECTION

1 ˆ Do you expect the personal data to move internationally? How will the data be processed? (This
may possible require information notice or determine need for particular legal clauses.)
2 ˆ Who will have access to personal data? Who will the data be shared with? How will the data
be shared? (This may possibly require information notice or may determine new required legal
agreements.)
3 ˆ For how many years do you currently anticipate keeping the personal data?

Annexes 159
Annex 6.1 Linking digital health implementations
to a national digital health enterprise
architecture
This template aims to transition the digital health implementation that you have designed to one that links to the
broader architectural requirements within an exchanged digital health system architecture.

The current state depicts how different systems are currently implemented, which may be as disparate applications
that are siloed or at best paired directly with other applications.

In the future-state diagram, highlight planned new and emerging digital components that others are implementing, as
well as the common and programme-specific functionalities that your system will focus on, specifying the applications
as common services and interoperability requirements that your system will leverage or contribute towards.

160 Digital implementation investment guide

Annexes

CURRENT STATE

SHARED SERVICES DATA SOURCES DATA SERVICES

Digital Health Platform

Shared Services Institution-Based Analytics,

HIS & Data Sources Dashboards, &
Business Domain Registry Services Digital Aids
Services
Terminology
Services
Health Mgmt EXTERNAL
info system Client SYSTEMS
Registry
Logistics Mgmt Facility Population-Based
info system Registry HIS & Data Sources

Shared Product Registry

Health Record Health Worker
Registry
ENTERPRISE

INTER-
OPERABILITY
LAYER
Enabling
Components

DIGITAL HEALTH APPLICATIONS WITH DIGITAL INTERVENTION FOR:

Health programmes / Use cases
A Health Programme B Use Case
C Health Programme
D Use Case
Point of Service Applications

LEGEND

Investment for Funded

new software software under
development

Investment for Funded fully

strengthening functional
software software

Investment for Software for

scaling functional decommissioning
software

Reused POS Unique POS

application application
161
162

FUTURE STATE

SHARED SERVICES DATA SOURCES DATA SERVICES

Digital Health Platform

Shared Services Institution-Based Analytics,

HIS & Data Sources Dashboards, &
Business Domain Registry Services Digital Aids
Services
Terminology
Services EXTERNAL
Health Mgmt SYSTEMS
info system Client
Registry
Logistics Mgmt Facility Population-Based
info system Registry HIS & Data Sources
Product
Shared Registry
Health Record Health Worker
Registry
ENTERPRISE

INTER-
OPERABILITY
LAYER
Enabling
Components

DIGITAL HEALTH APPLICATIONS WITH DIGITAL INTERVENTION FOR:

Health programmes / Use cases
A Health Programme B Use Case
C Health Programme
D Use Case
Point of Service Applications
Digital implementation investment guide

LEGEND

Investment for Funded

new software software under
development

Investment for Funded fully

strengthening functional
software software

Investment for Software for

scaling functional decommissioning
software

Reused POS Unique POS

application application
Annex 7.1 Budget template
This template can be used to estimate costs across common cost categories outlined in Chapter 7. You may also
consider reviewing Annex 5.3 to identify costs for interventions for specific digital health interventions. A digital version
may be downloaded here: https://tinyurl.com/DIIGBudgetTemplate.

Up-front
Phase Cost driver versus Year 1 Year 2 Year 3 Year 4 Year 5 TOTAL
recurring
Management and staffing Recurring
Governance Recurring
ONGOING/
ALL PHASES
Software licensing Up-front
cost per environment and
per end-user
Software customization, Up-front
including adding additional
languages
DEVELOPMENT Application installation Up-front
AND SETUP and configuration
Interoperability Recurring
with other systems
Hardware Recurring
End-user testing Recurring
Cost and availability of Recurring
data connectivity and
power
DEPLOYMENT Training Recurring
Roll-out Up-front
Data Recurring
collection
and use
INTEGRATION
AND INTER-
OPERABILITY

Any category that will be Recurring

affected by expanding
reach

SCALE

Annexes 163
 Up-front
Phase Cost driver versus Year 1 Year 2 Year 3 Year 4 Year 5 TOTAL
recurring
Voice and data services Recurring
(mobile data plan, Internet,
number of text messages)
Hardware maintenance, Recurring
ongoing administration
and replacement rate
Subscriptions Recurring
Software maintenance Recurring
(fixing bugs, adding features,
maintaining customizations)
SUSTAINED
OPERATIONS Transfer of ownership Recurring
Refresher training and Recurring
additional training
activities
M&E and data-use Recurring
activities
Collective benefit, such as Recurring
sharing learnings
TOTAL

164 Digital implementation investment guide

Annex 8.1 Adaptive management checklist
This checklist includes specific adaptive management practices that you can integrate into your planning and
implementation processes.

PLANNING

ˆ Ensure there is a dedicated budget for time to implement your adaptive

Budget for management processes.
adaptive
management ˆ Allow for some flexibility in the overall budget to enable course correction
as needed.

ˆ Identify key stakeholders and decision-makers, including data generators,

data analysts, decision-makers reviewing progress and authorities who can
Engage authorize changes in plans and/or redirect funds as needed.
stakeholders ˆ Clarify mechanisms for coordination between stakeholders/decision-
makers (such as technical working groups).
ˆ Develop a communications plan.

Design a ˆ Design a learning log and other knowledge management platforms based
learning
log on the communications plan.

Establish and ˆ Articulate the expected outcomes, goals and objectives of your digital
refine digital health health intervention; this process typically takes place during M&E planning
intervention goals
and objectives and can usually be taken directly from the M&E plan.

ˆ Develop an evidence-based theory of change articulating your hypothesis

for how change will happen throughout the life of your digital health
Develop and refine a intervention in order to achieve each goal and objective. Clearly map your
theory of change evidence-based assumptions on how inputs and activities will lead to
expected outputs and outcomes. This step is also typically part of designing
a monitoring plan, and you may not need to develop this from scratch.

Map areas of ˆ Identify areas where there may be risks to implementation fidelity or
uncertainty within where achieving desired outcomes may be uncertain given implementation
the theory of or contextual factors.
change for each ˆ Identify specific stakeholders and decision-makers to engage in discussions
objective on these areas of uncertainty.

Plan intentional ˆ Schedule regular times to pause and reflect on implementation data and
pause-and-reflect progress.
cycles. Identify
time points and ˆ Schedule appropriate data review meetings or technical working group
milestones when meetings well in advance to ensure that necessary stakeholders will be
progress will have to able to attend. These may include routine meetings (like quarterly team
be verified and course meetings) prior to work planning, at a point in time when an identified risk
corrections will need
to be made may occur or directly after major deliverables have been completed.

Annexes 165
 From the M&E
plan, identify and
map monitoring ˆ Identify the feedback frequency that is feasible to allow for rapid
measures and
specific assessments identification of potential issues.
required to assess ˆ Find the appropriate balance between rigorous and rapid methodologies
implementation for feedback. Frequency and rapid feedback need to be balanced with
fidelity, whether understanding the burden of collecting, analysing and reporting back those
outputs are being
realized and if risks data.
are arising that need
to be mitigated

ˆ For areas of uncertainty or risk, identify the appropriate decision-makers

to engage, alternative implementation options and critical costs associated
Develop matrix of with the alternatives.
alternative options
and costs ˆ Costing may be time-consuming, so if resources are constrained at
implementation-planning stages, at least clarify the process for developing
this matrix of alternative options and costing those options.

Develop adaptive ˆ Map decision-flow processes, identifying who needs to be informed, how
management flow; and if budgets need to be adjusted, who has authority to make decisions
articulate the steps to
get from decision to and when those decisions will be acted upon regarding different areas of
action uncertainty.

IMPLEMENTATION

Monitor and assess

interventions ˆ Implement the routine monitoring and assessments articulated in the M&E
to determine and adaptive management plans.
performance

ˆ Conduct data review meetings.

ˆ Provide feedback to appropriate decision-makers at the appropriate
Pause and reflect decision milestones to verify if things are on track or determine if course
on data regularly
corrections are needed.
ˆ Define recommendations and action steps needed based on data review.

ˆ Engage necessary actors to make decisions and approve any needed

adjustments.
Take evidence- ˆ Make evidence-based adjustments and course corrections.
based action ˆ Adapt the implementation as required, and be sure to update any
necessary assessments or monitoring measures needed to track the new
implementation plan.

Document findings ˆ Keep a record of lessons learned along the way. A learning log can be
and learnings in a used to track issues identified, data reviewed, decisions made and course
learning log corrections needed and acted upon.

Repeat and continue to monitor, reflect, adapt, document and learn

throughout the life cycle of the implementation

166 Digital implementation investment guide

Also see the following additional resources:
» Guidelines for adaptive management: outcome of the OzAM 2003 workshop, Brisbane. University of Queensland
School of Natural and Rural Systems Management; 2004 (https://espace.library.uq.edu.au/view/UQ:84380).
» PIMPAC adaptive management guidance. Pacific Islands Managed and Protected Area Community; 2018
(https://data.nodc.noaa.gov/coris/library/NOAA/CRCP/NMFS/PIRO/Projects/427/PIMPAC2018_Adaptive_
Management_Guidance.pdf).

Annexes 167
 Annex 8.2 Logic model template
168

Logic models link inputs (programme resources), with processes (activities undertaken in the delivery of services), outputs (products of processes), outcomes (intermediate
changes) and impact. You can use the template below to map out the different inputs, processes, outputs and outcomes, including the specific indicators you will use to
measure your outputs and outcomes.

INPUTS PROCESSES OUTPUTS OUTCOMES IMPACT

Digital implementation investment guide
Digital Implementation Investment Guide (DIIG) 169