A design and evaluation framework for digital health interventions

As digital health interventions are becoming more popular and important, members of the Care4Saxony team in cooperation with members of the Center for digital health interventions from the University of St Gallen decided to provide a framework for the evaluation of digital health interventions and the barriers they might face.

As chronic diseases are affecting more and more people in developed countries, having a negative influence on the quality of life of the patients and on the economy in those countries , the demand for technology rises that helps the individual cope with their condition. One approach to support patients is the implementation of information technology that monitors health conditions or delivers health interventions . Developing such technology and implementing it successfully demands careful consideration of design criteria and possible barriers. Even though each digital health intervention (DHI) is highly individual, an overall framework could be developed that illustrates the most important evaluation criteria and implementation barriers for all DHI.

Quelle: https://pixabay.com/de/photos/diabetes-blutzucker-diabetiker-528678/

13 categories of evaluation criteria could be identified with a literature review, ranging from privacy and security aspects to ethics and effectiveness (see table 1).

Evaluation Criteria Description
1 Ease of use How much effort does it take to use the DHI?
2 Content quality Is the content accurate, timely, complete, relevant and consistent?
3 Privacy & security How well were legal requirements, privacy aspects and security aspect considered?
4 Accountability How accessible is information about the DHI?
5 Adherence How high is the actual usage of the DHI compared to the intended usage?
6 Aesthetics Does the Interface apply design elements, colors and fonts in a logical way?
7 Perceived benefit To what extend does the person believe will the DHI improve their health?
8 Effectiveness How much does the DHI contribute to the enhancement of an individual’s health, behavior or condition?
9 Service quality Is enough support of the DHI provided?
10 Personalization To what extend does the DHI adapt to the needs of an individual?
11 Perceived enjoyment How engaging does an individual believe the DHI to be?
12 Ethics To what degree does the DHI address ethical aspects?
13 Safety How safe it the DHI with respect to side effects?

Furthermore, 25 categories of implementation barriers could be identified in a second literature review. They need to be addressed during the whole DHI life cycle as well.

Implementation Barriers Description
1 Individual characteristics of end users Characteristics of the patient and the healthcare provider

(e.g.  health status, demographics, resistance, lack of willingness to cooperate…)

2 Usability of technology Effort needed to use the DHI according to its purpose
3 Planning Administrative activities
4 Funding No or insufficient funding or lack of long-term funding
5 Regional Infrastructure Infrastructure available to the end user
6 Regulatory issues Regulatory issues by governmental and non-governmental organizations which need to be followed
7 Cooperation Lacking cooperation between healthcare providers
8 Disease Requirement due to the patient’s disease
9 Expectations Expectations of the end user regarding the functionality of the technology
10 Human technical support Lack of support for end users by humans
11 Individual resources of end users Financial or temporal resources of the end user
12 Methodology Lacking proof of evidence
13 Outcome expectations Outcome expectations regarding the technology
14 Standards Lack of standardized development or procedures
15 Guidelines Missing guidelines
16 Integration Integration of technology into healthcare system
17 Negative associations Associations of healthcare providers regarding the technology
18 Social support Lack of community support, missing consideration of patient’s environment
19 Workforce Professional people involved in the usage of the DHI
20 Costs Costs for start-up and maintenance, high cost of technology
21 Culture Characteristics o the end user which are inherent in a group of users belonging together
22 Interoperability Ability of technology to be used within existing infrastructure
23 Reimbursement Missing reimbursement policies
24 Responsibilities Different legal responsibilities
25 Social interaction Low interaction between patient and healthcare provider

Based on the MOST (multiphase-optimization strategy) framework, each category of evaluation criteria and barriers was added to form the framework for Design and Evaluation of DHIs (DEDHI). The framework consists of the original four MOST phases of which each was enriched with a description as well as evaluation criteria and barriers to be considered.

Quelle: https://pixabay.com/de/vectors/ehr-emr-elektronische-patientenakte-1476525/

1. Preparation phase (adapted from MOST )

The preparation phase mainly consists of defining the conceptual and technological foundation of a DHI. This means outlining the whished for outcomes and component of the DHI, testing those regarding their feasibility and acceptability and developing a research prototype. This prototype should provide basic functionality so that the feasibility of the DHI can be evaluated. An optimization criterion should be identified that provides the best expected outcome within technical and health economic constraints.

Evaluation criteria in this phase:

Ease of use, adherence, personalization, safety, privacy and security

Implementation barriers in this phase:

Social interaction, individual characteristics of end user, usability, expectations, negative associations, workforce, planning, funding, costs, standard, regulatory issues

2. Optimization Phase (adapted from MOST )

Here the goal is to build an optimized DHI by selecting effective components. This should be done by conducting optimization trials and identifying the best configuration that meets the set criteria. The prototype should now be an elaborated research prototype that provides the full functionality. (If those criteria are not met or the components turn out not to be effective it is suggested to revisit the preparation phase and make appropriate adaptions.)

Evaluation criteria in this phase:

Effectiveness, perceived benefit, content quality, personalization, perceived enjoyment, aesthetics, adherence, service quality, safety, privacy and security

Implementation barriers in this phase:

Social support, outcome expectations, usability, funding, costs, integration

3. Evaluation Phase (adapted from MOST )

In this phase the confirmation of the effectiveness of a DHI is important. This is done by conducting a randomized controlled trial of the DHI and comparing it with a control condition, i.e. treatment as usual. If the DHI outperforms the control condition it can be implemented at large scale. Otherwise it is advised to revisit phase two and to adjust the DHI.

Evaluation criteria in this phase:

Effectiveness, perceived benefit, adherence, personalization, service quality, safety, privacy and security, accountability

Implementation barriers in this phase:

Funding, costs, guidelines, methodology

4. Implementation phase (adapted from MOST )

Here the focus lies on the implementation and maintenance of the DHI. This primarily consists of developing a DHI that can be implemented at large scale in the healthcare market, and secondary of monitoring reach, impact and side effect of the DHI, so that the long-term outreach and efficiency can be determined. Thirdly, updating the DHI with respect to content and technology is essential.

Evaluation criteria in this phase:

Adherence, personalization, perceived benefit, content quality, ethics, service quality, safety, privacy and security, accountability

Implementation barriers in this phase:

Individual resources of end users, expectations, usability of technology, interoperability, human technical support, regional infrastructure, individual characteristics of healthcare providers, negative associations, accessibility, reimbursement, funding, costs, culture

Quelle: https://www.pexels.com/photo/background-board-chart-data-590041/

Not all identified barriers could be allocated to the DEDHI framework. This stems from the fact that some of the barriers do not relate to the life cycle of the DHI but rather the framing conditions. Not allocated barriers include missing benefits, cooperation and responsibilities, as well as characteristics of the disease involved, which hinder the usage of DHI in general.

All in all, the framework provided aims to support researchers as well as practitioners while designing and evaluating DHIs in the future.



Renne, I., & Gobbens, R. J. (2018). Effects of frailty and chronic diseases on quality of life in Dutch community-dwelling older adults: a cross-sectional study. Clinical Interventions in Aging, Volume 13, 325–334. https://doi.org/10.2147/CIA.S156116
Garin, N., Olaya, B., Moneta, M. V., Miret, M., Lobo, A., Ayuso-Mateos, J. L., & Haro, J. M. (2014). Impact of Multimorbidity on Disability and Quality of Life in the Spanish Older Population. PLoS ONE, 9(11), e111498. https://doi.org/10.1371/journal.pone.0111498
Collins, L. M., Baker, T. B., Mermelstein, R. J., Piper, M. E., Jorenby, D. E., Smith, S. S., Christiansen, B. A., Schlam, T. R., Cook, J. W., & Fiore, M. C. (2011). The Multiphase Optimization Strategy for Engineering Effective Tobacco Use Interventions. Annals of Behavioral Medicine, 41(2), 208–226. https://doi.org/10.1007/s12160-010-9253-x
Collins, L. M. (2018). Optimization of Behavioral, Biobehavioral, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST). Springer International Publishing. https://doi.org/10.1007/978-3-319-72206-1
Brennan, P., Perola, M., van Ommen, G.-J., Riboli, E., & On behalf of the European Cohort Consortium. (2017). Chronic disease research in Europe and the need for integrated population cohorts. European Journal of Epidemiology, 32(9), 741–749. https://doi.org/10.1007/s10654-017-0315-2
Kvedar, J. C., Fogel, A. L., Elenko, E., & Zohar, D. (2016). Digital medicine’s march on chronic disease. Nature Biotechnology, 34(3), 239–246. https://doi.org/10.1038/nbt.3495
Katz, D. L., Frates, E. P., Bonnet, J. P., Gupta, S. K., Vartiainen, E., & Carmona, R. H. (2018). Lifestyle as Medicine: The Case for a True Health Initiative. American Journal of Health Promotion, 32(6), 1452–1458. https://doi.org/10.1177/0890117117705949
Husky, M. M., Ferdous Farin, F., Compagnone, P., Fermanian, C., & Kovess-Masfety, V. (2018). Chronic back pain and its association with quality of life in a large French population survey. Health and Quality of Life Outcomes, 16(1), 195. https://doi.org/10.1186/s12955-018-1018-4
Agarwal, R., Gao, G. (Gordon), DesRoches, C., & Jha, A. K. (2010). Research Commentary-The Digital Transformation of Healthcare: Current Status and the Road Ahead. Information Systems Research, 21(4), 796–809. https://doi.org/https://doi.org/10.1287/isre.1100.0327