As digital health interventions are becoming more popular and important, members of the Care4Saxony team in cooperation with members of the Center for digital health interventions from the University of St Gallen decided to provide a framework for the evaluation of digital health interventions and the barriers they might face.
As chronic diseases are affecting more and more people in developed countries, having a negative influence on the quality of life of the patients and on the economy in those countries , the demand for technology rises that helps the individual cope with their condition. One approach to support patients is the implementation of information technology that monitors health conditions or delivers health interventions . Developing such technology and implementing it successfully demands careful consideration of design criteria and possible barriers. Even though each digital health intervention (DHI) is highly individual, an overall framework could be developed that illustrates the most important evaluation criteria and implementation barriers for all DHI.

13 categories of evaluation criteria could be identified with a literature review, ranging from privacy and security aspects to ethics and effectiveness (see table 1).
Evaluation Criteria | Description | |
1 | Ease of use | How much effort does it take to use the DHI? |
2 | Content quality | Is the content accurate, timely, complete, relevant and consistent? |
3 | Privacy & security | How well were legal requirements, privacy aspects and security aspect considered? |
4 | Accountability | How accessible is information about the DHI? |
5 | Adherence | How high is the actual usage of the DHI compared to the intended usage? |
6 | Aesthetics | Does the Interface apply design elements, colors and fonts in a logical way? |
7 | Perceived benefit | To what extend does the person believe will the DHI improve their health? |
8 | Effectiveness | How much does the DHI contribute to the enhancement of an individual’s health, behavior or condition? |
9 | Service quality | Is enough support of the DHI provided? |
10 | Personalization | To what extend does the DHI adapt to the needs of an individual? |
11 | Perceived enjoyment | How engaging does an individual believe the DHI to be? |
12 | Ethics | To what degree does the DHI address ethical aspects? |
13 | Safety | How safe it the DHI with respect to side effects? |
Furthermore, 25 categories of implementation barriers could be identified in a second literature review. They need to be addressed during the whole DHI life cycle as well.
Implementation Barriers | Description | |
1 | Individual characteristics of end users | Characteristics of the patient and the healthcare provider
(e.g. health status, demographics, resistance, lack of willingness to cooperate…) |
2 | Usability of technology | Effort needed to use the DHI according to its purpose |
3 | Planning | Administrative activities |
4 | Funding | No or insufficient funding or lack of long-term funding |
5 | Regional Infrastructure | Infrastructure available to the end user |
6 | Regulatory issues | Regulatory issues by governmental and non-governmental organizations which need to be followed |
7 | Cooperation | Lacking cooperation between healthcare providers |
8 | Disease | Requirement due to the patient’s disease |
9 | Expectations | Expectations of the end user regarding the functionality of the technology |
10 | Human technical support | Lack of support for end users by humans |
11 | Individual resources of end users | Financial or temporal resources of the end user |
12 | Methodology | Lacking proof of evidence |
13 | Outcome expectations | Outcome expectations regarding the technology |
14 | Standards | Lack of standardized development or procedures |
15 | Guidelines | Missing guidelines |
16 | Integration | Integration of technology into healthcare system |
17 | Negative associations | Associations of healthcare providers regarding the technology |
18 | Social support | Lack of community support, missing consideration of patient’s environment |
19 | Workforce | Professional people involved in the usage of the DHI |
20 | Costs | Costs for start-up and maintenance, high cost of technology |
21 | Culture | Characteristics o the end user which are inherent in a group of users belonging together |
22 | Interoperability | Ability of technology to be used within existing infrastructure |
23 | Reimbursement | Missing reimbursement policies |
24 | Responsibilities | Different legal responsibilities |
25 | Social interaction | Low interaction between patient and healthcare provider |
Based on the MOST (multiphase-optimization strategy) framework, each category of evaluation criteria and barriers was added to form the framework for Design and Evaluation of DHIs (DEDHI). The framework consists of the original four MOST phases of which each was enriched with a description as well as evaluation criteria and barriers to be considered.

1. Preparation phase (adapted from MOST )
The preparation phase mainly consists of defining the conceptual and technological foundation of a DHI. This means outlining the whished for outcomes and component of the DHI, testing those regarding their feasibility and acceptability and developing a research prototype. This prototype should provide basic functionality so that the feasibility of the DHI can be evaluated. An optimization criterion should be identified that provides the best expected outcome within technical and health economic constraints.
Evaluation criteria in this phase:
Ease of use, adherence, personalization, safety, privacy and security
Implementation barriers in this phase:
Social interaction, individual characteristics of end user, usability, expectations, negative associations, workforce, planning, funding, costs, standard, regulatory issues
2. Optimization Phase (adapted from MOST )
Here the goal is to build an optimized DHI by selecting effective components. This should be done by conducting optimization trials and identifying the best configuration that meets the set criteria. The prototype should now be an elaborated research prototype that provides the full functionality. (If those criteria are not met or the components turn out not to be effective it is suggested to revisit the preparation phase and make appropriate adaptions.)
Evaluation criteria in this phase:
Effectiveness, perceived benefit, content quality, personalization, perceived enjoyment, aesthetics, adherence, service quality, safety, privacy and security
Implementation barriers in this phase:
Social support, outcome expectations, usability, funding, costs, integration
3. Evaluation Phase (adapted from MOST )
In this phase the confirmation of the effectiveness of a DHI is important. This is done by conducting a randomized controlled trial of the DHI and comparing it with a control condition, i.e. treatment as usual. If the DHI outperforms the control condition it can be implemented at large scale. Otherwise it is advised to revisit phase two and to adjust the DHI.
Evaluation criteria in this phase:
Effectiveness, perceived benefit, adherence, personalization, service quality, safety, privacy and security, accountability
Implementation barriers in this phase:
Funding, costs, guidelines, methodology
4. Implementation phase (adapted from MOST )
Here the focus lies on the implementation and maintenance of the DHI. This primarily consists of developing a DHI that can be implemented at large scale in the healthcare market, and secondary of monitoring reach, impact and side effect of the DHI, so that the long-term outreach and efficiency can be determined. Thirdly, updating the DHI with respect to content and technology is essential.
Evaluation criteria in this phase:
Adherence, personalization, perceived benefit, content quality, ethics, service quality, safety, privacy and security, accountability
Implementation barriers in this phase:
Individual resources of end users, expectations, usability of technology, interoperability, human technical support, regional infrastructure, individual characteristics of healthcare providers, negative associations, accessibility, reimbursement, funding, costs, culture

Not all identified barriers could be allocated to the DEDHI framework. This stems from the fact that some of the barriers do not relate to the life cycle of the DHI but rather the framing conditions. Not allocated barriers include missing benefits, cooperation and responsibilities, as well as characteristics of the disease involved, which hinder the usage of DHI in general.
All in all, the framework provided aims to support researchers as well as practitioners while designing and evaluating DHIs in the future.
Referenzen