As digital health interventions are becoming more popular and important, members of the Care4Saxony team in cooperation with members of the Center for digital health interventions from the University of St Gallen decided to provide a framework for the evaluation of digital health interventions and the barriers they might face.

As chronic diseases are affecting more and more people in developed countries, having a negative influence on the quality of life of the patients and on the economy in those countries , the demand for technology rises that helps the individual cope with their condition. One approach to support patients is the implementation of information technology that monitors health conditions or delivers health interventions . Developing such technology and implementing it successfully demands careful consideration of design criteria and possible barriers. Even though each digital health intervention (DHI) is highly individual, an overall framework could be developed that illustrates the most important evaluation criteria and implementation barriers for all DHI.

13 categories of evaluation criteria could be identified with a literature review, ranging from privacy and security aspects to ethics and effectiveness (see table 1).

Furthermore, 25 categories of implementation barriers could be identified in a second literature review. They need to be addressed during the whole DHI life cycle as well.

Based on the MOST (multiphase-optimization strategy) framework, each category of evaluation criteria and barriers was added to form the framework for Design and Evaluation of DHIs (DEDHI). The framework consists of the original four MOST phases of which each was enriched with a description as well as evaluation criteria and barriers to be considered.

1. Preparation phase (adapted from MOST )

The preparation phase mainly consists of defining the conceptual and technological foundation of a DHI. This means outlining the whished for outcomes and component of the DHI, testing those regarding their feasibility and acceptability and developing a research prototype. This prototype should provide basic functionality so that the feasibility of the DHI can be evaluated. An optimization criterion should be identified that provides the best expected outcome within technical and health economic constraints.

Evaluation criteria in this phase:

Ease of use, adherence, personalization, safety, privacy and security

Implementation barriers in this phase:

Social interaction, individual characteristics of end user, usability, expectations, negative associations, workforce, planning, funding, costs, standard, regulatory issues

2. Optimization Phase (adapted from MOST )

Here the goal is to build an optimized DHI by selecting effective components. This should be done by conducting optimization trials and identifying the best configuration that meets the set criteria. The prototype should now be an elaborated research prototype that provides the full functionality. (If those criteria are not met or the components turn out not to be effective it is suggested to revisit the preparation phase and make appropriate adaptions.)

Evaluation criteria in this phase:

Effectiveness, perceived benefit, content quality, personalization, perceived enjoyment, aesthetics, adherence, service quality, safety, privacy and security

Implementation barriers in this phase:

Social support, outcome expectations, usability, funding, costs, integration

3. Evaluation Phase (adapted from MOST )

In this phase the confirmation of the effectiveness of a DHI is important. This is done by conducting a randomized controlled trial of the DHI and comparing it with a control condition, i.e. treatment as usual. If the DHI outperforms the control condition it can be implemented at large scale. Otherwise it is advised to revisit phase two and to adjust the DHI.

Evaluation criteria in this phase:

Effectiveness, perceived benefit, adherence, personalization, service quality, safety, privacy and security, accountability

Implementation barriers in this phase:

Funding, costs, guidelines, methodology

4. Implementation phase (adapted from MOST )

Here the focus lies on the implementation and maintenance of the DHI. This primarily consists of developing a DHI that can be implemented at large scale in the healthcare market, and secondary of monitoring reach, impact and side effect of the DHI, so that the long-term outreach and efficiency can be determined. Thirdly, updating the DHI with respect to content and technology is essential.

Evaluation criteria in this phase:

Adherence, personalization, perceived benefit, content quality, ethics, service quality, safety, privacy and security, accountability

Implementation barriers in this phase:

Individual resources of end users, expectations, usability of technology, interoperability, human technical support, regional infrastructure, individual characteristics of healthcare providers, negative associations, accessibility, reimbursement, funding, costs, culture

Not all identified barriers could be allocated to the DEDHI framework. This stems from the fact that some of the barriers do not relate to the life cycle of the DHI but rather the framing conditions. Not allocated barriers include missing benefits, cooperation and responsibilities, as well as characteristics of the disease involved, which hinder the usage of DHI in general.

All in all, the framework provided aims to support researchers as well as practitioners while designing and evaluating DHIs in the future.

Referenzen