Development of an evidence-based evaluation framework for digital health software products

Needs assessment

The first cohort of SME interviews identified usability, equity and inclusion, clinical and technical evidence, market and end-user evidence, and privacy and security as the major high-level topics, i.e., domains. Additionally, these interviews identified 4 stakeholder groups that would stand to benefit from a comprehensive EF: DHSP adopters (including healthcare systems, payers, and patients), DHSP developers, regulators, and industry associations.

For each interview in the second cohort, we sorted feedback by domain and employed an inductive approach for thematic analysis. The feedback received quickly reached saturation, as the participants shared similar needs and challenges that could be addressed with an EF. We moved to analysis after 15 interviews.

The domains were validated according to how many responses corresponded to a given domain or the number of comments that included a detail relevant to that domain. For adopters, usability received the most comments, followed by evidence, equity & inclusion, and privacy & security. Respondents did not separate out market and end-user evidence from clinical and technical evidence, but instead spoke of evidence as a single domain. Developer responses ranked the domains in a slightly different order: evidence, usability, privacy & security, and equity & inclusion. This analysis confirmed that the domains identified in the first cohort covered the areas that adopters and developers focus on when vetting or designing DHSP.

We used an inductive approach to review responses within each domain to identify themes and key trends for evaluation of DHSPs, spanning responses from adopters and developers (STable 1). This analysis also identified workflow integration, outcomes, and the overall business model as relevant context for assessing the quality of DHSPs; SMEs preferred that the domains serve as the central organizing unit for the EF.

The top three domains that both adopters and developers wanted to prioritize were evidence, usability, and privacy & security, with equity & inclusion as a 4th theme that applies to all domains. The themes for evidence varied slightly; adopters prioritized evidence vetted by clinicians and supporting workflow integrations, whereas developers prioritized evidence to support clinical claims and ROI. For usability, both stakeholder groups prioritized demonstrating knowledge of user needs for and value of using the DHSP. For privacy & security, adopters prioritized clearly defined measures without specifying a particular method, whereas developers stated that the reference standards of HITRUST²⁰, SOC 2 Type II²¹, and HIPAA²² should be in place (STable 1).

We developed content for focus group discussions around the finding that three domains are foundational. We began these discussions with prompts to learn about the need for an EF for developers and adopters. We then asked participants to assess each domain in the context areas: outcomes, equity & inclusion, workflow, and business model. We also asked participants to identify any gaps they believe exist in the ability of adopters to efficiently evaluate DHSPs. Participants were aware of or are currently using numerous frameworks, certifications, and standards for the privacy & security domain (such as HITRUST, HIPAA, SOC 2 Type II, SMART on FHIR, HITECH, SaMD, NCQQ, VA/DoD, FedRAMP, ONC, KLAS reports, GDPR)^{20,21,22,23,24,25,26,27,28,29,30}. Fig. 1

We then presented the adopter and developer focus-group participants with a grid depicting the domains and context areas. When presented with the question, “What value would an EF for DHSPs bring to healthcare?”, participants’ answers were grouped thematically (Fig. 2; STable 4).

Across the needs assessment activities, participants were asked for each domain if they believe “good exists and there is very little need for improvement” or “the current state is insufficient and there are many opportunities for improvement” (Fig. 1 and STable 2).

Survey development and deployment

We synthesized findings from the interviews and focus groups to design a survey aimed at quantifying the impact of the gaps identified in the focus groups. Specific questions were developed around the three domains.

The survey was sent to the DiMe community and completed by 93 participants: 45 adopters, 32 developers, and 16 regulators, investors, and industry association representatives. The top roles represented were executive leadership, research, data science, or analytics profiles (STable 5). All 3 groups ranked “clinical outcomes clearly defined” as the top criterion when evaluating a new DHSP (Table 1). All 3 groups also ranked “easier to tell which products are fit for my purpose” as the most valuable aspect of such a framework. The evidence domain was ranked as most important when evaluating DHSPs, and “outcomes” was ranked far above equity & inclusion or workflow integration when asked about context.

Evaluation framework development

Several outcomes from the needs assessment were used for the next research phase. This included condensing to 3 domains—evidence, privacy/security, and usability—for 4 stakeholder groups: adopters, developers, regulators, and industry associations. The theme of equity was ubiquitous and woven throughout the domains. The thematic analysis and survey results provided content for criteria to evaluate each domain. We also organized DHSPs into 3 user types based on the intended user group: patients or consumers only, patients and clinical teams, and clinical teams or administrators only.

For the literature review, we screened the titles and abstracts of 4504 unique publications. We reviewed the full text of 1551 (34%), of which 1053 (68%) provided recommendations or best practices to evaluate DHSP quality (SFig 2). We extracted information from these publications that could inform DHSP quality within the defined domains and identified the intended user group for the subject DHSP.

Through inductive thematic analysis, we identified three recommendations for improving quality for the evidence domain: engage a variety of stakeholders when developing or validating a DHSP, conduct a research study for evidence generation, and build consensus around evidence guidelines (STable 3). Within the domain of privacy & security, there was a call for more comprehensive guidelines specific to DHSPs, and more resources and information on data privacy for end-users. The themes we identified for the usability domain were also focused on providing more information and conducting user testing.

The landscape analysis revealed 160 professional sources for alignment with the three domains, product types, and recommendations or best practices that could inform an EF (STable 6): 47 regulatory guidances, 32 frameworks, 32 guidelines, 34 industry standards, and 15 tools. From these, 92 provided information relevant to the evidence domain, 81 related to privacy and security, and 62 related to usability.

Data from the needs assessment, survey, literature review, and landscape analysis were integrated into an EF organized around each domain. For each domain, we identified and defined criteria groups as the first level of organization (Table 2). We further divided each criteria group into criteria and associated benchmarks.

We interviewed SMEs to validate and refine this initial version of the EF. We asked them to approach their review of the framework through the lens of the stakeholder group most closely tied to their role. After interviewing 49 SMEs (16 adopters, 21 developers, and 12 industry association representatives) and observing saturation with the feedback, we moved to analysis and synthesis to integrate the feedback in the framework.

SMEs from all stakeholder groups expressed that the framework was comprehensive and contained important details. They recommended approaches to attesting to the benchmarks that ranged from federal regulations such as FDA 510(k) clearance to a document summarizing the work conducted or processes in place. They thought that the evidence domain could benefit most from an EF, as very few requirements exist for what good evidence should look like, especially for products that are not subject to FDA oversight. The primary feedback for privacy & security was that many privacy and security regulations exist for DHSPs. For usability, the primary feedback was that the usability criteria are important and should be addressed; however, very few developers give this the amount of attention it needs. For each domain, several criteria groups were collapsed, and criteria and benchmarks were combined to reflect SMEs feedback. We then validated the new criteria and benchmarks against industry standards, regulatory guidances, frameworks, and tools.

Finally, we conducted user testing with developers who were working on commercial products that fit ≥ 1 user group. We collected feedback from 9 developers, across the three domains, before reaching saturation. They indicated that the criteria and benchmarks were informative and applicable to their products, and suggested minor edits for evidence and privacy & security. The primary concern with usability was that the benchmarks were more detailed than those required by industry standards or regulatory bodies.