An assessment of how clinicians and staff members use a diabetes artificial intelligence prediction tool: Mixed methods study

Authors: LIAW, WINSTON - University Of Houston; RAMOS SILVA, YESSENIA - Rice University; SOLTERO, ERICA - Children'S Nutrition Research Center (CNRC); KRIST, ALEX - Virginia Commonwealth University; STOTTS, ANGELA - University Of Texas Health Science Center

Submitted to: JMIR AI
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 4/22/2023
Publication Date: 5/29/2023
Citation: Liaw, W.R., Ramos Silva, Y., Soltero, E.G., Krist, A., Stotts, A.L. 2023. An assessment of how clinicians and staff members use a diabetes artificial intelligence prediction tool: Mixed methods study. JMIR AI. 2. Article e45032. https://doi.org/10.2196/45032.
DOI: https://doi.org/10.2196/45032

Interpretive Summary: Nearly one third of those with type 2 diabetes have poorly controlled diabetes (hemoglobin A1c of >=9.0%), placing them at high risk for diabetes related complications such as kidney disease or neuropathy. It is critical to identify those at-risk for uncontrolled diabetes to intervene with treatment strategies. The purpose of this study was to assess how clinicians would use an artificial intelligence (AI) tool to identify patients at high-risk for uncontrolled type 2 diabetes. Twenty-two individuals, including clinicians (N=15) or employees from academic health centers (N=7), participated in an interview and a follow-up survey to understand the potential usefulness of the AI tool, ease of use, and factors that would affect tool adoption. During the interview, participants reviewed a sample electronic health record alert and were shown how the AI tool could be used to identify those at high risk for poor control. Participants discussed how they would use the tool, whether it would contribute to care, and the factors affecting its implementation. Data from this study revealed that most participants found the tool to be potentially useful, participants were concerned about how the tool would affect patient-oriented outcomes and clinic workflows, and many felt that adoption of the tool would be dependent on its validation, transparency, actionability, and design. Findings from this study suggest that an AI tool that could leverage information from the electronic health record to predict individuals at risk for uncontrolled diabetes would be useful and most agreed (77.3%) that they would use the tool. The information provided from these interviews will be used to guide the development and design the AI tool.

Technical Abstract: Nearly one-third of patients with diabetes are poorly controlled (hemoglobin A1c>=9%). Identifying at-risk individuals and providing them with effective treatment is an important strategy for preventing poor control. This study aims to assess how clinicians and staff members would use a clinical decision support tool based on artificial intelligence (AI) and identify factors that affect adoption. This was a mixed methods study that combined semistructured interviews and surveys to assess the perceived usefulness and ease of use, intent to use, and factors affecting tool adoption. We recruited clinicians and staff members from practices that manage diabetes. During the interviews, participants reviewed a sample electronic health record alert and were informed that the tool uses AI to identify those at high risk for poor control. Participants discussed how they would use the tool, whether it would contribute to care, and the factors affecting its implementation. In a survey, participants reported their demographics; rank-ordered factors influencing the adoption of the tool; and reported their perception of the tool's usefulness as well as their intent to use, ease of use, and organizational support for use. Qualitative data were analyzed using a thematic content analysis approach. We used descriptive statistics to report demographics and analyze the findings of the survey. In total, 22 individuals participated in the study. Two-thirds (14/22, 63%) of respondents were physicians. Overall, 36% (8/22) of respondents worked in academic health centers, whereas 27% (6/22) of respondents worked in federally qualified health centers. The interviews identified several themes: this tool has the potential to be useful because it provides information that is not currently available and can make care more efficient and effective; clinicians and staff members were concerned about how the tool affects patient-oriented outcomes and clinical workflows; adoption of the tool is dependent on its validation, transparency, actionability, and design and could be increased with changes to the interface and usability; and implementation would require buy-in and need to be tailored to the demands and resources of clinics and communities. Survey findings supported these themes, as 77% (17/22) of participants somewhat, moderately, or strongly agreed that they would use the tool, whereas these figures were 82% (18/22) for usefulness, 82% (18/22) for ease of use, and 68% (15/22) for clinic support. The 2 highest ranked factors affecting adoption were whether the tool improves health and the accuracy of the tool. Most participants found the tool to be easy to use and useful, although they had concerns about alert fatigue, bias, and transparency. These data will be used to enhance the design of an AI tool.