User-Centered Evaluation and Design Recommendations for an Internal Medicine Resident Competency Assessment Dashboard.

OBJECTIVES: Clinical Competency Committee (CCC) members employ varied approaches to the review process. This makes the design of a competency assessment dashboard that fits the needs of all members difficult. This work details a user-centered evaluation of a dashboard currently utilized by the Internal Medicine Clinical Competency Committee (IM CCC) at the University of Cincinnati College of Medicine and generated design recommendations. METHODS: Eleven members of the IM CCC participated in semistructured interviews with the research team. These interviews were recorded and transcribed for analysis. The three design research methods used in this study included process mapping (workflow diagrams), affinity diagramming, and a ranking experiment. RESULTS: Through affinity diagramming, the research team identified and organized opportunities for improvement about the current system expressed by study participants. These areas include a time-consuming preprocessing step, lack of integration of data from multiple sources, and different workflows for each step in the review process. Finally, the research team categorized nine dashboard components based on rankings provided by the participants. CONCLUSION: We successfully conducted user-centered evaluation of an IM CCC dashboard and generated four recommendations. Programs should integrate quantitative and qualitative feedback, create multiple views to display these data based on user roles, work with designers to create a usable, interpretable dashboard, and develop a strong informatics pipeline to manage the system. To our knowledge, this type of user-centered evaluation has rarely been attempted in the medical education domain. Therefore, this study provides best practices for other residency programs to evaluate current competency assessment tools and to develop new ones.

Adopting a metaverse-based workspace to support research team collaboration: a pilot study from an academic health informatics laboratory.

Objective: This study aimed to understand how a metaverse-based (virtual) workspace can be used to support the communication and collaboration in an academic health informatics lab. Materials and Methods: A survey of lab members (n = 14) was analyzed according to a concurrent triangulation mixed methods design. The qualitative survey data were organized according to the Capability, Opportunity, Motivation, Behavior (COM-B) model and combined to generate personas that represent the overall types of lab members. Additionally, scheduled work hours were analyzed quantitatively to complement the findings of the survey feedback. Results: Four personas, representative of different types of virtual workers, were developed using the survey responses. These personas reflected the wide variety of opinions about virtual work among the participants and helped to categorize the most common feedback. The Work Hours Schedule Sheet analysis showed the low number of possible collaboration opportunities that were utilized compared to the number available. Discussion: We found that informal communication and co-location were not supported by the virtual workplace as we had originally planned. To solve this issue, we offer 3 design recommendations for those looking to implement their own virtual informatics lab. First, labs should establish common goals and norms for virtual workplace interactions. Second, labs should carefully plan the virtual space layout to maximize communication opportunities. Finally, labs should work with their platform of choice to address technical limitations for their lab members to improve user experience. Future work includes a formal, theory-guided experiment with consideration on ethical and behavioral impact.

Usability Testing of an Interactive Dashboard for Surgical Quality Improvement in a Large Congenital Heart Center.

BACKGROUND: Interactive data visualization and dashboards can be an effective way to explore meaningful patterns in large clinical data sets and to inform quality improvement initiatives. However, these interactive dashboards may have usability issues that undermine their effectiveness. These usability issues can be attributed to mismatched mental models between the designers and the users. Unfortunately, very few evaluation studies in visual analytics have specifically examined such mismatches between these two groups. OBJECTIVES: We aimed to evaluate the usability of an interactive surgical dashboard and to seek opportunities for improvement. We also aimed to provide empirical evidence to demonstrate the mismatched mental models between the designers and the users of the dashboard. METHODS: An interactive dashboard was developed in a large congenital heart center. This dashboard provides real-time, interactive access to clinical outcomes data for the surgical program. A mixed-method, two-phase study was conducted to collect user feedback. A group of designers (N = 3) and a purposeful sample of users (N = 12) were recruited. The qualitative data were analyzed thematically. The dashboards were compared using the System Usability Scale (SUS) and qualitative data. RESULTS: The participating users gave an average SUS score of 82.9 on the new dashboard and 63.5 on the existing dashboard (p = 0.006). The participants achieved high task accuracy when using the new dashboard. The qualitative analysis revealed three opportunities for improvement. The data analysis and triangulation provided empirical evidence to the mismatched mental models. CONCLUSION: We conducted a mixed-method usability study on an interactive surgical dashboard and identified areas of improvements. Our study design can be an effective and efficient way to evaluate visual analytics systems in health care. We encourage researchers and practitioners to conduct user-centered evaluation and implement education plans to mitigate potential usability challenges and increase user satisfaction and adoption.