Evaluating Large Language Models for the National Premedical Exam in India: Comparative Analysis of GPT-3.5, GPT-4, and Bard.

BACKGROUND: Large language models (LLMs) have revolutionized natural language processing with their ability to generate human-like text through extensive training on large data sets. These models, including Generative Pre-trained Transformers (GPT)-3.5 (OpenAI), GPT-4 (OpenAI), and Bard (Google LLC), find applications beyond natural language processing, attracting interest from academia and industry. Students are actively leveraging LLMs to enhance learning experiences and prepare for high-stakes exams, such as the National Eligibility cum Entrance Test (NEET) in India. OBJECTIVE: This comparative analysis aims to evaluate the performance of GPT-3.5, GPT-4, and Bard in answering NEET-2023 questions. METHODS: In this paper, we evaluated the performance of the 3 mainstream LLMs, namely GPT-3.5, GPT-4, and Google Bard, in answering questions related to the NEET-2023 exam. The questions of the NEET were provided to these artificial intelligence models, and the responses were recorded and compared against the correct answers from the official answer key. Consensus was used to evaluate the performance of all 3 models. RESULTS: It was evident that GPT-4 passed the entrance test with flying colors (300/700, 42.9%), showcasing exceptional performance. On the other hand, GPT-3.5 managed to meet the qualifying criteria, but with a substantially lower score (145/700, 20.7%). However, Bard (115/700, 16.4%) failed to meet the qualifying criteria and did not pass the test. GPT-4 demonstrated consistent superiority over Bard and GPT-3.5 in all 3 subjects. Specifically, GPT-4 achieved accuracy rates of 73% (29/40) in physics, 44% (16/36) in chemistry, and 51% (50/99) in biology. Conversely, GPT-3.5 attained an accuracy rate of 45% (18/40) in physics, 33% (13/26) in chemistry, and 34% (34/99) in biology. The accuracy consensus metric showed that the matching responses between GPT-4 and Bard, as well as GPT-4 and GPT-3.5, had higher incidences of being correct, at 0.56 and 0.57, respectively, compared to the matching responses between Bard and GPT-3.5, which stood at 0.42. When all 3 models were considered together, their matching responses reached the highest accuracy consensus of 0.59. CONCLUSIONS: The study's findings provide valuable insights into the performance of GPT-3.5, GPT-4, and Bard in answering NEET-2023 questions. GPT-4 emerged as the most accurate model, highlighting its potential for educational applications. Cross-checking responses across models may result in confusion as the compared models (as duos or a trio) tend to agree on only a little over half of the correct responses. Using GPT-4 as one of the compared models will result in higher accuracy consensus. The results underscore the suitability of LLMs for high-stakes exams and their positive impact on education. Additionally, the study establishes a benchmark for evaluating and enhancing LLMs' performance in educational tasks, promoting responsible and informed use of these models in diverse learning environments.

RACares: a conceptual design to guide mHealth relational agent development based on a systematic review.

Background: A relational agent (RA) is a digital tool tailored to communicate with users, aiming to establish a sense of social ease and emotional bond, particularly focusing on their health and well-being concerns. A mobile health (mHealth) RA is particularly crafted to communicate with users within their mobile devices. As healthcare becomes increasingly digital, these mHealth RAs can serve as personal health assistants, e.g., guiding users through medical regimens, offering reminders for medication, providing emotional support during health crises, or even aiding in mental well-being exercises. Their accessibility, especially for those in remote areas, can bridge the gap between patients and immediate health assistance, revolutionizing the way healthcare is approached and delivered. Methods: In this paper, our primary focus is introducing a conceptual design for mHealth RAs with the aim of enhanced user engagement, personalized health interventions, consistent support, data collection and monitoring, and enhanced multimodal accessibility. To develop this conceptual design, we employed an inductive approach. This involved conducting a qualitative analysis on data gathered from a systematic literature review of RAs. Consequently, this analysis allowed us to identify a taxonomy of key design features essential for RAs. Results: This paper provides a conceptual design of mHealth RAs which includes five stages: user input receiving stage, input processing stage, data analysis stage, output processing stage, and output generation stage. A stage is a logical assembly of interconnected functionalities (components) that work together to accomplish a certain objective or set of goals. Each stage's outputs are used as inputs in the stages that follow after it. There is also a Data and Personalization Controller for aiding the data analysis stage. The stages are logically arranged one after another as follows: input, process, analysis, and output. Conclusions: The conceptual design aims to create RAs for various mHealth applications, including patient education, mental health counseling, and chronic disease management. This design is crucial in digital health research as it enhances patient-RA interactions, potentially improving health outcomes and experiences in non-life-threatening scenarios where RAs can be an alternative to human healthcare professionals (HCPs).

Design Validation of a Relational Agent by COVID-19 Patients: Mixed Methods Study.

BACKGROUND: Relational agents (RAs) have shown effectiveness in various health interventions with and without doctors and hospital facilities. In situations such as a pandemic like the COVID-19 pandemic when health care professionals (HCPs) and facilities are unable to cope with increased demands, RAs may play a major role in ameliorating the situation. However, they have not been well explored in this domain. OBJECTIVE: This study aimed to design a prototypical RA in collaboration with COVID-19 patients and HCPs and test it with the potential users, for its ability to deliver services during a pandemic. METHODS: The RA was designed and developed in collaboration with people with COVID-19 (n=21) and 2 groups of HCPs (n=19 and n=16, respectively) to aid COVID-19 patients at various stages by performing 4 main tasks: testing guidance, support during self-isolation, handling emergency situations, and promoting postrecovery mental well-being. A design validation survey was conducted with 98 individuals to evaluate the usability of the prototype using the System Usability Scale (SUS), and the participants provided feedback on the design. In addition, the RA's usefulness and acceptability were rated by the participants using Likert scales. RESULTS: In the design validation survey, the prototypical RA received an average SUS score of 58.82. Moreover, 90% (88/98) of participants perceived it to be helpful, and 69% (68/98) of participants accepted it as a viable alternative to HCPs. The prototypical RA received favorable feedback from the participants, and they were inclined to accept it as an alternative to HCPs in non-life-threatening scenarios despite the usability rating falling below the acceptable threshold. CONCLUSIONS: Based on participants' feedback, we recommend further development of the RA with improved automation and emotional support, ability to provide information, tracking, and specific recommendations.

A Mobile Application-Based Relational Agent as a Health Professional for COVID-19 Patients: Design, Approach, and Implications.

Relational Agents' (RAs) ability to maintain socio-emotional relationships with users can be an asset to COVID-19 patients. The goal of this research was to identify principles for designing an RA that can act as a health professional for a COVID-19 patient. We first identified tasks that such an RA can provide by interviewing 33 individuals, who had recovered from COVID-19. The transcribed interviews were analyzed using qualitative thematic analysis. Based on the findings, four sets of hypothetical conversations were handcrafted to illustrate how the proposed RA will execute the identified tasks. These conversations were then evaluated by 43 healthcare professionals in a qualitative study. Thematic analysis was again used to identify characteristics that would be suitable for the proposed RA. The results suggest that the RA must: model clinical protocols; incorporate evidence-based interventions; inform, educate, and remind patients; build trusting relationships, and support their socio-emotional needs. The findings have implications for designing RAs for other healthcare contexts beyond the pandemic.

Successful Aging for Community-Dwelling Older Adults: An Experimental Study with a Tablet App.

Mobile health (mHealth) technologies offer an opportunity to enable the care and support of community-dwelling older adults, however, research examining the use of mHealth in delivering quality of life (QoL) improvements in the older population is limited. We developed a tablet application (eSeniorCare) based on the Successful Aging framework and investigated its feasibility among older adults with low socioeconomic status. Twenty five participants (females = 14, mean age = 65 years) used the app to set and track medication intake reminders and health goals, and to play selected casual mobile games for 24 weeks. The Older person QoL and Short Health (SF12v2) surveys were administered before and after the study. The Wilcoxon rank tests were used to determine differences from baseline, and thematic analysis was used to analyze post-study interview data. The improvements in health-related QoL (HRQoL) scores were statistically significant (V=41.5, p=0.005856) across all participants. The frequent eSeniorCare users experienced statistically significant improvements in their physical health (V=13, p=0.04546) and HRQoL (V=7.5, p=0.0050307) scores. Participants reported that the eSeniorCare app motivated timely medication intake and health goals achievement, whereas tablet games promoted mental stimulation. Participants were willing to use mobile apps to self-manage their medications (70%) and adopt healthy activities (72%), while 92% wanted to recommend eSeniorCare to a friend. This study shows the feasibility and possible impact of an mHealth tool on the health-related QoL in older adults with a low socioeconomic status. mHealth support tools and future research to determine their effects are warranted for this population.

A Relational Agent for the COVID-19 Patients: Design, Approach, and Implications.

BACKGROUND: Relational agents (RAs) have shown effectiveness in various health interventions with and without doctors and hospital facilities. We suggest that in situations such as a pandemic like the COVID-19 when healthcare professionals (HCPs) and facilities are unable to cope with increased demands, RAs can play a major role in ameliorating the situation. OBJECTIVE: The goal of this research was to seek design validation on a prototypical RA to address healthcare needs of the COVID-19 patients. METHODS: Therefore, RAs can deliver health interventions during COVID-19 pandemic, but they have not been well-explored in this domain. To address this gap, a prototypical RA is iteratively designed and developed in collaboration with infected patients (n=21) and two groups of HCPs (n=19 and n=16 respectively) to aid COVID-19 patients at various stages by performing four main tasks: testing guidance, support during self-isolation, handling emergency situations, and promoting post-recovery mental well-being. RESULTS: A survey with 98 individuals was used to evaluate the usability of the prototype by system usability scale (SUS) and it received an average score of 58.82. Moreover, participants indicated perceived usefulness and acceptability of the system on Likert Scales where 89.65% perceived it to be helpful, 68.97% accepted it as a viable alternative to HCPs. CONCLUSIONS: The prototypical RA received favorable feedback from the participants and they were inclined to accept it as an alternative to HCPs in non-life-threatening scenarios despite the usability rating falling below the acceptable threshold. Based on participants' feedback, we recommend further development of the RA with improved automation and emotional support, ability to provide information, tracking, and specific recommendations.

Correction: Toward Designs of Workplace Stress Management Mobile Apps for Frontline Health Workers During the COVID-19 Pandemic and Beyond: Mixed Methods Qualitative Study.

[This corrects the article DOI: 10.2196/30640.].

Toward Designs of Workplace Stress Management Mobile Apps for Frontline Health Workers During the COVID-19 Pandemic and Beyond: Mixed Methods Qualitative Study.

BACKGROUND: In recent years, mobile apps have been developed to prevent burnout, promote anxiety management, and provide health education to workers in various workplace settings. However, there remains a paucity of such apps for frontline health workers (FHWs), even though FHWs are the most susceptible to stress due to the nature of their jobs. OBJECTIVE: The goal of this study was to provide suggestions for designing stress management apps to address workplace stressors of FHWs based on the understanding of their needs from FHWs' own perspectives and theories of stress. METHODS: A mixed methods qualitative study was conducted. Using a variety of search strings, we first collected 41 relevant web-based news articles published between December 2019 and May 2020 through the Google search engine. We then conducted a cross-sectional survey with 20 FHWs. Two researchers independently conducted qualitative analysis of all the collected data using a deductive followed by an inductive approach. RESULTS: Prevailing uncertainty and fear of contracting the infection was causing stress among FHWs. Moral injury associated with seeing patients die from lack of care and lack of experience in handling various circumstances were other sources of stress. FHWs mentioned 4 coping strategies. Quick coping strategies such as walking away from stressful situations, entertainment, and exercise were the most common ways to mitigate the impact of stress at work. Peer support and counseling services were other popular methods. Building resilience and driving oneself forward using internal motivation were also meaningful ways of overcoming stressful situations. Time constraints and limited management support prevented FHWs from engaging in stress management activities. CONCLUSIONS: Our study identified stressors, coping strategies, and challenges with applying coping strategies that can guide the design of stress management apps for FHWs. Given that the pandemic is ongoing and health care crises continue, FHWs remain a vulnerable population in need of attention.