Evaluating Large Language Models for the National Premedical Exam in India: Comparative Analysis of GPT-3.5, GPT-4, and Bard.

BACKGROUND: Large language models (LLMs) have revolutionized natural language processing with their ability to generate human-like text through extensive training on large data sets. These models, including Generative Pre-trained Transformers (GPT)-3.5 (OpenAI), GPT-4 (OpenAI), and Bard (Google LLC), find applications beyond natural language processing, attracting interest from academia and industry. Students are actively leveraging LLMs to enhance learning experiences and prepare for high-stakes exams, such as the National Eligibility cum Entrance Test (NEET) in India. OBJECTIVE: This comparative analysis aims to evaluate the performance of GPT-3.5, GPT-4, and Bard in answering NEET-2023 questions. METHODS: In this paper, we evaluated the performance of the 3 mainstream LLMs, namely GPT-3.5, GPT-4, and Google Bard, in answering questions related to the NEET-2023 exam. The questions of the NEET were provided to these artificial intelligence models, and the responses were recorded and compared against the correct answers from the official answer key. Consensus was used to evaluate the performance of all 3 models. RESULTS: It was evident that GPT-4 passed the entrance test with flying colors (300/700, 42.9%), showcasing exceptional performance. On the other hand, GPT-3.5 managed to meet the qualifying criteria, but with a substantially lower score (145/700, 20.7%). However, Bard (115/700, 16.4%) failed to meet the qualifying criteria and did not pass the test. GPT-4 demonstrated consistent superiority over Bard and GPT-3.5 in all 3 subjects. Specifically, GPT-4 achieved accuracy rates of 73% (29/40) in physics, 44% (16/36) in chemistry, and 51% (50/99) in biology. Conversely, GPT-3.5 attained an accuracy rate of 45% (18/40) in physics, 33% (13/26) in chemistry, and 34% (34/99) in biology. The accuracy consensus metric showed that the matching responses between GPT-4 and Bard, as well as GPT-4 and GPT-3.5, had higher incidences of being correct, at 0.56 and 0.57, respectively, compared to the matching responses between Bard and GPT-3.5, which stood at 0.42. When all 3 models were considered together, their matching responses reached the highest accuracy consensus of 0.59. CONCLUSIONS: The study's findings provide valuable insights into the performance of GPT-3.5, GPT-4, and Bard in answering NEET-2023 questions. GPT-4 emerged as the most accurate model, highlighting its potential for educational applications. Cross-checking responses across models may result in confusion as the compared models (as duos or a trio) tend to agree on only a little over half of the correct responses. Using GPT-4 as one of the compared models will result in higher accuracy consensus. The results underscore the suitability of LLMs for high-stakes exams and their positive impact on education. Additionally, the study establishes a benchmark for evaluating and enhancing LLMs' performance in educational tasks, promoting responsible and informed use of these models in diverse learning environments.

RACares: a conceptual design to guide mHealth relational agent development based on a systematic review.

Background: A relational agent (RA) is a digital tool tailored to communicate with users, aiming to establish a sense of social ease and emotional bond, particularly focusing on their health and well-being concerns. A mobile health (mHealth) RA is particularly crafted to communicate with users within their mobile devices. As healthcare becomes increasingly digital, these mHealth RAs can serve as personal health assistants, e.g., guiding users through medical regimens, offering reminders for medication, providing emotional support during health crises, or even aiding in mental well-being exercises. Their accessibility, especially for those in remote areas, can bridge the gap between patients and immediate health assistance, revolutionizing the way healthcare is approached and delivered. Methods: In this paper, our primary focus is introducing a conceptual design for mHealth RAs with the aim of enhanced user engagement, personalized health interventions, consistent support, data collection and monitoring, and enhanced multimodal accessibility. To develop this conceptual design, we employed an inductive approach. This involved conducting a qualitative analysis on data gathered from a systematic literature review of RAs. Consequently, this analysis allowed us to identify a taxonomy of key design features essential for RAs. Results: This paper provides a conceptual design of mHealth RAs which includes five stages: user input receiving stage, input processing stage, data analysis stage, output processing stage, and output generation stage. A stage is a logical assembly of interconnected functionalities (components) that work together to accomplish a certain objective or set of goals. Each stage's outputs are used as inputs in the stages that follow after it. There is also a Data and Personalization Controller for aiding the data analysis stage. The stages are logically arranged one after another as follows: input, process, analysis, and output. Conclusions: The conceptual design aims to create RAs for various mHealth applications, including patient education, mental health counseling, and chronic disease management. This design is crucial in digital health research as it enhances patient-RA interactions, potentially improving health outcomes and experiences in non-life-threatening scenarios where RAs can be an alternative to human healthcare professionals (HCPs).

Uncertainty in healthcare and health decision making: Five methodological and conceptual research recommendations from an interdisciplinary team.

Uncertainty is prevalent in various health contexts. It is imperative to understand how health-related uncertainty can impact individuals' healthcare experiences and health decision making. The purpose of the present paper is to provide five overarching recommendations from an interdisciplinary team of experts to address gaps in the literature on health-related uncertainty. We present a case study of health-related uncertainty within the specific context of alcohol use to demonstrate these gaps and provide context for the recommendations. The five recommendations concerning health-related uncertainty include: (1) use common, consistent terminology to discuss uncertainty, (2) clarify measures of individual differences in response to uncertainty, (3) increase research on uncertainty and affect, (4) investigate the impact of the channel through which uncertainty is communicated, and (5) develop theory-driven interventions to improve uncertainty management. We conclude by reviewing health contexts in which health-related uncertainty exists and note how our recommendations complement existing reviews and data.

Design Validation of a Relational Agent by COVID-19 Patients: Mixed Methods Study.

BACKGROUND: Relational agents (RAs) have shown effectiveness in various health interventions with and without doctors and hospital facilities. In situations such as a pandemic like the COVID-19 pandemic when health care professionals (HCPs) and facilities are unable to cope with increased demands, RAs may play a major role in ameliorating the situation. However, they have not been well explored in this domain. OBJECTIVE: This study aimed to design a prototypical RA in collaboration with COVID-19 patients and HCPs and test it with the potential users, for its ability to deliver services during a pandemic. METHODS: The RA was designed and developed in collaboration with people with COVID-19 (n=21) and 2 groups of HCPs (n=19 and n=16, respectively) to aid COVID-19 patients at various stages by performing 4 main tasks: testing guidance, support during self-isolation, handling emergency situations, and promoting postrecovery mental well-being. A design validation survey was conducted with 98 individuals to evaluate the usability of the prototype using the System Usability Scale (SUS), and the participants provided feedback on the design. In addition, the RA's usefulness and acceptability were rated by the participants using Likert scales. RESULTS: In the design validation survey, the prototypical RA received an average SUS score of 58.82. Moreover, 90% (88/98) of participants perceived it to be helpful, and 69% (68/98) of participants accepted it as a viable alternative to HCPs. The prototypical RA received favorable feedback from the participants, and they were inclined to accept it as an alternative to HCPs in non-life-threatening scenarios despite the usability rating falling below the acceptable threshold. CONCLUSIONS: Based on participants' feedback, we recommend further development of the RA with improved automation and emotional support, ability to provide information, tracking, and specific recommendations.

A Mobile Application-Based Relational Agent as a Health Professional for COVID-19 Patients: Design, Approach, and Implications.

Relational Agents' (RAs) ability to maintain socio-emotional relationships with users can be an asset to COVID-19 patients. The goal of this research was to identify principles for designing an RA that can act as a health professional for a COVID-19 patient. We first identified tasks that such an RA can provide by interviewing 33 individuals, who had recovered from COVID-19. The transcribed interviews were analyzed using qualitative thematic analysis. Based on the findings, four sets of hypothetical conversations were handcrafted to illustrate how the proposed RA will execute the identified tasks. These conversations were then evaluated by 43 healthcare professionals in a qualitative study. Thematic analysis was again used to identify characteristics that would be suitable for the proposed RA. The results suggest that the RA must: model clinical protocols; incorporate evidence-based interventions; inform, educate, and remind patients; build trusting relationships, and support their socio-emotional needs. The findings have implications for designing RAs for other healthcare contexts beyond the pandemic.

Successful Aging for Community-Dwelling Older Adults: An Experimental Study with a Tablet App.

Mobile health (mHealth) technologies offer an opportunity to enable the care and support of community-dwelling older adults, however, research examining the use of mHealth in delivering quality of life (QoL) improvements in the older population is limited. We developed a tablet application (eSeniorCare) based on the Successful Aging framework and investigated its feasibility among older adults with low socioeconomic status. Twenty five participants (females = 14, mean age = 65 years) used the app to set and track medication intake reminders and health goals, and to play selected casual mobile games for 24 weeks. The Older person QoL and Short Health (SF12v2) surveys were administered before and after the study. The Wilcoxon rank tests were used to determine differences from baseline, and thematic analysis was used to analyze post-study interview data. The improvements in health-related QoL (HRQoL) scores were statistically significant (V=41.5, p=0.005856) across all participants. The frequent eSeniorCare users experienced statistically significant improvements in their physical health (V=13, p=0.04546) and HRQoL (V=7.5, p=0.0050307) scores. Participants reported that the eSeniorCare app motivated timely medication intake and health goals achievement, whereas tablet games promoted mental stimulation. Participants were willing to use mobile apps to self-manage their medications (70%) and adopt healthy activities (72%), while 92% wanted to recommend eSeniorCare to a friend. This study shows the feasibility and possible impact of an mHealth tool on the health-related QoL in older adults with a low socioeconomic status. mHealth support tools and future research to determine their effects are warranted for this population.

A Relational Agent for the COVID-19 Patients: Design, Approach, and Implications.

BACKGROUND: Relational agents (RAs) have shown effectiveness in various health interventions with and without doctors and hospital facilities. We suggest that in situations such as a pandemic like the COVID-19 when healthcare professionals (HCPs) and facilities are unable to cope with increased demands, RAs can play a major role in ameliorating the situation. OBJECTIVE: The goal of this research was to seek design validation on a prototypical RA to address healthcare needs of the COVID-19 patients. METHODS: Therefore, RAs can deliver health interventions during COVID-19 pandemic, but they have not been well-explored in this domain. To address this gap, a prototypical RA is iteratively designed and developed in collaboration with infected patients (n=21) and two groups of HCPs (n=19 and n=16 respectively) to aid COVID-19 patients at various stages by performing four main tasks: testing guidance, support during self-isolation, handling emergency situations, and promoting post-recovery mental well-being. RESULTS: A survey with 98 individuals was used to evaluate the usability of the prototype by system usability scale (SUS) and it received an average score of 58.82. Moreover, participants indicated perceived usefulness and acceptability of the system on Likert Scales where 89.65% perceived it to be helpful, 68.97% accepted it as a viable alternative to HCPs. CONCLUSIONS: The prototypical RA received favorable feedback from the participants and they were inclined to accept it as an alternative to HCPs in non-life-threatening scenarios despite the usability rating falling below the acceptable threshold. Based on participants' feedback, we recommend further development of the RA with improved automation and emotional support, ability to provide information, tracking, and specific recommendations.

Global COVID-19 Vaccine Acceptance: A Systematic Review of Associated Social and Behavioral Factors.

COVID-19 vaccines have met varying levels of acceptance and hesitancy in different parts of the world, which has implications for eliminating the COVID-19 pandemic. The aim of this systematic review is to examine how and why the rates of COVID-19 vaccine acceptance and hesitancy differ across countries and continents. PubMed, Web of Science, IEEE Xplore and Science Direct were searched between 1 January 2020 and 31 July 2021 using keywords such as "COVID-19 vaccine acceptance". 81 peer-reviewed publications were found to be eligible for review. The analysis shows that there are global variations in vaccine acceptance among different populations. The vaccine-acceptance rates were the highest amongst adults in Ecuador (97%), Malaysia (94.3%) and Indonesia (93.3%) and the lowest amongst adults in Lebanon (21.0%). The general healthcare workers (HCWs) in China (86.20%) and nurses in Italy (91.50%) had the highest acceptance rates, whereas HCWs in the Democratic Republic of Congo had the lowest acceptance (27.70%). A nonparametric one-way ANOVA showed that the differences in vaccine-acceptance rates were statistically significant (H (49) = 75.302, p = 0.009*) between the analyzed countries. However, the reasons behind vaccine hesitancy and acceptance were similar across the board. Low vaccine acceptance was associated with low levels of education and awareness, and inefficient government efforts and initiatives. Furthermore, poor influenza-vaccination history, as well as conspiracy theories relating to infertility and misinformation about the COVID-19 vaccine on social media also resulted in vaccine hesitancy. Strategies to address these concerns may increase global COVID-19 vaccine acceptance and accelerate our efforts to eliminate this pandemic.

Correction: Toward Designs of Workplace Stress Management Mobile Apps for Frontline Health Workers During the COVID-19 Pandemic and Beyond: Mixed Methods Qualitative Study.

[This corrects the article DOI: 10.2196/30640.].

Toward Designs of Workplace Stress Management Mobile Apps for Frontline Health Workers During the COVID-19 Pandemic and Beyond: Mixed Methods Qualitative Study.

BACKGROUND: In recent years, mobile apps have been developed to prevent burnout, promote anxiety management, and provide health education to workers in various workplace settings. However, there remains a paucity of such apps for frontline health workers (FHWs), even though FHWs are the most susceptible to stress due to the nature of their jobs. OBJECTIVE: The goal of this study was to provide suggestions for designing stress management apps to address workplace stressors of FHWs based on the understanding of their needs from FHWs' own perspectives and theories of stress. METHODS: A mixed methods qualitative study was conducted. Using a variety of search strings, we first collected 41 relevant web-based news articles published between December 2019 and May 2020 through the Google search engine. We then conducted a cross-sectional survey with 20 FHWs. Two researchers independently conducted qualitative analysis of all the collected data using a deductive followed by an inductive approach. RESULTS: Prevailing uncertainty and fear of contracting the infection was causing stress among FHWs. Moral injury associated with seeing patients die from lack of care and lack of experience in handling various circumstances were other sources of stress. FHWs mentioned 4 coping strategies. Quick coping strategies such as walking away from stressful situations, entertainment, and exercise were the most common ways to mitigate the impact of stress at work. Peer support and counseling services were other popular methods. Building resilience and driving oneself forward using internal motivation were also meaningful ways of overcoming stressful situations. Time constraints and limited management support prevented FHWs from engaging in stress management activities. CONCLUSIONS: Our study identified stressors, coping strategies, and challenges with applying coping strategies that can guide the design of stress management apps for FHWs. Given that the pandemic is ongoing and health care crises continue, FHWs remain a vulnerable population in need of attention.

Classification Framework for Healthy Hairs and Alopecia Areata: A Machine Learning (ML) Approach.

Alopecia areata is defined as an autoimmune disorder that results in hair loss. The latest worldwide statistics have exhibited that alopecia areata has a prevalence of 1 in 1000 and has an incidence of 2%. Machine learning techniques have demonstrated potential in different areas of dermatology and may play a significant role in classifying alopecia areata for better prediction and diagnosis. We propose a framework pertaining to the classification of healthy hairs and alopecia areata. We used 200 images of healthy hairs from the Figaro1k dataset and 68 hair images of alopecia areata from the Dermnet dataset to undergo image preprocessing including enhancement and segmentation. This was followed by feature extraction including texture, shape, and color. Two classification techniques, i.e., support vector machine (SVM) and k-nearest neighbor (KNN), are then applied to train a machine learning model with 70% of the images. The remaining image set was used for the testing phase. With a 10-fold cross-validation, the reported accuracies of SVM and KNN are 91.4% and 88.9%, respectively. Paired sample T-test showed significant differences between the two accuracies with a p < 0.001. SVM generated higher accuracy (91.4%) as compared to KNN (88.9%). The findings of our study demonstrate potential for better prediction in the field of dermatology.

Stigma, biomarkers, and algorithmic bias: recommendations for precision behavioral health with artificial intelligence.

Effective implementation of artificial intelligence in behavioral healthcare delivery depends on overcoming challenges that are pronounced in this domain. Self and social stigma contribute to under-reported symptoms, and under-coding worsens ascertainment. Health disparities contribute to algorithmic bias. Lack of reliable biological and clinical markers hinders model development, and model explainability challenges impede trust among users. In this perspective, we describe these challenges and discuss design and implementation recommendations to overcome them in intelligent systems for behavioral and mental health.

Food for thought.

Healthy eating is essential to avoid many health problems, but identifying health-promoting foods and behaviors is difficult for most individuals. Fooducate is a diet tracking app that comes with a variety of tools, features and support mechanisms that claim to simplify the task of eating healthy. Based on our analysis of the user reviews, we identified five benefits that Fooducate users have reported experiencing in relation to achieving good health: improved food choices, increased awareness, weight loss personalized care and "ever-present" human support. Fooducate can align itself more closely with users' goal of achieving healthy lifestyle by expanding its database, making its interfaces flexible, improving its ability to provide personalized care and revising its ideological choices.

Expecting great expectations when expecting.

Pregnancy is a time when a woman experiences a number of unexpected biological, psychological and social changes. Traditionally, woman sought guidance and help from their mothers and/or social circles to learn how to tackle these changes. With the advent of the mobile revolution, women are now turning to mobile applications to do the same. In this article, we review a pregnancy tracking mobile application called Ovia, in the light of the comments made by mothers using it from around the world. We learn that women like Ovia because it allows them to track, monitor and explore their pregnancy journey by receiving personalized guidance.

Onto the bull's eye.

People set goals to improve their overall health and well-being. Unfortunately, people often lose motivation to achieve their goals because they fail to set proper goals. Goalify is a free mobile app with paid features that can help people set, achieve and maintain goals. It follows Locke and Latham strategies for successful goal setting. To gain greater control over their lives, we recommend readers to use a theory-based mobile app to set, log and achieve their goals.

Mind (is) your body.

The majority of people who experience mental health issues also have poor physical health resulting in decreased life expectancy. Fortunately, many physical health issues can be identified and rectified by monitoring various health indicators over a time period. The Physical Health Diary is a tool that people can use by themselves and/or with others to track, monitor and improve their physical health over time.

Sleeping with an Android.

Sleep quality and duration are strong indicators of an individual's health and quality of lifebut they are difficult to track in everyday life. Mobile apps such as Sleep as Android leverage smartphone sensors to track sleep patterns and make recommendations to improve sleeping habits.

Design and Evaluation of a Medication Adherence Application with Communication for Seniors in Independent Living Communities.

Medication non-adherence is a pressing concern among seniors, leading to a lower quality of life and higher healthcare costs. While mobile applications provide a viable medium for medication management, their utility can be limited without tackling the specific needs of seniors and facilitating the active involvement of care providers. To address these limitations, we are developing a tablet-based application designed specifically for seniors to track their medications and a web portal for their care providers to track medication adherence. In collaboration with a local Aging in Place program, we conducted a three-month study with sixteen participants from an independent living facility. Our study found that the application helped participants to effectively track their medications and improved their sense of wellbeing. Our findings highlight the importance of catering to the needs of seniors and of involving care providers in this process, with specific recommendations for the development of future medication management applications.

Heart rate monitoring mobile applications.

Total number of times a heart beats in a minute is known as the heart rate. Traditionally, heart rate was measured using clunky gadgets but these days it can be measured with a smartphone's camera. This can help you measure your heart rate anywhere and at anytime, especially during workouts so you can adjust your workout intensity to achieve maximum health benefits. With simple and easy to use mobile app, 'Unique Heart Rate Monitor', you can also maintain your heart rate history for personal reflection and sharing with a provider.

Apps for Highlights: our new column for your health management needs.

Millions of health and fitness apps are available in online App stores but people are confused about which ones to use and what are the possible benefits of using them. To broaden your understanding about uses and advantages of these mobile apps, every month in this issue, we will focus on a particular health issue and review related mobile apps.