Examining the Threat of ChatGPT to the Validity of Short Answer Assessments in an Undergraduate Medical Program.

OBJECTIVES: ChatGPT is an artificial intelligence model that can interpret free-text prompts and return detailed, human-like responses across a wide domain of subjects. This study evaluated the extent of the threat posed by ChatGPT to the validity of short-answer assessment problems used to examine pre-clerkship medical students in our undergraduate medical education program. METHODS: Forty problems used in prior student assessments were retrieved and stratified by levels of Bloom's Taxonomy. Thirty of these problems were submitted to ChatGPT-3.5. For the remaining 10 problems, we retrieved past minimally passing student responses. Six tutors graded each of the 40 responses. Comparison of performance between student-generated and ChatGPT-generated answers aggregated as a whole and grouped by Bloom's levels of cognitive reasoning, was done using t-tests, ANOVA, Cronbach's alpha, and Cohen's d. Scores for ChatGPT-generated responses were also compared to historical class average performance. RESULTS: ChatGPT-generated responses received a mean score of 3.29 out of 5 (n = 30, 95% CI 2.93-3.65) compared to 2.38 for a group of students meeting minimum passing marks (n = 10, 95% CI 1.94-2.82), representing higher performance (P = .008, η2 = 0.169), but was outperformed by historical class average scores on the same 30 problems (mean 3.67, P = .018) when including all past responses regardless of student performance level. There was no statistically significant trend in performance across domains of Bloom's Taxonomy. CONCLUSION: While ChatGPT was able to pass short answer assessment problems spanning the pre-clerkship curriculum, it outperformed only underperforming students. We remark that tutors in several cases were convinced that ChatGPT-produced responses were produced by students. Risks to assessment validity include uncertainty in identifying struggling students and inability to intervene in a timely manner. The performance of ChatGPT on problems requiring increasing demands of cognitive reasoning warrants further research.

Spiralling Pre-clerkship Concepts into the Clinical Phase: Augmenting Knowledge Transfer Using Innovative Technology-Enhanced Curriculum Activities.

Background: Clerkship is a challenging transition for medical students where they learn to apply functional knowledge and diagnostic reasoning skills learned in the pre-clinical phase into the clinical environment. Rather than a smooth continuum to facilitate application of knowledge, clerkship blocks are discrete, fragmented structures with little integration. Developments in cognitive psychology and increasing attention to the student learning environment are driving more purposeful integration in medical education. We sought to enhance knowledge transfer in the Family Medicine clerkship by developing an e-learning pathway with both asynchronous and synchronous components to integrate pre-clerkship problem-based learning (PBL) cases into more complex clinical scenarios. Methods: A parallel-convergent mixed methods evaluation was conducted, which included comparison of learning outcomes (exit exams) relative to the prior class, knowledge check quiz performance, and qualitative analysis of student and faculty perceptions. Results: Analyses revealed no significant difference between exit exam scores of the intervention and pre-intervention clerks (p = 0.30). There were statistically significant differences in mean quiz scores over the rotation (p = 0.0001). Moreover, learners and faculty each perceived the integration components as facilitating the transfer of pre-clinical learning into clerkship activities. Conclusion: The novel e-learning pathway firmly anchored FM clerkship learning and will continue to ensure learners are ideally primed to optimize their direct clinical learning opportunities.

Early generalist placements are associated with family medicine career choice: A systematic review and meta-analysis.

INTRODUCTION: Many developed countries have reported shortages of Primary Care (PC) physicians. The lack of a regular primary physician is associated with inferior health outcomes. One strategy to address this shortage is to increase the proportion of medical students selecting a Family Medicine (FM) or PC career. The purpose of this systematic review is to identify whether pre-clerkship general practice placements increase students' interest in, and selection of FM or PC residencies. METHODS: Three databases (PubMed, Embase, Web of Science) searched for interventional studies of pre-clerkship generalist placements in medical school. Pooled statistical analysis and meta-analysis were performed, along with narrative summaries when possible. Intervention participants (IP) were compared to controls matched (MC) for baseline interest in FM and an unmatched sample (UC) of contemporary students. FINDINGS: A 11 studies were identified including a total of 5430 students (2428 intervention participants and 3002 controls). IPs were more likely to match to FM than both MC (Risk Ratio: 1.62 [95% CI: 1.03-2.55]) and UC (RR: 2.04 [1.46-2.86]). Participation in long interventions (4-11 weeks) matched to FM at higher rates than short (25-40 hours) interventions (RR: 3.15 [2.28-4.35]). The percentage of students with FM/PC as their top specialty of interest increased after the placements (mean difference: +12.8%, n = 586). CONCLUSIONS: Medical students who participated in pre-clerkship general practice placements were more likely to match to a FM residency. Longer pre-clerkship placements had a stronger association with FM specialty choice. The implementation of longitudinal block generalist placements in the pre-clerkship years is one strategy for increasing interest in generalist careers. LEVEL OF EVIDENCE: IV, systematic review of level III and IV studies.

Structured diet and exercise guidance in pregnancy to improve health in women and their offspring: study protocol for the Be Healthy in Pregnancy (BHIP) randomized controlled trial.

BACKGROUND: Evidence from epidemiological and animal studies support the concept of programming fetal, neonatal, and adult health in response to in utero exposures such as maternal obesity and lifestyle variables. Excess gestational weight gain (GWG), maternal physical activity, and sub-optimal and excess nutrition during pregnancy may program the offspring's risk of obesity. Maternal intake of dairy foods rich in high-quality proteins, calcium, and vitamin D may influence later bone health status. Current clinical practice guidelines for managing GWG are not founded on randomized trials and lack specific "active intervention ingredients." The Be Healthy in Pregnancy (BHIP) study is a randomized controlled trial (RCT) designed to test the effectiveness of a novel structured and monitored Nutrition + Exercise intervention in pregnant women of all pre-pregnancy weight categories (except extreme obesity), delivered through prenatal care in community settings (rather than in hospital settings), on the likelihood of women achieving recommended GWG and a benefit to bone status of offspring and mother at birth and six months postpartum. METHODS: The BHIP study is a two-site RCT that will recruit up to 242 participants aged > 18 years at 12-17 weeks of gestation. After baseline measures, participants are randomized to either a structured and monitored Nutrition + Exercise (intervention) or usual care (control) program for the duration of their pregnancy. The primary outcome of the study is the percent of women who achieve GWG within the Institute of Medicine (IOM) guidelines. The secondary outcomes include: (1) maternal bone status via blood bone biomarkers during pregnancy; (2) infant bone status in cord blood; (3) mother and infant bone status measured by dual-energy absorptiometry scanning (DXA scan) at six months postpartum; (4) other measures including maternal blood pressure, blood glucose and lipid profiles, % body fat, and postpartum weight retention; and (5) infant weight z-scores and fat mass at six months of age. DISCUSSION: If effective, this RCT will generate high-quality evidence to refine the nutrition guidelines during pregnancy to improve the likelihood of women achieving recommended GWG. It will also demonstrate the importance of early nutrition on bone health in the offspring. TRIAL REGISTRATION: ClinicalTrials.gov, NCT01689961 Registered on 21 September 2012.

Update on the new 9-valent vaccine for human papillomavirus prevention.

OBJECTIVE: To provide family physicians with information on the efficacy, safety, public health effects, and cost-effectiveness of the 9-valent human papillomavirus (HPV) vaccine. QUALITY OF EVIDENCE: Relevant publications in PubMed up to May 2015 were reviewed and analyzed. Most evidence cited is level I (randomized controlled trials and meta-analyses) or level II (cross-sectional, case-control, and epidemiologic studies). Government reports and recommendations are also referenced. MAIN MESSAGE: The 9-valent HPV vaccine, which protects against HPV types 6, 11, 16, 18, 31, 33, 45, 52, and 58, is safe and effective and will further reduce the incidence of HPV infection, as well as HPV-related cancers. It can also indirectly protect unvaccinated individuals through herd immunity. With an effective vaccination program, most cervical cancers can be prevented. Analyses show that the cost-effectiveness of the 9-valent HPV vaccine in female patients is comparable to the original quadrivalent HPV vaccine (which protects against HPV types 6, 11, 16, and 18) currently in use. However, the usefulness of vaccinating male patients with the 9-valent HPV vaccine needs further investigation. CONCLUSION: The 9-valent HPV vaccine offers more protection against HPV than the quadrivalent HPV vaccine does and is as safe. Analysis of cost-effectiveness favours its use, at least in adolescent girls. Therefore, physicians should recommend the 9-valent HPV vaccine to patients instead of the quadrivalent HPV vaccine.

The relationship between perceived preceptor power use and student empowerment during clerkship rotations: a study of hidden curriculum.

CONTEXT: Medical learners are vulnerable to the social power used by preceptors. Furthermore, an individual's perceived level of personal empowerment has been identified as a predictor of positive behaviour in education environments. The degree to which medical students feel empowered in their clinical environments factors largely into how vulnerable they are to having their professional values influenced negatively by structural and cultural aspects of these environments. OBJECTIVES: The goal of this study was to explore the relationship between clerks' perceptions of personal empowerment and the social power employed by their preceptors. This research also investigates the prevalence of negative power use as a function of the hierarchical organisation of a clerkship rotation. METHODS: Validated power use and empowerment surveys were modified for clinical learning environments and administered to clerkship learners across six clerkship rotations. The outcomes of the two surveys were subjected to correlational analyses. Outcomes associated with the use of each type of perceived power were analysed to determine the influence of the relative involvement of residents and staff physicians in preceptorship. RESULTS: Correlational analyses revealed strong relationships between clerks' perceptions of preceptor power use and their own personal empowerment. Furthermore, although participants perceived significantly more pro-social preceptor uses of power, clerks perceived a higher prevalence of coercive power on rotations with high involvement of residents as preceptors. CONCLUSIONS: Clerks' perceptions of empowerment correlate positively with positive power bases and negatively with negative dimensions of preceptor power. This research has implications for the importance of the development of resident and faculty staff as educators, the identification of clerks who are vulnerable to ethical violation, and for a refined understanding of the transactional way in which power is experienced in a medical education context.

Knowledge translation tool to improve pregnant women's awareness of gestational weight gain goals and risks of gaining outside recommendations: a non-randomized intervention study.

BACKGROUND: There is an urgent need to prevent excessive pregnancy weight gain, a contributor to both maternal and child obesity. However, the majority of women had reported not being counseled to gain an appropriate amount of gestational weight by their health care providers. We developed a knowledge translation (KT) tool designed to facilitate the clinical interaction between pregnant women and their health care providers (HCPs). We piloted the tool on the impact on women's knowledge of gestational weight gain (GWG) goals, and evaluated its potential in promoting appropriate knowledge about GWG within the 2009 Institute of Medicine guidelines. METHODS: We conducted a prospective cohort study, comparing women's knowledge about GWG after the KT tool to women from the same clinics and care providers the year prior. Our primary outcome was the proportion of women who reported receiving an appropriate GWG recommendation from their care provider. We evaluated knowledge on a survey conducted at enrollment in the cohort at ≤ 20 weeks gestation and evaluated participant satisfaction with the KT tool in the third trimester. We performed univariate and multivariable logistic regression analyses for differences in outcomes with historical controls from the same clinics. Our a priori sample size calculation required 130 participants to demonstrate a 15% increase in reported counseling about gestational weight gain. RESULTS: One hundred and forty-six women were recruited and 131 (90%) completed the enrollment survey. Women who received the KT tool were more likely to report receiving a specific GWG recommendation from their HCP (adjusted odds ratio [AOR] 3.45, 95% confidence interval [CI] 2.22-5.37) and discussing GWG topics with their HCP (AOR 7.96, 95% CI 4.41-14.37), and believing that there were risks to their infants with inadequate GWG (AOR 2.48, 95% CI 1.14-5.37). Half of women (49.5%) indicated that they would recommend the tool to a friend. CONCLUSIONS: Women who received the KT tool reported receiving more counseling on GWG from their HCPs and were more aware of the risks of gaining outside appropriate GWG recommendations. The association between GWG education and GWG requires further research.

Modeling mental health information preferences during the early adult years: a discrete choice conjoint experiment.

Although most young adults with mood and anxiety disorders do not seek treatment, those who are better informed about mental health problems are more likely to use services. The authors used conjoint analysis to model strategies for providing information about anxiety and depression to young adults. Participants (N = 1,035) completed 17 choice tasks presenting combinations of 15 four-level attributes of a mental health information strategy. Latent class analysis yielded 3 segments. The virtual segment (28.7%) preferred working independently on the Internet to obtain information recommended by young adults who had experienced anxiety or depression. Self-assessment options and links to service providers were more important to this segment. Conventional participants (30.1%) preferred books or pamphlets recommended by a doctor, endorsed by mental health professionals, and used with a doctor's support. They would devote more time to information acquisition but were less likely to use Internet social networking options. Brief sources of information were more important to the low interest segment (41.2%). All segments preferred information about alternative ways to reduce anxiety or depression rather than psychological approaches or medication. Maximizing the use of information requires active and passive approaches delivered through old-media (e.g., books) and new-media (e.g., Internet) channels.

Little congruence between health care provider and patient perceptions of counselling on gestational weight gain.

OBJECTIVE: To determine the self-reported counselling practices of health care providers with regard to prenatal weight gain and the risks of inappropriate gain. METHODS: We conducted a cross-sectional survey using a self-administered questionnaire at obstetrician, midwifery, and family medicine clinics in Hamilton, Ontario. Health care providers were eligible to participate if they provided prenatal care and could read English sufficiently well to complete the survey. RESULTS: Forty-two health care providers completed the survey; of these, 95% reported counselling women to gain a specific amount of weight, and 81% reported that they recommended values that were in accordance with the 2009 Institute of Medicine/Health Canada guidelines. The risks of excess and inadequate gain were reported as being discussed with their patients by 87% and 76% of health care providers, respectively. CONCLUSION: In this first study to the best of our knowledge of gestational weight gain counselling since the publication of the 2009 guidelines, most health care providers reported discussing weight gain and the risks of inappropriate gain, which is incongruent with previously published information on their patients' reports of counselling.

Comparison of midwifery, family medicine, and obstetric patients' understanding of weight gain during pregnancy: a minority of women report correct counselling.

OBJECTIVE: We hypothesized that differences in models of care between health care providers would result in variations in patients' reports of counselling. Our objective was to compare what women reported being advised about weight gain during pregnancy and the risks of inappropriate weight gain according to their type of health care provider. METHODS: A cross-sectional survey was conducted using a self-administered questionnaire at obstetric, midwifery, and family medicine clinics in Hamilton, Ontario. Women were eligible to participate if they had had at least one prenatal visit, could read English, and had a live, singleton pregnancy. RESULTS: Three hundred and eight women completed the survey, a 93% response rate. Care for 90% of the group was divided approximately evenly between midwives, family physicians, and obstetricians. A minority of women looked after by any of the types of care providers reported being counselled correctly about how much weight to gain during pregnancy (16.3%, 10.3%, 9.2%, and 5.7% of patients of midwives, family physicians, obstetricians, or other types of care providers, respectively, P = 0.349). A minority of women with any category of care provider was planning to gain an amount of weight that fell within the guidelines or reported being told that there were risks to themselves or their babies with inappropriate gain. CONCLUSION: In this study comparing reported counselling between patients of obstetricians, midwives, family physicians, and other health care providers, low rates of counselling about gestational weight gain were universally reported. There is a common need for more effective counselling.

Despite 2009 guidelines, few women report being counseled correctly about weight gain during pregnancy.

OBJECTIVE: The purpose of this study was to determine the information that pregnant women report receiving when being counseled about weight gain and the risks of inappropriate gain. STUDY DESIGN: With the use of a self-administered questionnaire at prenatal clinics in Hamilton, Ontario, Canada, a cross-sectional survey was conducted of women who had had at least 1 prenatal visit, who could read English, and who had a live singleton gestation. RESULTS: Three hundred ten women completed the survey, which was a 93.6% response rate. Although 28.5% (95% confidence interval, 23.5-33.6%) reported that their health care provider had made a recommendation about how much weight they should gain, only 12.0% (95% confidence interval, 8-16.1%) of the women reported having achieved the recommended weight gain in accordance with the 2009 guidelines. One quarter of the women reported being told that there were risks with inappropriate gain. CONCLUSION: Despite the recent 2009 publication of the gestational weight gain guidelines, only 12% of women reported being counseled correctly, which suggests an urgent need for improved patient education.