Risk Stratification in Primary Care: Value-Based Contributions of Provider Adjudication.

BACKGROUND: In primary care risk stratification, automated algorithms do not consider the same factors as providers. The process of adjudication, in which providers review and adjust algorithm-derived risk scores, may improve the prediction of adverse outcomes. OBJECTIVE: We assessed the patient factors that influenced provider adjudication behavior and evaluated the performance of an adjudicated risk model against a commercial algorithm. DESIGN: (1) Structured interviews with primary care providers (PCP) and multivariable regression analysis and (2) receiver operating characteristic curves (ROC) with sensitivity analyses. PARTICIPANTS: Primary care patients aged 18 years and older with an adjudicated risk score. APPROACH AND MAIN MEASURES: (1) Themes from structured interviews and discrete variables associated with provider adjudication behavior; (2) comparison of concordance statistics and sensitivities between risk models. KEY RESULTS: 47,940 patients were adjudicated by PCPs in 2018. Interviews revealed that, in adjudication, providers consider disease severity, presence of self-management skills, behavioral health, and whether a risk score is actionable. Provider up-scoring from the algorithmic risk score was significantly associated with patient male sex (OR 1.24, CI 1.15-1.34), age > 65 (OR 2.55, CI 2.24-2.91), Black race (1.26, CI 1.02-1.55), polypharmacy >10 medications (OR 4.87, CI 4.27-5.56), a positive depression screen (OR 1.57, CI 1.43-1.72), and hemoglobin A1c >9 (OR 1.89, CI 1.52-2.33). Overall, the adjudicated risk model performed better than the commercial algorithm for all outcomes: ED visits (c-statistic 0.689 vs. 0.684, p < 0.01), hospital admissions (c-statistic 0.663 vs. 0.649, p < 0.01), and death (c-statistic 0.753 vs. 0.721, p < 0.01). When limited to males or seniors, the adjudicated models displayed either improved or non-inferior performance compared to the commercial model. CONCLUSIONS: Provider adjudication of risk stratification improves model performance because providers have a personal understanding of their patients and are able to apply their training to clinical decision-making.

Building Health System Capacity through Medical Education: A Targeted Needs Assessment to Guide Development of a Structured Internal Medicine Curriculum for Medical Interns in Botswana.

BACKGROUND: Medical internship is the final year of training before independent practice for most doctors in Botswana. Internship training in Botswana faces challenges including variability in participants' level of knowledge and skill related to their completion of medical school in a variety of settings (both foreign and domestic), lack of planned curricular content, and limited time for structured educational activities. Data on trainees' opinions regarding the content and delivery of graduate medical education in settings like Botswana are limited, which makes it difficult to revise programs in a learner-centered way. OBJECTIVE: To understand the perceptions and experiences of a group of medical interns in Botswana, in order to inform a large curriculum initiative. METHODS: We conducted a targeted needs assessment using structured interviews at one district hospital. The interview script included demographic, quantitative, and free- response questions. Fourteen interns were asked their opinions about the content and format of structured educational activities, and provided feedback on the preferred characteristics of a new curriculum. Descriptive statistics were calculated. FINDINGS: In the current curriculum, training workshops were the highest-scored teaching format, although most interns preferred lectures overall. Specialists were rated as the most useful teachers, and other interns and medical officers were rated as average. Interns felt they had adequate exposure to content such as HIV and tuberculosis, but inadequate exposure to areas including medical emergencies, non-communicable diseases, pain management, procedural skills, X-ray and EKG interpretation, disclosing medical information, and identifying career goals. For the new curriculum, interns preferred a structured case discussion format, and a focus on clinical reasoning and procedural skills. CONCLUSIONS: This needs assessment identified several foci for development, including a shift toward interactive sessions focused on skill development, the need to empower interns and medical officers to improve teaching skills, and the value of shifting curricular content to mirror the epidemiologic transition occurring in Botswana. Interns' input is being used to initiate a large curriculum intervention that will be piloted and scaled nationally over the next several years. Our results underscore the value of seeking the opinion of trainees, both to aid educators in building programs that serve them and in empowering them to direct their education toward their needs and goals.