Leveraging a Large Language Model to Assess Quality-of-Care: Monitoring ADHD Medication Side Effects.

Objective: To assess the accuracy of a large language model (LLM) in measuring clinician adherence to practice guidelines for monitoring side effects after prescribing medications for children with attention-deficit/hyperactivity disorder (ADHD). Methods: Retrospective population-based cohort study of electronic health records. Cohort included children aged 6-11 years with ADHD diagnosis and ≥2 ADHD medication encounters (stimulants or non-stimulants prescribed) between 2015-2022 in a community-based primary healthcare network (n=1247). To identify documentation of side effects inquiry, we trained, tested, and deployed an open-source LLM (LLaMA) on all clinical notes from ADHD-related encounters (ADHD diagnosis or ADHD medication prescription), including in-clinic/telehealth and telephone encounters (n=15,593 notes). Model performance was assessed using holdout and deployment test sets, compared to manual chart review. Results: The LLaMA model achieved excellent performance in classifying notes that contain side effects inquiry (sensitivity= 87.2%, specificity=86.3/90.3%, area under curve (AUC)=0.93/0.92 on holdout/deployment test sets). Analyses revealed no model bias in relation to patient age, sex, or insurance. Mean age (SD) at first prescription was 8.8 (1.6) years; patient characteristics were similar across patients with and without documented side effects inquiry. Rates of documented side effects inquiry were lower in telephone encounters than in-clinic/telehealth encounters (51.9% vs. 73.0%, p<0.01). Side effects inquiry was documented in 61% of encounters following stimulant prescriptions and 48% of encounters following non-stimulant prescriptions (p<0.01). Conclusions: Deploying an LLM on a variable set of clinical notes, including telephone notes, offered scalable measurement of quality-of-care and uncovered opportunities to improve psychopharmacological medication management in primary care.

Measuring quality-of-care in treatment of young children with attention-deficit/hyperactivity disorder using pre-trained language models.

OBJECTIVE: To measure pediatrician adherence to evidence-based guidelines in the treatment of young children with attention-deficit/hyperactivity disorder (ADHD) in a diverse healthcare system using natural language processing (NLP) techniques. MATERIALS AND METHODS: We extracted structured and free-text data from electronic health records (EHRs) of all office visits (2015-2019) of children aged 4-6 years in a community-based primary healthcare network in California, who had ≥1 visits with an ICD-10 diagnosis of ADHD. Two pediatricians annotated clinical notes of the first ADHD visit for 423 patients. Inter-annotator agreement (IAA) was assessed for the recommendation for the first-line behavioral treatment (F-measure = 0.89). Four pre-trained language models, including BioClinical Bidirectional Encoder Representations from Transformers (BioClinicalBERT), were used to identify behavioral treatment recommendations using a 70/30 train/test split. For temporal validation, we deployed BioClinicalBERT on 1,020 unannotated notes from other ADHD visits and well-care visits; all positively classified notes (n = 53) and 5% of negatively classified notes (n = 50) were manually reviewed. RESULTS: Of 423 patients, 313 (74%) were male; 298 (70%) were privately insured; 138 (33%) were White; 61 (14%) were Hispanic. The BioClinicalBERT model trained on the first ADHD visits achieved F1 = 0.76, precision = 0.81, recall = 0.72, and AUC = 0.81 [0.72-0.89]. Temporal validation achieved F1 = 0.77, precision = 0.68, and recall = 0.88. Fairness analysis revealed low model performance in publicly insured patients (F1 = 0.53). CONCLUSION: Deploying pre-trained language models on a variable set of clinical notes accurately captured pediatrician adherence to guidelines in the treatment of children with ADHD. Validating this approach in other patient populations is needed to achieve equitable measurement of quality of care at scale and improve clinical care for mental health conditions.

Anxiety and Depression Treatment in Primary Care Pediatrics.

BACKGROUND AND OBJECTIVES: Primary care pediatricians (PCP) are often called on to manage child and adolescent anxiety and depression. The objective of this study was to describe PCP care practices around prescription of selective serotonin reuptake inhibitors (SSRI) for patients with anxiety and/or depression by using medical record review. METHODS: We identified 1685 patients who had at least 1 visit with a diagnosis of anxiety and/or depression in a large primary care network and were prescribed an SSRI by a network PCP. We randomly selected 110 for chart review. We reviewed the visit when the SSRI was first prescribed (medication visit), immediately previous visit, and immediately subsequent visit. We abstracted rationale for prescribing medication, subspecialist involvement, referral for psychotherapy, and medication monitoring practices. RESULTS: At the medication visit, in 82% (n = 90) of cases, PCPs documented reasons for starting an SSRI, most commonly clinical change (57%, n = 63). Thirty percent (n = 33) of patients had documented involvement of developmental-behavioral pediatrics or psychiatry subspecialists at 1 of the 3 visits reviewed. Thirty-three percent (n = 37) were referred to unspecified psychotherapy; 4% (n = 4) were referred specifically for cognitive behavioral therapy. Of 69 patients with a subsequent visit, 48% (n = 33) had documentation of monitoring for side effects. CONCLUSIONS: When prescribing SSRIs for children with anxiety and/or depression, PCPs in this network documented appropriate indications for starting medication and prescribed without subspecialist involvement. Continuing medical education for PCPs who care for children with these conditions should include information about evidence-based psychotherapy and strategies for monitoring potential side effects.

Detection of Speech-Language Delay in the Primary Care Setting: An Electronic Health Record Investigation.

OBJECTIVE: The purpose of this study is to determine the rate and age at first identification of speech-language delay in relation to child sociodemographic variables among a pediatric primary care network. METHODS: We analyzed a deidentified data set of electronic health records of children aged 1- to 5-years-old seen between 2015 and 2019 at 10 practices of a community-based pediatric primary health care network. Primary outcomes were numbers (proportions) of patients with relevant ICD-10 visit-diagnosis codes and patient age (months) at first documentation of speech-language delay. Regression models estimated associations between outcomes and patient characteristics, adjusting for practice affiliation. RESULTS: Of 14,559 included patients, 2063 (14.1%) had speech-language delay: 68.4% males, 74.4% with private insurance, and 96.1% with English as a primary household language. Most patients (60%) were first identified at the 18- or 24-month well-child visit. The mean age at first documentation was 25.4 months (SD = 9.3), which did not differ between practices reporting the use of standardized developmental screener and those using surveillance questionnaires. Regression models showed that males were more than twice as likely than females to be identified with speech-language delay (adjusted odds ratio [aOR] = 2.05, 95% CI: [1.86-2.25]); publicly insured were more likely than privately insured patients to be identified with speech-language delay (aOR = 1.48, 95% CI: [1.30-1.68]). Females were older than males at first identification (+1.2 months, 95% CI: [0.3-2.1]); privately insured were older than military insured patients (private +3.3 months, 95% CI: [2.2-4.4]). CONCLUSION: Pediatricians in this network identified speech-language delays at similar rates to national prevalence. Further investigation is needed to understand differences in speech-language delay detection across patient subgroups in practices that use developmental screening and/or surveillance.

Continuity of Care in Primary Care for Young Children With Chronic Conditions.

OBJECTIVES: 1) To assess continuity of care (CoC) within primary-care practices for children with asthma and autism spectrum disorder (ASD) compared to children without chronic conditions, and 2) to determine patient and clinical-care factors associated with CoC. METHODS: Retrospective cohort study of electronic health records from office visits of children <9 years, seen ≥4 times between 2015 and 2019 in 10 practices of a community-based primary health care network in California. Three cohorts were constructed: 1) Asthma: ≥2 visits with asthma visit diagnoses; 2) ASD: same method; 3) Controls: no chronic conditions. CoC, using Usual Provider of Care measure (range > 0-1), was calculated for 1) all visits (overall) and 2) well-care visits. Fractional regression models examined CoC adjusting for patient age, medical insurance, practice affiliation, and number of visits. RESULTS: Of 30,678 children, 1875 (6.1%) were classified with Asthma, 294 (1.0%) with ASD, and 15,465 (50.4%) as Controls. Overall CoC was lower for Asthma (Mean = 0.58, SD 0.21) and ASD (M = 0.57, SD = 0.20) than Controls (M = 0.66, SD = 0.21); differences in well-care CoC were minimal. In regression models, lower overall CoC was found for Asthma (aOR = 0.90, 95% CI, 0.85-0.94). Lower overall and well-care CoC were associated with public insurance (aOR = 0.77, CI, 0.74-0.81; aOR = 0.64, CI, 0.59-0.69). CONCLUSION: After accounting for patient and clinical-care factors, children with asthma, but not with ASD, in this primary-care network had significantly lower CoC compared to children without chronic conditions. Public insurance was the most prominent patient factor associated with low CoC, emphasizing the need to address disparities in CoC.

Primary Care Diagnosis and Treatment of Attention-Deficit/Hyperactivity Disorder in School-Age Children: Trends and Disparities During the COVID-19 Pandemic.

OBJECTIVE: The aim of this study was to assess rates of primary care provider (PCP) diagnosis and treatment of school-age children with attention-deficit/hyperactivity disorder (ADHD) during the COVID-19 pandemic compared with prepandemic years and to investigate disparities in care. METHOD: We retrospectively analyzed electronic health records from all primary care visits (in-person and telehealth) of children aged 6 to 17 years seen between January 2016 and March 2021 in a community-based primary health care network (n = 77,298 patients). Study outcomes are as follows: (1) number of primary care visits, (2) number of visits with ADHD diagnosis (ADHD-related visits), (3) number of PCP prescriptions for ADHD medications, (4) number of patients with first ADHD diagnoses, and (5) number of first PCP prescriptions of ADHD medications. Interrupted time series analysis evaluated changes in rates of study outcomes during 4 quarters of the pandemic year (March 15, 2020-March 15, 2021) compared with prepandemic years (January 1, 2016-March 14, 2020). Patient demographic characteristics during prepandemic and pandemic years were compared. RESULTS: ADHD-related visits dropped in the first quarter of the pandemic year by 33% (95% confidence interval, 22.2%-43.6%), returning to prepandemic rates in subsequent quarters. ADHD medication prescription rates remained stable throughout the pandemic year. Conversely, rates of first ADHD diagnoses and first medication prescriptions remained significantly lower than prepandemic rates. The proportion of ADHD-related visits for patients living in low-income neighborhoods was lower in the pandemic year compared with prepandemic years. CONCLUSION: Ongoing treatment for school-age children with ADHD was maintained during the pandemic, especially in high-income families. Socioeconomic differences in ADHD-related care emphasize the need to improve access to care for all children with ADHD in the ongoing pandemic and beyond.

Medication Management of Anxiety and Depression by Primary Care Pediatrics Providers: A Retrospective Electronic Health Record Study.

Objectives: To describe medication management of children diagnosed with anxiety and/or depression by primary care providers within a primary care network. Study Design/Methods: We performed a retrospective cross-sectional analysis of electronic health record (EHR) structured data from all children seen at least twice in a 4-year observation period within a network of primary care clinics in Northern California. For children who had visit diagnoses of anxiety, depression, anxiety+depression or symptoms characteristic of these conditions, we analyzed the rates and types of medications prescribed. A logistic regression model considered patient variables for the combined sample. Results: Of all patients 6-18 years old (N = 59,484), 4.4% (n = 2,635) had a diagnosis of anxiety only, 2.4% (n = 1,433) depression only, and 1.2% (n = 737) both anxiety and depression (anxiety + depression); 18% of children with anxiety and/or depression had comorbid ADHD. A total of 15.0% with anxiety only (n = 357), 20.5% with depression only (n = 285), and 47.4% with anxiety+depression (n=343) were prescribed a psychoactive non-stimulant medication. For anxiety and depression only, the top three medications prescribed were sertraline, fluoxetine, and citalopram. For anxiety + depression, the top three medications prescribed were citalopram, sertraline, and escitalopram. Frequently prescribed medications also included benzodiazepines. Logistic regression modeling showed that the depression only and anxety + depression categories had increased likelihood of medication prescription. Older age and mental health comorbidities were independently associated with increased likelihood of medication prescription. Conclusions: In this network, ~8% of children carried a diagnosis of anxiety and/or depression. Medication choices generally aligned with current recommendations with the exception of use of benzodiazepines.

Attention-Deficit/Hyperactivity Disorder in 2- to 5-Year-Olds: A Primary Care Network Experience.

OBJECTIVE: To assess 1) rates of primary care provider (PCP) diagnosis of attention-deficit/hyperactivity disorder (ADHD) in young children, 2) documented PCP adherence to ADHD clinical practice guidelines, and 3) patient factors influencing PCP variation in diagnosis and management. METHODS: Retrospective cohort study of electronic health records from all office visits of children aged 2 to 5 years, seen ≥2 times between 2015 and 2019, in 10 practices of a community-based primary health care network. Outcomes included ADHD diagnosis (symptom or disorder), and adherence to guidelines in 1) comorbidity documentation at or after ADHD diagnosis, 2) ADHD medication choice, and 3) follow-up of medicated patients. Logistic regressions assessed associations between outcomes and patient characteristics. RESULTS: Of 29,408 eligible children, 195 (0.7%) had ADHD diagnoses. Of those, 56% had solely symptom-level diagnoses (eg, hyperactivity); 54% had documented comorbidities. ADHD medications were prescribed only to 4- to 5-year olds (40 of 195 [21%]); 85% received stimulants as first-line medication; 48% had follow-up visits within 2 months. Likelihood of ADHD diagnosis was higher for children with public or military insurance (odds ratio [OR] 1.94; 95% confidence interval [CI] 1.40-2.66; OR 3.17; 95% CI 1.93-4.96). Likelihood of comorbidity documentation was lower for older ADHD patients (OR 0.48; 95% CI 0.32-0.71) and higher for those with military insurance (OR 3.11; 95% CI 1.13-9.58). CONCLUSION: PCPs in this network frequently used symptom-level ADHD diagnoses in 2- to 5-year olds; ADHD diagnosis rates were below estimated population prevalence, with evidence for sociodemographic disparities. PCP comorbidity documentation and choice of stimulant medications were consistent with guidelines. Rates of timely follow-up were low.

Variation in Rate of Attention-Deficit/Hyperactivity Disorder Management by Primary Care Providers.

OBJECTIVE: To describe variation in rates of attention-deficit/hyperactivity disorder (ADHD) management by pediatrics primary care providers (PCPs) and to assess influence of clinician characteristics on variation. METHODS: Retrospective cohort study of electronic health records from all office visits of patients aged 4 to 17 years seen at least twice between 2015 and 2017 by 73 clinicians in 9 pediatrics practices of a community-based primary health care network in California. Outcomes per clinician: 1) percent patients seen for ADHD management; (2) percent ADHD patients with diagnosed comorbid conditions. Logistic random-effects regression models examined practice- and clinician-level variation. RESULTS: Of 40,323 patients in the cohort, 2039 (5.1%) carried an ADHD diagnosis, of which 1142 (56%) received ADHD medication. Percent of patients seen for ADHD management varied by clinician from 0.0% to 8.3% (median 3.0%). After accounting for practice-level variation and patient characteristics (ie, sex, age, insurance), clinician characteristics explained 28% of clinician variation in ADHD management. ADHD management rate was associated with high-percent full-time equivalent (odds ratio 1.17; 95% confidence interval 1.07-1.27). Percent of ADHD patients with diagnoses of comorbidities varied by clinician from 0.0% to 100% (median 35%). Association between ADHD management rate and comorbidity diagnosis was minimal (R = 0.10). CONCLUSIONS: Objective electronic health records measures showed that PCPs in this network varied widely in their involvement in ADHD management. For most PCPs, percent of patients with ADHD and diagnosis of comorbidities was lower than estimated prevalence rates. Exploration of modifiable factors associated with PCP variation is needed to inform strategies for implementation of evidence-based practices.

Amnesia for traumatic events among recent survivors: a pilot study.

OBJECTIVE: Traumatic amnesia has been amply documented in the psychoanalytic literature but inconsistently in the research literature. METHOD: Six trauma were followed prospectively. Survivors were interviewed 7, 30, and 120 days following the traumatic event. Each interview documented in detail their recollections of the day of their trauma. RESULTS: In four subjects who did not develop posttraumatic stress disorder (PTSD), we found brief, stable, and persistent memory gaps, which coincided with the moment of greatest emotional intensity. In two subjects who developed PTSD, we found, in addition to the previous form of amnesia, longer, progressive, and unstable memory gaps. DISCUSSION: Neurobiological research offers two explanatory mechanisms for the observations: A failure of acquisition of episodic memories may account for the stable deficits seen in all subjects. This could coincide with stress-induced malfunction of the hippocampal declarative memory system. A failure of spontaneous recall may account for the more extended traumatic amnesia that was observed in PTSD patients. This resembles the psychoanalytic description of repression. CONCLUSION: These preliminary findings suggest that brief, irreversible memory gaps are common in trauma survivors, whereas longer, progressive, and potentially reversible amnesia occurs among survivors who develop PTSD.