Using Concurrent Complication Reporting to Evaluate Resident Critical Thinking and Enhance Adult Learning.

OBJECTIVE: Critical thinking and accurate case analysis is difficult to quantify even within the context of routine morbidity and mortality reporting. We designed and implemented a HIPAA-compliant adverse outcome reporting system that collects weekly resident assessments of clinical care across multiple domains (case summary, complications, error analysis, Clavien-Dindo Harm, cognitive bias, standard of care, and ACGME core competencies). We hypothesized that incorporation of this system into the residency program's core curriculum would allow for identification of areas of cognitive weakness or strength and provide a longitudinal evaluation of critical thinking development. DESIGN: A validated, password-protected electronic platform linked to our electronic medical record was used to collect cases weekly in which surgical adverse events occurred. General surgery residents critiqued 1932 cases over a 4-year period from 3 major medical centers within our system. These data were reviewed by teaching faculty, corrected for accuracy and graded utilizing the software's critique algorithm. Grades were emailed to the residents at the time of the review, collected prospectively, stratified, and analyzed by post-graduate year (PGY). Evaluation of the resident scores for each domain and the resultant composite scores allowed for comparison of critical thinking skills across post-graduate year (PGY) over time. SETTING: Data was collected from 3 independently ACGME-accredited surgery residency programs over 3 tertiary hospitals within our health system. PARTICIPANTS: General surgery residents in clinical PGY 1-5. RESULTS: Residents scored highest in properly identifying ACGME core competencies and determining Clavien-Dindo scores (p < 0.006) with no improvement in providing accurate and concise clinical summaries. However, residents improved in recording data sufficient to identify error (p < 0.00001). A positive linear trend in median scores for all remaining domains except for cognitive bias was demonstrated (p < 0.001). Senior residents scored significantly higher than junior residents in all domains. Scores > 90% were never achieved. CONCLUSIONS: The use of an electronic standardized critique algorithm in the evaluation and assessment of adverse surgical case outcomes enabled the measure of residents' critical thinking skills. Feedback in the form of teaching faculty-facilitated discussion and emailed grades enhanced adult learning with a steady improvement in performance over PGY. Although residents improved with PGY, the data suggest that further improvement in all categories is possible. Implementing this standardized critique algorithm across PGY allows for evaluation of areas of individual resident weakness vs. strength, progression over time, and comparisons to peers. These data suggest that routine complication reporting may be enhanced as a critical thinking assessment tool and that improvement in critical thinking can be quantified. Incorporation of this platform into M&M conference has the potential to augment executive function and professional identity development.

How trace plots help interpret meta-analysis results.

The trace plot is seldom used in meta-analysis, yet it is a very informative plot. In this article, we define and illustrate what the trace plot is, and discuss why it is important. The Bayesian version of the plot combines the posterior density of τ , the between-study standard deviation, and the shrunken estimates of the study effects as a function of τ . With a small or moderate number of studies, τ is not estimated with much precision, and parameter estimates and shrunken study effect estimates can vary widely depending on the correct value of τ . The trace plot allows visualization of the sensitivity to τ along with a plot that shows which values of τ are plausible and which are implausible. A comparable frequentist or empirical Bayes version provides similar results. The concepts are illustrated using examples in meta-analysis and meta-regression; implementation in R is facilitated in a Bayesian or frequentist framework using the bayesmeta and metafor packages, respectively.

Practice-Based Learning and Improvement: Improving Morbidity and Mortality Review Using Natural Language Processing.

INTRODUCTION: Practice-Based Learning and Improvement, a core competency identified by the Accreditation Council for Graduate Medical Education, carries importance throughout a physician's career. Practice-Based Learning and Improvement is cultivated by a critical review of complications, yet methods to accurately identify complications are inadequate. Machine-learning algorithms show promise in improving identification of complications. We compare a manual-supplemented natural language processing (ms-NLP) methodology against a validated electronic morbidity and mortality (MM) database, the Morbidity and Mortality Adverse Event Reporting System (MARS) to understand the utility of NLP in MM review. METHODS: The number and severity of complications were compared between MARS and ms-NLP of surgical hospitalization discharge summaries among three academic medical centers. Clavien-Dindo (CD) scores were assigned to cases with identified complications and classified into minor (CD I-II) or major (CD III-IV) harm. RESULTS: Of 7774 admissions, 987 cases were identified to have 1659 complications by MARS and 1296 by ms-NLP. MARS identified 611 (62%) cases, whereas ms-NLP identified 670 (68%) cases. Less than one-third of cases (299, 30.3%) were detected by both methods. MARS identified a greater number of complications with major harm (457, 46.30%) than did ms-NLP (P < 0.0001). CONCLUSIONS: Both a prospectively maintained MM database and ms-NLP review of discharge summaries fail to identify a significant proportion of postoperative complications and overlap 1/3 of the time. ms-NLP more frequently identifies cases with minor complications, whereas prospective voluntary reporting more frequently identifies major complications. The educational benefit of reporting and analysis of complication data may be supplemented by ms-NLP but not replaced by it at this time.

Cognitive bias and severity of harm following surgery: Plan for workflow debiasing strategy.

INTRODUCTION: This study analyzes the relationship between cognitive bias (CB) and harm severity as measured by Clavien-Dindo Scores (CD). METHODS: A prospectively collected series of 655 severity matched general surgical cases with complications were analyzed. Cases were evaluated for CB and assigned harm scores as defined by CD grade. Potentially mitigating "debiasing" strategies were identified for each bias attribution. RESULTS: Among cases with CB, 24% (55/232) were CD(I-II) and 76% (177/232) were CD(III-V). Odds ratio suggests that serious complications occur nearly 60% more frequently when CB is identified. The CBs identified with severe harm were Overconfidence, Commission, Anchoring, Confirmation, and Diagnosis Momentum. Preliminary data on debiasing strategies suggest diagnosis review, linear reasoning and Type II thinking may be relevant in over 85% of complications. CONCLUSION: The incidence of CB is increased in patients sustaining severe harm. Understanding the specific CBs identified and their mitigating debiasing strategies may improve outcomes.

Reporting Bayesian Results.

Because of the different philosophy of Bayesian statistics, where parameters are random variables and data are considered fixed, the analysis and presentation of results will differ from that of frequentist statistics. Most importantly, the probabilities that a parameter is in certain regions of the parameter space are crucial quantities in Bayesian statistics that are not calculable (or considered important) in the frequentist approach that is the basis of much of traditional statistics. In this article, I discuss the implications of these differences for presentation of the results of Bayesian analyses. In doing so, I present more detailed guidelines than are usually provided and explain the rationale for my suggestions.

Overview of Bayesian Statistics.

Bayesian statistics is becoming a popular approach to handling complex statistical modeling. This special issue of Evaluation Review features several Bayesian contributions. In this overview, I present the basics of Bayesian inference. Bayesian statistics is based on the principle that parameters have a distribution of beliefs about them that behave exactly like probability distributions. We can use Bayes' Theorem to update our beliefs about values of the parameters as new information becomes available. Even better, we can make statements that frequentists do not, such as "the probability that an effect is larger than 0 is .93," and can interpret 95% (e.g.) intervals as people naturally want, that there is a 95% probability that the parameter is in that interval. I illustrate the basic concepts of Bayesian statistics through a simple example of predicting admissions to a PhD program.

Using Bayesian Correspondence Criteria to Compare Results From a Randomized Experiment and a Quasi-Experiment Allowing Self-Selection.

BACKGROUND: Randomized experiments yield unbiased estimates of treatment effect, but such experiments are not always feasible. So researchers have searched for conditions under which randomized and nonrandomized experiments can yield the same answer. This search requires well-justified and informative correspondence criteria, that is, criteria by which we can judge if the results from an appropriately adjusted nonrandomized experiment well-approximate results from randomized experiments. Past criteria have relied exclusively on frequentist statistics, using criteria such as whether results agree in sign or statistical significance or whether results differ significantly from each other. OBJECTIVES: In this article, we show how Bayesian correspondence criteria offer more varied, nuanced, and informative answers than those from frequentist approaches. RESEARCH DESIGN: We describe the conceptual bases of Bayesian correspondence criteria and then illustrate many possibilities using an example that compares results from a randomized experiment to results from a parallel nonequivalent comparison group experiment in which participants could choose their condition. RESULTS: Results suggest that, in this case, the quasi-experiment reasonably approximated the randomized experiment. CONCLUSIONS: We conclude with a discussion of the advantages (computation of relevant quantities, interpretation, and estimation of quantities of interest for policy), disadvantages, and limitations of Bayesian correspondence criteria. We believe that in most circumstances, the advantages of Bayesian approaches far outweigh the disadvantages.

Multilevel modeling of single-case data: A comparison of maximum likelihood and Bayesian estimation.

The focus of this article is to describe Bayesian estimation, including construction of prior distributions, and to compare parameter recovery under the Bayesian framework (using weakly informative priors) and the maximum likelihood (ML) framework in the context of multilevel modeling of single-case experimental data. Bayesian estimation results were found similar to ML estimation results in terms of the treatment effect estimates, regardless of the functional form and degree of information included in the prior specification in the Bayesian framework. In terms of the variance component estimates, both the ML and Bayesian estimation procedures result in biased and less precise variance estimates when the number of participants is small (i.e., 3). By increasing the number of participants to 5 or 7, the relative bias is close to 5% and more precise estimates are obtained for all approaches, except for the inverse-Wishart prior using the identity matrix. When a more informative prior was added, more precise estimates for the fixed effects and random effects were obtained, even when only 3 participants were included. (PsycINFO Database Record

Synthesizing Evidence in Public Policy Contexts: The Challenge of Synthesis When There Are Only a Few Studies.

For a variety of reasons, researchers and evidence-based clearinghouses synthesizing the results of multiple studies often have very few studies that are eligible for any given research question. This situation is less than optimal for meta-analysis as it is usually practiced, that is, by employing inverse variance weights, which allows more informative studies to contribute relatively more to the analysis. This article outlines the choices available for synthesis when there are few studies to synthesize. As background, we review the synthesis practices used in several projects done at the behest of governmental agencies and private foundations. We then discuss the strengths and limitations of different approaches to meta-analysis in a limited information environment. Using examples from the U.S. Department of Education's What Works Clearinghouse as case studies, we conclude with a discussion of Bayesian meta-analysis as a potential solution to the challenges encountered when attempting to draw inferences about the effectiveness of interventions from a small number of studies.

Predictors of Depression Among Adult Women With Diabetes in the United States: An Analysis Using National Health and Nutrition Examination Survey Data From 2007 to 2012.

PURPOSE: The purpose of the study was to identify the sex-specific characteristics that predict depression among adult women with diabetes. METHODS: Data from the 2007-2012 National Health and Nutrition Examination Survey in the United States were used to identify the predictors of depression in a large sample of women ages 20 years and older with diabetes (n = 946). RESULTS: When extrapolated to almost 9 million women in the United States ≥ 20 years of age with diabetes, 19.0% had depression. Female-specific significant predictors of depression included younger age (< 65 years old), less than high school graduation, self-rated fair or poor health, inactivity due to poor health, and pain that interferes with usual activities. Marital status and diabetes-related factors (years living with diabetes, use of insulin, parent or sibling with diabetes) were not significant predictors of depression in adult women with diabetes. CONCLUSION: When educating and counseling women with diabetes, diabetes educators should be aware that some of the predictors of depression in women with diabetes differ from those of populations that include both sexes. Depression screening, although important for all women with diabetes, should especially be performed among women with female-specific depression predictors.

Single-case experimental design yielded an effect estimate corresponding to a randomized controlled trial.

OBJECTIVES: We reanalyzed data from a previous randomized crossover design that administered high or low doses of intravenous immunoglobulin (IgG) to 12 patients with hypogammaglobulinaemia over 12 time points, with crossover after time 6. The objective was to see if results corresponded when analyzed as a set of single-case experimental designs vs. as a usual randomized controlled trial (RCT). STUDY DESIGN AND SETTINGS: Two blinded statisticians independently analyzed results. One analyzed the RCT comparing mean outcomes of group A (high dose IgG) to group B (low dose IgG) at the usual trial end point (time 6 in this case). The other analyzed all 12 time points for the group B patients as six single-case experimental designs analyzed together in a Bayesian nonlinear framework. RESULTS: In the randomized trial, group A [M = 794.93; standard deviation (SD) = 90.48] had significantly higher serum IgG levels at time six than group B (M = 283.89; SD = 71.10) (t = 10.88; df = 10; P < 0.001), yielding a mean difference of MD = 511.05 [standard error (SE) = 46.98]. For the single-case experimental designs, the effect from an intrinsically nonlinear regression was also significant and comparable in size with overlapping confidence intervals: MD = 495.00, SE = 54.41, and t = 495.00/54.41 = 9.10. Subsequent exploratory analyses indicated that how trend was modeled made a difference to these conclusions. CONCLUSIONS: The results of single-case experimental designs accurately approximated results from an RCT, although more work is needed to understand the conditions under which this holds.

Computerized cognitive training and functional recovery in major depressive disorder: A meta-analysis.

OBJECTIVE: Depression is common, frequently resistant to antidepressant treatment, and associated with impairments in cognition and everyday functioning. Computerized cognitive training (CCT) paradigms offer potential to improve cognition, mood and everyday functioning, but their effectiveness is not well established. The goal of this article was to conduct a systematic review and meta-analysis to determine the efficacy of CCT in depressive disorders. METHOD: A search was conducted to identify high quality randomized controlled CCT trials per PRISMA guidelines using PsycINFO and MEDLINE with the keywords "Cognitive training" or "Cognitive remediation" or "Cognitive rehabilitation" and "Depression". 9 randomized trials for depressed adults met inclusion criteria. Effect sizes (Hedge's g) were calculated for key outcome measures of mood symptom severity, daily functioning, and cognition. A 3-level Bayesian hierarchical linear model was used to estimate effect sizes for each domain and study. Publication bias was assessed using Classic Fail Safe N's and homogeneity was evaluated using Q and I(2) indexes. RESULTS: Significant small-moderate effects for Symptom Severity (0.43) and Daily Functioning (0.72), and moderate-large effects for Attention (0.67), Working Memory (0.72), and Global Functioning (1.05) were found. No significant effects were found for Executive Functioning or Verbal Memory. Moderator variable analysis revealed decreased effect of CCT with age. Gender and concurrent medication treatment did not affect the results. LIMITATIONS: Small sample size, short duration, pseudo-specificity, and high heterogeneity for Verbal Memory measures. CONCLUSIONS: CCT is associated with improvement in depressive symptoms and everyday functioning, though produces inconsistent effects on cognition.

A Meta-Analysis of Executive Dysfunction and Antidepressant Treatment Response in Late-Life Depression.

OBJECTIVE: Depressed older adults with executive dysfunction (ED) may respond poorly to antidepressant treatment. ED is a multifaceted construct and different studies have measured different aspects of ED, making it unclear which aspects predict poor response. Meta-analytic methods were used to determine whether ED predicts poor antidepressant treatment response in late-life depression and to determine which domains of executive functioning are responsible for this relationship. METHODS: A Medline search was conducted to identify regimented treatment trials contrasting executive functioning between elderly responders and nonresponders; only regimented treatment trials for depressed outpatients aged 50 and older were included. Following the most recent PRISMA guidelines, 25 measures of executive functioning were extracted from eight studies. Six domains were identified: cognitive flexibility, planning and organization, response inhibition, selective attention, verbal fluency, and the Dementia Rating Scale Initiation/Perseveration composite score (DRS I/P). Hedge's g was calculated for each measure of executive functioning. A three-level Bayesian hierarchical linear model (HLM) was used to estimate effect sizes for each domain of executive functioning. RESULTS: The effect of planning and organization was significantly different from zero (Bayesian HLM estimate of domain effect size: 0.91; 95% CI: 0.32-1.58), whereas cognitive flexibility, response inhibition, selective attention, verbal fluency, and the DRS I/P composite score were not. CONCLUSION: The domain of planning and organization is meaningfully associated with poor antidepressant treatment response in late-life depression. These findings suggest that therapies that focus on planning and organization may provide effective augmentation strategies for antidepressant nonresponders with late-life depression.

The potential for glycemic control monitoring and screening for diabetes at dental visits using oral blood.

OBJECTIVES: We examined the potential for glycemic control monitoring and screening for diabetes in a dental setting among adults (n = 408) with or at risk for diabetes. METHODS: In 2013 and 2014, we performed hemoglobin A1c (HbA1c) tests on dried blood samples of gingival crevicular blood and compared these with paired "gold-standard" HbA1c tests with dried finger-stick blood samples in New York City dental clinic patients. We examined differences in sociodemographics and diabetes-related risk and health care characteristics for 3 groups of at-risk patients. RESULTS: About half of the study sample had elevated HbA1c values in the combined prediabetes and diabetes ranges, with approximately one fourth of those in the diabetes range. With a correlation of 0.991 between gingival crevicular and finger-stick blood HbA1c, measures of concurrence between the tests were extremely high for both elevated HbA1c and diabetes-range HbA1c levels. Persons already diagnosed with diabetes and undiagnosed persons aged 45 years or older could especially benefit from HbA1c testing at dental visits. CONCLUSIONS: Gingival crevicular blood collected at the dental visit can be used to screen for diabetes and monitor glycemic control for many at-risk patients.

Latinas with elevated fasting plasma glucose: an analysis using NHANES 2009-2010 data.

For Latinas with fasting plasma glucose (FPG) levels in the prediabetes and diabetes ranges, early detection can support steps to optimize their health. Data collected in 2009-2010 indicate that 36.7% of Latinas in the United States had elevated FPG levels. Latinas with elevated FPG who were unaware of their diabetes status were significantly less likely than non-Hispanic White and non-Hispanic Black women to have seen a health care provider in the past year (75.8%, 92.9%, and 90.2%, respectively; p = .018). With almost 1 million Latinas in the United States with elevated FPG unaware of their diabetes risk, and less likely than other at-risk women to see health care providers, there is an urgent need to establish alternate sites of opportunity for their diabetes screening.

Nonlinear Bayesian analysis for single case designs.

Several authors have suggested the use of multilevel models for the analysis of data from single case designs. Multilevel models are a logical approach to analyzing such data, and deal well with the possible different time points and treatment phases for different subjects. However, they are limited in several ways that are addressed by Bayesian methods. For small samples Bayesian methods fully take into account uncertainty in random effects when estimating fixed effects; the computational methods now in use can fit complex models that represent accurately the behavior being modeled; groups of parameters can be more accurately estimated with shrinkage methods; prior information can be included; and interpretation is more straightforward. The computer programs for Bayesian analysis allow many (nonstandard) nonlinear models to be fit; an example using floor and ceiling effects is discussed here.

Bayesian analysis of data from single case designs.

Bayesian statistical methods have great potential advantages for the analysis of data from single case designs. Bayesian methods combine prior information with data from a study to form a posterior distribution of information about their parameters and functions. The interpretation of results from a Bayesian analysis is more natural than those from classical methods, and there are interpretations of useful quantities that are not possible in classical statistics, such as the probability that an effect size is small, or is greater than zero, or is large enough to be considered important. They are not based on asymptotic theory, so small sample size is not a problem for inference. These methods are implemented on free software, and are similar to non-Bayesian software, so analysts familiar with frequentist methods for multilevel data should find the transition relatively painless.

Neuropsychological functioning and severity of ADHD in early childhood: a four-year cross-lagged study.

Children with attention deficit/hyperactivity disorder (ADHD) have poorer neuropsychological functioning relative to their typically developing peers. However, it is unclear whether early neuropsychological functioning predicts later ADHD severity and/or the latter is longitudinally associated with subsequent neuropsychological functioning; and whether these relations are different in children with and without early symptoms of ADHD. This study aimed to examine the longitudinal associations between ADHD severity and neuropsychological functioning among children at high and low risk of developing ADHD. Hyperactive/Inattentive (H/I; n = 140) and Typically developing (TD; n = 76) preschoolers (age 3-4 years) were recruited (BL) and followed annually for 3 years (F1, F2, and F3). Teachers rated the children's ADHD severity and impairment using the Behavior Assessment System for Children-2 and the Children's Problem Checklist, respectively. Parent reports of children's ADHD severity were obtained using the Kiddie-Schedule for Affective Disorders and Schizophrenia - Present and Lifetime version. Neuropsychological functioning was assessed using the NEPSY. In the full sample, there were bidirectional longitudinal associations between neuropsychological functioning and ADHD severity between F1 and F3. Among H/I children, neuropsychological functioning at F1 and F2 predicted ADHD severity at F2 and F3, respectively. In contrast, among TD children the only significant relationship observed was that elevated ADHD symptoms at F2 were associated with poorer neuropsychological functioning at F3. Improved neuropsychological functioning may attenuate ADHD symptoms and associated impairment among H/I children during the early school years. Interventions designed to improve neuropsychological functioning among young H/I children may be beneficial in reducing their ADHD severity.