Variations in senior medical student diagnostic justification ability.

PURPOSE: To determine the diagnostic justification proficiency of senior medical students across a broad spectrum of cases with common chief complaints and diagnoses. METHOD: The authors gathered diagnostic justification exercise data from the Senior Clinical Comprehensive Examination taken by Southern Illinois University School of Medicine's students from the classes of 2011 (n = 67), 2012 (n = 66), and 2013 (n = 79). After interviewing and examining standardized patients, students listed their key findings and diagnostic possibilities considered, and provided a written explanation of how they used key findings to move from their initial differential diagnoses to their final diagnosis. Two physician judges blindly rated responses. RESULTS: Student diagnostic justification performance was highly variable from case to case and often rated below expectations. Of the students in the classes of 2011, 2012, and 2013, 57% (38/67), 23% (15/66), and 33% (26/79) were judged borderline or poor on diagnostic justification performance for more than 50% of the cases on the examination. CONCLUSIONS: Student diagnostic justification performance was inconsistent across the range of cases, common chief complaints, and underlying diagnoses used in this study. More than 20% of students exhibited borderline or poor diagnostic justification performance on more than 50% of the cases. If these results are confirmed in other medical schools, attention needs to be directed to investigating new curricular methods that ensure deliberate practice of these competencies across the spectrum of common chief complaints and diagnoses and do not depend on the available mix of patients.

Theory development and application in medical education.

The role and status of theory is by no means a new topic in medical education. Yet summarizing where we have been and where we are going with respect to theory development and application is difficult because our community has not yet fully elucidated what constitutes medical education theory. In this article, we explore the idea of conceptualizing theory as an effect on scholarly dialogue among medical educators. We describe theory-enabled conversation as argumentation, which frames inquiry, permits the evaluation of evidence, and enables the acquisition of community understanding that has utility beyond investigators' local circumstances. We present ideas for assessing argumentation quality and suggest approaches to increasing the frequency and quality of argumentation in the exchange among diverse medical education scholars.

Pediatric reference ranges for proinflammatory and anti-inflammatory cytokines in cerebrospinal fluid and serum by multiplexed immunoassay.

To define cytokine concentrations and detectability in children with noninflammatory neurological disorders (NIND). The multiplex bead assay technology was used for simultaneous measurement of 34 soluble cytokines/chemokines in cerebrospinal fluid (CSF) from 73 NIND. Sera from 36 healthy children and 37 NIND also were analyzed. In CSF, CXCL10 had the highest concentration; CCL2, CXCL10, and interleukin (IL)-6 were detectable in all samples, and CXCL8, CCL22, CXCL1, IL-16, and IL-1 receptor antagonist were found in ≥50% of the samples. In serum, CXCL1 had the highest concentration; sIL-2Ra, CXCL1, CXCL10, and CCL22 were detectable in all samples, and CCL2, IL-12, CCL5, and granulocyte monocyte colony-stimulating factor (GM-CSF) were found in ≥50% of the samples. The mean CSF:serum ratio for CCL2 was several-fold higher than the rest, with the CXCL10 and CXCL8 ratios also >1. Intercorrelations between CSF cytokines included CCL2 versus CXCL8 and IL-6, and CXCL1 versus CCL22, reflecting both T-helper-1 (Th1)/Th1 and Th1/Th2 relations. Serum correlations included CCL11 versus CCL2, GM-CSF, and IL-4. For serum cytokines, the agreement between healthy children and NIND was good, with the exception of higher CCL4 in NIND. Cytokines in children varied greatly in concentration and detectability, with chemokines predominating in the CSF. These data allow investigators to select their own kit cytokines, instead of manufacturer-selected cytokines, for greater cost-effectiveness and interpretability.

Cytokines, cytokine antagonists, and soluble adhesion molecules in pediatric OMS and other neuroinflammatory disorders.

OBJECTIVE: To test for hypothesized disease- and treatment-induced changes in cytokines and adhesion molecules in children with opsoclonus-myoclonus syndrome (OMS). METHODS: Multiplex bead assay technology was used for simultaneous measurement of 34 soluble cytokines in cerebrospinal fluid (CSF) and serum. Soluble intercellular adhesion molecule-1 (sICAM-1) and vascular cell adhesion molecule-1 (sVCAM-1) were measured by ELISA. In total, there were 388 children (239 OMS, 114 controls, and 35 other inflammatory neurological disorders (OIND)). RESULTS: In untreated OMS, mean CSF IL-6 was elevated 2.3-fold, but 67-fold in OIND, without significant differences in other CSF cytokines. Mean serum concentrations of sIL-2Ra (+50%) and CXCL1 (+70%) (p<0.0001) were also raised. CSF CCL5 was more often detected in untreated OMS than controls (p=0.005), as was serum CCL11 and IL-13 in treated OMS. Mean CSF CCL4 and IL-1Ra were selectively higher in IVIg-treated OMS (p≤0.0001). CSF sICAM-1 was elevated only in OIND (3.3-fold); serum sICAM-1 was higher in untreated OMS (+21%); and sVCAM-1 was not affected. No correlations with OMS severity or duration were identified. CONCLUSIONS: Novel cytokine, cytokine antagonist, and soluble adhesion molecule abnormalities due to OMS or treatment were found. However, the normality of much of the data strengthens previous findings implicating B cell mechanisms.

CCR4 agonists CCL22 and CCL17 are elevated in pediatric OMS sera: rapid and selective down-regulation of CCL22 by ACTH or corticosteroids.

PURPOSE: To study the role of Th2-attracting chemokines in opsoclonus-myoclonus syndrome (OMS), a serious neurological paraneoplastic disorder in need of better immunological understanding and therapy. METHODS: The CCR4 agonists CCL22 and CCL17 were measured in serum by ELISA in children with OMS (238 and 260, respectively), pediatric controls (115 and 143), and other inflammatory neurological disorders (33 and 24). RESULTS: Both CCL22 (+55 %) and CCL17 (+121 %) were significantly elevated in untreated OMS compared to controls and inter-correlated (p < 0.0001). Their concentrations in untreated OMS also were higher than in OIND (21 %, 41 %). The concentration of CCL22 in ACTH and steroids groups (not IVIg) was 51 % lower than in controls, but only a smaller effect of ACTH on CCL17 was found. Prospective longitudinal studies revealed a precipitous 81 % drop in CCL22 even by the first week of high-dose ACTH therapy, staying below control mean for at least 12 weeks, and a 34 % reduction after 8 months of combined treatment. Response to ACTH was dose-related (r = -0.50, p < 0.0001). Luminex detection confirmed the ELISA results for CCL22, which were about 200 % higher. CONCLUSIONS: These data reveal an elevated serum concentration of Th2-attracting chemokines CCL22 and CCL17 in OMS. Marked and rapid reduction in CCL22, not CCL17, with either ACTH or steroid therapy suggests differential regulation and cellular sources of CCR4 ligands, and CCL22 as a potential candidate biomarker for ACTH or corticosteroid effect.

BAFF/APRIL system in pediatric OMS: relation to severity, neuroinflammation, and immunotherapy.

BACKGROUND: B-cell dysregulation has been implicated but not fully characterized in pediatric opsoclonus-myoclonus syndrome (OMS), a neuroblastoma-associated neuroinflammatory disorder. OBJECTIVE: To assess the role of B-cell activating factor (BAFF) and a proliferation-inducing ligand (APRIL), two critical B cell-modulating cytokines, as potential biomarkers of disease activity and treatment biomarkers in OMS. METHODS: Soluble BAFF and APRIL were measured in cerebrospinal fluid (CSF) and serum by ELISA in 433 children (296 OMS, 109 controls, 28 other inflammatory neurological disorders (OIND)). BAFF-R receptors on circulating CD19+ B cells were measured by flow cytometry. A blinded scorer rated motor severity on the OMS Evaluation Scale. Immunotherapies were evaluated cross-sectionally and longitudinally. RESULTS: The mean CSF BAFF concentration, which was elevated in untreated OMS and OIND, correlated with OMS severity category (P = 0.006), and reduction by adrenocorticotropic hormone or corticotropin (ACTH) (-61%) or corticosteroids (-38%) was seen at each level of severity. In contrast, CSF APRIL was normal in OMS and OIND and unaffected by immunotherapy. When the entire OMS dataset was dichotomized into 'high' versus 'normal' CSF BAFF concentration, the phenotype of the high group included greater motor severity and number of CSF oligoclonal bands, and a higher concentration of inflammatory chemokines CXCL13 and CXCL10 in CSF and CXCL9 and CCL21 in serum. Serum APRIL was 6.7-fold higher in the intravenous immunoglobulins (IVIg) group, whereas serum BAFF was 2.6-fold higher in the rituximab group. The frequency of B cell BAFF-R expression was similar in untreated and treated OMS. Longitudinal studies of CSF BAFF revealed a significant decline in ACTH-treated patients (with or without rituximab) (P < 0.0001). Longitudinal studies of serum APRIL showed a 2.9-fold increase after 1 to 2 g/kg IVIg monotherapy (P = 0.0003). CONCLUSIONS: Striking distinctions in BAFF/APRIL signaling were found. OMS displayed heterogeneity in CSF BAFF expression, which met many but not all criteria as a potential biomarker of disease activity. We speculate that CSF BAFF may have more utility in a biomarker panel than as a stand-alone biomarker, and that the selective upregulation of both serum APRIL by IVIg and BAFF by rituximab, as well as downregulation of CSF BAFF by ACTH/steroids, may have utility as treatment biomarkers.

A template for reliable assessment of resident operative performance: assessment intervals, numbers of cases and raters.

BACKGROUND: Operative performance rating (OPR) instruments have been developed to assess operative performance (OP). To guide program implementation, this study determined: 1) Appropriate intervals for OP progress decisions, 2) Number of OPRs and raters required per interval to achieve reproducible results. METHODS: 21 surgeons rated 897 OPs (3 procedures) by 36 residents. Six-month PGY intervals were compared to determine length of stable operative performance intervals. Variance component analyses established rating factor importance. Generalizability analyses and decision studies determined number of OPRs required for reproducible OP decisions (reliabilities = 0.80). RESULTS: Resident OPRs are stable across single PGY years. 2.3 OPRs/resident/month provided a dependable basis for annual or semi-annual resident OP decisions. Results were similar for all procedures and training years. Rater idiosyncrasies accounted for most score variation (63% when interaction effects involving rater idiosyncrasies were included). Resident ability was the next most important source of variation (12%). Procedure was a less important source (5%). CONCLUSION: Annual resident OP decisions are supported. 2.3 OPRs per month provide a dependable basis for judging resident OP. These numbers are sufficient regardless of training year or procedure mix though efforts should be made to balance procedure mix. Multiple raters should rate each resident to control for rater idiosyncrasies.

From test validity to construct validity and back?

CONTEXT: Major changes in thinking about validity have occurred during the past century, shifting the focus in thinking from the validity of the test to the validity of test score interpretations. These changes have resulted from the 'new' thinking about validity in which construct validity has emerged as the central or unifying idea of validity today. Construct validity was introduced by Cronbach and Meehl in the mid-1950s in an attempt to address the validity of those many psychological concepts that have no clear referent in reality. To do this, construct validity theory required a nomological network--an elaborate theoretical network of constructs and observations connected by scientific laws--to validate the constructs. However, nomological networks are hard to come by and none that would do the job required by construct validity has been forthcoming to date. Thus, the current construct validity approach has retreated to one of simply 'interpretation and argument', but this seems to be too general to tie down the constructs in the way a nomological network would do to give credibility to the validity of the construct. As a result, the concept of validity seems to have been watered down and the credibility of validity claims weakened. OBJECTIVES: The purpose of this paper is to encourage a discussion of the use of construct validity in medical education, and to suggest that test developers and users reconsider the use of abstract theoretical constructs that have no referent apart from theory. METHODS: We present a critical review of these concerns about construct validity and provide for contrast a brief overview of a recently proposed view of measurement based on scientific realism and causality analysis.

Key role of CXCL13/CXCR5 axis for cerebrospinal fluid B cell recruitment in pediatric OMS.

To study aberrant B cell trafficking into the CSF in opsoclonus-myoclonus syndrome (OMS), chemoattractants CXCL13 and CXCL12, and B cell frequency and CXCR5 expression, were evaluated. CSF CXCL13 concentration and the CSF/serum ratio were higher in untreated OMS than controls, related directly to OMS severity and inversely to OMS duration, and correlated with CSF B cell frequency and oligoclonal bands. CXCL12 showed the opposite pattern. Selective accumulation of CXCR5+ memory B cells in CSF was found. In ACTH-treated OMS, CXCL13, but not CXCL12, was lower. These data implicate the chemokine/chemoreceptor pair CXCL13/CXR5 in B cell recruitment to the CNS in OMS. CXCL13 and CXCL12 may serve as reciprocal biomarkers of disease activity, but CXCL13 also had utility as a treatment biomarker.

Cerebrospinal fluid oligoclonal bands in childhood opsoclonus-myoclonus.

Oligoclonal bands in cerebrospinal fluid reflect local B-cell responses associated with various neuroinflammatory disorders. In opsoclonus-myoclonus syndrome, cerebrospinal fluid B-cell expansion was demonstrated, but no studies of oligoclonal bands are available. In a prospective case-control study of 132 children (103 with opsoclonus-myoclonus, 29 neurologic control subjects), cerebrospinal fluid oligoclonal bands, measured by isoelectric focusing with immunofixation, were observed in 35% with opsoclonus-myoclonus and none of the control subjects, with the highest frequency in severe cases (56%). In oligoclonal band-positive patients, the mean band number was 5 ± 3 S.D. (range, 2-10) and the total severity score was significantly higher than in band-negative patients, whereas the frequency of CD19(+) B cells, opsoclonus-myoclonus duration, neuroblastoma detection, and relapse history did not differ. The cerebrospinal fluid immunoglobulin G synthesis rate, immunoglobulin index, and Q albumin were normal. In 17 untreated children receiving adrenocorticotropic hormone, intravenous immunoglobulins, and rituximab, the number of oligoclonal band-positive decreased by 75%, and the mean band count fell by 80%. Oligoclonal band detection adds useful information to neuroimmunologic "staging" in opsoclonus-myoclonus. However, flow cytometry provides a more sensitive measure of B-cell infiltration. Cerebrospinal fluid oligoclonal bands warrant monitoring in long-term follow-up studies of disease-modifying drugs for opsoclonus-myoclonus.

Systems-based practice assessed with a performance-based examination simulated and scored by standardized participants in the health care system: feasibility and psychometric properties.

BACKGROUND: Systems-based practice is one of the six general competencies proposed by the Accreditation Council for Graduate Medical Education in their Outcome Project. However, little has been published on its assessment--possibly because the systems-based practice competency has been viewed as difficult to define and measure. PURPOSE: The purpose of this study was to determine whether a full performance-based examination of systems-based practice cases simulated and scored by standardized participants in the health care system could feasibly be constructed and implemented that would provide reliable and valid measurements. METHODS: In the 1st year of the project (2008), four systems-based practice cases were developed and pilot tested with 13 residents. Videotapes of residents were studied to develop an instrument for subsequent assessment of performance by standardized participants. In the 2nd year (2009), the examination was expanded to a full 12 cases, which were completed by 11 second-year residents, and psychometric analyses were performed on the scores. RESULTS: The generalizability coefficient for the full 12-case examination based on scoring by standardized participants was .71, which is nearly equal to that based on scoring by faculty physician observers, which was .78. The correlation between total scores obtained with standardized participants and physician observers was .78. CONCLUSIONS: A performance-based examination can provide a feasible and reliable assessment of systems-based practice. However, attempts to evaluate convergent validity and discriminant validity-by correlating systems-based practice performance assessments with mean global ratings of residents on the 6 competencies by faculty throughout training-were unsuccessful, due to a lack of independence between the rated dimensions.

Research in assessment: consensus statement and recommendations from the Ottawa 2010 Conference.

Medical education research in general is a young scientific discipline which is still finding its own position in the scientific range. It is rooted in both the biomedical sciences and the social sciences, each with their own scientific language. A more unique feature of medical education (and assessment) research is that it has to be both locally and internationally relevant. This is not always easy and sometimes leads to purely ideographic descriptions of an assessment procedure with insufficient general lessons or generalised scientific knowledge being generated or vice versa. For medical educational research, a plethora of methodologies is available to cater to many different research questions. This article contains consensus positions and suggestions on various elements of medical education (assessment) research. Overarching is the position that without a good theoretical underpinning and good knowledge of the existing literature, good research and sound conclusions are impossible to produce, and that there is no inherently superior methodology, but that the best methodology is the one most suited to answer the research question unambiguously. Although the positions should not be perceived as dogmas, they should be taken as very serious recommendations. Topics covered are: types of research, theoretical frameworks, designs and methodologies, instrument properties or psychometrics, costs/acceptability, ethics, infrastructure and support.

Reports of the decline of empathy during medical education are greatly exaggerated: a reexamination of the research.

PURPOSE: Research is said to show that empathy declines during medical school and residency training. These studies and their results were examined to determine the extent of the decline and the plausibility of any alternative explanations. METHOD: Eleven studies published from 2000 to 2008 which reported empathy at various stages of physician training were reexamined. Their results were transformed back to the original units of the rating scales to make results more interpretable by reporting them in the metric of the original anchors. Next, the relationship between empathy ratings and response rates were examined to see whether response bias was a plausible threat to the validity of the empathy decline conclusion. RESULTS: The changes in mean empathy ranged across the 11 studies from a 0.1-point increase in empathy to a 0.5-point decrease, with an average of a 0.2-point decline for the 11 studies (ratings were on 5-point, 7-point, and 9-point scales). Mean ratings were similar in medical school and residency. Response rates were low and-where reported-declined on average about 26 percentage points. CONCLUSIONS: Reexamination revealed that the evidence does not warrant the strong, disturbing conclusion that empathy declines during medical education. Results show a very weak decline in mean ratings, and even the weak decline is questionable because of the low and varying response rates. Moreover, the empathy instruments are self-reports, and it isn't clear what they measure-or whether what they measure is indicative of patients' perceptions and the effectiveness of patient care.

B cell depletion therapy for new-onset opsoclonus-myoclonus.

Twelve immunotherapy-naïve children with opsoclonus-myoclonus syndrome and CSF B cell expansion received rituximab, adrenocorticotropic hormone (ACTH), and IVIg. Motor severity lessened 73% by 6 mo and 81% at 1 yr (P < 0.0001). Opsoclonus and action myoclonus disappeared rapidly, whereas gait ataxia and some other motor components improved more slowly. ACTH dose was tapered by 87%. Reduction in total CSF B cells was profound at 6 mo (-93%). By study end, peripheral B cells returned to 53% of baseline and serum IgM levels to 63%. Overall clinical response trailed peripheral B cell and IgM depletion, but improvement continued after their levels recovered. All but one non-ambulatory subject became ambulatory without additional chemotherapy; two relapsed and remitted; four had rituximab-related or possibly related adverse events; and two had low-titer human anti-chimeric antibody. Combination of rituximab with conventional agents as initial therapy was effective and safe. A controlled trial with long-term safety monitoring is indicated.

Long-term cerebrospinal fluid and blood lymphocyte dynamics after rituximab for pediatric opsoclonus-myoclonus.

INTRODUCTION: Opsoclonus-myoclonus syndrome (OMS) is an autoimmune paraneoplastic disorder characterized by B and T cell abnormalities in cerebrospinal fluid (CSF) and propensity for relapse. The study aim was to assess whether rituximab-induced B cell ablation in CSF outlasts repopulation in blood and if there are changes in other lymphocyte subsets. MATERIALS AND METHODS: In 25 children with OMS, the expression of CSF and blood lymphocyte surface antigens was evaluated by flow cytometry before and at intervals after rituximab therapy. RESULTS: The reduction in CSF CD27+ memory, CD38+ activated, CD5+, and other B cell subsets was profound (p < 0.0001), comparable across groups (-94%), and sustained over 12-18 months despite repopulation in blood. The observed lag in memory B cell pool recovery in the CSF compared to peripheral blood may be clinically relevant. T cell phenotypic changes involved frequency, not absolute counts, and were transient. Co-treatment with IVIg or ACTH did not significantly alter B cell depletion or repletion. DISCUSSION: These data indicate that rituximab affords long-term protection against CSF B cell expansion in OMS (ClinicalTrials.gov NCT00244361).

Therapeutic down-regulation of central and peripheral B-cell-activating factor (BAFF) production in pediatric opsoclonus-myoclonus syndrome.

Opsoclonus-myoclonus syndrome (OMS) is an autoimmune, paraneoplastic, central nervous system disorder, characterized by cerebrospinal fluid (CSF) B-cell expansion and various putative autoantibodies. To investigate the role of B-cell activating factor (BAFF) in OMS and the effect of disease-modifying immunotherapies used to treat it, BAFF was measured by enzyme-linked immunoadsorbent assay in the CSF and serum of 161 children with OMS and 116 pediatric controls. The mean concentration of CSF BAFF and the CSF/serum BAFF ratio were significantly higher in untreated OMS compared to neurological controls. CSF and serum BAFF levels were significantly lower in children treated with ACTH or corticosteroids, as was the CSF/serum BAFF ratio. There was a strong, negative correlation between CSF or serum BAFF levels and ACTH dose. Monthly IVIg infusions had no net impact on BAFF levels, and the combination of IVIg with ACTH or steroids did not reduce or enhance their anti-BAFF effects. These data indicate that BAFF production is increased centrally, not peripherally, in OMS, implying astrocytic over production. The novel dose-related central and peripheral anti-BAFF properties of ACTH, especially, have implications for other BAFF-related autoimmune disorders, infectious diseases, and cancers.