Education research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination.

OBJECTIVE: The American Board of Psychiatry and Neurology (ABPN) has recently replaced the traditional, centralized oral examination with the locally administered Neurology Clinical Skills Examination (NEX). The ABPN postulated the experience with the NEX would be similar to the Mini-Clinical Evaluation Exercise, a reliable and valid assessment tool. The reliability and validity of the NEX has not been established. METHODS: NEX encounters were videotaped at 4 neurology programs. Local faculty and ABPN examiners graded the encounters using 2 different evaluation forms: an ABPN form and one with a contracted rating scale. Some NEX encounters were purposely failed by residents. Cohen's kappa and intraclass correlation coefficients (ICC) were calculated for local vs ABPN examiners. RESULTS: Ninety-eight videotaped NEX encounters of 32 residents were evaluated by 20 local faculty evaluators and 18 ABPN examiners. The interrater reliability for a determination of pass vs fail for each encounter was poor (kappa 0.32; 95% confidence interval [CI] = 0.11, 0.53). ICC between local faculty and ABPN examiners for each performance rating on the ABPN NEX form was poor to moderate (ICC range 0.14-0.44), and did not improve with the contracted rating form (ICC range 0.09-0.36). ABPN examiners were more likely than local examiners to fail residents. CONCLUSIONS: There is poor interrater reliability between local faculty and American Board of Psychiatry and Neurology examiners. A bias was detected for favorable assessment locally, which is concerning for the validity of the examination. Further study is needed to assess whether training can improve interrater reliability and offset bias.

The neurology clerkship core curriculum.

Neurologic symptoms are common in all practice settings, and neurologic diseases comprise a large and increasing proportion of health care expenditures and global disease burden. Consequently, the training of all physicians should prepare them to recognize patients who may have neurologic disease, and to take the initial steps in evaluating and managing those patients. We present a core curriculum outlining the clinical neurology skills and knowledge necessary to achieve that degree of preparation. The curriculum emphasizes general principles and a systematic approach to patients with neurologic symptoms and signs. The ability to perform and interpret the neurologic examination is fundamental to that approach, so the curriculum delineates the essential components of the examination in three different clinical settings. The focus of the curriculum is on symptom-based rather than disease-based learning. The only specific diseases selected for inclusion are conditions that are common or require urgent management. This curriculum has been approved by the national organization of neurology clerkship directors and endorsed by the major national professional organizations of neurologists. It is intended as a template for planning a neurology clerkship and as a benchmark for evaluating existing clerkships. It should be especially helpful to clerkship directors, neurology chairs, deans of medical education, and members of external accreditation groups.

Measurement of progression in Alzheimer's disease: a clinician's perspective.

Patients with Alzheimer's disease (AD) and their families must confront two fundamental truths. First, AD is a uniformly progressive disease that ultimately results in debilitating cognitive impairment. Second, although there is now evidence that some medications may produce transient improvement or possibly even slowing of disease progression, there is currently no way to halt the progression of AD. Consequently, patients and their families consistently ask the following questions: 1. What new management issues can be anticipated, and when? 2. What clinical developments are atypical and merit evaluation for a superimposed problem? 3. Is the current treatment working? These questions can only be answered by referring to the natural course of AD, and specifically, information regarding measures of functional impairment and how they change over time. The information that is currently available on this topic is limited and often embodies implicit assumptions that have not been adequately tested. This information will be reviewed, and directions for future research will be outlined.

Diagnostic criteria for Parkinson disease.

The clinical diagnosis of Parkinson disease (PD) is based on the identification of some combination of the cardinal motor signs of bradykinesia, rigidity, tremor, and postural instability, but few attempts have been made to develop explicit diagnostic criteria. We propose a clinical diagnostic classification based on a comprehensive review of the literature regarding the sensitivity and specificity of the characteristic clinical features of PD. Three levels of diagnostic confidence are differentiated: Definite, Probable, and Possible. The diagnoses of Possible and Probable PD are based on clinical criteria alone. Neuropathologic confirmation is required for the diagnosis of Definite PD in patients with the clinical diagnosis of Possible or Probable PD. Criteria for histopathologic confirmation of PD are also presented.

Teaching neurology residents in the outpatient setting.

OBJECTIVE: To determine how residency programs are responding to the shift of neurological practice into the outpatient setting. DESIGN: A nine-item questionnaire was sent to the directors of all US neurology residency programs. Each item had two parts, the first describing the current program, and the second describing an "ideal" program designed to optimize resident education. The same questionnaire was also sent to all house officers and faculty associated with a single residency program to assess variability in perceptions. SETTING: United States neurology residency programs (mail survey). PARTICIPANTS: Directors of neurology residency programs and all house officers and faculty members at a single residency program. RESULTS: Eighty-one (70%) of the 116 questionnaires distributed were returned. There were four areas of general consensus among the program directors: (1) more time should be devoted to outpatient care during residency training; (2) more continuity at the resident level should be provided for patients seen in subspecialty clinics; (3) faculty should provide more supervision of residents when they see follow-up patients; and (4) conferences specifically directed at outpatient management issues should be developed. CONCLUSIONS: Neurology residency directors agree that current approaches to teaching in the outpatient setting fall short of an educationally ideal system. Four areas of perceived deficiency have been identified. Creative solutions will be necessary to correct these perceived deficiencies.

Alternative calculation of the global clinical dementia rating.

The Clinical Dementia Rating (CDR) is a widely used instrument for rating the global severity of dementia, with scores in six independently rated categories used as a basis for computing a global score. We have found that the algorithms currently used to calculate the global score from the six component scores produce results that are sometimes inconsistent with the goals of the rating scale. We systematically analyzed these incongruities and developed an alternative algorithm that eliminates them while retaining the fundamental features of the original method.

Change in pattern of muscle activity following botulinum toxin injections for torticollis.

Twenty patients with torticollis had electromyographic studies of their neck muscles performed before and after a series of local injections of botulinum toxin. The pattern of muscle activity changed after the injections, and this effect persisted even after head position had returned to baseline. Patients who did not experience any clinical benefit from the injections also demonstrated a change in the pattern of muscle activity. These results suggest that the underlying abnormality in torticollis usually involves a general motor program for head position, rather than the activity of individual neck muscles.

Controlled trial of botulinum toxin injections in the treatment of spasmodic torticollis.

We administered local injections of botulinum toxin to 20 patients with torticollis in a blinded, placebo-controlled study. Each patient received four sets of injections: three different doses of botulinum toxin and one placebo. The order of the sessions was random and unknown to the patients. Sixteen of the patients (80%) reported subjective improvement to at least one dose of botulinum toxin; 11 (55%) reported substantial improvement. No objective benefit was documented. Side effects were minor and transient, although dysphagia occurred in four. Some patients reported that the effect waned despite persistent relaxation or even flaccidity of previously overactive muscles, suggesting a change in the pattern of muscle activity after botulinum toxin injections.

Neuromuscular effects distant from the site of botulinum neurotoxin injection.

We assessed the severity and temporal profile of distant neuromuscular effects from a single dose (280 units) of botulinum neurotoxin injected into neck muscles for torticollis. We performed single-fiber EMG studies on the biceps brachii of six patients to measure jitter (20 pairs) and fiber density on the initial treatment day and then again, at least once more, after 2 to 12 weeks. No patient developed weakness beyond the neck muscles or decrement of the biceps response to repetitive 3-Hz nerve stimulation. Between the baseline and the last follow-up study, the average of mean MCD increased from 29 microseconds to 38 microseconds (31%). Mean fiber density increased concurrently or earlier from 1.35 to 1.79 (33%). There were no electrophysiologic signs of presynaptic blockade, even at 2 and 4 weeks. The effects we observed are compatible with stimulation of terminal sprouting by the neurotoxin, without significant presynaptic inhibition of acetylcholine release. We therefore believe that higher dosages of the neurotoxin may be used if clinically indicated.

Modified line-element theory for spatial-frequency and width discrimination.

Recent data from several laboratories have shown that spatial-frequency discrimination is not a smooth function of frequency but rather exhibits alternate peaks and troughs. A model for spatial-frequency discrimination analogous to line-element models for color discrimination is presented here and shown to provide a reasonable fit to the available data. This model is based on the predicted responses of six spatial-frequency-tuned mechanisms, whose sensitivity curves have been estimated in previously published masking experiments. In order to fit the data it is necessary to pool responses from units centered under the stimulus as well as from spatially neighboring units. Thus it appears that the visual system utilizes both spatial and spatial-frequency information in discrimination tasks.

Shifts in perceived size due to masking.

It has been suggested that perceived size depends upon the distribution of responses among a localized population of different size-tuned mechanisms. If so, then spatial frequency masking, which alters the distribution of responses, should produce shifts in a test pattern's perceived size. We have found this to be the case. The magnitude and direction of the shifts depend only on the frequency difference in octaves (delta omega) between mask and standard. The results are not antisymmetric about the point delta omega = 0, however, as they would be if perceived size were based purely on local size-tuning properties. Our results suggest that size perception depends both on these local properties and on the spatial distribution of mechanism responses.

Shifts in perceived size as a function of contrast and temporal modulation.

It has been suggested that perceived size depends upon the distribution of responses among a localized population of different size-tuned mechanisms. If so, then manipulations which alter this distribution should also affect perceived size. We therefore studied the effects of luminance contrast and mode of temporal presentation on size perception. Both manipulations produce significant shifts in perceived size. The results suggest that at least two factors contribute to the size percept. For small and intermediate size patterns, perceived size depends primarily on the local distribution of size-tuned mechanism responses, whereas for wide patterns, the spatial distribution of mechanism responses plays a major role.