Copy-and-Paste in Medical Student Notes: Extent, Temporal Trends, and Relationship to Scholastic Performance.

BACKGROUND: Medical students may observe and subsequently perpetuate redundancy in clinical documentation, but the degree of redundancy in student notes and whether there is an association with scholastic performance are unknown. OBJECTIVES: This study sought to quantify redundancy, defined generally as the proportion of similar text between two strings, in medical student notes and evaluate the relationship between note redundancy and objective indicators of student performance. METHODS: Notes generated by medical students rotating through their medicine clerkship during a single academic year at our institution were analyzed. A student-patient interaction (SPI) was defined as a history and physical and at least two contiguous progress notes authored by the same student during a single patient's hospitalization. For some students, SPI pairs were available from early and late in the clerkship. Redundancy between analogous sections of consecutive notes was calculated on a 0 to 100% scale and was derived from edit distance, the number of changes needed to transform one text string into another. Indicators of student performance included United States Medical Licensing Exam (USMLE) scores. RESULTS: Ninety-four single SPIs and 58 SPI pairs were analyzed. Redundancy in the assessment/plan section was high (40%) and increased within individual SPIs (to 60%; p < 0.001) and between SPI pairs over the course of the clerkship (by 30-40%; p < 0.001). Students in the lowest tertile of USMLE step II clinical knowledge scores had higher redundancy in the assessment/plan section than their classmates (67 ± 24% vs. 38 ± 22%; p = 0.002). CONCLUSION: During the medicine clerkship, the assessment/plan section of medical student notes became more redundant over a patient's hospital course and as students gained clinical experience. These trends may be indicative of deficiencies in clinical knowledge or reasoning, as evidenced by performance on some standardized evaluations.

"Yes, and " Exploring the Future of Learning Analytics in Medical Education.

This Conversations Starter article presents a selected research abstract from the 2017 Association of American Medical Colleges Northeastern Region Group on Educational Affairs annual spring meeting. The abstract is paired with the integrative commentary of three experts who shared their thoughts stimulated by the study. Commentators brainstormed "what's next" with learning analytics in medical education, including advancements in interaction metrics and the use of interactivity analysis to deepen understanding of perceptual, cognitive, and social learning and transfer processes.

Changing Medical School IT to Support Medical Education Transformation.

PROBLEM: Many medical schools are modifying curricula to reflect the rapidly evolving health care environment, but schools struggle to provide the educational informatics technology (IT) support to make the necessary changes. Often a medical school's IT support for the education mission derives from isolated work units employing separate technologies that are not interoperable. INTERVENTION: We launched a redesigned, tightly integrated, and novel IT infrastructure to support a completely revamped curriculum at the Vanderbilt School of Medicine. This system uses coordinated and interoperable technologies to support new instructional methods, capture students' effort, and manage feedback, allowing the monitoring of students' progress toward specific competency goals across settings and programs. CONTEXT: The new undergraduate medical education program at Vanderbilt, entitled Curriculum 2.0, is a competency-based curriculum in which the ultimate goal is medical student advancement based on performance outcomes and personal goals rather than a time-based sequence of courses. IT support was essential in the creation of Curriculum 2.0. In addition to typical learning and curriculum management functions, IT was needed to capture data in the learning workflow for analysis, as well as for informing individual and programmatic success. We aligned people, processes, and technology to provide the IT infrastructure for the organizational transformation. OUTCOMES: Educational IT personnel were successfully realigned to create the new IT system. The IT infrastructure enabled monitoring of student performance within each competency domain across settings and time via personal student electronic portfolios. Students use aggregated performance data, derived in real time from the portfolio, for mentor-guided performance assessment, and for creation of individual learning goals and plans. Poorly performing students were identified earlier through online communication systems that alert the appropriate instructor or coach of low quiz grades or missed learning goals. Graphical and narrative displays of a student's competency performance across courses and clinical experiences informed high-stake decisions made about student progress by the promotions committee. Similarly, graphical display of aggregate student outcomes provided education leaders with information needed to adjust and improve the curriculum. LESSONS LEARNED: With the alignment of people, processes, and technology, educational IT can facilitate transformational steps in the training of medical students.

Spanning a Decade of Physician Boundary Violations: Are We Improving?

Sexual boundary violations can negatively impact the culture of safety within a medical practice or healthcare institution and severely compromise the covenant of care and physician objectivity. Lack of education and training is one factor associated with physician misconduct that leads to high financial and personal cost. This paper presents a follow-up study of physicians referred to a professional development course in 2001 and presents demographic data from 2001 to present. The paper focuses on the education and remediation progress regarding sexual misconduct by physicians.

Using natural language processing to provide personalized learning opportunities from trainee clinical notes.

OBJECTIVE: Assessment of medical trainee learning through pre-defined competencies is now commonplace in schools of medicine. We describe a novel electronic advisor system using natural language processing (NLP) to identify two geriatric medicine competencies from medical student clinical notes in the electronic medical record: advance directives (AD) and altered mental status (AMS). MATERIALS AND METHODS: Clinical notes from third year medical students were processed using a general-purpose NLP system to identify biomedical concepts and their section context. The system analyzed these notes for relevance to AD or AMS and generated custom email alerts to students with embedded supplemental learning material customized to their notes. Recall and precision of the two advisors were evaluated by physician review. Students were given pre and post multiple choice question tests broadly covering geriatrics. RESULTS: Of 102 students approached, 66 students consented and enrolled. The system sent 393 email alerts to 54 students (82%), including 270 for AD and 123 for AMS. Precision was 100% for AD and 93% for AMS. Recall was 69% for AD and 100% for AMS. Students mentioned ADs for 43 patients, with all mentions occurring after first having received an AD reminder. Students accessed educational links 34 times from the 393 email alerts. There was no difference in pre (mean 62%) and post (mean 60%) test scores. CONCLUSIONS: The system effectively identified two educational opportunities using NLP applied to clinical notes and demonstrated a small change in student behavior. Use of electronic advisors such as these may provide a scalable model to assess specific competency elements and deliver educational opportunities.

Formation of a New Entity to Support Effective Use of Technology in Medical Education: The Student Technology Committee.

BACKGROUND: As technology in medical education expands from teaching tool to crucial component of curricular programming, new demands arise to innovate and optimize educational technology. While the expectations of today's digital native students are significant, their experience and unique insights breed new opportunities to involve them as stakeholders in tackling educational technology challenges. OBJECTIVE: The objective of this paper is to present our experience with a novel medical student-led and faculty-supported technology committee that was developed at Vanderbilt University School of Medicine to harness students' valuable input in a comprehensive fashion. Key lessons learned through the initial successes and challenges of implementing our model are also discussed. METHODS: A committee was established with cooperation of school administration, a faculty advisor with experience launching educational technologies, and a group of students passionate about this domain. Committee membership is sustained through annual selective recruitment of interested students. RESULTS: The committee serves 4 key functions: acting as liaisons between students and administration; advising development of institutional educational technologies; developing, piloting, and assessing new student-led educational technologies; and promoting biomedical and educational informatics within the school community. Participating students develop personally and professionally, contribute to program implementation, and extend the field's understanding by pursuing research initiatives. The institution benefits from rapid improvements to educational technologies that meet students' needs and enhance learning opportunities. Students and the institution also gain from fostering a campus culture of awareness and innovation in informatics and medical education. The committee's success hinges on member composition, school leadership buy-in, active involvement in institutional activities, and support for committee initiatives. CONCLUSIONS: Students should have an integral role in advancing medical education technology to improve training for 21st-century physicians. The student technology committee model provides a framework for this integration, can be readily implemented at other institutions, and creates immediate value for students, faculty, information technology staff, and the school community.

Automatic scoring of medical students' clinical notes to monitor learning in the workplace.

BACKGROUND: Educators need efficient and effective means to track students' clinical experiences to monitor their progress toward competency goals. AIM: To validate an electronic scoring system that rates medical students' clinical notes for relevance to priority topics of the medical school curriculum. METHOD: The Vanderbilt School of Medicine Core Clinical Curriculum enumerates 25 core clinical problems (CCP) that graduating medical students must understand. Medical students upload clinical notes pertinent to each CCP to a web-based dashboard, but criteria for determining relevance of a note and consistent uploading practices by students are lacking. The Vanderbilt Learning Portfolio (VLP) system automates both tasks by rating relevance for each CCP and uploading the note to the student's electronic dashboard. We validated this electronic scoring system by comparing the relevance of 265 clinical notes written by third year medical students to each of the 25 core patient problems as scored by VLP verses an expert panel of raters. RESULTS: We established the threshold score which yielded 75% positive prediction of relevance for 16 of the 25 clinical problems to expert opinion. DISCUSSION: Automated scoring of student's clinical notes provides a novel, efficient and standardized means of tracking student's progress toward institutional competency goals.

Automated Assessment of Medical Students' Clinical Exposures according to AAMC Geriatric Competencies.

Competence is essential for health care professionals. Current methods to assess competency, however, do not efficiently capture medical students' experience. In this preliminary study, we used machine learning and natural language processing (NLP) to identify geriatric competency exposures from students' clinical notes. The system applied NLP to generate the concepts and related features from notes. We extracted a refined list of concepts associated with corresponding competencies. This system was evaluated through 10-fold cross validation for six geriatric competency domains: "medication management (MedMgmt)", "cognitive and behavioral disorders (CBD)", "falls, balance, gait disorders (Falls)", "self-care capacity (SCC)", "palliative care (PC)", "hospital care for elders (HCE)" - each an American Association of Medical Colleges competency for medical students. The systems could accurately assess MedMgmt, SCC, HCE, and Falls competencies with F-measures of 0.94, 0.86, 0.85, and 0.84, respectively, but did not attain good performance for PC and CBD (0.69 and 0.62 in F-measure, respectively).

Teaching evidence-based medicine: Impact on students' literature use and inpatient clinical documentation.

BACKGROUND: Effective teaching of evidence-based medicine (EBM) to medical students is important for lifelong self-directed learning. AIMS: We implemented a brief workshop designed to teach literature searching skills to third-year medical students. We assessed its impact on students' utilization of EBM resources during their clinical rotation and the quality of EBM integration in inpatient notes. METHODS: We developed a physician-led, hands-on workshop to introduce EBM resources to all internal medicine clerks. Pre- and post-workshop measures included student's attitudes to EBM, citations of EBM resources in their clinical notes, and quality of the EBM component of the discussion in the note. Computer log analysis recorded students' online search attempts. RESULTS: After the workshop, students reported improved comfort using EBM and increased utilization of EBM resources. EBM integration into the discussion component of the notes also showed significant improvement. Computer log analysis of students' searches demonstrated increased utilization of EBM resources following the workshop. CONCLUSIONS: We describe the successful implementation of a workshop designed to teach third-year medical students how to perform an efficient EBM literature search. We demonstrated improvements in students' confidence regarding EBM, increased utilization of EBM resources, and improved integration of EBM into inpatient notes.

What factors create a humanistic doctor? A nationwide survey of fourth-year medical students.

PURPOSE: The authors sought to develop a conceptual framework of the factors that most influence medical students' development of humanism and to explore students' opinions regarding the role these factors play in developing or inhibiting humanism. METHOD: In 2006-2007, the authors conducted 16 focus groups with fourth-year students and first-year residents at four universities to design a conceptual framework. They used the framework to develop a survey, which they administered to fourth-year medical students at 20 U.S. medical schools in 2007-2008. RESULTS: Data from 80 focus-group participants suggested that the key influences on students' development of humanism were their authentic, unique, and participatory experiences before and during medical school, and the opportunity to process these experiences. Students who completed the survey (N = 1,170) reported that experiences of greatest intensity (e.g., being involved in a case where the patient dies), participatory learning experiences (e.g., volunteer work, international clinical rotations), and positive role models had the greatest effect on their development of humanism, whereas stressful conditions, such as a busy workload or being tired or postcall, inhibited their humanism. Women and students going into primary care placed significantly greater importance on experiences promoting humanism than did men and those not going into primary care. In addition, students with lower debt burdens viewed such experiences as more important than did those with higher debt burdens. CONCLUSIONS: Students viewed a variety of factors as influencing their development of humanism. This research provides a starting point for enhancing curricula to promote humanism.

Comparing content coverage in medical curriculum to trainee-authored clinical notes.

Accurate assessment and evaluation of medical curricula has long been a goal of medical educators. Current methods rely on manually-entered keywords and trainee-recorded logs of case exposure. In this study, we used natural language processing to compare the clinical content coverage in a four-year medical curriculum to the electronic medical record notes written by clinical trainees. The content coverage was compared for each of 25 agreed-upon core clinical problems (CCPs) and seven categories of infectious diseases. Most CCPs were covered in both corpora. Lecture curricula more frequently represented rare curricula, and several areas of low content coverage were identified, primarily related to outpatient complaints. Such methods may prove useful for future curriculum evaluations and revisions.

Evaluation of a method to identify and categorize section headers in clinical documents.

OBJECTIVE: Clinical notes, typically written in natural language, often contain substructure that divides them into sections, such as "History of Present Illness" or "Family Medical History." The authors designed and evaluated an algorithm ("SecTag") to identify both labeled and unlabeled (implied) note section headers in "history and physical examination" documents ("H&P notes"). DESIGN: The SecTag algorithm uses a combination of natural language processing techniques, word variant recognition with spelling correction, terminology-based rules, and naive Bayesian scoring methods to identify note section headers. Eleven physicians evaluated SecTag's performance on 319 randomly chosen H&P notes. MEASUREMENTS: The primary outcomes were the algorithm's recall and precision in identifying all document sections and a predefined list of twenty-nine major sections. A secondary outcome was to evaluate the algorithm's ability to recognize the correct start and end boundaries of identified sections. RESULTS: The SecTag algorithm identified 16,036 total sections and 7,858 major sections. Physician evaluators classified 15,329 as true positives and identified 160 sections omitted by SecTag. The recall and precision of the SecTag algorithm were 99.0 and 95.6% for all sections, 98.6 and 96.2% for major sections, and 96.6 and 86.8% for unlabeled sections. The algorithm determined the correct starting and ending text boundaries for 94.8% of labeled sections and 85.9% of unlabeled sections. CONCLUSIONS: The SecTag algorithm accurately identified both labeled and unlabeled sections in history and physical documents. This type of algorithm may assist in natural language processing applications, such as clinical decision support systems or competency assessment for medical trainees.

Tracking medical students' clinical experiences using natural language processing.

Graduate medical students must demonstrate competency in clinical skills. Current tracking methods rely either on manual efforts or on simple electronic entry to record clinical experience. We evaluated automated methods to locate 10 institution-defined core clinical problems from three medical students' clinical notes (n=290). Each note was processed with section header identification algorithms and the KnowledgeMap concept identifier to locate Unified Medical Language System (UMLS) concepts. The best performing automated search strategies accurately classified documents containing primary discussions to the core clinical problems with area under receiver operator characteristic curve of 0.90-0.94. Recall and precision for UMLS concept identification was 0.91 and 0.92, respectively. Of the individual note section, concepts found within the chief complaint, history of present illness, and assessment and plan were the strongest predictors of relevance. This automated method of tracking can provide detailed, pertinent reports of clinical experience that does not require additional work from medical trainees. The coupling of section header identification and concept identification holds promise for other natural language processing tasks, such as clinical research or phenotype identification.

Automated capture and assessment of medical student clinical experience.

Currently, many medical educators track trainee clinical experience using student-created manual logs. Using a web-based portfolio system that captures all notes written by trainees in the electronic medical record, we examined a graduating medical student's clinical notes to determine if we could automatically assess exposure to 10 institution-defined core clinical topics. We located all biomedical concepts in his clinical notes, divided by note section, using the KnowledgeMap concept identifier. Notes were ranked according to the concepts matching each core topic's concept list. Clinician educators then reviewed each note to determine relevance to the core topic. The student covered all core topics, with between 2 and 41 notes containing highly relevant discussions. The algorithm effectively predicted relevance (p<0.001). This method is a promising first step toward automated competency assessment.

Development and evaluation of a clinical note section header terminology.

Clinical documentation is often expressed in natural language text, yet providers often use common organizations that segment these notes in sections, such as history of present illness or physical examination. We developed a hierarchical section header terminology, supporting mappings to LOINC and other vocabularies; it contained 1109 concepts and 4332 synonyms. Physicians evaluated it compared to LOINC and the Evaluation and Management billing schema using a randomly selected corpus of history and physical notes. Evaluated documents contained a median of 54 sections and 27 major sections. There were 16,196 total sections in the evaluation note corpus. The terminology contained 99.9% of the clinical sections; LOINC matched 77% of section header concepts and 20% of section header strings in those documents. The section terminology may enable better clinical note understanding and interoperability. Future development and integration into natural language processing systems is needed.

Proposed standards for medical education submissions to the Journal of General Internal Medicine.

To help authors design rigorous studies and prepare clear and informative manuscripts, improve the transparency of editorial decisions, and raise the bar on educational scholarship, the Deputy Editors of the Journal of General Internal Medicine articulate standards for medical education submissions to the Journal. General standards include: (1) quality questions, (2) quality methods to match the questions, (3) insightful interpretation of findings, (4) transparent, unbiased reporting, and (5) attention to human subjects' protection and ethical research conduct. Additional standards for specific study types are described. We hope these proposed standards will generate discussion that will foster their continued evolution.

Automatic capture of student notes to augment mentor feedback and student performance on patient write-ups.

OBJECTIVE: To determine whether the integration of an automated electronic clinical portfolio into clinical clerkships can improve the quality of feedback given to students on their patient write-ups and the quality of students' write-ups. DESIGN: The authors conducted a single-blinded, randomized controlled study of an electronic clinical portfolio that automatically collects all students' clinical notes and notifies their teachers (attending and resident physicians) via e-mail. Third-year medical students were randomized to use the electronic portfolio or traditional paper means. Teachers in the portfolio group provided feedback directly on the student's write-up using a web-based application. Teachers in the control group provided feedback directly on the student's write-up by writing in the margins of the paper. Outcomes were teacher and student assessment of the frequency and quality of feedback on write-ups, expert assessment of the quality of student write-ups at the end of the clerkship, and participant assessment of the value of the electronic portfolio system. RESULTS: Teachers reported giving more frequent and detailed feedback using the portfolio system (p = 0.01). Seventy percent of students who used the portfolio system, versus 39% of students in the control group (p = 0.001), reported receiving feedback on more than half of their write-ups. Write-ups of portfolio students were rated of similar quality to write-ups of control students. Teachers and students agreed that the system was a valuable teaching tool and easy to use. CONCLUSIONS: An electronic clinical portfolio that automatically collects students' clinical notes is associated with improved teacher feedback on write-ups and similar quality of write-ups.

A CME course aimed at addressing disruptive physician behavior.

Examine the results of a study of physicians with disruptive behavior who went through a special training program to help them better control their anger and outbursts.