Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education.

PURPOSE: Learner development and promotion rely heavily on narrative assessment comments, but narrative assessment quality is rarely evaluated in medical education. Educators have developed tools such as the Quality of Assessment for Learning (QuAL) tool to evaluate the quality of narrative assessment comments; however, scoring the comments generated in medical education assessment programs is time intensive. The authors developed a natural language processing (NLP) model for applying the QuAL score to narrative supervisor comments. METHOD: Samples of 2,500 Entrustable Professional Activities assessments were randomly extracted and deidentified from the McMaster (1,250 comments) and Saskatchewan (1,250 comments) emergency medicine (EM) residency training programs during the 2019-2020 academic year. Comments were rated using the QuAL score by 25 EM faculty members and 25 EM residents. The results were used to develop and test an NLP model to predict the overall QuAL score and QuAL subscores. RESULTS: All 50 raters completed the rating exercise. Approximately 50% of the comments had perfect agreement on the QuAL score, with the remaining resolved by the study authors. Creating a meaningful suggestion for improvement was the key differentiator between high- and moderate-quality feedback. The overall QuAL model predicted the exact human-rated score or 1 point above or below it in 87% of instances. Overall model performance was excellent, especially regarding the subtasks on suggestions for improvement and the link between resident performance and improvement suggestions, which achieved 85% and 82% balanced accuracies, respectively. CONCLUSIONS: This model could save considerable time for programs that want to rate the quality of supervisor comments, with the potential to automatically score a large volume of comments. This model could be used to provide faculty with real-time feedback or as a tool to quantify and track the quality of assessment comments at faculty, rotation, program, or institution levels.

ASD/OCD-Linked Protocadherin-10 Regulates Synapse, But Not Axon, Development in the Amygdala and Contributes to Fear- and Anxiety-Related Behaviors.

The Protocadherin-10 (PCDH10) gene is associated with autism spectrum disorder (ASD), obsessive-compulsive disorder (OCD), and major depression (MD). The PCDH10 protein is a homophilic cell adhesion molecule that belongs to the δ2-protocadherin family. PCDH10 is highly expressed in the developing brain, especially in the basolateral nucleus of the amygdala (BLA). However, the role of PCDH10 in vivo has been debatable: one paper reported that a Pcdh10 mutant mouse line showed changes in axonal projections; however, another Pcdh10 mutant mouse line was reported to have failed to detect axonal phenotypes. Therefore, the actual roles of PCDH10 in the brain remain to be elucidated. We established a new Pcdh10 KO mouse line using the CRISPR/Cas9 system, without inserting gene cassettes to avoid nonspecific effects, examined the roles of PCDH10 in the brain, and studied the behavioral consequences of Pcdh10 inactivation. Here, we show that Pcdh10 KO mice do not show defects in axonal development. Instead, we find that Pcdh10 KO mice exhibit impaired development of excitatory synapses in the dorsal BLA. We further demonstrate that male Pcdh10 KO mice exhibit reduced anxiety-related behaviors, impaired fear conditioning, decreased stress-coping responses, and mildly impaired social recognition and communication. These results indicate that PCDH10 plays a critical role in excitatory synapse development, but not axon development, in the dorsal BLA and that PCDH10 regulates anxiety-related, fear-related, and stress-related behaviors. Our results reveal the roles of PCDH10 in the brain and its relationship to relevant psychiatric disorders such as ASD, OCD, and MD.SIGNIFICANCE STATEMENTProtocadherin-10 (PCDH10) encodes a cell adhesion molecule and is implicated in autism spectrum disorder (ASD), obsessive-compulsive disorder (OCD), and major depression (MD). PCDH10 is highly expressed in the basolateral nucleus of the amygdala (BLA). However, the phenotypes of previously published Pcdh10 mutant mice are debatable, and some are possibly because of the nonspecific effects of the LacZ/Neo cassette inserted in the mice. We have generated a new Pcdh10 mutant mouse line without the LacZ/Neo cassette. Using our new mouse line, we reveal the roles of PCDH10 for excitatory synapse development in the BLA. The mutant mice exhibit anxiety-related, fear-related, and stress-related behaviors, which are relevant to ASD, OCD, and MD, suggesting a possible treatment strategy for such psychiatric disorders.

Behavioral discrimination and time-series phenotyping of birdsong performance.

Variation in the acoustic structure of vocal signals is important to communicate social information. However, relatively little is known about the features that receivers extract to decipher relevant social information. Here, we took an expansive, bottom-up approach to delineate the feature space that could be important for processing social information in zebra finch song. Using operant techniques, we discovered that female zebra finches can consistently discriminate brief song phrases ("motifs") from different social contexts. We then applied machine learning algorithms to classify motifs based on thousands of time-series features and to uncover acoustic features for motif discrimination. In addition to highlighting classic acoustic features, the resulting algorithm revealed novel features for song discrimination, for example, measures of time irreversibility (i.e., the degree to which the statistical properties of the actual and time-reversed signal differ). Moreover, the algorithm accurately predicted female performance on individual motif exemplars. These data underscore and expand the promise of broad time-series phenotyping to acoustic analyses and social decision-making.