Testing a Communication Assessment Tool for Ethically Sensitive Scenarios: Protocol of a Validation Study.

BACKGROUND: Although well-designed instruments to assess communication during medical interviews and complex encounters exist, assessment tools that differentiate between communication, empathy, decision-making, and moral judgment are needed to assess different aspects of communication during situations defined by ethical conflict. To address this need, we developed an assessment tool that differentiates competencies associated with practice in ethically challenging situations. The competencies are grouped into three distinct categories: communication skills, civility and respectful behavior, clinical and ethical judgment and decision-making. OBJECTIVE: The overall objective of this project is to develop an assessment tool for ethically sensitive scenarios that measures the degree of respect for the attitudes and beliefs of patients and family members, the demands of clinical decision-making, and the success in dealing with ethical conflicts in the clinical context. In this article, we describe the research method we will use during the pilot-test study using the neonatal context to provide validity evidence to support the features of the Assessment Communication Tool for Ethics (ACT4Ethics) instrument. METHODS: This study is part of a multiphase project designed according to modern validity principles including content, response process, internal structure, relation to other variables, and social consequences. The design considers threats to validity such as construct underrepresentation and factors exerting nonrandom influence on scores. This study consists of two primary steps: (1) train the raters in the use of the new tool and (2) pilot-test a simulation using an Objective Structured Clinical Examination. We aim to obtain a total of 90 independent assessments based on the performance of 30 trainees rated by 15 trained raters for analysis. A comparison of raters' responses will allow us to compute a measure of interrater reliability. We will additionally compare the results of ACT4Ethics with another existing instrument. RESULTS: This study will take approximately 18 months to complete and the results should be available by September 2019. CONCLUSIONS: ACT4Ethics should allow clinician-teachers to assess and monitor the development of competency of trainees' judgments and communication skills when facing ethically sensitive clinical situations. The instrument will also guide the provision of meaningful feedback to ensure that trainees develop specific communication, empathy, decision-making, and ethical competencies. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/12039.

Expert Agreement in the Interpretation of Lung Ultrasound Studies Performed on Mechanically Ventilated Patients.

OBJECTIVES: Although lung ultrasound (US) has been shown to have high diagnostic accuracy in patients presenting with acute dyspnea, its precision in critically ill patients is unknown. We investigated common areas of agreement and disagreement by studying 6 experts as they interpreted lung US studies in a cohort of intensive care unit (ICU) patients. METHODS: A previous study by our group asked experts to rate the quality of 150 lung US studies performed by 10 novices in a population of mechanically ventilated patients. For this study, experts were asked to interpret them without the clinical context, reporting the presence of pneumothorax, interstitial syndrome, consolidation, atelectasis, or pleural effusion. RESULTS: The rate of expert agreement depended on how it was defined, ranging from 51% (with a strict definition of agreement) to 57% (with a more liberal definition). Removing cases involving lung consolidation (the most common source of disagreement) improved the rates of agreement to 69% and 86%, respectively. CONCLUSIONS: The frequency of agreement was lower than might have been expected in this study. Several potential reasons are identified, chief among them the fact that ICU patients often develop multiple pulmonary insults, making agreement on a specific primary diagnosis challenging. This finding suggests that the utility of lung US in identifying the main contributing lung condition in ICU patients may be lower than in dyspneic patients encountered in the emergency department. It also raises the possibility that the clinical context is more important for lung US than other imaging modalities.

The Assessment of Competency in Thoracic Sonography (ACTS) scale: validation of a tool for point-of-care ultrasound.

BACKGROUND: The rapid adoption of point-of-care ultrasound (POCUS) has created a need to develop assessment tools to ensure that learners can competently use these technologies. In this study, the authors developed and tested a rating scale to assess the quality of point-of-care thoracic ultrasound studies performed by novices. In Phase 1, the Assessment of Competency in Thoracic Sonography (ACTS) scale was developed based on structured interviews with subject matter experts. The tool was then piloted on a small series of ultrasound studies in Phase 2. In Phase 3 the tool was applied to a sample of 150 POCUS studies performed by ten learners; performance was then assessed by two independent raters. RESULTS: Evidence for the content validity of the ACTS scale was provided by a consensus exercise wherein experts agreed on the general principles and specific items that make up the scale. The tool demonstrated reasonable inter-rater reliability despite minimal requirements for evaluator training and displayed evidence of good internal structure, with related scale items correlating well with each other. Analysis of the aggregate learning curves suggested a rapid early improvement in learner performance with slower improvement after approximately 25-30 studies. CONCLUSIONS: The ACTS scale provides a straightforward means to assess learner performance. Our results support the conclusion that the tool is an effective means of making valid judgments regarding competency in point-of-care thoracic ultrasound, and that the majority of learner improvement occurs during their first 25-30 practice studies.

Outcomes from extensive training in critical care echocardiography: Identifying the optimal number of practice studies required to achieve competency.

PURPOSE: Optimal instruction and assessment of critical care ultrasound (CCUS) skills requires an assessment tool to measure learner competency and changes over time. In this study, a previously published tool was used to monitor the development of critical care echocardiography (CCE) competencies, the attainment of performance plateaus, and the extent to which previous experience influenced learning. MATERIALS AND METHODS: A group of experts used the Rapid Assessment of Competency in Echocardiography (RACE) scale to rate a large pool of CCE studies performed by novices in a longitudinal design. A total of 380 studies performed by twelve learners were assessed; each study was independently rated by two experts. RESULTS: Learners demonstrated improvement in mean RACE scores over time, with peak performance occurring early in training and a performance plateau thereafter. Learners with little experience received the greatest benefit from training, with an average performance plateau reached at the twentieth study. CONCLUSIONS: Supporting earlier results, the RACE scale provided a straightforward means to assess learner performance with minimal requirements for evaluator training. The results of the present study suggest that novices experience the greatest gains in competency during their first twenty practice studies, a threshold which should serve to guide training initiatives.

The Rapid Assessment of Competency in Echocardiography Scale: Validation of a Tool for Point-of-Care Ultrasound.

OBJECTIVES: Increased use of point-of-care ultrasound (US) requires the development of assessment tools that measure the competency of learners. In this study, we developed and tested a tool to assess the quality of point-of-care cardiac US studies performed by novices. METHODS: In phase 1, the Rapid Assessment of Competency in Echocardiography (RACE) scale was developed on the basis of structured interviews with subject matter experts; the tool was then piloted on a small series of US studies in phase 2. In phase 3, the tool was applied to a sample of 154 point-of-care US studies performed by 12 learners; each study was independently rated by 2 experts, with quantitative analysis subsequently performed. RESULTS: Evidence of the content validity of the RACE scale was supported by a consensus exercise, wherein experts agreed on the assessment dimensions and specific items that made up the RACE scale. The tool showed good inter-rater reliability. An analysis of inter-item correlations provided support for the internal structure of the scale, and the tool was able to discriminate between learners early in their point-of-care US learning and those who were more advanced in their training. CONCLUSIONS: The RACE scale provides a straightforward means to assess learner performance with minimal requirements for evaluator training. Our results support the conclusion that the tool is an effective means of making valid judgments regarding competency in point-of-care cardiac US.

A reappraisal of the uncanny valley: categorical perception or frequency-based sensitization?

The uncanny valley (UCV) hypothesis describes a non-linear relationship between perceived human-likeness and affective response. The "uncanny valley" refers to an intermediate level of human-likeness that is associated with strong negative affect. Recent studies have suggested that the uncanny valley might result from the categorical perception of human-like stimuli during identification. When presented with stimuli sharing human-like traits, participants attempt to segment the continuum in "human" and "non-human" categories. Due to the ambiguity of stimuli located at a category boundary, categorization difficulty gives rise to a strong, negative affective response. Importantly, researchers who have studied the UCV in terms of categorical perception have focused on categorization responses rather than affective ratings. In the present study, we examined whether the negative affect associated with the UCV might be explained in terms of an individual's degree of exposure to stimuli. In two experiments, we tested a frequency-based model against a categorical perception model using a category-learning paradigm. We manipulated the frequency of exemplars that were presented to participants from two categories during a training phase. We then examined categorization and affective responses functions, as well as the relationship between categorization and affective responses. Supporting previous findings, categorization responses suggested that participants acquired novel category structures that reflected a category boundary. These category structures appeared to influence affective ratings of eeriness. Crucially, participants' ratings of eeriness were additionally affected by exemplar frequency. Taken together, these findings suggest that the UCV is determined by both categorical properties as well as the frequency of individual exemplars retained in memory.

Selective attention and subjective confidence calibration.

In the present experiments, failures of selective visual attention were invoked using the B. A. Eriksen and C. W. Eriksen (1974) flanker task. On each trial, a three-letter stimulus array was flashed briefly, followed by a mask. The identity of the two flanking letters was response congruent, neutral, or incongruent with the identity of the middle target letter. On half of the trials, confidence ratings were obtained after each response. In the first three experiments, participants were highly overconfident in the accuracy of their responding to incongruent flanker stimulus arrays. In a final experiment, presenting a prestimulus target location cue greatly reduced both selective attention failure and overconfidence. The findings demonstrate that participants are often unaware of such selective attention failures and provide support for the notion that, in these cases, decisional processing is driven largely by the identities of the incongruent flankers. In addition, responding was invariably slower and sometimes more accurate when confidence was required than when it was not required, demonstrating that the need to provide posttrial confidence reports can affect decisional processing. Moreover, there was some evidence that the presence of neutral contextual flanking information can slow responding, suggesting that such nondiagnostic information can, indeed, contribute to decisional processing.