Use of sensors to quantify procedural idle time: Validity evidence for a new mastery metric.

BACKGROUND: Quantification of mastery is the first step in using objective metrics for teaching. We hypothesized that during orotracheal intubation, top tier performers have less idle time compared to lower tier performers. METHODS: At the Anesthesiology 2018 Annual Meeting, 82 participants intubated a normal airway simulator and a burnt airway simulator. The movements of the participant's laryngoscope were quantified using electromagnetic motion sensors. Top tier performers were defined as participants who intubated both simulators successfully in less than the median time for each simulator. Idle time was defined as the duration of time when the laryngoscope was not moving. RESULTS: Top performers showed less Idle Time when intubating the normal airway compared to lower tier performers (14.5 ± 9.8 seconds vs 34.0 ± 52.0 seconds, respectively P < .01). Likewise, top performers showed less Idle Time when intubating the burnt airway compared to lower tier performers (18.6 ± 15.2 seconds vs 63.4 ± 59.11 seconds; P < .01). Comparing performance on the burnt airway to the normal airway, there was a difference for lower tier performers (63.4 ± 59.1 seconds vs 34.0 ± 52.0 seconds; P < .01) but not for top tier performers (18.6 ± 15.2 seconds vs 14.5 ± 9.8 seconds; P = .07). CONCLUSION: Similar to our previous findings with other procedures, Idle Time was shown to have known group validity evidence when comparing top performers with lower tier performers. Further, Idle Time was correlated with procedure difficulty in our prior work. We observed statistically significant differences in Idle Times for lower tier performers when comparing the normal airway to the burnt airway but not for top tier performers. Our findings support the continued exploration of Idle Time for development of objective assessment and curricula.

Instructional scaffolding to improve students' skills in evaluating clinical literature.

OBJECTIVE: To implement and assess the effectiveness of an activity to teach pharmacy students to critically evaluate clinical literature using instructional scaffolding and a Clinical Trial Evaluation Rubric. DESIGN: The literature evaluation activity centered on a single clinical research article and involved individual, small group, and large group instruction, with carefully structured, evidence-based scaffolds and support materials centered around 3 educational themes: (1) the reader's awareness of text organization, (2) contextual/background information and vocabulary, and (3) questioning, prompting, and self-monitoring (metacognition). ASSESSMENT: Students initially read the article, scored it using the rubric, and wrote an evaluation. Students then worked individually using a worksheet to identify and define 4 to 5 vocabulary/concept knowledge gaps. They then worked in small groups and as a class to further improve their skills. Finally, they assessed the same article using the rubric and writing a second evaluation. Students' rubric scores for the article decreased significantly from a mean pre-activity score of 76.7% to a post-activity score of 61.7%, indicating that their skills in identifying weaknesses in the article's study design had improved. CONCLUSION: Use of instructional scaffolding in the form of vocabulary supports and the Clinical Trial Evaluation Rubric improved students' ability to critically evaluate a clinical study compared to lecture-based coursework alone.