Assessment of Radiation Therapy Technologists' Workload and Situation Awareness: Monitoring 2 Versus 3 Collocated Display Monitors.

PURPOSE: This study aimed to assess the effect of monitoring 2 versus 3 collocated displays on radiation therapist technologists' (RTTs) workload (WL) and situation awareness (SA) during routine treatment delivery tasks. METHODS AND MATERIALS: Seven RTTs completed 4 simulated treatment delivery scenarios (2 scenarios per experimental condition; 2 vs 3 collocated displays) in a within-subject experiment. WL was subjectively measured using the National Aeronautics and Space Administration (NASA) Task Load Index, and objectively measured using eye activity measures. SA was subjectively measured using the SA rating technique, and objectively measured using the SA global assessment technique. Two-tailed paired t tests were conducted to test for differences in means when parametric assumptions were satisfied, otherwise Wilcoxon signed-rank tests were conducted. A .05 level of significance was applied to all statistical tests. RESULTS: No statistically and clinically significant differences were observed between monitoring 2 versus 3 monitors on eye tracking measures (blink rate: 9.4 [4.8] vs 9.6 [4.0]; task evoked pupillary response: 0.16 [0.14] vs 0.21 [0.15]; NASA Task Load Index: 34.7 [19.8] vs 35.3 [20.4]; SA rating technique: 19.3 [6.2] vs 19.5 [7.0]; and SA global assessment technique scores: 100 [0] vs 100 [0]). CONCLUSIONS: Our preliminary findings suggest that monitoring 3 collocated displays by 1 RTT does not impact WL and SA compared with monitoring 2 collocated displays. Only 2 of many possible configurations were investigated. If institutions removed the 3rd display based on the results of this study, there could be unforeseen error(s) if that display helped in situations not assessed in this study.

Effect of Simulation-Based Training and Neurofeedback Interventions on Radiation Technologists' Workload, Situation Awareness, and Performance.

PURPOSE: Our purpose was to assess the effect of a combined intervention - simulation-based training supported by neurofeedback sessions - on radiation technologists' (RTs') workload, situation awareness, and performance during routine quality assurance and treatment delivery tasks. METHODS AND MATERIALS: As part of a prospective institutional review board approved study, 32 RTs previously randomized to receive versus not receive simulation-based training focused on patient safety were again randomized to receive versus not receive a 3-week neurofeedback intervention (8 sessions of alpha-theta protocol) focused on stress reduction as well as conscious precision, strong focus, and ability to solve arising problems. Perceived workload was quantified using the NASA Task Load Index. Situation awareness was quantified using the situation awareness rating technique. Performance score was calculated using procedural compliance with time-out components and error detection. RESULTS: RTs randomized to simulation-based training followed by neurofeedback sessions demonstrated no significant changes in perceived workload or situation awareness scores, but did have better performance compared with other study groups (P < .01). CONCLUSIONS: This finding is encouraging and provides basis for using neurofeedback as means to possibly augment performance improvements gained during simulation-based training.

Pilot Study Using Neurofeedback as a Tool to Reduce Surgical Resident Burnout.

BACKGROUND: Burnout is prevalent among surgical residents. Neurofeedback is a technique to train the brain in self-regulation skills. We aimed to assess the impact of neurofeedback on the cognitive workload and personal growth areas of surgery residents with burnout and depression. STUDY DESIGN: Fifteen surgical residents with burnout (Maslach Burnout Inventory [MBI] score > 27) and depression (Patient Health Questionnaire-9 Depression Screen [PHQ-9] score >10), from 1 academic institution, were enrolled and participated in this institutional review board-approved prospective study. Ten residents with more severe burnout and depression scores were assigned to receive 8 weeks of neurofeedback treatments, and 5 others with less severe symptoms were treated as controls. Each participant's cognitive workload (or mental effort) was assessed initially, and again after treatment via electroencephalogram (EEG) while the subjects performed n-back working memory tasks. Analysis of variance (ANOVA) tested for significance between the degree of change in the treatment and control groups. Each subject was also asked to rate changes in growth areas, such as sleep and stress. RESULTS: Both groups showed high cognitive workload in the pre-assessment. After the neurofeedback intervention, the treatment group showed a significant (p < 0.01) improvement in cognitive workload via EEG during the working memory task. These differences were not noted in the control group. There was significant correlation between time (NFB sessions) and average improvement in all growth areas (r = 0.98) CONCLUSIONS: Residents demonstrated high levels of burnout, correlating with EEG patterns indicative of post-traumatic stress disorder. There was a notable change in cognitive workload after the neurofeedback treatment, suggesting a return to a more efficient neural network.

Impact of Workspace Design on Radiation Therapist Technicians' Physical Stressors, Mental Workload, Situation Awareness, and Performance.

PURPOSE: Our purpose was to assess the effect of workspace configuration on radiation therapists' (RTs) physical stressors, mental workload (WL), situational awareness (SA), and performance during routine treatment delivery tasks in a simulated environment. METHODS AND MATERIALS: Fourteen RTs were randomized to 2 workspace configurations while performing 4 simulated scenarios: current (not ergonomically optimized; n = 7) and enhanced (ergonomically optimized, n = 7). Physical stressors were objectively assessed using a rapid upper limb assessment tool. Mental WL was measured at the end of each simulated scenario subjectively using the NASA Task-Load Index and objectively throughout the scenario using eye-tracking metrics (pupil diameter and blink rate). SA was measured at the end of each simulated scenario subjectively using the situation awareness and review technique. Performance was measured objectively via assessment of time-out compliance, error detection, and procedural compliance. Analysis of variance was used to test the effect of workspace configuration on physical stressors, mental WL, SA, and performance. RESULTS: The enhanced configuration significantly reduced physical stressors (rapid upper limb assessment; P < .01) and resulted in a higher rate of time-out compliance (P = .01) compared with current workspace configuration. No significant effect on other metrics was measured. CONCLUSIONS: Our results suggest that an ergonomically designed workspace may minimize physical stressors and improve the performance of RTs.

Impact of Simulation-Based Training on Radiation Therapists' Workload, Situation Awareness, and Performance.

PURPOSE: This study aimed to assess the impact of simulation-based training intervention on radiation therapy therapist (RTT) mental workload, situation awareness, and performance during routine quality assurance (QA) and treatment delivery tasks. METHODS AND MATERIALS: As part of a prospective institutional review board-approved study, 32 RTTs completed routine QA and treatment delivery tasks on clinical scenarios in a simulation laboratory. Participants, randomized to receive (n = 16) versus not receive (n = 16) simulation-based training had pre- and postintervention assessments of mental workload, situation awareness, and performance. We used linear regression models to compare the postassessment scores between the study groups while controlling for baseline scores. Mental workload was quantified subjectively using the NASA Task Load Index. Situation awareness was quantified subjectively using the situation awareness rating technique and objectively using the situation awareness global assessment technique. Performance was quantified based on procedural compliance (adherence to preset/standard QA timeout tasks) and error detection (detection and correction of embedded treatment planning errors). RESULTS: Simulation-based training intervention was associated with significant improvements in overall performance (P < .01), but had no significant impact on mental workload or subjective/objective quantifications of situation awareness. CONCLUSIONS: Simulation-based training might be an effective tool to improve RTT performance of QA-related tasks.

Incorporating Human Factors Analysis and Classification System (HFACS) Into Analysis of Reported Near Misses and Incidents in Radiation Oncology.

PURPOSE: Human factors analysis and classification system (HFACS) is a framework for investigation into causation of human errors. We herein assess whether radiation oncology professionals, with brief training, can conduct HFACS on reported near misses or safety incidents (NMSIs) in a reliable (eg, with a high level of agreement) and practical (eg, timely and with user satisfaction) manner. METHODS AND MATERIALS: We adapted a classical HFACS framework by selecting and modifying main headings, subheadings, and nano-codes that were most likely to apply to radiation oncology settings. The final modified HFACS included 3 main headings, 8 subheadings, and 20 nano-codes. The modified HFACS was first tested in a simulated trial on 8 NMSI and was analyzed by 5 to 10 radiation oncology professionals, with 2 endpoints: (1) agreement among participants at the main-heading, subheading, and nano-code level, and (2) time to complete the analysis. We then performed a prospective trial integrating this approach into a weekly NMSI review meeting, with 10 NMSIs analyzed by 8 to 13 radiation oncology professionals with the same endpoints, while also collecting survey data on participants' satisfaction. RESULTS: In the simulated trial, agreement among participants was 85% on the main headings, 73% on the subheadings, and 70% on the nano-codes. Participants needed, on average, 16.4 minutes (standard deviation, 5.7 minutes) to complete an analysis. In the prospective trial, agreement between participants was 81% on the main headings, 75% on the subheadings, and 74% on the nano-codes. Participants needed, on average, 8.3 minutes (standard deviation, 4.7 minutes) to complete an analysis. The average satisfaction with the proposed HFACS approach was 3.9 (standard deviation 1.0) on a scale from 1 to 5. CONCLUSIONS: This study demonstrates that, after relatively brief training, radiation oncology professionals were able to perform HFACS analysis in a reliable and timely manner and with a relatively high level of satisfaction.

Association of the Usability of Electronic Health Records With Cognitive Workload and Performance Levels Among Physicians.

Importance: Current electronic health record (EHR) user interfaces are suboptimally designed and may be associated with excess cognitive workload and poor performance. Objective: To assess the association between the usability of an EHR system for the management of abnormal test results and physicians' cognitive workload and performance levels. Design, Setting, and Participants: This quality improvement study was conducted in a simulated EHR environment. From April 1, 2016, to December 23, 2016, residents and fellows from a large academic institution were enrolled and allocated to use either a baseline EHR (n = 20) or an enhanced EHR (n = 18). Data analyses were conducted from January 9, 2017, to March 30, 2018. Interventions: The EHR with enhanced usability segregated in a dedicated folder previously identified critical test results for patients who did not appear for a scheduled follow-up evaluation and provided policy-based decision support instructions for next steps. The baseline EHR displayed all patients with abnormal or critical test results in a general folder and provided no decision support instructions for next steps. Main Outcomes and Measures: Cognitive workload was quantified subjectively using NASA-Task Load Index and physiologically using blink rates. Performance was quantified according to the percentage of appropriately managed abnormal test results. Results: Of the 38 participants, 25 (66%) were female. The 20 participants allocated to the baseline EHR compared with the 18 allocated to the enhanced EHR demonstrated statistically significantly higher cognitive workload as quantified by blink rate (mean [SD] blinks per minute, 16 [9] vs 24 [7]; blink rate, -8 [95% CI, -13 to -2]; P = .01). The baseline group showed statistically significantly poorer performance compared with the enhanced group who appropriately managed 16% more abnormal test results (mean [SD] performance, 68% [19%] vs 98% [18%]; performance rate, -30% [95% CI, -40% to -20%]; P < .001). Conclusions and Relevance: Relatively basic usability enhancements to the EHR system appear to be associated with better physician cognitive workload and performance; this finding suggests that next-generation systems should strip away non-value-added EHR interactions, which may help physicians eliminate the need to develop their own suboptimal workflows.

Identifying Factors and Root Causes Associated With Near-Miss or Safety Incidents in Patients Treated With Radiotherapy: A Case-Control Analysis.

PURPOSE: To identify factors associated with a near-miss or safety incident (NMSI) in patients undergoing radiotherapy and identify common root causes of NMSIs and their relationship with incident severity. METHODS: We retrospectively studied NMSIs filed between October 2014 and April 2016. We extracted patient-, treatment-, and disease-specific data from patients with an NMSI (n = 200; incident group) and a similar group of control patients (n = 200) matched in time, without an NMSI. A root cause and incident severity were determined for each NMSI. Univariable and multivariable analyses were performed to determine which specific factors were contributing to NMSIs. Multivariable logistic regression was used to determine root causes of NMSIs and their relationship with incident severity. RESULTS: NMSIs were associated with the following factors: head and neck sites (odds ratio [OR], 5.2; P = .01), image-guided intensity-modulated radiotherapy (OR, 3; P = .009), daily imaging (OR, 7; P < .001), and tumors staged as T2 (OR, 3.3; P = .004). Documentation and scheduling errors were the most common root causes (29%). Communication errors were more likely to affect patients ( P < .001), and technical treatment delivery errors were most associated with a higher severity score ( P = .005). CONCLUSION: Several treatment- and disease-specific factors were found to be associated with an NMSI. Overall, our results suggest that complexity (eg, head and neck, image-guided intensity-modulated radiotherapy, and daily imaging) might be a contributing factor for an NMSI. This promotes an idea of developing a more dedicated and robust quality assurance system for complex cases and highlights the importance of a strong reporting system to support a safety culture.

Improving radiation oncology providers' workload and performance: Can simulation-based training help?

PURPOSE: To help with ongoing safety challenges in radiation therapy (RT), the objective of this research was to develop and assess the impact of a simulation-based training intervention on radiation oncology providers' workload and performance during treatment planning and quality assurance (QA) tasks. METHODS AND MATERIALS: Eighteen radiation oncology professionals completed routine treatment planning and QA tasks on 2 clinical scenarios in a simulation laboratory as part of a prospective institutional review board-approved study. Workload was measured at the end of each assessment/scenario using the NASA Task-Load Index. Performance was quantified based on procedural compliance (adherence to preset/standard QA tasks), time-to-scenario completion, and clinically relevant performance. Participants were then randomized to receive (vs not receive) simulation-based training intervention (eg, standardized feedback on workload and performance) and underwent repeat measurements of workload and performance. Pre- and postintervention changes in workload and performance from participants who received (vs did not receive) were compared using 2-way analysis of variance. RESULTS: Simulation-based training was associated with significant improvements in procedural compliance (P = .01) and increases in time-to-scenario completion (P < .01) but had no significant impact on subjective workload or clinically relevant performance. CONCLUSION: Simulation-based training may be a tool to improve procedural compliance of RT professionals and to acquire new skills and knowledge to proactively maintain RT professionals' preoccupation with patient safety.

Quantification of baseline pupillary response and task-evoked pupillary response during constant and incremental task load.

The methods employed to quantify the baseline pupil size and task-evoked pupillary response (TEPR) may affect the overall study results. To test this hypothesis, the objective of this study was to assess variability in baseline pupil size and TEPR during two basic working memory tasks: constant load of 3-letters memorisation-recall (10 trials), and incremental load memorisation-recall (two trials of each load level), using two commonly used methods (1) change from trail/load specific baseline, (2) change from constant baseline. Results indicated that there was a significant shift in baseline between the trails for constant load, and between the load levels for incremental load. The TEPR was independent of shifts in baseline using method 1 only for constant load, and method 2 only for higher levels of incremental load condition. These important findings suggest that the assessment of both the baseline and methods to quantify TEPR are critical in ergonomics application, especially in studies with small number of trials per subject per condition. Practitioner Summary: Quantification of TEPR can be affected by shifts in baseline pupil size that are most likely affected by non-cognitive factors when other external factors are kept constant. Therefore, quantification methods employed to compute both baseline and TEPR are critical in understanding the information processing of humans in practical ergonomics settings.

Toward a better understanding of task demands, workload, and performance during physician-computer interactions.

OBJECTIVE: To assess the relationship between (1) task demands and workload, (2) task demands and performance, and (3) workload and performance, all during physician-computer interactions in a simulated environment. METHODS: Two experiments were performed in 2 different electronic medical record (EMR) environments: WebCIS (n = 12) and Epic (n = 17). Each participant was instructed to complete a set of prespecified tasks on 3 routine clinical EMR-based scenarios: urinary tract infection (UTI), pneumonia (PN), and heart failure (HF). Task demands were quantified using behavioral responses (click and time analysis). At the end of each scenario, subjective workload was measured using the NASA-Task-Load Index (NASA-TLX). Physiological workload was measured using pupillary dilation and electroencephalography (EEG) data collected throughout the scenarios. Performance was quantified based on the maximum severity of omission errors. RESULTS: Data analysis indicated that the PN and HF scenarios were significantly more demanding than the UTI scenario for participants using WebCIS (P < .01), and that the PN scenario was significantly more demanding than the UTI and HF scenarios for participants using Epic (P < .01). In both experiments, the regression analysis indicated a significant relationship only between task demands and performance (P < .01). DISCUSSION: Results suggest that task demands as experienced by participants are related to participants' performance. Future work may support the notion that task demands could be used as a quality metric that is likely representative of performance, and perhaps patient outcomes. CONCLUSION: The present study is a reasonable next step in a systematic assessment of how task demands and workload are related to performance in EMR-evolving environments.

Application of human factors analysis and classification system model to event analysis in radiation oncology.

PURPOSE: To assess the applicability of the human factors analysis and classification system (HFACS) model for analyzing events in a radiation oncology environment by comparing the HFACS analysis results between novices and experts. METHODS AND MATERIALS: Four novices (resident physicians, 2-4 postgraduate years) volunteered to participate and were asked to independently perform the HFACS analysis on the selected 30 events. The events were divided into the following 2 sets: (1) the description of events was given in detail (eg, the conditions under which the events occurred including information about root causes of error); (2) the description of events was given without detailed information. Each novice's categorization of events into 4 main and 12 sublevels were compared with expert categorization ("gold standard") and used for statistical analysis. Analysis of variance was performed to test the difference in mean agreement with the expert-defined gold standard between the novices across the 4 main levels; and across the 12 sublevels independently. RESULTS: There were no significant differences in mean agreement with the expert-defined gold standard among the 4 novices (novice versus novice; P > .05) across the 4 main and 12 sublevels, respectively. There was a significant difference in mean agreement with the expert-defined gold standard among the 4 main levels and 12 sublevels (P < .05) across the 4 novices. For the 4 main levels, there was a significant difference in agreement with the expert-defined gold standard for events with detailed information versus events without detailed information provided across the 4 novices. The additional information did not improve mean agreement on the 12 sublevels. CONCLUSIONS: Novices learned to use the HFACS model for higher level analysis (4 main levels) with 1 hour training. Regardless of the amount of detail provided in the event description, the study results indicate a need of formal training for novices to better understand the definition and their interpretation at the 12 sublevel analyses.

Relating physician's workload with errors during radiation therapy planning.

PURPOSE: To relate subjective workload (WL) levels to errors for routine clinical tasks. METHODS AND MATERIALS: Nine physicians (4 faculty and 5 residents) each performed 3 radiation therapy planning cases. The WL levels were subjectively assessed using National Aeronautics and Space Administration Task Load Index (NASA-TLX). Individual performance was assessed objectively based on the severity grade of errors. The relationship between the WL and performance was assessed via ordinal logistic regression. RESULTS: There was an increased rate of severity grade of errors with increasing WL (P value = .02). As the majority of the higher NASA-TLX scores, and the majority of the performance errors were in the residents, our findings are likely most pertinent to radiation oncology centers with training programs. CONCLUSIONS: WL levels may be an important factor contributing to errors during radiation therapy planning tasks.

Subjective and objective quantification of physician's workload and performance during radiation therapy planning tasks.

PURPOSE: To quantify, and compare, workload for several common physician-based treatment planning tasks using objective and subjective measures of workload. To assess the relationship between workload and performance to define workload levels where performance could be expected to decline. METHODS AND MATERIALS: Nine physicians performed the same 3 tasks on each of 2 cases ("easy" vs "hard"). Workload was assessed objectively throughout the tasks (via monitoring of pupil size and blink rate), and subjectively at the end of each case (via National Aeronautics and Space Administration Task Load Index; NASA-TLX). NASA-TLX assesses the 6 dimensions (mental, physical, and temporal demands, frustration, effort, and performance); scores > or ≈ 50 are associated with reduced performance in other industries. Performance was measured using participants' stated willingness to approve the treatment plan. Differences in subjective and objective workload between cases, tasks, and experience were assessed using analysis of variance (ANOVA). The correlation between subjective and objective workload measures were assessed via the Pearson correlation test. The relationships between workload and performance measures were assessed using the t test. RESULTS: Eighteen case-wise and 54 task-wise assessments were obtained. Subjective NASA-TLX scores (P < .001), but not time-weighted averages of objective scores (P > .1), were significantly lower for the easy vs hard case. Most correlations between the subjective and objective measures were not significant, except between average blink rate and NASA-TLX scores (r = -0.34, P = .02), for task-wise assessments. Performance appeared to decline at NASA-TLX scores of ≥55. CONCLUSIONS: The NASA-TLX may provide a reasonable method to quantify subjective workload for broad activities, and objective physiologic eye-based measures may be useful to monitor workload for more granular tasks within activities. The subjective and objective measures, as herein quantified, do not necessarily track each other, and more work is needed to assess their utilities. From a series of controlled experiments, we found that performance appears to decline at subjective workload levels ≥55 (as measured via NASA-TLX), which is consistent with findings from other industries.

Quantifying the impact of cross coverage on physician's workload and performance in radiation oncology.

PURPOSE: To quantitatively assess the difference in workload and performance of radiation oncology physicians during radiation therapy treatment planning tasks under the conditions of "cross coverage" versus planning a patient with whom they were familiar. METHODS AND MATERIALS: Eight physicians (3 experienced faculty physicians and 5 physician residents) performed 2 cases. The first case represented a "cross-coverage" scenario where the physicians had no prior information about the case to be planned. The second exposure represented a "regular-coverage" scenario where the physicians were familiar with the patient case to be planned. Each case involved 3 tasks to be completed systematically. Workload was assessed both subjectively (perceived) using National Aeronautics and Space Administration-Task Load Index (NASA-TLX), and objectively (physiological) throughout the task using eye data (via monitoring pupil size and blink rate). Performance of each task and the case was measured using completion time. Subjective willingness to approve or disapprove the generated plan was obtained after completion of the case only. RESULTS: Forty-eight perceived and 48 physiological workload assessments were obtained. Overall, results revealed a significant increase in perceived workload (high NASA-TLX score) and decrease in performance (longer completion time and reduced approval rate) during cross coverage. There were nonsignificant increases in pupil diameter and decreases in the blink rate during cross-coverage versus regular-coverage scenario. In both cross-coverage and regular-coverage scenarios the level of experience did not affect workload and performance. CONCLUSIONS: The cross-coverage scenario significantly increases perceived workload and degrades performance versus regular coverage. Hence, to improve patient safety, efforts must be made to develop policies, standard operating procedures, and usability improvements to electronic medical record and treatment planning systems for "easier" information processing to deal with cross coverage, while recognizing strengths and limitations of human performance.

Quantitative assessment of workload and stressors in clinical radiation oncology.

PURPOSE: Workload level and sources of stressors have been implicated as sources of error in multiple settings. We assessed workload levels and sources of stressors among radiation oncology professionals. Furthermore, we explored the potential association between workload and the frequency of reported radiotherapy incidents by the World Health Organization (WHO). METHODS AND MATERIALS: Data collection was aimed at various tasks performed by 21 study participants from different radiation oncology professional subgroups (simulation therapists, radiation therapists, physicists, dosimetrists, and physicians). Workload was assessed using National Aeronautics and Space Administration Task-Load Index (NASA TLX). Sources of stressors were quantified using observational methods and segregated using a standard taxonomy. Comparisons between professional subgroups and tasks were made using analysis of variance ANOVA, multivariate ANOVA, and Duncan test. An association between workload levels (NASA TLX) and the frequency of radiotherapy incidents (WHO incidents) was explored (Pearson correlation test). RESULTS: A total of 173 workload assessments were obtained. Overall, simulation therapists had relatively low workloads (NASA TLX range, 30-36), and physicists had relatively high workloads (NASA TLX range, 51-63). NASA TLX scores for physicians, radiation therapists, and dosimetrists ranged from 40-52. There was marked intertask/professional subgroup variation (P<.0001). Mental demand (P<.001), physical demand (P=.001), and effort (P=.006) significantly differed among professional subgroups. Typically, there were 3-5 stressors per cycle of analyzed tasks with the following distribution: interruptions (41.4%), time factors (17%), technical factors (13.6%), teamwork issues (11.6%), patient factors (9.0%), and environmental factors (7.4%). A positive association between workload and frequency of reported radiotherapy incidents by the WHO was found (r = 0.87, P value=.045). CONCLUSIONS: Workload level and sources of stressors vary among professional subgroups. Understanding the factors that influence these findings can guide adjustments to the workflow procedures, physical layout, and/or communication protocols to enhance safety. Additional evaluations are needed in order to better understand if these findings are systemic.