Implementation of an Institution-Wide Rules-Based Automated CT Protocoling System.

BACKGROUND. Examination protocoling is a noninterpretive task that increases radiologists' workload and can cause workflow inefficiencies. OBJECTIVE. The purpose of this study was to evaluate effects of an automated CT protocoling system on examination process times and protocol error rates. METHODS. This retrospective study included 317,597 CT examinations (mean age, 61.8 ± 18.1 [SD] years; male, 161,125; female, 156,447; unspecified sex, 25) from July 2020 to June 2022. A rules-based automated protocoling system was implemented institution-wide; the system evaluated all CT orders in the EHR and assigned a protocol or directed the order for manual radiologist protocoling. The study period comprised pilot (July 2020 to December 2020), implementation (January 2021 to December 2021), and postimplementation (January 2022 to June 2022) phases. Proportions of automatically protocoled examinations were summarized. Process times were recorded. Protocol error rates were assessed by counts of quality improvement (QI) reports and examination recalls and comparison with retrospectively assigned protocols in 450 randomly selected examinations. RESULTS. Frequency of automatic protocoling was 19,366/70,780 (27.4%), 68,875/163,068 (42.2%), and 54,045/83,749 (64.5%) in pilot, implementation, and postimplementation phases, respectively (p < .001). Mean (± SD) times from order entry to protocol assignment for automatically and manually protocoled examinations for emergency department examinations were 0.2 ± 18.2 and 2.1 ± 69.7 hours, respectively; mean inpatient examination times were 0.5 ± 50.0 and 3.5 ± 105.5 hours; and mean outpatient examination times were 361.7 ± 1165.5 and 1289.9 ± 2050.9 hours (all p < .001). Mean (± SD) times from order entry to examination completion for automatically and manually protocoled examinations for emergency department examinations were 2.6 ± 38.6 and 4.2 ± 73.0 hours, respectively (p < .001); for inpatient examinations were 6.3 ± 74.6 and 8.7 ± 109.3 hours (p = .001); and for outpatient examinations were 1367.2 ± 1795.8 and 1471.8 ± 2118.3 hours (p < .001). In the three phases, there were three, 19, and 25 QI reports and zero, one, and three recalls, respectively, for automatically protocoled examinations, versus nine, 19, and five QI reports and one, seven, and zero recalls for manually protocoled examinations. Retrospectively assigned protocols were concordant with 212/214 (99.1%) of automatically protocoled versus 233/236 (98.7%) of manually protocoled examinations. CONCLUSION. The automated protocoling system substantially reduced radiologists' protocoling workload and decreased times from order entry to protocol assignment and examination completion; protocol errors and recalls were infrequent. CLINICAL IMPACT. The system represents a solution for reducing radiologists' time spent performing noninterpretive tasks and improving care efficiency.

Addressing the Challenges of Implementing Artificial Intelligence Tools in Clinical Practice: Principles From Experience.

The multitude of artificial intelligence (AI)-based solutions, vendors, and platforms poses a challenging proposition to an already complex clinical radiology practice. Apart from assessing and ensuring acceptable local performance and workflow fit to improve imaging services, AI tools require multiple stakeholders, including clinical, technical, and financial, who collaborate to move potential deployable applications to full clinical deployment in a structured and efficient manner. Postdeployment monitoring and surveillance of such tools require an infrastructure that ensures proper and safe use. Herein, the authors describe their experience and framework for implementing and supporting the use of AI applications in radiology workflow.

Real-Time Electronic Dashboard Technology and Its Use to Improve Pediatric Radiology Workflow.

The purpose of our study was to create a real-time electronic dashboard in the pediatric radiology reading room providing a visual display of updated information regarding scheduled and in-progress radiology examinations that could help radiologists to improve clinical workflow and efficiency. To accomplish this, a script was set up to automatically send real-time HL7 messages from the radiology information system (Epic Systems, Verona, WI) to an Iguana Interface engine, with relevant data regarding examinations stored in an SQL Server database for visual display on the dashboard. Implementation of an electronic dashboard in the reading room of a pediatric radiology academic practice has led to several improvements in clinical workflow, including decreasing the time interval for radiologist protocol entry for computed tomography or magnetic resonance imaging examinations as well as fewer telephone calls related to unprotocoled examinations. Other advantages include enhanced ability of radiologists to anticipate and attend to examinations requiring radiologist monitoring or scanning, as well as to work with technologists and operations managers to optimize scheduling in radiology resources. We foresee increased utilization of electronic dashboard technology in the future as a method to improve radiology workflow and quality of patient care.

Utilization Management of High-Cost Imaging in an Outpatient Setting in a Large Stable Patient and Provider Cohort over 7 Years.

Purpose To quantify the effect of a comprehensive, long-term, provider-led utilization management (UM) program on high-cost imaging (computed tomography, magnetic resonance imaging, nuclear imaging, and positron emission tomography) performed on an outpatient basis. Materials and Methods This retrospective, 7-year cohort study included all patients regularly seen by primary care physicians (PCPs) at an urban academic medical center. The main outcome was the number of outpatient high-cost imaging examinations per patient per year ordered by the patient's PCP or by any specialist. The authors determined the probability of a patient undergoing any high-cost imaging procedure during a study year and the number of examinations per patient per year (intensity) in patients who underwent high-cost imaging. Risk-adjusted hierarchical models were used to directly quantify the physician component of variation in probability and intensity of high-cost imaging use, and clinicians were provided with regular comparative feedback on the basis of the results. Observed trends in high-cost imaging use and provider variation were compared with the same measures for outpatient laboratory studies because laboratory use was not subject to UM during this period. Finally, per-member per-year high-cost imaging use data were compared with statewide high-cost imaging use data from a major private payer on the basis of the same claim set. Results The patient cohort steadily increased in size from 88 959 in 2007 to 109 823 in 2013. Overall high-cost imaging utilization went from 0.43 examinations per year in 2007 to 0.34 examinations per year in 2013, a decrease of 21.33% (P < .0001). At the same time, similarly adjusted routine laboratory study utilization decreased by less than half that rate (9.4%, P < .0001). On the basis of unadjusted data, outpatient high-cost imaging utilization in this cohort decreased 28%, compared with a 20% decrease in statewide utilization (P = .0023). Conclusion Analysis of high-cost imaging utilization in a stable cohort of patients cared for by PCPs during a 7-year period showed that comprehensive UM can produce a significant and sustained reduction in risk-adjusted per-patient year outpatient high-cost imaging volume. © RSNA, 2017.

Discrepancies in reporting the vertebral level of abnormality in MR of the spine.

PURPOSE: The aim of this study was to assess discrepancies in the spinal levels of abnormalities stated in the findings or impression (or both) sections of radiology reports of magnetic resonance (MR) imaging. MATERIALS AND METHODS: Radiology reports from January 2006 through December 2007 (n = 2,097,966) were analyzed using an online radiology report search engine. Reports were searched for presence of the key words MR spine and addendum. The addended reports were then manually assessed for any discrepancies in the reported spinal levels between the body and impression sections; the addenda corrected these errors (identified errors). In addition, all reports with the search term MR spine from January 2006 (n = 1,183) and January 2007 (n = 1,354) were assessed manually to recognize unidentified errors in spinal locations of reported pathology. Two neuroradiologists independently graded the clinical significance of errors on a 5-point scale (1 = definitely not significant, 5 = definitely significant). RESULTS: Of the 11,427 spinal MR reports analyzed in 2006, 7 had identified errors in the sites (levels of the spine) of the lesions. In 2007 (n = 11,785 spinal MR reports), 4 reports were detected with identified errors in spinal levels. In January 2006 and January 2007, 8 and 12 reports, respectively, had unidentified erroneous vertebral levels. Errors were related to discrepant vertebral regions (eg, cervical vs thoracic) in 16% of cases (5 of 31), the wrong number of vertebrae (eg, L2 instead of L3) in 68% of cases (21 of 31), and both in 16% of cases (5 of 31). The average time taken to issue an addendum was 5 +/- 7 days in 2006 and 11 +/- 13 days in 2007. Fifteen reports (48%) scored <3 on the scale of clinical significance, 1 report scored 3, and 15 scored >3. CONCLUSIONS: Errors in lesion level on spinal MR do occur in radiology reports. The number of unidentified errors is substantially higher than that of identified errors. Care should be taken before signing off on radiology reports to identify erroneous mentions of the vertebral levels of abnormalities.

Use of Radcube for extraction of finding trends in a large radiology practice.

The purpose of our study was to demonstrate the use of Natural Language Processing (Leximer), along with Online Analytic Processing, (NLP-OLAP), for extraction of finding trends in a large radiology practice. Prior studies have validated the Natural Language Processing (NLP) program, Leximer for classifying unstructured radiology reports based on the presence of positive radiology findings (F (POS)) and negative radiology findings (F (NEG)). The F (POS) included new relevant radiology findings and any change in status from prior imaging. Electronic radiology reports from 1995-2002 and data from analysis of these reports with NLP-Leximer were saved in a data warehouse and exported to a multidimensional structure called the Radcube. Various relational queries on the data in the Radcube were performed using OLAP technique. Thus, NLP-OLAP was applied to determine trends of F (POS) in different radiology exams for different patient and examination attributes. Pivot tables were exported from NLP-OLAP interface to Microsoft Excel for statistical analysis. Radcube allowed rapid and comprehensive analysis of F (POS) and F (NEG) trends in a large radiology report database. Trends of F (POS) were extracted for different patient attributes such as age groups, gender, clinical indications, diseases with ICD codes, patient types (inpatient, ambulatory), imaging characteristics such as imaging modalities, referring physicians, radiology subspecialties, and body regions. Data analysis showed substantial differences between F (POS) rates for different imaging modalities ranging from 23.1% (mammography, 49,163/212,906) to 85.8% (nuclear medicine, 93,852/109,374; p < 0.0001). In conclusion, NLP-OLAP can help in analysis of yield of different radiology exams from a large radiology report database.

Natural language processing using online analytic processing for assessing recommendations in radiology reports.

PURPOSE: The study purpose was to describe the use of natural language processing (NLP) and online analytic processing (OLAP) for assessing patterns in recommendations in unstructured radiology reports on the basis of patient and imaging characteristics, such as age, gender, referring physicians, radiology subspecialty, modality, indications, diseases, and patient status (inpatient vs outpatient). MATERIALS AND METHODS: A database of 4,279,179 radiology reports from a single tertiary health care center during a 10-year period (1995-2004) was created. The database includes reports of computed tomography, magnetic resonance imaging, fluoroscopy, nuclear medicine, ultrasound, radiography, mammography, angiography, special procedures, and unclassified imaging tests with patient demographics. A clinical data mining and analysis NLP program (Leximer, Nuance Inc, Burlington, Massachusetts) in conjunction with OLAP was used for classifying reports into those with recommendations (I(REC)) and without recommendations (N(REC)) for imaging and determining I(REC) rates for different patient age groups, gender, imaging modalities, indications, diseases, subspecialties, and referring physicians. In addition, temporal trends for I(REC) were also determined. RESULTS: There was a significant difference in the I(REC) rates in different age groups, varying between 4.8% (10-19 years) and 9.5% (>70 years) (P <.0001). Significant variations in I(REC) rates were observed for different imaging modalities, with the highest rates for computed tomography (17.3%, 100,493/581,032). The I(REC) rates varied significantly for different subspecialties and among radiologists within a subspecialty (P < .0001). For most modalities, outpatients had a higher rate of recommendations when compared with inpatients. CONCLUSION: The radiology reports database analyzed with NLP in conjunction with OLAP revealed considerable differences between recommendation trends for different imaging modalities and other patient and imaging characteristics.