Full Resolution Simulation for Evaluation of Critical Care Imaging Interpretation; Part 2: Random Effects Reveal the Interplay Between Case Difficulty, Resident Competence, and the Training Environment.

RATIONALE AND OBJECTIVES: To further characterize empirical data from a full-resolution simulation of critical care imaging coupled with post hoc grading of resident's interpretations by senior radiologists. To present results from estimating the random effects terms in a comprehensive mixed (hierarchical) regression model. MATERIALS AND METHODS: After accounting for 9 fixed effects detailed in Part 1 of this paper, we estimated normally distributed random effects, expressed in terms of score offsets for each case, resident, program, and grader. RESULTS: The fixed effects alone explained 8.8% of score variation and adding the random effects increased explanatory power of the model to account for 36% of score variation. As quantified by intraclass correlation coefficient (ICC = 28.5%; CI: 25.1-31.6) the majority of score variation is directly attributable to the case at hand. This "case difficulty" measure has reliability of 95%. Individual residents accounted for much of the remaining score variation (ICC = 5.3%; CI: 4.6-5.9) after adjusting for all other effects including level of training. The reliability of this "resident competence" measure is 82%. Residency training program influence on scores was small (ICC = 1.1%; CI: 0.42-1.7). Although a few significantly high and low ones can be identified, reliability of 73% militates for caution. At the same time, low intraprogram variation is very encouraging. Variation attributable to differences between graders was minimal (ICC = 0.58%; CI: 0.0-1.2) which reassures us that the method of scoring is reliable, consistent, and likely extensible. CONCLUSION: Full resolution simulation based evaluation of critical care radiology interpretation is being conducted remotely and efficiently at large scale. A comprehensive mixed model of the resulting scores reliably quantifies case difficulty and resident competence.

Full Resolution Simulation for Evaluation of Critical Care Imaging Interpretation; Part 1: Fixed Effects Identify Influences of Exam, Specialty, Fatigue, and Training on Resident Performance.

RATIONALE AND OBJECTIVES: To describe our full-resolution simulation of critical care imaging coupled with posthoc grading of resident's interpretations and present results from the fixed effects terms in a comprehensive mixed regression model of the resulting scores. MATERIALS AND METHODS: The system delivered full resolution DICOM studies via clinical-grade viewing software integrated with a custom built web-based workflow and reporting system. The interpretations submitted by participating residents from 47 different programs were graded (scores of 0-10) on a case by case basis by a cadre of faculty members from our department. The data from 5 yearly (2014-2018) cycles consisting of 992 separate 65 case, 8 hour simulation sessions were collated from the transaction records. We used a mixed (hierarchical) statistical model with nine fixed and four random independent variables. In this paper, we present the results from the nine fixed effects. RESULTS: There were 19,916/63,839 (27.0%, CI 26.7%-27.4%) scores in the 0-2 range (i.e., clinically significant miss). Neurological cases were more difficult with adjusted scores 2.3 (CI 1.9-3.2) lower than body/musculoskeletal cases. There was a small (0.3, CI 0.20-0.38 points) but highly significant (p<0.0001) decrease in score for the final 13/65 cases (fifth quintile) as evidence of fatigue during the last hour of an 8 hour shift. By comparing adjusted scores from mid-R1 (quarter 3) to late-R3 (quarter 12) we estimate the training effect as an increase of 2.2 (CI 1.90-2.50) points. CONCLUSION: Full resolution simulation based evaluation of critical care radiology interpretation is being conducted remotely and efficiently at large scale. Analysis of the resulting scores yields multiple insights into the interpretative process.

Behavioral Health Factors as Predictors of Emergency Department Use in the High-Risk, High-Cost Medicare Population.

OBJECTIVE: This study measured the presence, extent, and type of behavioral health factors in a high-cost Medicare population and their association with the probability and intensity of emergency department (ED) use. METHODS: Retrospective claims analysis and a comprehensive electronic medical record-based review were conducted for patients enrolled in a 65-month prospective care management program at an academic tertiary medical center (N=3,620). A two-part model used multivariable logistic regression to evaluate the effect of behavioral health factors on the probability of ED use, complemented by a Poisson model to measure the number of ED visits. Control variables included demographic characteristics, poststudy survival, and hierarchical condition category risk score. RESULTS: After analyses controlled for comorbidities and other relevant variables, patients with two or more behavioral health diagnosis categories or two or more behavioral health medications were about twice as likely as those without such categories or medications to use the ED. Patients with a diagnosis category of psychosis, neuropsychiatric disorders, sleep disorders, or adjustment disorders were significantly more likely than those without these disorders to use the ED. Most primary ED diagnoses were not of behavioral health conditions. CONCLUSIONS: Behavioral health factors had a substantial and significant effect on the likelihood and number of ED visits in a population of high-cost Medicare patients. Attention to behavioral health factors as independent predictors of ED use may be useful in influencing ED use in high-cost populations.

Angiosomal radiopathologic analysis of transarterial radioembolization for the treatment of hepatocellular carcinoma.

PURPOSE: To assess the radiopathologic correlation following Yttrium-90 transarterial radioembolization (TARE) of hepatocellular carcinoma (HCC) using variable radiodosimetry to identify imaging surrogates of histologic response. METHODS: Twelve patients with HCC underwent ablative (≥ 190 Gy) and/or non-ablative (< 190 Gy) TARE delivered in a segmental, lobar, or combined fashion as a surgical neoadjuvant or bridge to transplantation. Both targeted tumor and treatment angiosome were analyzed before and after TARE utilizing hepatocyte-specific contrast-enhanced MRI or contrast-enhanced CT. Responses were graded using EASL and mRECIST criteria. Histologic findings including percent tumor necrosis and adjacent hepatic substrate effects were correlated with imaging features. RESULTS: Complete pathologic necrosis (CPN) was observed in 7/12 tumors post-TARE. Ablative and non-ablative dosing resulted in CPN in 5/6 and 2/6 tumors, respectively. Hyperintensity on T2-weighted imaging, the absence of hepatocyte-specific gadolinium contrast uptake, and plateau or persistent enhancement kinetics in the angiosome correlated with CPN and performed similarly to EASL and mRECIST criteria in predicting CPN. CONCLUSIONS: The absence of hepatocyte-specific contrast uptake, increased signal on T2-weighted sequences, and plateau or persistent enhancement in the angiosome may represent MRI surrogates of CPN following TARE of HCC. These findings correlated with EASL and mRECIST response criteria. Further investigation is needed to determine the role of these findings as possible adjuncts to conventional imaging criteria.

Images of imaging: how to process and display imaging utilization for large populations.

OBJECTIVE: We propose a method of processing and displaying imaging utilization data for large populations. CONCLUSION: The comprehensive and finely grained picture of imaging utilization yielded by our methods is a first step toward population-based imaging utilization management. We believe that our methods for the categorization and display of imaging utilization will prove to be widely useful.

Provider Feedback about Imaging Appropriateness by Using Scores from Order Entry Decision Support: Raw Rates Misclassify Outliers.

PURPOSE: To determine the relevant physician- and practice-related factors that jointly affect the rate of low-utility imaging examinations (score of 1-3 out of 9) ordered by means of an order entry system that provides normative appropriateness feedback. MATERIALS AND METHODS: This HIPAA-compliant study was approved by the institutional review board under an expedited protocol for analyzing anonymous aggregated administrative data. This is a retrospective study of approximately 250 000 consecutive scheduled outpatient advanced imaging examinations (computed tomography, magnetic resonance imaging, nuclear medicine) ordered by 164 primary care and 379 medical specialty physicians from 2008 to 2012. A hierarchical logistic regression model was used to identify multiple predictors of the probability that an examination received a low utility score. Physician- and practice-specific random effects were estimated to articulate (odds ratio) and quantify (intraclass correlation) interphysician variation. RESULTS: Fixed effects found to be statistically significant predictors of low-utility imaging included examination type, whether the examination was cancelled, status of the person entering the order, and the total number of examinations ordered by the clinician. Neither patient age nor sex had any effect, and there were no secular trends (year of study). The remaining amount of interphysician variation was moderate (intraclass correlation, 22%), whereas the variation between medical specialties and primary care practices was low (intraclass correlation, 5%). The estimated physician-specific effects had reliability of 70%, which makes them just suitable for identifying outliers. CONCLUSION: The authors found that 22% of the variation in the rate of low-utility examinations is attributable to ordering providers and 5% to their specialty or clinic.

Use of imaging in the emergency department: physicians have limited effect on variation.

PURPOSE: To quantify interphysician variation in imaging use during emergency department (ED) visits and examine the contribution of factors to this variation at the patient, visit, and physician level. MATERIALS AND METHODS: This study was HIPAA compliant and approved by the institutional review board of Partners Healthcare System (Boston, Mass), with waiver of informed consent. In this retrospective study of 88 851 consecutive ED visits during 2011 at a large urban teaching hospital, a hierarchical logistic regression model was used to identify multiple predictors for the probability that low- or high-cost imaging would be ordered during a given visit. Physician-specific random effects were estimated to articulate (by odds ratio) and quantify (by intraclass correlation coefficient [ICC]) interphysician variation. RESULTS: Patient- and visit-level factors found to be statistically significant predictors of imaging use included measures of ED busyness, prior ED visit, referral source to the ED, and ED arrival mode. Physician-level factors (eg, sex, years since graduation, annual workload, and residency training) did not correlate with imaging use. The remaining amount of interphysician variation was very low (ICC, 0.97% for low-cost imaging; ICC, 1.07% for high-cost imaging). These physician-specific odds ratios of imaging estimates were moderately reliable at 0.78 (95% confidence interval [CI]: 0.77, 0.79) for low-cost imaging and 0.76 (95% CI: 0.74, 0.78) for high-cost imaging. CONCLUSION: After careful and comprehensive case-mix adjustment by using hierarchical logistic regression, only about 1% of the variability in ED imaging utilization was attributable to physicians.

Face time versus test ordering: is there a trade-off?

BACKGROUND: As clinician-patient face time comes under pressure, clinicians might consider substituting testing for time spent in diagnostic reasoning, history, and physical exam. OBJECTIVES: To explore the relationship between clinician-patient time and medical resource utilization. METHODS: In the Massachusetts General Hospital/Massachusetts General Physician Organization outpatient radio frequency identification project, clinicians and patients wore real-time location system (RTLS) tags. "Face time" was defined as the duration patients and clinicians were colocated. Radiology testing was used as a proxy for medical resource use. A radiology test was determined to be associated with a clinical encounter if it involved the same patient and clinician and occurred less than 3 months after the index encounter. Radiologic data were derived from the electronic health record and test appropriateness proxy score from the radiology order entry system. Data were synthesized and analyzed using standard structured query language and statistical analysis. RESULTS: From July 2008 to October 2010, 2086 clinical encounter medical records and RTLSmeasured face times could be associated: 1957 for primary care (PC) and 129 for urgent care (UC). Of these, 471 met study criteria. In PC, shorter face time was associated with more testing, but shorter wait times and flow times. In UC, testing was not associated with shorter face times, but was associated with shorter wait times and longer flow times. CONCLUSION: Our pilot suggests RTLS can capture face time and trade-offs between face time and testing. Ongoing studies will elucidate how these trade-offs affect patients, clinicians, and healthcare systems.

Determinants of diagnostic imaging utilization in primary care.

OBJECTIVES: To examine patient and physician factors affecting utilization of diagnostic imaging in primary care. DATA SOURCES/STUDY SETTING: Patient-level data from a large academic group practice over the period July 1, 2007, through June 30, 2009. STUDY DESIGN: This is a retrospective cohort study of 85,277 patients cared for by 148 primary care physicians (PCPs). The dependent variable is the number of outpatient imaging exams ordered by each patient's PCP over the study period. Independent variables include 17 patient factors describing both clinical need and demographic characteristics and 7 physician factors. DATA COLLECTION: Data were collected from the electronic medical record and associated administrative databases. PRINCIPAL FINDINGS: Patient factors having a statistically significant effect on both the probability race, more than 10 medications, congestive heart failure, diabetes, hypertension, other problems, visits to the PCP, visits to specialists, and imaging exams ordered by specialists. For physician factors, experience, gender, and having another degree were statistically significant in both portions of the model. CONCLUSIONS: Both patient and physician factors have a substantial effect on primary care outpatient diagnostic imaging utilization. Several of these significantly influence both the probability that any images will be ordered and the intensity (number) of imaging.

Email notification combined with off site signing substantially reduces resident approval to faculty verification time.

RATIONALE AND OBJECTIVES: Attending radiologist signature time (AST) is a variable and modifiable component of overall report turnaround time. Delays in finalized reports have potential to undermine radiologists' value as consultants and adversely affect patient care. This study was performed to evaluate the impact of notebook computer distribution and daily automated e-mail notification on reducing AST. MATERIALS AND METHODS: Two simultaneous interventions were initiated in the authors' radiology department in February 2010. These included the distribution of a notebook computer with preloaded software for each attending radiologist to sign radiology reports and daily automated e-mail notifications for unsigned reports. The digital dictation system archive and the radiology information system were queried for all radiology reports produced from January 2009 through August 2010. The time between resident approval and attending radiologist signature before and after the intervention was analyzed. Potential unintended "side effects" of the intervention were also studied. RESULTS: Resident-authored reports were signed, on average, 2.53 hours sooner after the intervention. This represented a highly significant (P = .003) decrease in AST with all else held equal. Postintervention reports were authored by residents at the same rate (about 70%). An unintended "side effect" was that attending radiologists were less likely to make changes to resident-authored reports after the intervention. CONCLUSIONS: E-mail notification combined with offsite signing can reduce AST substantially. Notebook computers with preloaded software streamline the process of accessing, editing, and signing reports. The observed decrease in AST reflects a positive change in the timeliness of report signature.

Factors affecting attending agreement with resident early readings of computed tomography and magnetic resonance imaging of the head, neck, and spine.

RATIONALE AND OBJECTIVES: This study examines the joint effect of several factors on radiology resident performance in the task of interpreting after-hours neuroradiology examinations. MATERIALS AND METHODS: As part of a quality assessment process, we conducted a prospective evaluation of all (N = 21,796) after-hours preliminary readings of neuroradiology examinations performed by radiology residents over a 62-month period at our academic medical center. Each reading was scored by the interpreting neuroradiologist as "agree," "disagree with minimal clinical impact," and "disagree with significant clinical impact." Coded resident and attending identities were also recorded for each case along with modality, body area studied, and the date of examination. These raw data were used to create an analytic data set with level of resident/attending agreement as the outcome and six predictors, including two date-derived variables: months 1-62 representing when the case occurred during the study and quartiles 1-4 accounting for the timing of the case in each resident's own experience. Cross tabulations, plots, bivariate statistics, and logistic regression were used to examine the relationships between study variables and the outcome (level of agreement). RESULTS: Over about 5 years of the study, the absolute number of significant disagreements remained stable at about three per month. The total caseload increased at a rate of 4.1 per month with most of the increase falling into the agree category, whereas the minimal disagreements actually decreased slightly (0.2 per month). In the logistic model for disagreement, three of the factors accounted for most of the variance: attending (61%), resident (15%), and month (15%). Study type (modality and area examined) accounted for another 10%. There was no significant contribution from the variable (quartile) constructed to test for individual resident learning during the on-call experience. CONCLUSION: Although residents differ somewhat in the extent of attending agreement with their on-call work, evaluation or remediation made on the basis of simple comparison of these rates should be done with caution. Improved agreement over time seems to be a collective experience shared by residents.

Inference and uncertainty in radiology.

This paper seeks to enhance understanding of the philosophical underpinnings of our discipline and the resulting practical implications. Radiology reports exist in order to convey new knowledge about a patient's condition based on empiric observations from anatomic or functional images of the body. The route to explanation and prediction from empiric evidence is mostly through inference based on inductive (and sometimes abductive) arguments. The conclusions of inductive arguments are, by definition, contingent and provisional. Therefore, it is necessary to deal in some way with the uncertainty of inferential conclusions (i.e. interpretations) made in radiology reports. Two paradigms for managing uncertainty in natural sciences exist in dialectic tension with each other. These are the frequentist and Bayesian theories of probability. Tension between them is mirrored during routine interactions among radiologists and clinicians. I will describe these core issues and argue that they are quite relevant to routine image interpretation and reporting.

Free text versus structured format: information transfer efficiency of radiology reports.

OBJECTIVE: We discuss the effect of radiology report format on the accuracy and speed with which reviewers can extract case-specific information. MATERIALS AND METHODS: A Web-based testing mechanism was used to present radiology reports to each of 16 senior medical students and record their answers to 10 multiple choice questions about specific medical content for each of 12 cases. Subjects were randomly assigned to view the reports in either free text or structured format. In addition to number of answers correct for each case, we recorded the time taken for each case and an efficiency score (correctly answered questions per minute). These three outcomes were tested for differences on report format using multifactorial analysis of variance. A postexperimental questionnaire and a mediated focus group elicited subject preference as to radiology report format. RESULTS: There were no significant differences in the three outcomes (score, time, and efficiency) between the free text and structured format conditions. The power of the experiment was sufficient to detect small differences in these outcomes by format. Subjects strongly and consistently expressed a preference for the structured version. CONCLUSION: We assert that free text and itemized (structured) forms of radiology reports are equally efficient and accurate for transmitting case-specific interpretative content to reviewers of the document.

Assessment of the understanding of the risks and benefits of hormone replacement therapy (HRT) in primary care physicians.

OBJECTIVE: This study was designed to assess the understanding of primary care physicians who practice in the state of Florida of the risks and benefits of hormone replacement therapy as described in the Women's Health Initiative clinical trial. STUDY DESIGN: All primary care physicians in the state of Florida, including family practice, internal medicine, and Ob/Gyn were identified and mailed an anonymous survey asking about their understanding of the risks and benefits of hormone replacement therapy (HRT), including heart disease, breast cancer, osteoporotic fractures, colon cancer, stroke, and death. Univariate statistics stratified by specialty were performed with Kruskal-Wallis one-way analysis of variance. RESULTS: Six thousand one hundred twenty-five surveys were mailed, with the return of 600 completed surveys, including 203 Ob/Gyn, 145 internal medicine, 219 family practice, and 33 other. Overall, Ob/Gyns had a more positive view of HRT, and internal medicine had a relatively negative view of HRT. In general, respondents overestimated the risk attributable to hormone replacement therapy. Ob/Gyns were significantly more likely to provide accurate assessments of these risks and benefits compared with the other specialties. Respondents correctly identified the magnitude of risks and benefits 28% of the time, 67% of the time overestimated risks and benefits, and 5% of the time misunderstood the direction of risk vs benefit of HRT. CONCLUSION: This study suggests that the findings of the Women's Health Initiative are misunderstood by the majority of primary care specialists, although Ob/Gyns have a better understanding of the risks and benefits compared to other specialties. We hypothesize that respondents that overestimate the increase or decrease in risk were making the error of confusing relative risk with absolute risk difference. There is a great need for physician education about the attributable risks and benefits of HRT.

Optic neuropathy secondary to cat scratch disease: distinguishing MR imaging features from other types of optic neuropathies.

BACKGROUND AND PURPOSE: MR imaging characteristics of optic neuropathy caused by cat scratch disease have not yet been described; this lack of information may result in incorrect diagnosis and may contribute to initiation of inappropriate therapy. Our study was based on the hypothesis that cat scratch disease-related optic neuropathy has distinct MR imaging features compared with those of other types of optic neuropathies. METHODS: Eighty-two patients with various causes of optic neuropathy and available MR imaging examinations were included in this study. Two readers blinded to the diagnosis reviewed the MR images independently in regard to presence, location, and extent of optic nerve enhancement. The MR imaging findings were correlated with the final diagnosis. RESULTS: Eleven percent (9/82) of the patients received a final diagnosis of cat scratch disease. Optic nerve enhancement in patients with cat scratch disease (5/37) was localized to a 3- to 4-mm segment at the optic nerve-globe junction. All other patients with optic neuropathy (31/37) with one exception showed enhancement away from the optic nerve-globe junction or a long-segment enhancement when the optic nerve-globe junction was also involved. Four patients with cat scratch disease did not show any optic nerve MR abnormalities. CONCLUSION: Unilateral, short-segment enhancement localized to the optic nerve-globe junction is highly specific for cat scratch disease as the underlying cause of optic neuropathy and may help in establishing the diagnosis of this condition.

Conceptual approach for the design of radiology reporting interfaces: the talking template.

Within the coming decade, traditional dictation supported by human transcription for radiology reports will be replaced by one or more computerized methods. This paper discusses the cognitive and process efficiency problems arising from currently available technology including speech recognition and menu-driven interfaces. A specific concept for interaction with the reporting interface is proposed. This is called the "talking template" and departs from other designs by providing for all interactions to be mediated through audible prompts and microphone controls. The radiologist can recapture efficiency and cognitive focus by dictating while viewing images without the "look away" problem inherent in other interfaces.

The socioeconomic aspects of information technology for health care with emphasis on radiology.

RATIONALE AND OBJECTIVES: Information technology is the key to cost effective and error free medical care in the United States and the only problem is that there is not enough of it yet. During the past 15 years, billions of dollars have been spent on information technology for health care with very little benefit but significant adverse effects on patients, physicians, and nurses. The truth about health care information technology (HIT) probably lies somewhere between these extreme statements, representing technophile and skeptical views, respectively. MATERIALS AND METHODS: There is no doubt that computer and communication hardware has reached a state of sophistication and availability in which any and all necessary information can be generated, stored, and distributed to health care workers in support of their patient care tasks. The barriers to rapid and widespread development and diffusion of cost effective and practically useful HIT are exclusively related to human factors. RESULTS: This article explores some of the organizational, cultural, cognitive, and economic forces that interact to influence success of HIT initiatives in health care organizations. A key point to be recognized is that the intrinsically handcrafted nature of health care work combined with high degrees of complexity and contingency make it impossible to "computerize" with the same ease and completeness of other industries. The major thrust of the argument is that designers of information systems and health care informatics managers must meet needs of patients and care providers. The software they create and implement should promote, support, and enhance the existing processes of health care rather than seeking to dictate how direct care providers should do their work. CONCLUSIONS: Instead of looking for "buy in" from physicians and nurses, the informatics community must return the authority over functional specification of patient care information systems to them--where it belonged in the first place. This same lesson about computer technology and organizational politics is also being learned in the business community, where executives are reclaiming responsibility for mission critical informatics decisions.

The ACR appropriateness criteria: translation to practice and research.

The ACR Appropriateness Criteria (ACRAC) are now just about a decade old. Along with a continued commitment to update and add to these guidelines, the author advocates that the ACR undertake the task of converting the current text-based contents into a comprehensive, multipurpose relational database. A candidate structure for such a database is described, and several uses for it are articulated. The accompanying policy discussion advocates efforts to widely disseminate the resulting ACRAC database for use by payers in setting reimbursement and by hospitals in radiology order entry software. Also described are some ways that the ACRAC, formatted as a database, can be used for health services research.