Detecting Spurious Correlations With Sanity Tests for Artificial Intelligence Guided Radiology Systems.

Artificial intelligence (AI) has been successful at solving numerous problems in machine perception. In radiology, AI systems are rapidly evolving and show progress in guiding treatment decisions, diagnosing, localizing disease on medical images, and improving radiologists' efficiency. A critical component to deploying AI in radiology is to gain confidence in a developed system's efficacy and safety. The current gold standard approach is to conduct an analytical validation of performance on a generalization dataset from one or more institutions, followed by a clinical validation study of the system's efficacy during deployment. Clinical validation studies are time-consuming, and best practices dictate limited re-use of analytical validation data, so it is ideal to know ahead of time if a system is likely to fail analytical or clinical validation. In this paper, we describe a series of sanity tests to identify when a system performs well on development data for the wrong reasons. We illustrate the sanity tests' value by designing a deep learning system to classify pancreatic cancer seen in computed tomography scans.

Quality control of radiomic features using 3D-printed CT phantoms.

Purpose: The lack of standardization in quantitative radiomic measures of tumors seen on computed tomography (CT) scans is generally recognized as an unresolved issue. To develop reliable clinical applications, radiomics must be robust across different CT scan modes, protocols, software, and systems. We demonstrate how custom-designed phantoms, imprinted with human-derived patterns, can provide a straightforward approach to validating longitudinally stable radiomic signature values in a clinical setting. Approach: Described herein is a prototype process to design an anatomically informed 3D-printed radiomic phantom. We used a multimaterial, ultra-high-resolution 3D printer with voxel printing capabilities. Multiple tissue regions of interest (ROIs), from four pancreas tumors, one lung tumor, and a liver background, were extracted from digital imaging and communication in medicine (DICOM) CT exam files and were merged together to develop a multipurpose, circular radiomic phantom (18 cm diameter and 4 cm width). The phantom was scanned 30 times using standard clinical CT protocols to test repeatability. Features that have been found to be prognostic for various diseases were then investigated for their repeatability and reproducibility across different CT scan modes. Results: The structural similarity index between the segment used from the patients' DICOM image and the phantom CT scan was 0.71. The coefficient variation for all assessed radiomic features was < 1.0 % across 30 repeat scans of the phantom. The percent deviation (pDV) from the baseline value, which was the mean feature value determined from repeat scans, increased with the application of the lung convolution kernel, changes to the voxel size, and increases in the image noise. Gray level co-occurrence features, contrast, dissimilarity, and entropy were particularly affected by different scan modes, presenting with pDV > ± 15 % . Conclusions: Previously discovered prognostic and popular radiomic features are variable in practice and need to be interpreted with caution or excluded from clinical implementation. Voxel-based 3D printing can reproduce tissue morphology seen on CT exams. We believe that this is a flexible, yet practical, way to design custom phantoms to validate and compare radiomic metrics longitudinally, over time, and across systems.

Computed tomography x-ray tube life analysis: a multiyear study.

PURPOSE: To assess and compare actual computed tomography (CT) x-ray tube life with manufacturer warranty coverage limits because prolonging the tube life helps to lower operating costs. METHODS: Ten GE LightSpeed CT scanners with 40 Performix Ultra tube changes and 3 GE VCT scanners with 10 Performix Pro tube changes were followed for 6 years. CT x-ray tube life measurements were performed by analyzing log files of the units after a tube change. RESULTS: The Ultra tubes warranty coverage limit is 70 kAs or 12 months, whichever comes first. For Pro tubes, it is 6000 examinations or 12 months, whichever comes first. Measurements for the Performix Ultra CT x-ray tubes showed a range of 7 to 48 months and 16.7 to 239.9 kAs. Mean values for the Ultra CT x-ray tubes were 19.2 ± 12.5 months and 81.0 ± 45.4 kAs. Seven Ultra CT x-ray tubes did not meet the warranty coverage limits, with an average life of 8 months and 48.1 kAs. For the Pro CT x-ray tubes, the measured logs indicated 22.4 ± 9.6 months of CT x-ray tube life. All 10 Pro CT x-ray tubes exceeded company warranty coverage limits. DISCUSSION: Although Pro tubes lasted longer, they acquired fewer scans than did Ultra tubes. A similar result was shown for current output. CONCLUSION: Because the clinical demand for a CT scanner varies, it is difficult to determine the reason for the failed tubes. Mechanical, environmental, and usage factors can reduce the life expectancy of an x-ray tube.

Dose uniformity analysis among ten 16-slice same-model CT scanners.

PURPOSE: With the introduction of multislice scanners, computed tomographic (CT) dose optimization has become important. The patient-absorbed dose may differ among the scanners although they are the same type and model. To investigate the dose output variation of the CT scanners, we designed the study to analyze dose outputs of 10 same-model CT scanners using 3 clinical protocols. MATERIALS AND METHODS: Ten GE Lightspeed (GE Healthcare, Waukesha, Wis) 16-slice scanners located at main campus and various satellite locations of our institution have been included in this study. All dose measurements were performed using poly (methyl methacrylate) (PMMA) head (diameter, 16 cm) and body (diameter, 32 cm) phantoms manufactured by Radcal (RadCal Corp, Monrovia, Calif) using a 9095 multipurpose analyzer with 10 × 9-3CT ion chamber both from the same manufacturer. Ion chamber is inserted into the peripheral and central axis locations and volume CT dose index (CTDIvol) is calculated as weighted average of doses at those locations. Three clinical protocol settings for adult head, high-resolution chest, and adult abdomen are used for dose measurements. RESULTS: We have observed up to 9.4% CTDIvol variation for the adult head protocol in which the largest variation occurred among the protocols. However, head protocol uses higher milliampere second values than the other 2 protocols. Most of the measured values were less than the system-stored CTDIvol values. It is important to note that reduction in dose output from tubes as they age is expected in addition to the intrinsic radiation output fluctuations of the same scanner. CONCLUSION: Although the same model CT scanners were used in this study, it is possible to see CTDIvol variation in standard patient scanning protocols of head, chest, and abdomen. The compound effect of the dose variation may be larger with higher milliampere and multiphase and multilocation CT scans.

Limits of Tumor Detectability in Nuclear Medicine and PET.

OBJECTIVE: Nuclear medicine is becoming increasingly important in the early detection of malignancy. The advantage of nuclear medicine over other imaging modalities is the high sensitivity of the gamma camera. Nuclear medicine counting equipment has the capability of detecting levels of radioactivity which exceed background levels by as little as 2.4 to 1. This translates to only a few hundred counts per minute on a regular gamma camera or as few as 3 counts per minute when using coincidence detection on a positron emission tomography (PET) camera. MATERIAL AND METHODS: We have experimentally measured the limits of detectability using a set of hollow spheres in a Jaszczak phantom at various tumor-to-background ratios. Imaging modalities for this work were (1) planar, (2) SPECT, (3) PET, and (4) planar camera with coincidence detection capability (MCD). RESULTS: When there is no background (infinite contrast) activity present, the detectability of tumors is similar for PET and planar imaging. With the presence of the background activity , PET can detect objects in an order of magnitude smaller in size than that can be seen by conventional planar imaging especially in the typical clinical low (3:1) T/B ratios. The detection capability of the MCD camera lies between a conventional nuclear medicine (planar / SPECT) scans and the detection capability of a dedicated PET scanner. CONCLUSION: Among nuclear medicine's armamentarium, PET is the closest modality to CT or MR imaging in terms of limits of detection. Modern clinical PET scanners have a resolution limit of 4 mm, corresponding to the detection of tumors with a volume of 0.2 ml (7 mm diameter) in 5:1 T/B ratio. It is also possible to obtain better resolution limits with dedicated brain and animal scanners. The future holds promise in development of new detector materials, improved camera design, and new reconstruction algorithms which will improve sensitivity, resolution, contrast, and thereby further diminish the limits of tumor detectability. CONFLICT OF INTEREST: None declared.

Deep-inspiration breath-hold PET/CT: clinical findings with a new technique for detection and characterization of thoracic lesions.

UNLABELLED: Respiratory motion during PET/CT acquisition can cause misregistration and inaccuracies in calculation of standardized uptake values (SUVs). Our aim was to compare the detection and characterization of thoracic lesions on PET/CT with and without a deep-inspiration protocol. METHODS: We studied 15 patients with suspected pulmonary lesions who underwent clinical PET/CT, followed by deep-inspiration breath-hold (BH) PET/CT. In BH CT, the whole chest of the patient was scanned in 15 s at the end of deep inspiration. For BH PET, patients were asked to hold their breath 9 times for 20-s intervals. One radiologist reviewed images, aiming to detect and characterize pulmonary, nodal, and skeletal abnormalities. Clinical CT and BH CT were compared for number, size, and location of lesions. Lesion SUVs were compared between clinical PET and BH PET. Images were also visually assessed for accuracy of fusion and registration. RESULTS: All patients had lesions on clinical CT and BH CT. Pulmonary BH CT detected more lesions than clinical CT in 13 of 15 patients (86.7%). The total number of lung lesions detected increased from 53 with clinical CT to 82 with BH CT (P<0.001). Eleven patients showed a total of 31 lesions with abnormal (18)F-FDG uptake. BH PET/CT had the advantage of reducing misregistration and permitted a better localization of sites with (18)F-FDG uptake. A higher SUV was noted in 22 of 31 lesions on BH PET compared with clinical PET, with an average increase in SUV of 14%. CONCLUSION: BH PET/CT enabled an increased detection and better characterization of thoracic lesions compared with a standard PET/CT protocol, in addition to more precise localization and quantification of the findings. The technique is easy to implement in clinical practice and requires only a minor increase in the examination time.