Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition.

BACKGROUND AND PURPOSE: Artificial intelligence (AI) models in radiology are frequently developed and validated using datasets from a single institution and are rarely tested on independent, external datasets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multi-center AI competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency. MATERIALS AND METHODS: In total, 1201 anonymized, full-head NCCT clinical scans from five institutions were pooled to form the dataset. The dataset encompassed normal studies as well as pathologies including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these-task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2: age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3: evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed each model's performance. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each AI model, along with the area under the ROC curve (AUROC). RESULTS: 1177 studies were processed by four teams. The median age of patients was 62, with an interquartile range of 33. 19 teams from various academic institutions registered for the competition. Of these, four teams submitted their final results. No commercial entities participated in the competition. For task 1, AUROCs ranged from 0.49 to 0.59. For task 2, two teams completed the task with AUROC values of 0.57 and 0.52. For task 3, teams had little to no agreement with the ground truth. CONCLUSIONS: To assess the performance of AI models in real-world clinical scenarios, we analyzed their performance in the ASFNR AI Competition. The first ASFNR Competition underscored the gap between expectation and reality; the models largely fell short in their assessments. As the integration of AI tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.ABBREVIATIONS: AI = artificial intelligence; ASFNR = American Society of Functional Neuroradiology; AUROC = area under the receiver operating characteristic curve; DICOM = Digital Imaging and Communications in Medicine; GEE = generalized estimation equation; IQR = interquartile range; NPV = negative predictive value; PPV = positive predictive value; ROC = receiver operating characteristic; TBI = traumatic brain injury.

Imaging of Atypical and Complicated Posterior Reversible Encephalopathy Syndrome.

Posterior reversible encephalopathy syndrome (PRES) is a condition clinically characterized by headache, altered mental status, seizures, and visual loss and may be associated with systemic hypertension, preeclampsia/eclampsia, chemotherapy, immunosuppressive therapies in the setting of organ transplantation, and uremic encephalopathy. While brain imaging in patients with PRES typically reveals symmetric vasogenic edema within the parietal and occipital lobes, PRES may present with atypical imaging findings such as central brainstem and deep gray involvement without subcortical edema, and even spinal cord involvement. Additionally, PRES may be complicated in some cases by the presence of cytotoxic edema and hemorrhage. This review will serve to summarize the pathophysiologic theories and controversies underlying PRES, imaging features encountered in atypical and complicated PRES, and the implications these findings may have on patient prognosis.

Intracranial Hemorrhage Imaging.

Intracranial hemorrhage is a medical event frequently encountered in the clinical practice of radiology that has significant potential for patient morbidity and mortality. The expedient and accurate identification of intracranial hemorrhage as well as elucidation of the underlying cause can assist in optimizing the care of these patients. In this review, we attempt to familiarize the reader with the imaging appearance of multiple types of intracranial hemorrhage, both intra-axial and extra-axial and utilizing both computed tomography and magnetic resonance imaging, as well as to provide a framework for assessment of the underlying cause of the hemorrhage.

Characteristics of Rib Fractures in Child Abuse-The Role of Low-Dose Chest Computed Tomography.

OBJECTIVES: Our aim is to describe the radiologic characteristics of rib fractures in clinically diagnosed cases of child abuse and suggest a complementary imaging for radiographically occult injuries in highly suspicious cases of child abuse. METHODS: Retrospective analysis of initial and follow-up skeletal surveys and computed tomography (CT) scans of 16 patients younger than 12 months were reviewed after obtaining approval from our institutional review board. The number, location, displacement, and age of the rib fractures were recorded. RESULTS: Out of a total 105 rib fractures, 84% (87/105) were detected on the initial skeletal survey. Seventeen percent (18/105) were seen only after follow-up imaging, more than half of which (11/18) were detected on a subsequent CT. Majority of the fractures were posterior (43%) and anterior (30%) in location. An overwhelming majority (96%) of the fractures are nondisplaced. CONCLUSIONS: Seventeen percent of rib fractures analyzed in the study were not documented on the initial skeletal survey. Majority of fractures are nondisplaced and located posteriorly or anteriorly, areas that are often difficult to assess especially in the acute stage. The CT scan is more sensitive in evaluating these types of fractures. Low-dose chest CT can be an important imaging modality for suspicious cases of child abuse when initial radiographic findings are inconclusive.

Perfusion Imaging in Acute Traumatic Brain Injury.

Traumatic brain injury (TBI) is a significant problem worldwide and neuroimaging plays a critical role in diagnosis and management. Recently, perfusion neuroimaging techniques have been explored in TBI to determine and characterize potential perfusion neuroimaging biomarkers to aid in diagnosis, treatment, and prognosis. In this article, computed tomography (CT) bolus perfusion, MR imaging bolus perfusion, MR imaging arterial spin labeling perfusion, and xenon CT are reviewed with a focus on their applications in acute TBI. Future research directions are also discussed.

DUBA: a deubiquitinase that regulates type I interferon production.

Production of type I interferon (IFN-I) is a critical host defense triggered by pattern-recognition receptors (PRRs) of the innate immune system. Deubiquitinating enzyme A (DUBA), an ovarian tumor domain-containing deubiquitinating enzyme, was discovered in a small interfering RNA-based screen as a regulator of IFN-I production. Reduction of DUBA augmented the PRR-induced IFN-I response, whereas ectopic expression of DUBA had the converse effect. DUBA bound tumor necrosis factor receptor-associated factor 3 (TRAF3), an adaptor protein essential for the IFN-I response. TRAF3 is an E3 ubiquitin ligase that preferentially assembled lysine-63-linked polyubiquitin chains. DUBA selectively cleaved the lysine-63-linked polyubiquitin chains on TRAF3, resulting in its dissociation from the downstream signaling complex containing TANK-binding kinase 1. A discrete ubiquitin interaction motif within DUBA was required for efficient deubiquitination of TRAF3 and optimal suppression of IFN-I. Our data identify DUBA as a negative regulator of innate immune responses.