Clinical application of radiological AI for pulmonary nodule evaluation: Replicability and susceptibility to the population shift caused by the COVID-19 pandemic.

PURPOSE: replicability and generalizability of medical AI are the recognized challenges that hinder a broad AI deployment in clinical practice. Pulmonary nodes detection and characterization based on chest CT images is one of the demanded use cases for automatization by means of AI, and multiple AI solutions addressing this task are becoming available. Here, we evaluated and compared the performance of several commercially available radiological AI with the same clinical task on the same external datasets acquired before and during the pandemic of COVID-19. APPROACH: 5 commercially available AI models for pulmonary nodule detection were tested on two external datasets labelled by experts according to the intended clinical task. Dataset1 was acquired before the pandemic and did not contain radiological signs of COVID-19; dataset2 was collected during the pandemic and did contain radiological signs of COVID-19. ROC-analysis was applied separately for the dataset1 and dataset2 to select probability thresholds for each dataset separately. AUROC, sensitivity and specificity metrics were used to assess and compare the results of AI performance. RESULTS: Statistically significant differences in AUROC values were observed between the AI models for the dataset1. Whereas for the dataset2 the differences of AUROC values became statistically insignificant. Sensitivity and specificity differed statistically significantly between the AI models for the dataset1. This difference was insignificant for the dataset2 when we applied the probability threshold initially selected for the dataset1. An update of the probability threshold based on the dataset2 created statistically significant differences of sensitivity and specificity between AI models for the dataset2. For 3 out of 5 AI models, the update of the probability threshold was valuable to compensate for the degradation of AI model performances with the population shift caused by the pandemic. CONCLUSIONS: Population shift in the data is able to deteriorate differences of AI models performance. Update of the probability threshold together with the population shift seems to be valuable to preserve AI models performance without retraining them.

Routine Brain MRI Findings on the Long-Term Effects of COVID-19: A Scoping Review.

RATIONALE AND OBJECTIVES: Post-COVID condition (PCC) is associated with long-term neuropsychiatric symptoms. Magnetic resonance imaging (MRI) in PCC examines the brain metabolism, connectivity, and morphometry. Such techniques are not easily available in routine practice. We conducted a scoping review to determine what is known about the routine MRI findings in PCC patients. MATERIALS AND METHODS: The PubMed database was searched up to 11 April 2023. We included cohort, cross-sectional, and before-after studies in English. Articles with only advanced MRI sequences (DTI, fMRI, VBM, PWI, ASL), preprints, and case reports were excluded. The National Heart, Lung, and Blood Institute and PRISMA Extension tools were used for quality assurance. RESULTS: A total of 7 citations out of 167 were included. The total sample size was 451 patients (average age 51 ± 8 years; 67% female). Five studies followed a single recovering cohort, while two studies compared findings between two severity groups. The MRI findings were perivascular spaces (47%), microbleeds (27%) and white matter lesions (10%). All the studies agreed that PCC manifestations are not associated with specific MRI findings. CONCLUSION: The results of the included studies are heterogeneous due to the low agreement on the types of MRI abnormalities in PCC. Our findings indicate that the routine brain MRI protocol has little value for long COVID diagnostics.

An International Non-Inferiority Study for the Benchmarking of AI for Routine Radiology Cases: Chest X-ray, Fluorography and Mammography.

An international reader study was conducted to gauge an average diagnostic accuracy of radiologists interpreting chest X-ray images, including those from fluorography and mammography, and establish requirements for stand-alone radiological artificial intelligence (AI) models. The retrospective studies in the datasets were labelled as containing or not containing target pathological findings based on a consensus of two experienced radiologists, and the results of a laboratory test and follow-up examination, where applicable. A total of 204 radiologists from 11 countries with various experience performed an assessment of the dataset with a 5-point Likert scale via a web platform. Eight commercial radiological AI models analyzed the same dataset. The AI AUROC was 0.87 (95% CI:0.83-0.9) versus 0.96 (95% CI 0.94-0.97) for radiologists. The sensitivity and specificity of AI versus radiologists were 0.71 (95% CI 0.64-0.78) versus 0.91 (95% CI 0.86-0.95) and 0.93 (95% CI 0.89-0.96) versus 0.9 (95% CI 0.85-0.94) for AI. The overall diagnostic accuracy of radiologists was superior to AI for chest X-ray and mammography. However, the accuracy of AI was noninferior to the least experienced radiologists for mammography and fluorography, and to all radiologists for chest X-ray. Therefore, an AI-based first reading could be recommended to reduce the workload burden of radiologists for the most common radiological studies such as chest X-ray and mammography.

AI-Based CXR First Reading: Current Limitations to Ensure Practical Value.

We performed a multicenter external evaluation of the practical and clinical efficacy of a commercial AI algorithm for chest X-ray (CXR) analysis (Lunit INSIGHT CXR). A retrospective evaluation was performed with a multi-reader study. For a prospective evaluation, the AI model was run on CXR studies; the results were compared to the reports of 226 radiologists. In the multi-reader study, the area under the curve (AUC), sensitivity, and specificity of the AI were 0.94 (CI95%: 0.87-1.0), 0.9 (CI95%: 0.79-1.0), and 0.89 (CI95%: 0.79-0.98); the AUC, sensitivity, and specificity of the radiologists were 0.97 (CI95%: 0.94-1.0), 0.9 (CI95%: 0.79-1.0), and 0.95 (CI95%: 0.89-1.0). In most regions of the ROC curve, the AI performed a little worse or at the same level as an average human reader. The McNemar test showed no statistically significant differences between AI and radiologists. In the prospective study with 4752 cases, the AUC, sensitivity, and specificity of the AI were 0.84 (CI95%: 0.82-0.86), 0.77 (CI95%: 0.73-0.80), and 0.81 (CI95%: 0.80-0.82). Lower accuracy values obtained during the prospective validation were mainly associated with false-positive findings considered by experts to be clinically insignificant and the false-negative omission of human-reported "opacity", "nodule", and calcification. In a large-scale prospective validation of the commercial AI algorithm in clinical practice, lower sensitivity and specificity values were obtained compared to the prior retrospective evaluation of the data of the same population.

Diagnostic Accuracy of AI for Opportunistic Screening of Abdominal Aortic Aneurysm in CT: A Systematic Review and Narrative Synthesis.

In this review, we focused on the applicability of artificial intelligence (AI) for opportunistic abdominal aortic aneurysm (AAA) detection in computed tomography (CT). We used the academic search system PubMed as the primary source for the literature search and Google Scholar as a supplementary source of evidence. We searched through 2 February 2022. All studies on automated AAA detection or segmentation in noncontrast abdominal CT were included. For bias assessment, we developed and used an adapted version of the QUADAS-2 checklist. We included eight studies with 355 cases, of which 273 (77%) contained AAA. The highest risk of bias and level of applicability concerns were observed for the "patient selection" domain, due to the 100% pathology rate in the majority (75%) of the studies. The mean sensitivity value was 95% (95% CI 100-87%), the mean specificity value was 96.6% (95% CI 100-75.7%), and the mean accuracy value was 95.2% (95% CI 100-54.5%). Half of the included studies performed diagnostic accuracy estimation, with only one study having data on all diagnostic accuracy metrics. Therefore, we conducted a narrative synthesis. Our findings indicate high study heterogeneity, requiring further research with balanced noncontrast CT datasets and adherence to reporting standards in order to validate the high sensitivity value obtained.

Changes in software as a medical device based on artificial intelligence technologies.

PURPOSE: to develop a procedure for registering changes, notifying users about changes made, unifying software as a medical device based on artificial intelligence technologies (SaMD-AI) changes, as well as requirements for testing and inspections-quality control before and after making changes. METHODS: The main types of changes, divided into two groups-major and minor. Major changes imply a subsequent change of a SaMD-AI version to improve efficiency and safety, to change the functionality, and to ensure the processing of new data types. Minor changes imply those that SaMD-AI developers can make due to errors in the program code. Three types of SaMD-AI testings are proposed to use: functional testing, calibration testing or control, and technical testing. RESULTS: The presented approaches for validation SaMD-AI changes were introduced. The unified requirements for the request for changes and forms of their submission made this procedure understandable for SaMD-AI developers, and also adjusted the workload for the Experiment experts who checked all the changes made to SaMD-AI. CONCLUSION: This article discusses the need to control changes in the module of SaMD-AI, as innovative products influencing medical decision making. It justifies the need to control a module operation of SaMD-AI after making changes. To streamline and optimize the necessary and sufficient control procedures, a systematization of possible changes in SaMD-AI and testing methods was carried out.

Diffusion processes modeling in magnetic resonance imaging.

BACKGROUND: The paper covers modern approaches to the evaluation of neoplastic processes with diffusion-weighted imaging (DWI) and proposes a physical model for monitoring the primary quantitative parameters of DWI and quality assurance. Models of hindered and restricted diffusion are studied. MATERIAL AND METHOD: To simulate hindered diffusion, we used aqueous solutions of polyvinylpyrrolidone with concentrations of 0 to 70%. We created siloxane-based water-in-oil emulsions that simulate restricted diffusion in the intracellular space. To obtain a high signal on DWI in the broadest range of b values, we used silicon oil with high T2: cyclomethicone and caprylyl methicone. For quantitative assessment of our phantom, we performed DWI on 1.5T magnetic resonance scanner with various fat suppression techniques. We assessed water-in-oil emulsion as an extracorporeal source signal by simultaneously scanning a patient in whole-body DWI sequence. RESULTS: We developed phantom with control substances for apparent diffusion coefficient (ADC) measurements ranging from normal tissue to benign and malignant lesions: from 2.29 to 0.28 mm2/s. The ADC values of polymer solutions are well relevant to the mono-exponential equation with the mean relative difference of 0.91%. CONCLUSION: The phantom can be used to assess the accuracy of the ADC measurements, as well as the effectiveness of fat suppression. The control substances (emulsions) can be used as a body marker for quality assurance in whole-body DWI with a wide range of b values.

Telemedicine-based system for quality management and peer review in radiology.

OBJECTIVES: Quality assurance is the key component of modern radiology. A telemedicine-based quality assurance system helps to overcome the "scoring" approach and makes the quality control more accessible and objective. METHODS: A concept for quality assurance in radiology is developed. Its realization is a set of strategies, actions, and tools. The latter is based on telemedicine-based peer review of 23,199 computed tomography (CT) and magnetic resonance imaging (MRI) images. RESULTS: The conception of the system for quality management in radiology represents a chain of actions: "discrepancies evaluation - routine support - quality improvement activity - discrepancies evaluation". It is realized by an audit methodology, telemedicine, elearning, and other technologies. After a year of systemic telemedicine-based peer reviews, the authors have estimated that clinically significant discrepancies were detected in 6% of all cases, while clinically insignificant ones were found in 19% of cases. Most often, problems appear in musculoskeletal records; 80% of the examinations have diagnostic or technical imperfections. The presence of routine telemedicine support and personalized elearning allowed improving the diagnostics quality. The level of discrepancies has decreased significantly (p < 0.05). CONCLUSION: The telemedicine-based peer review system allows improving radiology departments' network effectiveness. MAIN MESSAGES: • "Scoring" approach to radiologists' performance assessment must be changed. • Telemedicine peer review and personalized elearning significantly decrease the number of discrepancies. • Teleradiology allows linking all primary-level hospitals to a common peer review network.

Feasibility of a clearing house for improved cooperation between telemedicine networks delivering humanitarian services: acceptability to network coordinators.

BACKGROUND: Telemedicine networks, which deliver humanitarian services, sometimes need to share expertise to find particular experts in other networks. It has been suggested that a mechanism for sharing expertise between networks (a 'clearing house') might be useful. OBJECTIVE: To propose a mechanism for implementing the clearing house concept for sharing expertise, and to confirm its feasibility in terms of acceptability to the relevant networks. DESIGN: We conducted a needs analysis among eight telemedicine networks delivering humanitarian services. A small proportion of consultations (5-10%) suggested that networks may experience difficulties in finding the right specialists from within their own resources. With the assistance of key stakeholders, many of whom were network coordinators, various methods of implementing a clearing house were considered. One simple solution is to establish a central database holding information about consultants who have agreed to provide help to other networks; this database could be made available to network coordinators who need a specialist when none was available in their own network. RESULTS: The proposed solution was examined in a desktop simulation exercise, which confirmed its feasibility and probable value. CONCLUSIONS: This analysis informs full-scale implementation of a clearing house, and an associated examination of its costs and benefits.

Comparative performance of seven long-running telemedicine networks delivering humanitarian services.

Seven long-running telemedicine networks were surveyed. The networks provided humanitarian services (clinical and educational) in developing countries, and had been in operation for periods of 5-15 years. The number of experts serving each network ranged from 15 to 513. The smallest network had a total of 10 requesters and the largest one had more than 500 requesters. The networks operated in nearly 60 countries. The seven networks managed a total of 1857 cases in 2011, i.e. an average of 265 cases per year per network. There was a significant growth in total activity, amounting to 100.3 cases per year during the 15 year study period. In 2011, network activity was 50-700 teleconsultations per network. There were clear differences in the patterns of activity, with some networks managing an increasing caseload, and others managing a slowly reducing caseload. The seven networks had published a total of 44 papers listed in Medline which summarized the evidence resulting from the delivery of services by telemedicine. There was a dearth of information about clinical and cost-effectiveness. Nevertheless, the services were widely appreciated by referring doctors, considered to be clinically useful, and there were indications that clinical outcomes for telemedicine patients were often improved. Despite a lack of formal evidence, the present study suggests that telemedicine can provide clinically useful services in developing countries.

Long-running telemedicine networks delivering humanitarian services: experience, performance and scientific output.

OBJECTIVE: To summarize the experience, performance and scientific output of long-running telemedicine networks delivering humanitarian services. METHODS: Nine long-running networks--those operating for five years or more--were identified and seven provided detailed information about their activities, including performance and scientific output. Information was extracted from peer-reviewed papers describing the networks' study design, effectiveness, quality, economics, provision of access to care and sustainability. The strength of the evidence was scored as none, poor, average or good. FINDINGS: The seven networks had been operating for a median of 11 years (range: 5-15). All networks provided clinical tele-consultations for humanitarian purposes using store-and-forward methods and five were also involved in some form of education. The smallest network had 15 experts and the largest had more than 500. The clinical caseload was 50 to 500 cases a year. A total of 59 papers had been published by the networks, and 44 were listed in Medline. Based on study design, the strength of the evidence was generally poor by conventional standards (e.g. 29 papers described non-controlled clinical series). Over half of the papers provided evidence of sustainability and improved access to care. Uncertain funding was a common risk factor. CONCLUSION: Improved collaboration between networks could help attenuate the lack of resources reported by some networks and improve sustainability. Although the evidence base is weak, the networks appear to offer sustainable and clinically useful services. These findings may interest decision-makers in developing countries considering starting, supporting or joining similar telemedicine networks.

Experience with low-cost telemedicine in three different settings. Recommendations based on a proposed framework for network performance evaluation.

BACKGROUND: Telemedicine has been used for many years to support doctors in the developing world. Several networks provide services in different settings and in different ways. However, to draw conclusions about which telemedicine networks are successful requires a method of evaluating them. No general consensus or validated framework exists for this purpose. OBJECTIVE: To define a basic method of performance measurement that can be used to improve and compare teleconsultation networks; to employ the proposed framework in an evaluation of three existing networks; to make recommendations about the future implementation and follow-up of such networks. METHODS: Analysis based on the experience of three telemedicine networks (in operation for 7-10 years) that provide services to doctors in low-resource settings and which employ the same basic design. FINDINGS: Although there are many possible indicators and metrics that might be relevant, five measures for each of the three user groups appear to be sufficient for the proposed framework. In addition, from the societal perspective, information about clinical- and cost-effectiveness is also required. The proposed performance measurement framework was applied to three mature telemedicine networks. Despite their differences in terms of activity, size and objectives, their performance in certain respects is very similar. For example, the time to first reply from an expert is about 24 hours for each network. Although all three networks had systems in place to collect data from the user perspective, none of them collected information about the coordinator's time required or about ease of system usage. They had only limited information about quality and cost. CONCLUSION: Measuring the performance of a telemedicine network is essential in understanding whether the network is working as intended and what effect it is having. Based on long-term field experience, the suggested framework is a practical tool that will permit organisations to assess the performance of their own networks and to improve them by comparison with others. All telemedicine systems should provide information about setup and running costs because cost-effectiveness is crucial for sustainability.

Four years' experience of teleconsultations in daily clinical practice.

In the four-year period from 2000, the Department of Informatics and Telemedicine of the Donetsk R&D Institute of Traumatology and Orthopaedics organized 210 teleconsultations. In 91 cases the Institute was the enquiring party and in 119 the consulting one. Teleconsultations were carried out for 137 male and 73 female patients aged between one month and 85 years. A review of the results showed that the reliability of diagnosis of different traumas and diseases made using digital images was 88%. The efficiency of implementing the recommendations provided by remote consultants was 88%. We developed an algorithm to select the most suitable telemedical technique for a clinical situation. We also developed a list of indications for clinical teleconsultation. The optimum equipment for clinical teleconsultations consists of a PC, digital camera, dial-up Internet line and printer. Asynchronous formal and informal Internet-based teleconsultations are most expedient for routine clinical practice, supplemented by realtime teleconsultations where necessary.

Our experience with telemedicine in traumatology and orthopedics.

BACKGROUND: Telemedicine widely takes root in all branches of modern medicine including traumatology and orthopedics. The main goal of this work was to present our experience with asynchronic teleconsultation in daily clinical practice, in particular in the treatment of polytrauma patients. METHODS: Throughout 2000 and 2003, we carried out 144 teleconsultations for 92 men and 52 women (age range three months to 80 years). Of these, we were the inquiring party in 51 cases, the consulting one in 88 cases, and the mediator in five cases. Time passed till the completion of consultations ranged from 12 to 24 hours. RESULTS: The number of consultants was one, two, three, and more in 99, 22, 3, and 15 teleconsultations, respectively. The most common questions (n=128) were those of treatment tactics. In the majority of cases, the consultant approved of the diagnosis suggested by the inquirer and formulated or corrected the scheme of the treatment. The majority of teleconsultations were concerned with various problems of traumatology (n=83) and orthopedics (n=31). For each clinical case, we received a mean of 2.6 replies (range 1 to 8). The effectiveness of the suggested treatment methods accounted for approximately 80% in final decision making. Teleconsultations provided considerable benefits in the treatment of polytrauma patients, including decreases in in-hospital treatment necessities (16%), in complication rates (9.2%) and their severity, the relative risk of developing complications (10%), and in the need for re-hospitalization (0.4%). CONCLUSION: In view of our experience, we recommend that asynchronic consultations on the basis of the Internet-technology be more commonly used in the treatment of polytrauma patients.