Software with artificial intelligence-derived algorithms for analysing CT brain scans in people with a suspected acute stroke: a systematic review and cost-effectiveness analysis.

Background: Artificial intelligence-derived software technologies have been developed that are intended to facilitate the review of computed tomography brain scans in patients with suspected stroke. Objectives: To evaluate the clinical and cost-effectiveness of using artificial intelligence-derived software to support review of computed tomography brain scans in acute stroke in the National Health Service setting. Methods: Twenty-five databases were searched to July 2021. The review process included measures to minimise error and bias. Results were summarised by research question, artificial intelligence-derived software technology and study type. The health economic analysis focused on the addition of artificial intelligence-derived software-assisted review of computed tomography angiography brain scans for guiding mechanical thrombectomy treatment decisions for people with an ischaemic stroke. The de novo model (developed in R Shiny, R Foundation for Statistical Computing, Vienna, Austria) consisted of a decision tree (short-term) and a state transition model (long-term) to calculate the mean expected costs and quality-adjusted life-years for people with ischaemic stroke and suspected large-vessel occlusion comparing artificial intelligence-derived software-assisted review to usual care. Results: A total of 22 studies (30 publications) were included in the review; 18/22 studies concerned artificial intelligence-derived software for the interpretation of computed tomography angiography to detect large-vessel occlusion. No study evaluated an artificial intelligence-derived software technology used as specified in the inclusion criteria for this assessment. For artificial intelligence-derived software technology alone, sensitivity and specificity estimates for proximal anterior circulation large-vessel occlusion were 95.4% (95% confidence interval 92.7% to 97.1%) and 79.4% (95% confidence interval 75.8% to 82.6%) for Rapid (iSchemaView, Menlo Park, CA, USA) computed tomography angiography, 91.2% (95% confidence interval 77.0% to 97.0%) and 85.0 (95% confidence interval 64.0% to 94.8%) for Viz LVO (Viz.ai, Inc., San Fransisco, VA, USA) large-vessel occlusion, 83.8% (95% confidence interval 77.3% to 88.7%) and 95.7% (95% confidence interval 91.0% to 98.0%) for Brainomix (Brainomix Ltd, Oxford, UK) e-computed tomography angiography and 98.1% (95% confidence interval 94.5% to 99.3%) and 98.2% (95% confidence interval 95.5% to 99.3%) for Avicenna CINA (Avicenna AI, La Ciotat, France) large-vessel occlusion, based on one study each. These studies were not considered appropriate to inform cost-effectiveness modelling but formed the basis by which the accuracy of artificial intelligence plus human reader could be elicited by expert opinion. Probabilistic analyses based on the expert elicitation to inform the sensitivity of the diagnostic pathway indicated that the addition of artificial intelligence to detect large-vessel occlusion is potentially more effective (quality-adjusted life-year gain of 0.003), more costly (increased costs of £8.61) and cost-effective for willingness-to-pay thresholds of £3380 per quality-adjusted life-year and higher. Limitations and conclusions: The available evidence is not suitable to determine the clinical effectiveness of using artificial intelligence-derived software to support the review of computed tomography brain scans in acute stroke. The economic analyses did not provide evidence to prefer the artificial intelligence-derived software strategy over current clinical practice. However, results indicated that if the addition of artificial intelligence-derived software-assisted review for guiding mechanical thrombectomy treatment decisions increased the sensitivity of the diagnostic pathway (i.e. reduced the proportion of undetected large-vessel occlusions), this may be considered cost-effective. Future work: Large, preferably multicentre, studies are needed (for all artificial intelligence-derived software technologies) that evaluate these technologies as they would be implemented in clinical practice. Study registration: This study is registered as PROSPERO CRD42021269609. Funding: This award was funded by the National Institute for Health and Care Research (NIHR) Evidence Synthesis programme (NIHR award ref: NIHR133836) and is published in full in Health Technology Assessment; Vol. 28, No. 11. See the NIHR Funding and Awards website for further award information.

Stroke is a serious life-threatening medical condition caused by a blood clot or haemorrhage in the brain. Quick and effective management, including a brain scan, of the patients with suspected stroke can make a big difference in their outcome. Artificial intelligence-derived computer programmes exist that are intended to help with the interpretation of computed tomography scans of the brain in stroke. We undertook a thorough review of the existing research into the effectiveness and value for money of using these programmes to help doctors and other specialists to interpret computed tomography brain scans. We found very little evidence to tell us how well artificial intelligence-derived computer programmes work in practice. Some studies have looked at artificial intelligence-derived computer programmes on their own (i.e. not taken together with a doctor's judgement, as they were designed to be used). Other studies have looked at what happens to patients who are treated for stroke when artificial intelligence-derived computer programmes are used; these studies provide no information about whether using artificial intelligence-derived computer programmes may have led to patients who could have benefitted from treatment being missed. It is unclear how well artificial intelligence-derived software-assisted review works when added to current clinical practice.

Systematic review search methods evaluated using the Preferred Reporting of Items for Systematic Reviews and Meta-Analyses and the Risk Of Bias In Systematic reviews tool.

OBJECTIVES: To evaluate the methodological and reporting characteristics of search methods of systematic reviews (SRs) using the Preferred Reporting of Systematic Reviews and Meta-Analyses (PRISMA) checklist and the Risk Of Bias In Systematic reviews (ROBIS) tool. METHODS: A sample of 505 SRs published in 2016 was taken from KSR Evidence, a database of SRs, and analyzed to assess compliance with Information sources and Search of the PRISMA checklist. Domain 2 (D2) (Identification and Selection of Studies) of the ROBIS tool was used to judge the risk of bias in search methods. RESULTS: Regarding Information sources and Search of PRISMA, twenty percent of SRs which claimed to be PRISMA-compliant in their methods, were compliant; twenty-four percent of SRs published in journals that require PRISMA reporting were compliant; nineteen percent in total were found to be compliant. Twenty-eight percent of SRs were judged to be at a low risk of bias in D2 and so searched widely with an effective strategy and, finally, ten percent were both compliant with the reporting of Information sources and with Search of PRISMA and were judged to be at a low risk of bias in D2. CONCLUSIONS: Ninety percent of SRs are failing to report search methods adequately and to conduct comprehensive searches using a wide range of resources. Editors of journals and peer reviewers need to ensure that they understand the requirements of PRISMA and that compliance is adhered to. Additionally, the comprehensiveness of search methods for SRs needs to be given more critical consideration.

A systematic review of international guidelines and recommendations for the genetic screening, diagnosis, genetic counseling, and treatment of BRCA-mutated breast cancer.

PURPOSE: To conduct a systematic review of international guidelines on screening and management of patients with BRCA-mutated breast cancer (BC). METHODS: Major electronic databases (MEDLINE and Embase; N=8) and gray literature sources were searched (January 2007 to February 2018). Latest guideline recommendations on genetic screening, counseling, and BC treatment of BRCA mutation carriers were summarized. Guidelines specific to germline BRCA (gBRCA) mutation were captured where available. RESULTS: A total of 3,775 records were retrieved and 32 guidelines were included; Europe (n=16), USA (n=11), Canada (n=3), Australia (n=1), and Japan (n=1) were included. Across and within guidelines, genetic counseling was recommended at multiple points in the care pathway, though the format was not always clearly defined. US guidelines emphasized that BRCA mutation testing should occur after specialized genetic counseling; other European guidelines are less prescriptive. BRCA testing eligibility criteria differed, with some guidelines being less restrictive; US National Comprehensive Cancer Network (NCCN) BC guidelines specified that HER2-negative BC patients eligible for single-agent therapy are eligible for gBRCA testing. Fast-track BRCA testing is recommended in the Netherlands if treatment choice will affect survival, but in the UK only as part of clinical trials. More recent European (European School of Oncology-European Society for Medical Oncology 3rd International Consensus Guidelines for Breast Cancer in Young Women 2017, Arbeitsgemeinschaft Gynäkologische Onkologie 2017 in Germany) and US (NCCN) guidelines have updated recommendations regarding gBRCA-targeted poly(ADP-ribose) polymerase (PARP) inhibitor therapy in BC. CONCLUSION: Regional and organizational guidelines differ for genetic screening, counseling, and treatment of patients with BRCA-mutated BC. Guideline harmonization would optimize identification and management of these patients.

Clinically inappropriate post hoc exclusion of study participants from test accuracy calculations: the ROMA score, an example from a recent NICE diagnostic assessment.

OBJECTIVE: To explore how the definition of the target condition and post hoc exclusion of participants can limit the usefulness of diagnostic accuracy studies. METHODS: We used data from a systematic review, conducted for a NICE diagnostic assessment of risk scores to inform secondary care decisions about specialist referral for women with suspected ovarian cancer, to explore how the definition of the target condition and post hoc exclusion of participants can limit the usefulness of diagnostic accuracy studies to inform clinical practice. RESULTS: Fourteen of the studies evaluated the ROMA score, nine used Abbott ARCHITECT tumour marker assays, five used Roche Elecsys. The summary sensitivity estimate (Abbott ARCHITECT) was highest, 95.1% (95% CI: 92.4 to 97.1%), where analyses excluded participants with borderline tumours or malignancies other than epithelial ovarian cancer and lowest, 75.0% (95% CI: 60.4 to 86.4%), where all participants were included. Results were similar for Roche Elecsys tumour marker assays. Although the number of patients involved was small, data from studies that reported diagnostic accuracy for both the whole study population and with post hoc exclusion of those with borderline or non-epithelial malignancies suggested that patients with borderline or malignancies other than epithelial ovarian cancer accounts for between 50 and 85% of false-negative ROMA scores. CONCLUSIONS: Our results illustrate the potential consequences of inappropriate population selection in diagnostic studies; women with non-epithelial ovarian cancers or non-ovarian primaries, and those borderline tumours may be disproportionately represented among those with false negative, 'low risk' ROMA scores. These observations highlight the importance of giving careful consideration to how the target condition has been defined when assessing whether the diagnostic accuracy estimates reported in clinical studies will translate into clinical utility in real-world settings.

Risk scores to guide referral decisions for people with suspected ovarian cancer in secondary care: a systematic review and cost-effectiveness analysis.

BACKGROUND: Ovarian cancer is the sixth most common cancer in UK women and can be difficult to diagnose, particularly in the early stages. Risk-scoring can help to guide referral to specialist centres. OBJECTIVES: To assess the clinical and cost-effectiveness of risk scores to guide referral decisions for women with suspected ovarian cancer in secondary care. METHODS: Twenty-one databases, including MEDLINE and EMBASE, were searched from inception to November 2016. Review methods followed published guidelines. The meta-analysis using weighted averages and random-effects modelling was used to estimate summary sensitivity and specificity with 95% confidence intervals (CIs). The cost-effectiveness analysis considered the long-term costs and quality-adjusted life-years (QALYs) associated with different risk-scoring methods, and subsequent care pathways. Modelling comprised a decision tree and a Markov model. The decision tree was used to model short-term outcomes and the Markov model was used to estimate the long-term costs and QALYs associated with treatment and progression. RESULTS: Fifty-one diagnostic cohort studies were included in the systematic review. The Risk of Ovarian Malignancy Algorithm (ROMA) score did not offer any advantage over the Risk of Malignancy Index 1 (RMI 1). Patients with borderline tumours or non-ovarian primaries appeared to account for disproportionately high numbers of false-negative, low-risk ROMA scores. (Confidential information has been removed.) To achieve similar levels of sensitivity to the Assessment of Different NEoplasias in the adneXa (ADNEX) model and the International Ovarian Tumour Analysis (IOTA) group's simple ultrasound rules, a very low RMI 1 decision threshold (25) would be needed; the summary sensitivity and specificity estimates for the RMI 1 at this threshold were 94.9% (95% CI 91.5% to 97.2%) and 51.1% (95% CI 47.0% to 55.2%), respectively. In the base-case analysis, RMI 1 (threshold of 250) was the least effective [16.926 life-years (LYs), 13.820 QALYs] and the second cheapest (£5669). The IOTA group's simple ultrasound rules (inconclusive, assumed to be malignant) were the cheapest (£5667) and the second most effective [16.954 LYs, 13.841 QALYs], dominating RMI 1. The ADNEX model (threshold of 10%), costing £5699, was the most effective (16.957 LYs, 13.843 QALYs), and compared with the IOTA group's simple ultrasound rules, resulted in an incremental cost-effectiveness ratio of £15,304 per QALY gained. At thresholds of up to £15,304 per QALY gained, the IOTA group's simple ultrasound rules are cost-effective; the ADNEX model (threshold of 10%) is cost-effective for higher thresholds. LIMITATIONS: Information on the downstream clinical consequences of risk-scoring was limited. CONCLUSIONS: Both the ADNEX model and the IOTA group's simple ultrasound rules may offer increased sensitivity relative to current practice (RMI 1); that is, more women with malignant tumours would be referred to a specialist multidisciplinary team, although more women with benign tumours would also be referred. The cost-effectiveness model supports prioritisation of sensitivity over specificity. Further research is needed on the clinical consequences of risk-scoring. STUDY REGISTRATION: This study is registered as PROSPERO CRD42016053326. FUNDING DETAILS: The National Institute for Health Research Health Technology Assessment programme.

A systematic review to assess adherence and persistence with statins.

OBJECTIVE: To identify and assess studies published over a 10 year period up to February 2016 which measure adherence or persistence with statins, to summarize their methods, strengths and weaknesses and to summarize evidence linking statin adherence/persistence with risk of cardiovascular events. METHODS: Electronic databases and abstracts from four major cardiovascular disease conferences were searched from January 2005 to February 2016. The study selection process was performed by two reviewers working independently. Studies were included if they reported data regarding patient adherence or persistence with statins in adults with primary hypercholesterolemia, using any type of study design or length of follow-up. One reviewer extracted the study data and assessed study quality, which was checked by a second reviewer independently. Given the heterogeneity between the included studies a narrative critique and summary is presented. RESULTS: We report on 84 real world studies which aimed to assess adherence or persistence with statins. The majority of studies concluded that good adherence/persistence was associated with reduction in cardiovascular events and mortality. In two studies high intensity statin regimens were associated with poorer patient adherence when compared to low intensity statins. Adherence and persistence with statin therapy also has an impact on hospitalization costs and other cardiovascular disease (CVD) related costs. CONCLUSIONS: Adherence and persistence are associated with a reduction in CVD events and mortality. There was limited evidence to suggest that high intensity statin regimens are associated with poorer treatment adherence when compared to lower intensity regimens. Hence, more robust studies are required to establish this association. As recommended by the 2013 ACC/AHA, 2016 ESC and several other clinical guidelines, clinicians and pharmacy managers should regularly monitor statin therapy adherence.

Supplementary searches of PubMed to improve currency of MEDLINE and MEDLINE In-Process searches via Ovid.

OBJECTIVE: The research investigated whether conducting a supplementary search of PubMed in addition to the main MEDLINE (Ovid) search for a systematic review is worthwhile and to ascertain whether this PubMed search can be conducted quickly and if it retrieves unique, recently published, and ahead-of-print studies that are subsequently considered for inclusion in the final systematic review. METHODS: Searches of PubMed were conducted after MEDLINE (Ovid) and MEDLINE In-Process (Ovid) searches had been completed for seven recent reviews. The searches were limited to records not in MEDLINE or MEDLINE In-Process (Ovid). RESULTS: Additional unique records were identified for all of the investigated reviews. Search strategies were adapted quickly to run in PubMed, and reviewer screening of the results was not time consuming. For each of the investigated reviews, studies were ordered for full screening; in six cases, studies retrieved from the supplementary PubMed searches were included in the final systematic review. CONCLUSION: Supplementary searching of PubMed for studies unavailable elsewhere is worthwhile and improves the currency of the systematic reviews.

ImmunoCAP® ISAC and Microtest for multiplex allergen testing in people with difficult to manage allergic disease: a systematic review and cost analysis.

BACKGROUND: Allergy is a form of immune-mediated exaggerated sensitivity (hypersensitivity) to a substance that is either inhaled, swallowed, injected or comes into contact with the skin. Foreign substances that provoke allergies are called allergens. It has been claimed that multiplex allergen testing may help in diagnosing the cause of symptoms in patients with an unclear cause of allergy or who are allergic to more than one substance. OBJECTIVES: To evaluate multiplex allergen testing [devices that can measure the presence of multiple immunoglobulin E (IgE) antibodies in a patient's blood at the same time], by assessing (1) clinical effectiveness (allergy symptoms, incidence of acute exacerbations, mortality, adverse events of testing and treatment, health-care presentations or admissions, health-related quality of life); (2) effects on treatment (diet, immunotherapy medications, other potential testing); (3) any additional diagnostic information provided by multiplex allergen testing; and (4) cost-effectiveness (cost of different assessment strategies). METHODS: Fifteen databases were searched from 2005 to April 2015, including MEDLINE (via OvidSp), MEDLINE In-Process Citations, MEDLINE Daily Update, PubMed (National Library of Medicine), EMBASE, Cochrane Database of Systematic Reviews (CDSR), Cochrane Central Register of Controlled Trials (CENTRAL), Database of Abstracts of Reviews of Effects (DARE), Health Technology Assessment (HTA) database, Science Citation Index (SCI), Conference Proceedings Citation Index-Science (CPCI-S), BIOSIS Previews, Latin American and Caribbean Health Sciences Literature (LILACS), National Institute for Health Research (NIHR) HTA programme, and the US Food and Drug Administration (FDA); supplementary searches of conference proceedings and trials registries were performed. Review methods followed published guidance from the Cochrane Collaboration and the Centre for Reviews and Dissemination, University of York, UK. The methodological quality of included studies was assessed using appropriate published tools or a review-specific tool designed by the project team. Studies were summarised in a narrative synthesis. Owing to a lack of data on the clinical effectiveness of multiplex allergen testing, no long-term cost-effectiveness model was developed. A conceptual model structure was developed and cost analyses were performed to examine the short-term costs of various possible diagnostic pathways. RESULTS: Fifteen studies were included in the review. The very limited available data indicated that the addition of multiplex allergen testing [ImmunoCAP(®) Immuno Solid-phase Allergen Chip (ISAC), Thermo Fisher Scientific/Phadia AB, Uppsala, Sweden] to standard diagnostic work-up can change the clinicians' views on the diagnosis, management and treatment of patients. There was some indication that the use of ImmunoCAP ISAC testing may be useful to guide decisions on the discontinuation of restrictive diets, the content of allergen-specific immunotherapy (SIT) prescriptions, and whether or not patients should receive SIT. However, none of the studies that we identified reported any information on clinical outcomes subsequent to changes in treatment or management. There was some evidence that ImmunoCAP ISAC may be useful for discriminating allergens that are structurally similar and are recognised by the same IgE antibody (cross-immunoreactive). No data were available for Microtest (Microtest Matrices Ltd, London, UK). Detailed cost analyses suggested that multiplex allergen testing would have to result in a substantial reduction of the proportions of patients receiving single IgE testing and oral food challenge tests in order to be cost-saving in the short term. CONCLUSIONS: No recommendations for service provision can be made based on the analyses included in this report. It is suggested that a consensus-based protocol for the use of multiplex allergen testing be developed. The clinical effectiveness and cost-effectiveness of the proposed protocol should then be assessed by comparing long-term clinical and quality of life outcomes and resource use in patients managed using the protocol with those managed using a standard diagnostic pathway. STUDY REGISTRATION: This study is registered as PROSPERO CRD42015019739. FUNDING: This project was a Diagnostic Assessment Report commissioned by the NIHR HTA programme on behalf of the National Institute for Health and Care Excellence.