Automatic de-identification of French electronic health records: a cost-effective approach exploiting distant supervision and deep learning models.

BACKGROUND: Electronic health records (EHRs) contain valuable information for clinical research; however, the sensitive nature of healthcare data presents security and confidentiality challenges. De-identification is therefore essential to protect personal data in EHRs and comply with government regulations. Named entity recognition (NER) methods have been proposed to remove personal identifiers, with deep learning-based models achieving better performance. However, manual annotation of training data is time-consuming and expensive. The aim of this study was to develop an automatic de-identification pipeline for all kinds of clinical documents based on a distant supervised method to significantly reduce the cost of manual annotations and to facilitate the transfer of the de-identification pipeline to other clinical centers. METHODS: We proposed an automated annotation process for French clinical de-identification, exploiting data from the eHOP clinical data warehouse (CDW) of the CHU de Rennes and national knowledge bases, as well as other features. In addition, this paper proposes an assisted data annotation solution using the Prodigy annotation tool. This approach aims to reduce the cost required to create a reference corpus for the evaluation of state-of-the-art NER models. Finally, we evaluated and compared the effectiveness of different NER methods. RESULTS: A French de-identification dataset was developed in this work, based on EHRs provided by the eHOP CDW at Rennes University Hospital, France. The dataset was rich in terms of personal information, and the distribution of entities was quite similar in the training and test datasets. We evaluated a Bi-LSTM + CRF sequence labeling architecture, combined with Flair + FastText word embeddings, on a test set of manually annotated clinical reports. The model outperformed the other tested models with a significant F1 score of 96,96%, demonstrating the effectiveness of our automatic approach for deidentifying sensitive information. CONCLUSIONS: This study provides an automatic de-identification pipeline for clinical notes, which can facilitate the reuse of EHRs for secondary purposes such as clinical research. Our study highlights the importance of using advanced NLP techniques for effective de-identification, as well as the need for innovative solutions such as distant supervision to overcome the challenge of limited annotated data in the medical domain.

Phenotyping of heart failure with preserved ejection faction using electronic health records and echocardiography.

Aims: Patients presenting symptoms of heart failure with preserved ejection fraction (HFpEF) are not a homogenous population. Different phenotypes can differ in prognosis and optimal management strategies. We sought to identify phenotypes of HFpEF by using the medical information database from a large university hospital centre using machine learning. Methods and results: We explored the use of clinical variables from electronic health records in addition to echocardiography to identify different phenotypes of patients with HFpEF. The proposed methodology identifies four phenotypic clusters based on both clinical and echocardiographic characteristics, which have differing prognoses (death and cardiovascular hospitalization). Conclusion: This work demonstrated that artificial intelligence-derived phenotypes could be used as a tool for physicians to assess risk and to target therapies that may improve outcomes.

Leveraging Clinical Data Warehouses to Measure Impact of Update Prescription Guidelines of Polyvalent Immunoglobulins in 2018 in France.

In France and in other countries, we observed a significant growth in human polyvalent immunoglobulins (PvIg) usage. PvIg is manufactured from plasma collected from numeral donors, and its production is complex. Supply tensions have been observed for several years, and it is necessary to limit their consumption. Therefore, French Health Authority (FHA) provided guidelines in June 2018 to restrict their usage. This research aims to assess the guidelines' impact of the FHA on the use of PvIg. We analyzed data from Rennes University Hospital, where all PvIg prescriptions are reported electronically with quantity, rhythm, and indication. From the clinical data warehouses of RUH, we extracted comorbidities and lab results to evaluate the more complex guidelines. We globally noticed a reduction in the consumption of PvIg after the guidelines. Compliance with the recommended quantities and rhythms have also been observed. By combining two sources of data, we have been able to show an impact of FHA's guidelines on the consumption of PvIg.

Prescreening in oncology trials using medical records. Natural language processing applied on lung cancer multidisciplinary team meeting reports.

Defining profiles of patients that could benefit from relevant anti-cancer treatments is essential. An increasing number of specific criteria are necessary to be eligible to specific anti-cancer therapies. This study aimed to develop an automated algorithm able to detect patient and tumor characteristics to reduce the time-consuming prescreening for trial inclusions without delay. Hence, 640 anonymized multidisciplinary team meetings (MTM) reports concerning lung cancers from one French teaching hospital data warehouse between 2018 and 2020 were annotated. To automate the extraction of eight major eligibility criteria, corresponding to 52 classes, regular expressions were implemented. The RegEx's evaluation gave a F1-score of 93% in average, a positive predictive value (precision) of 98% and sensitivity (recall) of 92%. However, in MTM, fill rates variabilities among patient and tumor information remained important (from 31% to 100%). Genetic mutations and rearrangement test results were the least reported characteristics and also the hardest to automatically extract. To ease prescreening in clinical trials, the PreScIOUs study demonstrated the additional value of rule based and machine learning based methods applied on lung cancer MTM reports.

Gastroenteritis Forecasting Assessing the Use of Web and Electronic Health Record Data With a Linear and a Nonlinear Approach: Comparison Study.

BACKGROUND: Disease surveillance systems capable of producing accurate real-time and short-term forecasts can help public health officials design timely public health interventions to mitigate the effects of disease outbreaks in affected populations. In France, existing clinic-based disease surveillance systems produce gastroenteritis activity information that lags real time by 1 to 3 weeks. This temporal data gap prevents public health officials from having a timely epidemiological characterization of this disease at any point in time and thus leads to the design of interventions that do not take into consideration the most recent changes in dynamics. OBJECTIVE: The goal of this study was to evaluate the feasibility of using internet search query trends and electronic health records to predict acute gastroenteritis (AG) incidence rates in near real time, at the national and regional scales, and for long-term forecasts (up to 10 weeks). METHODS: We present 2 different approaches (linear and nonlinear) that produce real-time estimates, short-term forecasts, and long-term forecasts of AG activity at 2 different spatial scales in France (national and regional). Both approaches leverage disparate data sources that include disease-related internet search activity, electronic health record data, and historical disease activity. RESULTS: Our results suggest that all data sources contribute to improving gastroenteritis surveillance for long-term forecasts with the prominent predictive power of historical data owing to the strong seasonal dynamics of this disease. CONCLUSIONS: The methods we developed could help reduce the impact of the AG peak by making it possible to anticipate increased activity by up to 10 weeks.

The Role of Heterogenous Real-world Data for Dengue Surveillance in Martinique: Observational Retrospective Study.

BACKGROUND: Traditionally, dengue prevention and control rely on vector control programs and reporting of symptomatic cases to a central health agency. However, case reporting is often delayed, and the true burden of dengue disease is often underestimated. Moreover, some countries do not have routine control measures for vector control. Therefore, researchers are constantly assessing novel data sources to improve traditional surveillance systems. These studies are mostly carried out in big territories and rarely in smaller endemic regions, such as Martinique and the Lesser Antilles. OBJECTIVE: The aim of this study was to determine whether heterogeneous real-world data sources could help reduce reporting delays and improve dengue monitoring in Martinique island, a small endemic region. METHODS: Heterogenous data sources (hospitalization data, entomological data, and Google Trends) and dengue surveillance reports for the last 14 years (January 2007 to February 2021) were analyzed to identify associations with dengue outbreaks and their time lags. RESULTS: The dengue hospitalization rate was the variable most strongly correlated with the increase in dengue positivity rate by real-time reverse transcription polymerase chain reaction (Pearson correlation coefficient=0.70) with a time lag of -3 weeks. Weekly entomological interventions were also correlated with the increase in dengue positivity rate by real-time reverse transcription polymerase chain reaction (Pearson correlation coefficient=0.59) with a time lag of -2 weeks. The most correlated query from Google Trends was the "Dengue" topic restricted to the Martinique region (Pearson correlation coefficient=0.637) with a time lag of -3 weeks. CONCLUSIONS: Real-word data are valuable data sources for dengue surveillance in smaller territories. Many of these sources precede the increase in dengue cases by several weeks, and therefore can help to improve the ability of traditional surveillance systems to provide an early response in dengue outbreaks. All these sources should be better integrated to improve the early response to dengue outbreaks and vector-borne diseases in smaller endemic territories.

Linking Biomedical Data Warehouse Records With the National Mortality Database in France: Large-scale Matching Algorithm.

BACKGROUND: Often missing from or uncertain in a biomedical data warehouse (BDW), vital status after discharge is central to the value of a BDW in medical research. The French National Mortality Database (FNMD) offers open-source nominative records of every death. Matching large-scale BDWs records with the FNMD combines multiple challenges: absence of unique common identifiers between the 2 databases, names changing over life, clerical errors, and the exponential growth of the number of comparisons to compute. OBJECTIVE: We aimed to develop a new algorithm for matching BDW records to the FNMD and evaluated its performance. METHODS: We developed a deterministic algorithm based on advanced data cleaning and knowledge of the naming system and the Damerau-Levenshtein distance (DLD). The algorithm's performance was independently assessed using BDW data of 3 university hospitals: Lille, Nantes, and Rennes. Specificity was evaluated with living patients on January 1, 2016 (ie, patients with at least 1 hospital encounter before and after this date). Sensitivity was evaluated with patients recorded as deceased between January 1, 2001, and December 31, 2020. The DLD-based algorithm was compared to a direct matching algorithm with minimal data cleaning as a reference. RESULTS: All centers combined, sensitivity was 11% higher for the DLD-based algorithm (93.3%, 95% CI 92.8-93.9) than for the direct algorithm (82.7%, 95% CI 81.8-83.6; P<.001). Sensitivity was superior for men at 2 centers (Nantes: 87%, 95% CI 85.1-89 vs 83.6%, 95% CI 81.4-85.8; P=.006; Rennes: 98.6%, 95% CI 98.1-99.2 vs 96%, 95% CI 94.9-97.1; P<.001) and for patients born in France at all centers (Nantes: 85.8%, 95% CI 84.3-87.3 vs 74.9%, 95% CI 72.8-77.0; P<.001). The DLD-based algorithm revealed significant differences in sensitivity among centers (Nantes, 85.3% vs Lille and Rennes, 97.3%, P<.001). Specificity was >98% in all subgroups. Our algorithm matched tens of millions of death records from BDWs, with parallel computing capabilities and low RAM requirements. We used the Inseehop open-source R script for this measurement. CONCLUSIONS: Overall, sensitivity/recall was 11% higher using the DLD-based algorithm than that using the direct algorithm. This shows the importance of advanced data cleaning and knowledge of a naming system through DLD use. Statistically significant differences in sensitivity between groups could be found and must be considered when performing an analysis to avoid differential biases. Our algorithm, originally conceived for linking a BDW with the FNMD, can be used to match any large-scale databases. While matching operations using names are considered sensitive computational operations, the Inseehop package released here is easy to run on premises, thereby facilitating compliance with cybersecurity local framework. The use of an advanced deterministic matching algorithm such as the DLD-based algorithm is an insightful example of combining open-source external data to improve the usage value of BDWs.

Standardized Description of the Feature Extraction Process to Transform Raw Data Into Meaningful Information for Enhancing Data Reuse: Consensus Study.

BACKGROUND: Despite the many opportunities data reuse offers, its implementation presents many difficulties, and raw data cannot be reused directly. Information is not always directly available in the source database and needs to be computed afterwards with raw data for defining an algorithm. OBJECTIVE: The main purpose of this article is to present a standardized description of the steps and transformations required during the feature extraction process when conducting retrospective observational studies. A secondary objective is to identify how the features could be stored in the schema of a data warehouse. METHODS: This study involved the following 3 main steps: (1) the collection of relevant study cases related to feature extraction and based on the automatic and secondary use of data; (2) the standardized description of raw data, steps, and transformations, which were common to the study cases; and (3) the identification of an appropriate table to store the features in the Observation Medical Outcomes Partnership (OMOP) common data model (CDM). RESULTS: We interviewed 10 researchers from 3 French university hospitals and a national institution, who were involved in 8 retrospective and observational studies. Based on these studies, 2 states (track and feature) and 2 transformations (track definition and track aggregation) emerged. "Track" is a time-dependent signal or period of interest, defined by a statistical unit, a value, and 2 milestones (a start event and an end event). "Feature" is time-independent high-level information with dimensionality identical to the statistical unit of the study, defined by a label and a value. The time dimension has become implicit in the value or name of the variable. We propose the 2 tables "TRACK" and "FEATURE" to store variables obtained in feature extraction and extend the OMOP CDM. CONCLUSIONS: We propose a standardized description of the feature extraction process. The process combined the 2 steps of track definition and track aggregation. By dividing the feature extraction into these 2 steps, difficulty was managed during track definition. The standardization of tracks requires great expertise with regard to the data, but allows the application of an infinite number of complex transformations. On the contrary, track aggregation is a very simple operation with a finite number of possibilities. A complete description of these steps could enhance the reproducibility of retrospective studies.

Drug-Drug Interactions with Oral Anticoagulants as Potentially Inappropriate Medications: Prevalence and Outcomes in Elderly Patients in Primary Care and Hospital Settings.

Direct oral anticoagulants and vitamin K antagonists are considered as potentially inappropriate medications (PIM) in several situations according to Beers Criteria. Drug-drug interactions (DDI) occurring specifically with these oral anticoagulants considered PIM (PIM-DDI) is an issue since it could enhance their inappropriate character and lead to adverse drug events, such as bleeding events. The aim of this study was (1) to describe the prevalence of oral anticoagulants as PIM, DDI and PIM-DDI in elderly patients in primary care and during hospitalization and (2) to evaluate their potential impact on the clinical outcomes by predicting hospitalization for bleeding events using machine learning methods. This retrospective study based on the linkage between a primary care database and a hospital data warehouse allowed us to display the oral anticoagulant treatment pathway. The prevalence of PIM was similar between primary care and hospital setting (22.9% and 20.9%), whereas the prevalence of DDI and PIM-DDI were slightly higher during hospitalization (47.2% vs. 58.9% and 19.5% vs. 23.5%). Concerning mechanisms, combined with CYP3A4-P-gp interactions as PIM-DDI, were among the most prevalent in patients with bleeding events. Although PIM, DDI and PIM-DDI did not appeared as major predictors of bleeding events, they should be considered since they are the only factors that can be optimized by pharmacist and clinicians.

How to Optimize Connection Between PACS and Clinical Data Warehouse: A Web Service Approach Based on Full Metadata Integration.

Clinical image data analysis is an active area of research. Integrating such data in a Clinical Data Warehouse (CDW) implies to unlock the PACS and RIS and to address interoperability and semantics issues. Based on specific functional and technical requirements, our goal was to propose a web service (I4DW) that allows users to query and access pixel data from a CDW by fully integrating and indexing imaging metadata. Here, we present the technical implementation of this workflow as well as the evaluation we carried out using a prostate cancer cohort use case. The query mechanism relies on a Dicom metadata hierarchy dynamically generated during the ETL Process. We evaluated the Dicom data transfer performance of I4DW, and found mean retrieval times of 5.94 seconds and 0.9 seconds to retrieve a complete DICOM series from the PACS and all metadata of a series. We could retrieve all patients and imaging tests of the prostate cancer cohort with a precision of 0.95 and a recall of 1. By leveraging the CMOVE method, our approach based on the Dicom protocol is scalable and domain-neutral. Future improvement will focus on performance optimization and de identification.

"Book Music" Representation for Temporal Data, as a Part of the Feature Extraction Process: A Novel Approach to Improve the Handling of Time-Dependent Data in Secondary Use of Healthcare Structured Data.

Book music is extensively used in street organs. It consists of thick cardboard, containing perforated holes specifying the musical notes. We propose to represent clinical time-dependent data in a tabular form inspired from this principle. The sheet represents a statistical individual, each row represents a binary time-dependent variable, and each hole denotes the "true" value. Data from electronic health records or nationwide medical-administrative databases can then be represented: demographics, patient flow, drugs, laboratory results, diagnoses, and procedures. This data representation is suitable for survival analysis (e.g., Cox model with repeated outcomes and changing covariates) and different types of temporal association rules. Quantitative continuous variables can be discretized, as in clinical studies. The "book music" approach could become an intermediary step in feature extraction from structured data. It would enable to better account for time in analyses, notably for historical cohort analyses based on healthcare data reuse.

The Challenges of Implementing Comprehensive Clinical Data Warehouses in Hospitals.

Digital health, e-health, telemedicine-this abundance of terms illustrates the scientific and technical revolution at work, made possible by high-speed processing of health data, artificial intelligence (AI), and the profound upheavals currently taking place and yet to come in health systems [...].

Phenotyping of Heart Failure with Preserved Ejection Faction Using Health Electronic Records and Echocardiography.

Patients suffering from heart failure (HF) symptoms and a normal left ventricular ejection fraction (LVEF 50%) present very different clinical phenotypes that could influence their survival. This study aims to identify phenotypes of this type of HF by using the medical information database from Rennes University Hospital Center. We present a preliminary work, where we explore the use of clinical variables from health electronic records (HER) in addition to echocardiography to identify several phenotypes of patients suffering from heart failure with preserved ejection fraction. The proposed methodology identifies 4 clusters with various characteristics (both clinical and echocardiographic) that are linked to survival (death, surgery, hospitalization). In the future, this work could be deployed as a tool for the physician to assess risks and contribute to support better care for patients.

Indexing Imaging Reports for Data Sharing: A Study of Mapping Using RadLex Playbook and LOINC.

New use cases and the need for quality control and imaging data sharing in health studies require the capacity to align them to reference terminologies. We are interested in mapping the local terminology used at our center to describe imaging procedures to reference terminologies for imaging procedures (RadLex Playbook and LOINC/RSNA Radiology Playbook). We performed a manual mapping of the 200 most frequent imaging report titles at our center (i.e. 73.2% of all imaging exams). The mapping method was based only on information explicitly stated in the titles. The results showed 57.5% and 68.8% of exact mapping to the RadLex and LOINC/RSNA Radiology Playbooks, respectively. We identified the reasons for the mapping failure and analyzed the issues encountered.

Out-of-Hospital Cardiac Arrest Detection by Machine Learning Based on the Phonetic Characteristics of the Caller's Voice.

INTRODUCTION: Out-of-hospital cardiac arrest (OHCA) is a major public health issue. The prognosis is closely related to the time from collapse to return of spontaneous circulation. Resuscitation efforts are frequently initiated at the request of emergency call center professionals who are specifically trained to identify critical conditions over the phone. However, 25% of OHCAs are not recognized during the first call. Therefore, it would be interesting to develop automated computer systems to recognize OHCA on the phone. The aim of this study was to build and evaluate machine learning models for OHCA recognition based on the phonetic characteristics of the caller's voice. METHODS: All patients for whom a call was done to the emergency call center of Rennes, France, between 01/01/2017 and 01/01/2019 were eligible. The predicted variable was OHCA presence. Predicting variables were collected by computer-automatized phonetic analysis of the call. They were based on the following voice parameters: fundamental frequency, formants, intensity, jitter, shimmer, harmonic to noise ratio, number of voice breaks, and number of periods. Three models were generated using binary logistic regression, random forest, and neural network. The area under the curve (AUC) was the primary outcome used to evaluate each model performance. RESULTS: 820 patients were included in the study. The best model to predict OHCA was random forest (AUC=74.9, 95% CI=67.4-82.4). CONCLUSION: Machine learning models based on the acoustic characteristics of the caller's voice can recognize OHCA. The integration of the acoustic parameters identified in this study will help to design decision-making support systems to improve OHCA detection over the phone.

Data-driven methods for dengue prediction and surveillance using real-world and Big Data: A systematic review.

BACKGROUND: Traditionally, dengue surveillance is based on case reporting to a central health agency. However, the delay between a case and its notification can limit the system responsiveness. Machine learning methods have been developed to reduce the reporting delays and to predict outbreaks, based on non-traditional and non-clinical data sources. The aim of this systematic review was to identify studies that used real-world data, Big Data and/or machine learning methods to monitor and predict dengue-related outcomes. METHODOLOGY/PRINCIPAL FINDINGS: We performed a search in PubMed, Scopus, Web of Science and grey literature between January 1, 2000 and August 31, 2020. The review (ID: CRD42020172472) focused on data-driven studies. Reviews, randomized control trials and descriptive studies were not included. Among the 119 studies included, 67% were published between 2016 and 2020, and 39% used at least one novel data stream. The aim of the included studies was to predict a dengue-related outcome (55%), assess the validity of data sources for dengue surveillance (23%), or both (22%). Most studies (60%) used a machine learning approach. Studies on dengue prediction compared different prediction models, or identified significant predictors among several covariates in a model. The most significant predictors were rainfall (43%), temperature (41%), and humidity (25%). The two models with the highest performances were Neural Networks and Decision Trees (52%), followed by Support Vector Machine (17%). We cannot rule out a selection bias in our study because of our two main limitations: we did not include preprints and could not obtain the opinion of other international experts. CONCLUSIONS/SIGNIFICANCE: Combining real-world data and Big Data with machine learning methods is a promising approach to improve dengue prediction and monitoring. Future studies should focus on how to better integrate all available data sources and methods to improve the response and dengue management by stakeholders.

Leveraging National Claims and Hospital Big Data: Cohort Study on a Statin-Drug Interaction Use Case.

BACKGROUND: Linking different sources of medical data is a promising approach to analyze care trajectories. The aim of the INSHARE (Integrating and Sharing Health Big Data for Research) project was to provide the blueprint for a technological platform that facilitates integration, sharing, and reuse of data from 2 sources: the clinical data warehouse (CDW) of the Rennes academic hospital, called eHOP (entrepôt Hôpital), and a data set extracted from the French national claim data warehouse (Système National des Données de Santé [SNDS]). OBJECTIVE: This study aims to demonstrate how the INSHARE platform can support big data analytic tasks in the health field using a pharmacovigilance use case based on statin consumption and statin-drug interactions. METHODS: A Spark distributed cluster-computing framework was used for the record linkage procedure and all analyses. A semideterministic record linkage method based on the common variables between the chosen data sources was developed to identify all patients discharged after at least one hospital stay at the Rennes academic hospital between 2015 and 2017. The use-case study focused on a cohort of patients treated with statins prescribed by their general practitioner or during their hospital stay. RESULTS: The whole process (record linkage procedure and use-case analyses) required 88 minutes. Of the 161,532 and 164,316 patients from the SNDS and eHOP CDW data sets, respectively, 159,495 patients were successfully linked (98.74% and 97.07% of patients from SNDS and eHOP CDW, respectively). Of the 16,806 patients with at least one statin delivery, 8293 patients started the consumption before and continued during the hospital stay, 6382 patients stopped statin consumption at hospital admission, and 2131 patients initiated statins in hospital. Statin-drug interactions occurred more frequently during hospitalization than in the community (3800/10,424, 36.45% and 3253/14,675, 22.17%, respectively; P<.001). Only 121 patients had the most severe level of statin-drug interaction. Hospital stay burden (length of stay and in-hospital mortality) was more severe in patients with statin-drug interactions during hospitalization. CONCLUSIONS: This study demonstrates the added value of combining and reusing clinical and claim data to provide large-scale measures of drug-drug interaction prevalence and care pathways outside hospitals. It builds a path to move the current health care system toward a Learning Health System using knowledge generated from research on real-world health data.

Pilot Study of an e-Cohort to Monitor Adverse Event for Patient with Hip Prostheses from Clinical Data Warehouse.

Hip arthroplasty represents a large proportion of orthopaedic activity, constantly increasing. Automating monitoring from clinical data warehouses is an opportunity to dynamically monitor devices and patient outcomes allowing improve clinical practices. Our objective was to assess quantitative and qualitative concordance between claim data and device supply data in order to create an e-cohort of patients undergoing a hip replacement. We performed a single-centre cohort pilot study, from one clinical data warehouse of a French University Hospital, from January 1, 2010 to December 31, 2019. We included all adult patients undergoing a hip arthroplasty, and with at least one hip medical device provided. Patients younger than 18 years or opposed to the reuse of their data were excluded from the analysis. Our primary outcome was the percentage of hospital stays with both hip arthroplasty and hip device provided. The patient and stay characteristics assessed in this study were: age, sex, length of stay, surgery procedure (replacement, repositioning, change, or reconstruction), medical motif for surgery (osteoarthritis, fracture, cancer, infection, or other) and device provided (head, stem, shell, or other). We found 3,380 stays and 2,934 patients, 96.4% of them had both a hip surgery procedure and a hip device provided. These data from different sources are close enough to be integrated in a common clinical data warehouse.

A framework for validating AI in precision medicine: considerations from the European ITFoC consortium.

BACKGROUND: Artificial intelligence (AI) has the potential to transform our healthcare systems significantly. New AI technologies based on machine learning approaches should play a key role in clinical decision-making in the future. However, their implementation in health care settings remains limited, mostly due to a lack of robust validation procedures. There is a need to develop reliable assessment frameworks for the clinical validation of AI. We present here an approach for assessing AI for predicting treatment response in triple-negative breast cancer (TNBC), using real-world data and molecular -omics data from clinical data warehouses and biobanks. METHODS: The European "ITFoC (Information Technology for the Future Of Cancer)" consortium designed a framework for the clinical validation of AI technologies for predicting treatment response in oncology. RESULTS: This framework is based on seven key steps specifying: (1) the intended use of AI, (2) the target population, (3) the timing of AI evaluation, (4) the datasets used for evaluation, (5) the procedures used for ensuring data safety (including data quality, privacy and security), (6) the metrics used for measuring performance, and (7) the procedures used to ensure that the AI is explainable. This framework forms the basis of a validation platform that we are building for the "ITFoC Challenge". This community-wide competition will make it possible to assess and compare AI algorithms for predicting the response to TNBC treatments with external real-world datasets. CONCLUSIONS: The predictive performance and safety of AI technologies must be assessed in a robust, unbiased and transparent manner before their implementation in healthcare settings. We believe that the consideration of the ITFoC consortium will contribute to the safe transfer and implementation of AI in clinical settings, in the context of precision oncology and personalized care.