AI-based denoising of head impact kinematics measurements with convolutional neural network for traumatic brain injury prediction.

OBJECTIVE: Wearable devices are developed to measure head impact kinematics but are intrinsically noisy because of the imperfect interface with human bodies. This study aimed to improve the head impact kinematics measurements obtained from instrumented mouthguards using deep learning to enhance traumatic brain injury (TBI) risk monitoring. METHODS: We developed one-dimensional convolutional neural network (1D-CNN) models to denoise mouthguard kinematics measurements for tri-axial linear acceleration and tri-axial angular velocity from 163 laboratory dummy head impacts. The performance of the denoising models was evaluated on three levels: kinematics, brain injury criteria, and tissue-level strain and strain rate. Additionally, we performed a blind test on an on-field dataset of 118 college football impacts and a test on 413 post-mortem human subject (PMHS) impacts. RESULTS: On the dummy head impacts, the denoised kinematics showed better correlation with reference kinematics, with relative reductions of 36% for pointwise root mean squared error and 56% for peak absolute error. Absolute errors in six brain injury criteria were reduced by a mean of 82%. For maximum principal strain and maximum principal strain rate, the mean error reduction was 35% and 69%, respectively. On the PMHS impacts, similar denoising effects were observed and the peak kinematics after denoising were more accurate (relative error reduction for 10% noisiest impacts was 75.6%). CONCLUSION: The 1D-CNN denoising models effectively reduced errors in mouthguard-derived kinematics measurements on dummy and PMHS impacts. SIGNIFICANCE: This study provides a novel approach for denoising head kinematics measurements in dummy and PMHS impacts, which can be further validated on more real-human kinematics data before real-world applications.

Brain Deformation Estimation With Transfer Learning for Head Impact Datasets Across Impact Types.

OBJECTIVE: The machine-learning head model (MLHM) to accelerate the calculation of brain strain and strain rate, which are the predictors for traumatic brain injury (TBI), but the model accuracy was found to decrease sharply when the training/test datasets were from different head impacts types (i.e., car crash, college football), which limits the applicability of MLHMs to different types of head impacts and sports. Particularly, small sizes of target dataset for specific impact types with tens of impacts may not be enough to train an accurate impact-type-specific MLHM. METHODS: To overcome this, we propose data fusion and transfer learning to develop a series of MLHMs to predict the maximum principal strain (MPS) and maximum principal strain rate (MPSR). RESULTS: The strategies were tested on American football (338), mixed martial arts (457), reconstructed car crash (48) and reconstructed American football (36) and we found that the MLHMs developed with transfer learning are significantly more accurate in estimating MPS and MPSR than other models, with a mean absolute error (MAE) smaller than 0.03 in predicting MPS and smaller than [Formula: see text] in predicting MPSR on all target impact datasets. High performance in concussion detection was observed based on the MPS and MPSR estimated by the transfer-learning-based models. CONCLUSION: The MLHMs can be applied to various head impact types for rapidly and accurately calculating brain strain and strain rate. SIGNIFICANCE: This study enables developing MLHMs for the head impact type with limited availability of data, and will accelerate the applications of MLHMs.

Toward more accurate and generalizable brain deformation estimators for traumatic brain injury detection with unsupervised domain adaptation.

Machine learning head models (MLHMs) are developed to estimate brain deformation for early detection of traumatic brain injury (TBI). However, the overfitting to simulated impacts and the lack of generalizability caused by distributional shift of different head impact datasets hinders the broad clinical applications of current MLHMs. We propose brain deformation estimators that integrates unsupervised domain adaptation with a deep neural network to predict whole-brain maximum principal strain (MPS) and MPS rate (MPSR). With 12,780 simulated head impacts, we performed unsupervised domain adaptation on on-field head impacts from 302 college football (CF) impacts and 457 mixed martial arts (MMA) impacts using domain regularized component analysis (DRCA) and cycle-GAN-based methods. The new model improved the MPS/MPSR estimation accuracy, with the DRCA method significantly outperforming other domain adaptation methods in prediction accuracy (p<0.001): MPS RMSE: 0.027 (CF) and 0.037 (MMA); MPSR RMSE: 7.159 (CF) and 13.022 (MMA). On another two hold-out testsets with 195 college football impacts and 260 boxing impacts, the DRCA model significantly outperformed the baseline model without domain adaptation in MPS and MPSR estimation accuracy (p<0.001). The DRCA domain adaptation reduces the MPS/MPSR estimation error to be well below TBI thresholds, enabling accurate brain deformation estimation to detect TBI in future clinical applications.

Padded Helmet Shell Covers in American Football: A Comprehensive Laboratory Evaluation with Preliminary On-Field Findings.

Protective headgear effects measured in the laboratory may not always translate to the field. In this study, we evaluated the impact attenuation capabilities of a commercially available padded helmet shell cover in the laboratory and on the field. In the laboratory, we evaluated the padded helmet shell cover's efficacy in attenuating impact magnitude across six impact locations and three impact velocities when equipped to three different helmet models. In a preliminary on-field investigation, we used instrumented mouthguards to monitor head impact magnitude in collegiate linebackers during practice sessions while not wearing the padded helmet shell covers (i.e., bare helmets) for one season and whilst wearing the padded helmet shell covers for another season. The addition of the padded helmet shell cover was effective in attenuating the magnitude of angular head accelerations and two brain injury risk metrics (DAMAGE, HARM) across most laboratory impact conditions, but did not significantly attenuate linear head accelerations for all helmets. Overall, HARM values were reduced in laboratory impact tests by an average of 25% at 3.5 m/s (range: 9.7 to 39.6%), 18% at 5.5 m/s (range: - 5.5 to 40.5%), and 10% at 7.4 m/s (range: - 6.0 to 31.0%). However, on the field, no significant differences in any measure of head impact magnitude were observed between the bare helmet impacts and padded helmet impacts. Further laboratory tests were conducted to evaluate the ability of the padded helmet shell cover to maintain its performance after exposure to repeated, successive impacts and across a range of temperatures. This research provides a detailed assessment of padded helmet shell covers and supports the continuation of in vivo helmet research to validate laboratory testing results.

Machine-learning-based head impact subtyping based on the spectral densities of the measurable head kinematics.

BACKGROUND: Traumatic brain injury can be caused by head impacts, but many brain injury risk estimation models are not equally accurate across the variety of impacts that patients may undergo, and the characteristics of different types of impacts are not well studied. We investigated the spectral characteristics of different head impact types with kinematics classification. METHODS: Data were analyzed from 3262 head impacts from lab reconstruction, American football, mixed martial arts, and publicly available car crash data. A random forest classifier with spectral densities of linear acceleration and angular velocity was built to classify head impact types (e.g., football, car crash, mixed martial arts). To test the classifier robustness, another 271 lab-reconstructed impacts were obtained from 5 other instrumented mouthguards. Finally, with the classifier, type-specific, nearest-neighbor regression models were built for brain strain. RESULTS: The classifier reached a median accuracy of 96% over 1000 random partitions of training and test sets. The most important features in the classification included both low- and high-frequency features, both linear acceleration features and angular velocity features. Different head impact types had different distributions of spectral densities in low- and high-frequency ranges (e.g., the spectral densities of mixed martial arts impacts were higher in the high-frequency range than in the low-frequency range). The type-specific regression showed a generally higher R2 value than baseline models without classification. CONCLUSION: The machine-learning-based classifier enables a better understanding of the impact kinematics spectral density in different sports, and it can be applied to evaluate the quality of impact-simulation systems and on-field data augmentation.

An Anomaly Detection Algorithm Based on Ensemble Learning for 5G Environment.

With the advent of the digital information age, new data services such as virtual reality, industrial Internet, and cloud computing have proliferated in recent years. As a result, it increases operator demand for 5G bearer networks by providing features such as high transmission capacity, ultra-long transmission distance, network slicing, and intelligent management and control. Software-defined networking, as a new network architecture, intends to increase network flexibility and agility and can better satisfy the demands of 5G networks for network slicing. Nevertheless, software-defined networking still faces the challenge of network intrusion. We propose an abnormal traffic detection method based on the stacking method and self-attention mechanism, which makes up for the shortcoming of the inability to track long-term dependencies between data samples in ensemble learning. Our method utilizes a self-attention mechanism and a convolutional network to automatically learn long-term associations between traffic samples and provide them to downstream tasks in sample embedding. In addition, we design a novel stacking ensemble method, which computes the sample embedding and the predicted values of the heterogeneous base learner through the fusion module to obtain the final outlier results. This paper conducts experiments on abnormal traffic datasets in the software-defined network environment, calculates precision, recall and F1-score, and compares and analyzes them with other algorithms. The experimental results show that the method designed in this paper achieves 0.9972, 0.9996, and 0.9984 in multiple indicators of precision, recall, and F1-score, respectively, which are better than the comparison methods.

Piecewise Multivariate Linearity Between Kinematic Features and Cumulative Strain Damage Measure (CSDM) Across Different Types of Head Impacts.

In a previous study, we found that the relationship between brain strain and kinematic features cannot be described by a generalized linear model across different types of head impacts. In this study, we investigate if such a linear relationship exists when partitioning head impacts using a data-driven approach. We applied the K-means clustering method to partition 3161 impacts from various sources including simulation, college football, mixed martial arts, and car crashes. We found piecewise multivariate linearity between the cumulative strain damage (CSDM; assessed at the threshold of 0.15) and head kinematic features. Compared with the linear regression models without partition and the partition according to the types of head impacts, K-means-based data-driven partition showed significantly higher CSDM regression accuracy, which suggested the presence of piecewise multivariate linearity across types of head impacts. Additionally, we compared the piecewise linearity with the partitions based on individual features used in clustering. We found that the partition with maximum angular acceleration magnitude at 4706 rad/s2 led to the highest piecewise linearity. This study may contribute to an improved method for the rapid prediction of CSDM in the future.

Reliably Filter Drug-Induced Liver Injury Literature With Natural Language Processing and Conformal Prediction.

Drug-induced liver injury describes the adverse effects of drugs that damage the liver. Life-threatening results were also reported in severe cases. Therefore, liver toxicity is an important assessment for new drug candidates. These reports are documented in research papers that contain preliminary in vitro and in vivo experiments. Conventionally, data extraction from publications relies on resource-demanding manual labeling, which restricts the efficiency of the information extraction. The development of natural language processing techniques enables the automatic processing of biomedical texts. Herein, based on around 28,000 papers (titles and abstracts) provided by the Critical Assessment of Massive Data Analysis challenge, this study benchmarked model performances on filtering liver-damage-related literature. Among five text embedding techniques, the model using term frequency-inverse document frequency (TF-IDF) and logistic regression outperformed others with an accuracy of 0.957 on the validation set. Furthermore, an ensemble model with similar overall performances was developed with a logistic regression model on the predicted probability given by separate models with different vectorization techniques. The ensemble model achieved a high accuracy of 0.954 and an F1 score of 0.955 in the hold-out validation data in the challenge. Moreover, important words in positive/negative predictions were identified via model interpretation. The prediction reliability was quantified with conformal prediction, which provides users with a control over the prediction uncertainty. Overall, the ensemble model and TF-IDF model reached satisfactory classification results, which can be used by researchers to rapidly filter literature that describes events related to liver injury induced by medications.

Finding the Spatial Co-Variation of Brain Deformation With Principal Component Analysis.

OBJECTIVE: Strain and strain rate are effective traumatic brain injury metrics. In finite element (FE) head model, thousands of elements were used to represent the spatial distribution of these metrics. Owing that these metrics are resulted from brain inertia, their spatial distribution can be represented in more concise pattern. Since head kinematic features and brain deformation vary largely across head impact types (Zhan et al., 2021), we applied principal component analysis (PCA) to find the spatial co-variation of injury metrics (maximum principal strain (MPS), MPS rate (MPSR) and MPS × MPSR) in four impact types: simulation, football, mixed martial arts and car crashes, and used the PCA to find patterns in these metrics and improve the machine learning head model (MLHM). METHODS: We applied PCA to decompose the injury metrics for all impacts in each impact type, and investigate the spatial co-variation using the first principal component (PC1). Furthermore, we developed a MLHM to predict PC1 and then inverse-transform to predict for all brain elements. The accuracy, the model complexity and the size of training dataset of PCA-MLHM are compared with previous MLHM (Zhan et al., 2021). RESULTS: PC1 explained variance on the datasets. Based on PC1 coefficients, the corpus callosum and midbrain exhibit high variance on all datasets. Finally, the PCA-MLHM reduced model parameters by 74% with a similar MPS estimation accuracy. CONCLUSION: The brain injury metric in a dataset can be decomposed into mean components and PC1 with high explained variance. SIGNIFICANCE: The spatial co-variation analysis enables better interpretation of the patterns in brain injury metrics. It also improves the efficiency of MLHM.

Toward a Comprehensive Delineation of White Matter Tract-Related Deformation.

Finite element (FE) models of the human head are valuable instruments to explore the mechanobiological pathway from external loading, localized brain response, and resultant injury risks. The injury predictability of these models depends on the use of effective criteria as injury predictors. The FE-derived normal deformation along white matter (WM) fiber tracts (i.e., tract-oriented strain) recently has been suggested as an appropriate predictor for axonal injury. However, the tract-oriented strain only represents a partial depiction of the WM fiber tract deformation. A comprehensive delineation of tract-related deformation may improve the injury predictability of the FE head model by delivering new tract-related criteria as injury predictors. Thus, the present study performed a theoretical strain analysis to comprehensively characterize the WM fiber tract deformation by relating the strain tensor of the WM element to its embedded fiber tract. Three new tract-related strains with exact analytical solutions were proposed, measuring the normal deformation perpendicular to the fiber tracts (i.e., tract-perpendicular strain), and shear deformation along and perpendicular to the fiber tracts (i.e., axial-shear strain and lateral-shear strain, respectively). The injury predictability of these three newly proposed strain peaks along with the previously used tract-oriented strain peak and maximum principal strain (MPS) were evaluated by simulating 151 impacts with known outcome (concussion or non-concussion). The results preliminarily showed that four tract-related strain peaks exhibited superior performance than MPS in discriminating concussion and non-concussion cases. This study presents a comprehensive quantification of WM tract-related deformation and advocates the use of orientation-dependent strains as criteria for injury prediction, which may ultimately contribute to an advanced mechanobiological understanding and enhanced computational predictability of brain injury.

Identifying Factors Associated with Head Impact Kinematics and Brain Strain in High School American Football via Instrumented Mouthguards.

Repeated head impact exposure and concussions are common in American football. Identifying the factors associated with high magnitude impacts aids in informing sport policy changes, improvements to protective equipment, and better understanding of the brain's response to mechanical loading. Recently, the Stanford Instrumented Mouthguard (MiG2.0) has seen several improvements in its accuracy in measuring head kinematics and its ability to correctly differentiate between true head impact events and false positives. Using this device, the present study sought to identify factors (e.g., player position, helmet model, direction of head acceleration, etc.) that are associated with head impact kinematics and brain strain in high school American football athletes. 116 athletes were monitored over a total of 888 athlete exposures. 602 total impacts were captured and verified by the MiG2.0's validated impact detection algorithm. Peak values of linear acceleration, angular velocity, and angular acceleration were obtained from the mouthguard kinematics. The kinematics were also entered into a previously developed finite element model of the human brain to compute the 95th percentile maximum principal strain. Overall, impacts were (mean ± SD) 34.0 ± 24.3 g for peak linear acceleration, 22.2 ± 15.4 rad/s for peak angular velocity, 2979.4 ± 3030.4 rad/s2 for peak angular acceleration, and 0.262 ± 0.241 for 95th percentile maximum principal strain. Statistical analyses revealed that impacts resulting in Forward head accelerations had higher magnitudes of peak kinematics and brain strain than Lateral or Rearward impacts and that athletes in skill positions sustained impacts of greater magnitude than athletes in line positions. 95th percentile maximum principal strain was significantly lower in the observed cohort of high school football athletes than previous reports of collegiate football athletes. No differences in impact magnitude were observed in athletes with or without previous concussion history, in athletes wearing different helmet models, or in junior varsity or varsity athletes. This study presents novel information on head acceleration events and their resulting brain strain in high school American football from our advanced, validated method of measuring head kinematics via instrumented mouthguard technology.

Structuring clinical text with AI: Old versus new natural language processing techniques evaluated on eight common cardiovascular diseases.

Free-text clinical notes in electronic health records are more difficult for data mining while the structured diagnostic codes can be missing or erroneous. To improve the quality of diagnostic codes, this work extracts diagnostic codes from free-text notes: five old and new word vectorization methods were used to vectorize Stanford progress notes and predict eight ICD-10 codes of common cardiovascular diseases with logistic regression. The models showed good performance, with TF-IDF as the best vectorization model showing the highest AUROC (0.9499-0.9915) and AUPRC (0.2956-0.8072). The models also showed transferability when tested on MIMIC-III data with AUROC from 0.7952 to 0.9790 and AUPRC from 0.2353 to 0.8084. Model interpretability was shown by the important words with clinical meanings matching each disease. This study shows the feasibility of accurately extracting structured diagnostic codes, imputing missing codes, and correcting erroneous codes from free-text clinical notes for information retrieval and downstream machine-learning applications.

Predictive Factors of Kinematics in Traumatic Brain Injury from Head Impacts Based on Statistical Interpretation.

Brain tissue deformation resulting from head impacts is primarily caused by rotation and can lead to traumatic brain injury. To quantify brain injury risk based on measurements of kinematics on the head, finite element (FE) models and various brain injury criteria based on different factors of these kinematics have been developed, but the contribution of different kinematic factors has not been comprehensively analyzed across different types of head impacts in a data-driven manner. To better design brain injury criteria, the predictive power of rotational kinematics factors, which are different in (1) the derivative order (angular velocity, angular acceleration, angular jerk), (2) the direction and (3) the power (e.g., square-rooted, squared, cubic) of the angular velocity, were analyzed based on different datasets including laboratory impacts, American football, mixed martial arts (MMA), NHTSA automobile crashworthiness tests and NASCAR crash events. Ordinary least squares regressions were built from kinematics factors to the 95% maximum principal strain (MPS95), and we compared zero-order correlation coefficients, structure coefficients, commonality analysis, and dominance analysis. The angular acceleration, the magnitude and the first power factors showed the highest predictive power for the majority of impacts including laboratory impacts, American football impacts, with few exceptions (angular velocity for MMA and NASCAR impacts). The predictive power of rotational kinematics about three directions (x: posterior-to-anterior, y: left-to-right, z: superior-to-inferior) of kinematics varied with different sports and types of head impacts.

Time Window of Head Impact Kinematics Measurement for Calculation of Brain Strain and Strain Rate in American Football.

Wearable devices have been shown to effectively measure the head's movement during impacts in sports like American football. When a head impact occurs, the device is triggered to collect and save the kinematic measurements during a predefined time window. Then, based on the collected kinematics, finite element (FE) head models can calculate brain strain and strain rate, which are used to evaluate the risk of mild traumatic brain injury. To find a time window that can provide a sufficient duration of kinematics for FE analysis, we investigated 118 on-field video-confirmed football head impacts collected by the Stanford Instrumented Mouthguard. The simulation results based on the kinematics truncated to a shorter time window were compared with the original to determine the minimum time window needed for football. Because the individual differences in brain geometry influence these calculations, we included six representative brain geometries and found that larger brains need a longer time window of kinematics for accurate calculation. Among the different sizes of brains, a pre-trigger time of 40 ms and a post-trigger time of 70 ms were found to yield calculations of brain strain and strain rate that were not significantly different from calculations using the original 200 ms time window recorded by the mouthguard. Therefore, approximately 110 ms is recommended for complete modeling of impacts for football.

The relationship between brain injury criteria and brain strain across different types of head impacts can be different.

Multiple brain injury criteria (BIC) are developed to quickly quantify brain injury risks after head impacts. These BIC originated from different head impact types (e.g. sports and car crashes) are widely used in risk evaluation. However, the accuracy of using the BIC on brain injury risk estimation across head impact types has not been evaluated. Physiologically, brain strain is often considered the key parameter of brain injury. To evaluate the BIC's risk estimation accuracy across five datasets comprising different head impact types, linear regression was used to model 95% maximum principal strain, 95% maximum principal strain at the corpus callosum and cumulative strain damage (15%) on 18 BIC. The results show significantly different relationships between BIC and brain strain across datasets, indicating the same BIC value may suggest different brain strain across head impact types. The accuracy of brain strain regression is generally decreasing if the BIC regression models are fitted on a dataset with a different type of head impact rather than on the dataset with the same type. Given this finding, this study raises concerns for applying BIC to estimate the brain injury risks for head impacts different from the head impacts on which the BIC was developed.

Rapid Estimation of Entire Brain Strain Using Deep Learning Models.

OBJECTIVE: Many recent studies suggest that brain deformation resulting from head impacts are linked to the corresponding clinical outcome, such as mild traumatic brain injury (mTBI). Even if several finite element (FE) head models have been developed and validated to calculate brain deformation based on impact kinematics, the clinical application of these FE head models is limited due to the time-consuming nature of FE simulations. This work aims to accelerate the brain deformation calculation and thus improve the potential for clinical applications. METHODS: We propose a deep learning head model with a five-layer deep neural network and feature engineering, and trained and tested the model on 2511 head impacts from a combination of head model simulations and on-field college football and mixed martial arts impacts. RESULTS: The proposed deep learning head model can calculate the maximum principal strain (Green Lagrange) for every element in the entire brain in less than 0.001 s with an average root mean squared error of 0.022 and a standard deviation of 0.001 over twenty repeats with random data partition and model initialization. CONCLUSION: Trained and tested using the dataset of 2511 head impacts, this model can be applied to various sports in the calculation of brain strain with accuracy, and its applicability can even further be extended by incorporating data from other types of head impacts. SIGNIFICANCE: In addition to the potential clinical application in real-time brain deformation monitoring, this model will help researchers estimate the brain strain from a large number of head impacts more efficiently than using FE models.

AI-based analysis of CT images for rapid triage of COVID-19 patients.

The COVID-19 pandemic overwhelms the medical resources in the stressed intensive care unit (ICU) capacity and the shortage of mechanical ventilation (MV). We performed CT-based analysis combined with electronic health records and clinical laboratory results on Cohort 1 (n = 1662 from 17 hospitals) with prognostic estimation for the rapid stratification of PCR confirmed COVID-19 patients. These models, validated on Cohort 2 (n = 700) and Cohort 3 (n = 662) constructed from nine external hospitals, achieved satisfying performance for predicting ICU, MV, and death of COVID-19 patients (AUROC 0.916, 0.919, and 0.853), even on events happened two days later after admission (AUROC 0.919, 0.943, and 0.856). Both clinical and image features showed complementary roles in prediction and provided accurate estimates to the time of progression (p < 0.001). Our findings are valuable for optimizing the use of medical resources in the COVID-19 pandemic. The models are available here: https://github.com/terryli710/COVID_19_Rapid_Triage_Risk_Predictor .

CT-based Rapid Triage of COVID-19 Patients: Risk Prediction and Progression Estimation of ICU Admission, Mechanical Ventilation, and Death of Hospitalized Patients.

The wave of COVID-19 continues to overwhelm the medical resources, especially the stressed intensive care unit (ICU) capacity and the shortage of mechanical ventilation (MV). Here we performed CT-based analysis combined with electronic health records and clinical laboratory results on Cohort 1 (n = 1662 from 17 hospitals) with prognostic estimation for the rapid stratification of PCR confirmed COVID-19 patients. These models, validated on Cohort 2 (n = 700) and Cohort 3 (n = 662) constructed from 9 external hospitals, achieved satisfying performance for predicting ICU, MV and death of COVID-19 patients (AUROC 0.916, 0.919 and 0.853), even on events happened two days later after admission (AUROC 0.919, 0.943 and 0.856). Both clinical and image features showed complementary roles in events prediction and provided accurate estimates to the time of progression (p<.001). Our findings are valuable for delivering timely treatment and optimizing the use of medical resources in the pandemic of COVID-19.

Discrimination between Alternative Herbal Medicines from Different Categories with the Electronic Nose.

As alternative herbal medicine gains soar in popularity around the world, it is necessary to apply a fast and convenient means for classifying and evaluating herbal medicines. In this work, an electronic nose system with seven classification algorithms is used to discriminate between 12 categories of herbal medicines. The results show that these herbal medicines can be successfully classified, with support vector machine (SVM) and linear discriminant analysis (LDA) outperforming other algorithms in terms of accuracy. When principal component analysis (PCA) is used to lower the number of dimensions, the time cost for classification can be reduced while the data is visualized. Afterwards, conformal predictions based on 1NN (1-Nearest Neighbor) and 3NN (3-Nearest Neighbor) (CP-1NN and CP-3NN) are introduced. CP-1NN and CP-3NN provide additional, yet significant and reliable, information by giving the confidence and credibility associated with each prediction without sacrificing of accuracy. This research provides insight into the construction of a herbal medicine flavor library and gives methods and reference for future works.