Assessing the effects of data drift on the performance of machine learning models used in clinical sepsis prediction.

BACKGROUND: Data drift can negatively impact the performance of machine learning algorithms (MLAs) that were trained on historical data. As such, MLAs should be continuously monitored and tuned to overcome the systematic changes that occur in the distribution of data. In this paper, we study the extent of data drift and provide insights about its characteristics for sepsis onset prediction. This study will help elucidate the nature of data drift for prediction of sepsis and similar diseases. This may aid with the development of more effective patient monitoring systems that can stratify risk for dynamic disease states in hospitals. METHODS: We devise a series of simulations that measure the effects of data drift in patients with sepsis, using electronic health records (EHR). We simulate multiple scenarios in which data drift may occur, namely the change in the distribution of the predictor variables (covariate shift), the change in the statistical relationship between the predictors and the target (concept shift), and the occurrence of a major healthcare event (major event) such as the COVID-19 pandemic. We measure the impact of data drift on model performances, identify the circumstances that necessitate model retraining, and compare the effects of different retraining methodologies and model architecture on the outcomes. We present the results for two different MLAs, eXtreme Gradient Boosting (XGB) and Recurrent Neural Network (RNN). RESULTS: Our results show that the properly retrained XGB models outperform the baseline models in all simulation scenarios, hence signifying the existence of data drift. In the major event scenario, the area under the receiver operating characteristic curve (AUROC) at the end of the simulation period is 0.811 for the baseline XGB model and 0.868 for the retrained XGB model. In the covariate shift scenario, the AUROC at the end of the simulation period for the baseline and retrained XGB models is 0.853 and 0.874 respectively. In the concept shift scenario and under the mixed labeling method, the retrained XGB models perform worse than the baseline model for most simulation steps. However, under the full relabeling method, the AUROC at the end of the simulation period for the baseline and retrained XGB models is 0.852 and 0.877 respectively. The results for the RNN models were mixed, suggesting that retraining based on a fixed network architecture may be inadequate for an RNN. We also present the results in the form of other performance metrics such as the ratio of observed to expected probabilities (calibration) and the normalized rate of positive predictive values (PPV) by prevalence, referred to as lift, at a sensitivity of 0.8. CONCLUSION: Our simulations reveal that retraining periods of a couple of months or using several thousand patients are likely to be adequate to monitor machine learning models that predict sepsis. This indicates that a machine learning system for sepsis prediction will probably need less infrastructure for performance monitoring and retraining compared to other applications in which data drift is more frequent and continuous. Our results also show that in the event of a concept shift, a full overhaul of the sepsis prediction model may be necessary because it indicates a discrete change in the definition of sepsis labels, and mixing the labels for the sake of incremental training may not produce the desired results.

Assessing the effects of data drift on the performance of machine learning models used in clinical sepsis prediction.

Background: Data drift can negatively impact the performance of machine learning algorithms (MLAs) that were trained on historical data. As such, MLAs should be continuously monitored and tuned to overcome the systematic changes that occur in the distribution of data. In this paper, we study the extent of data drift and provide insights about its characteristics for sepsis onset prediction. This study will help elucidate the nature of data drift for prediction of sepsis and similar diseases. This may aid with the development of more effective patient monitoring systems that can stratify risk for dynamic disease states in hospitals. Methods: We devise a series of simulations that measure the effects of data drift in patients with sepsis. We simulate multiple scenarios in which data drift may occur, namely the change in the distribution of the predictor variables (covariate shift), the change in the statistical relationship between the predictors and the target (concept shift), and the occurrence of a major healthcare event (major event) such as the COVID-19 pandemic. We measure the impact of data drift on model performances, identify the circumstances that necessitate model retraining, and compare the effects of different retraining methodologies and model architecture on the outcomes. We present the results for two different MLAs, eXtreme Gradient Boosting (XGB) and Recurrent Neural Network (RNN). Results: Our results show that the properly retrained XGB models outperform the baseline models in all simulation scenarios, hence signifying the existence of data drift. In the major event scenario, the area under the receiver operating characteristic curve (AUROC) at the end of the simulation period is 0.811 for the baseline XGB model and 0.868 for the retrained XGB model. In the covariate shift scenario, the AUROC at the end of the simulation period for the baseline and retrained XGB models is 0.853 and 0.874 respectively. In the concept shift scenario and under the mixed labeling method, the retrained XGB models perform worse than the baseline model for most simulation steps. However, under the full relabeling method, the AUROC at the end of the simulation period for the baseline and retrained XGB models is 0.852 and 0.877 respectively. The results for the RNN models were mixed, suggesting that retraining based on a fixed network architecture may be inadequate for an RNN. We also present the results in the form of other performance metrics such as the ratio of observed to expected probabilities (calibration) and the normalized rate of positive predictive values (PPV) by prevalence, referred to as lift, at a sensitivity of 0.8. Conclusion: Our simulations reveal that retraining periods of a couple of months or using several thousand patients are likely to be adequate to monitor machine learning models that predict sepsis. This indicates that a machine learning system for sepsis prediction will probably need less infrastructure for performance monitoring and retraining compared to other applications in which data drift is more frequent and continuous. Our results also show that in the event of a concept shift, a full overhaul of the sepsis prediction model may be necessary because it indicates a discrete change in the definition of sepsis labels, and mixing the labels for the sake of incremental training may not produce the desired results.

Mapping Driver Mutations to Histopathological Subtypes in Papillary Thyroid Carcinoma: Applying a Deep Convolutional Neural Network.

Papillary thyroid carcinoma (PTC) is the most common subtype of thyroid cancers and informative biomarkers are critical for risk stratification and treatment guidance. About half of PTCs harbor BRAFV600E and 10%-15% have RAS mutations. In the current study, we trained a deep learning convolutional neural network (CNN) model (Google Inception v3) on histopathology images obtained from The Cancer Genome Atlas (TCGA) to classify PTCs into BRAFV600E or RAS mutations. We aimed to answer whether CNNs can predict driver gene mutations using images as the only input. The performance of our method is comparable to that of recent publications of other cancer types using TCGA tumor slides with area under the curve (AUC) of 0.878-0.951. Our model was tested on separate tissue samples from the same cohort. On the independent testing subset, the accuracy rate using the cutoff of truth rate 0.8 was 95.2% for BRAF and RAS mutation class prediction. Moreover, we showed that the image-based classification correlates well with mRNA-derived expression pattern (Spearman correlation, rho = 0.63, p = 0.002 on validation data and rho = 0.79, p = 2 × 10-5 on final testing data). The current study demonstrates the potential of deep learning approaches for histopathologically classifying cancer based on driver mutations. This information could be of value assisting clinical decisions involving PTCs.

Serum Vitamin E Levels of Adults with Nonalcoholic Fatty Liver Disease: An Inverse Relationship with All-Cause Mortality in Non-Diabetic but Not in Pre-Diabetic or Diabetic Subjects.

Nonalcoholic fatty liver disease (NAFLD) is a growing health threat worldwide. Vitamin E supplementation is recommended for nonalcoholic steatohepatitis (NASH) patients, but only for non-diabetic subjects. We aimed to investigate whether serum vitamin E levels differently impact long-term prognosis in diabetic versus non-diabetic NAFLD individuals. A total of 2404 ultrasonographically defined NAFLD individuals from National Health and Nutrition Examination Survey (NHANES) III were stratified by their glycemic statuses into diabetic (N = 662), pre-diabetic (N = 836) and non-diabetic (N = 906), and the relationship between serum vitamin E levels and all-cause mortality was analyzed. The serum vitamin E concentrations were 31.1 ± 14.1, 26.7 ± 9.6, and 24.7 ± 9.8 µmol/L and vitamin E: total cholesterol ratios were 5.16 ± 1.70, 4.81 ± 1.46, and 4.80 ± 1.34 µmol/mmol in in diabetic, pre-diabetic, and non-diabetic groups, respectively. Of 2404 NAFLD subjects, 2403 have mortality information and 152 non-diabetic, 244 pre-diabetic, and 342 diabetic participants died over a median follow-up period of 18.8 years. Both serum vitamin E levels and vitamin E: total cholesterol ratios were negatively associated with all-cause mortality after adjusting for possible confounders in non-diabetic subjects (HR = 0.483, and 0.451, respectively, p < 0.005), but not in either diabetic or pre-diabetic subjects. In NAFLD individuals, both serum vitamin E and lipid-corrected vitamin E were (1) higher in the diabetic group; and (2) negatively associated with all-cause mortality only in the non-diabetic group. Further investigations are warranted to elucidate the underlying mechanism of this inverse association of serum vitamin E concentration with all-cause mortality in non-diabetic but not pre-diabetic or diabetic subjects.

A plasma protein derived TGFß signature is a prognostic indicator in triple negative breast cancer.

We investigated the potential of in-depth quantitative plasma proteome analysis to uncover proteins predictive of progression and metastasis in triple negative breast cancer (TNBC). Analysis of samples from 24 pre-menopausal and 24 post-menopausal women with newly diagnosed TNBC who subsequently developed metastasis or remained metastasis free were utilized in the proteomic discovery set, which resulted in 43 proteins associated with tumor progression. These proteins were found to form a hierarchical network with TGFß. The signature was further confirmed and refined by integrating plasma protein data from a murine TNBC model that encompassed mice with rapid- versus slow-growing tumors. Three genes consisting of CLIC1, MAPRE1, and SERPINA3 in the refined TGFß signature significantly stratified overall survival (log-rank p = 0.0141) in a larger validation cohort irrespective of menopausal status, tumor stage, grade, and size.

Sex-Dimorphic Association of Plasma Fatty Acids with Cardiovascular Fitness in Young and Middle-Aged General Adults: Subsamples from NHANES 2003⁻2004.

To explore the potential association of plasma fatty acids (FAs) and cardiovascular fitness level (CVFL), data of 449 subjects from 2003⁻2004 National Health and Nutrition Examination Survey (NHANES) were analyzed. Among these 249 men and 200 women, aged 20⁻50 years (33.4 ± 8.4 year, mean ± Standard Deviation), 79 low, 166 moderate and 204 high CVFL were categorized by age- and gender- specific percentile, respectively. Twenty-four fatty acids were quantified from fasting plasma. Higher levels of 2 very long-chain saturated FAs (VLSFAs): Arachidic acid (AR1, C20:0) and Docosanoic acid (DA1, C22:0) as well as 2 n-6 polyunsaturated FAs (PUFAs): Arachidonic acid (AA, C20:4n-6) and Docosatetraenoic acid (DTA, C22:4n-6) were observed in the subjects with low CVFL. Notably this association exists only in men. Estimated maximal oxygen consumption (VO2max), the marker for cardiorespiratory fitness, was used for further regression analysis. After the adjustment of potential confounding factors (age, smoking, hypertension status, body mass index (BMI), insulin resistance status, and C-reactive protein (CRP), AA was the only FA correlated with low VO2max in women; while in men AR1, DA1, AA, and DTA remain negatively associated with VO2max. This preliminary analysis suggests a sex-dimorphic relationship between these plasma VLSFAs and n-6 PUFAs with CVFL and merits further investigation.

The Emerging Role of B Cells in Tumor Immunity.

There is increasing evidence supporting a role for B cells in tumor immunology. Paraneoplastic syndromes occurring before a cancer diagnosis have pointed to the potential for harnessing the humoral immune response for early cancer detection. The presence of tumor-infiltrating B lymphocytes has been linked to a favorable clinical outcome in many types of cancers. However, B cells represent a heterogeneous population with functionally distinct subsets, and the balance among subtypes impacts tumor development. Here, we review recent findings related to B cells and to the humoral immune response in cancer and their translational significance. Cancer Res; 76(19); 5597-601. ©2016 AACR.

Cell-cycle-regulated activation of Akt kinase by phosphorylation at its carboxyl terminus.

Akt, also known as protein kinase B, plays key roles in cell proliferation, survival and metabolism. Akt hyperactivation contributes to many pathophysiological conditions, including human cancers, and is closely associated with poor prognosis and chemo- or radiotherapeutic resistance. Phosphorylation of Akt at S473 (ref. 5) and T308 (ref. 6) activates Akt. However, it remains unclear whether further mechanisms account for full Akt activation, and whether Akt hyperactivation is linked to misregulated cell cycle progression, another cancer hallmark. Here we report that Akt activity fluctuates across the cell cycle, mirroring cyclin A expression. Mechanistically, phosphorylation of S477 and T479 at the Akt extreme carboxy terminus by cyclin-dependent kinase 2 (Cdk2)/cyclin A or mTORC2, under distinct physiological conditions, promotes Akt activation through facilitating, or functionally compensating for, S473 phosphorylation. Furthermore, deletion of the cyclin A2 allele in the mouse olfactory bulb leads to reduced S477/T479 phosphorylation and elevated cellular apoptosis. Notably, cyclin A2-deletion-induced cellular apoptosis in mouse embryonic stem cells is partly rescued by S477D/T479E-Akt1, supporting a physiological role for cyclin A2 in governing Akt activation. Together, the results of our study show Akt S477/T479 phosphorylation to be an essential layer of the Akt activation mechanism to regulate its physiological functions, thereby providing a new mechanistic link between aberrant cell cycle progression and Akt hyperactivation in cancer.

Chemical genetic screen for AMPKα2 substrates uncovers a network of proteins involved in mitosis.

The energy-sensing AMP-activated protein kinase (AMPK) is activated by low nutrient levels. Functions of AMPK, other than its role in cellular metabolism, are just beginning to emerge. Here we use a chemical genetics screen to identify direct substrates of AMPK in human cells. We find that AMPK phosphorylates 28 previously unidentified substrates, several of which are involved in mitosis and cytokinesis. We identify the residues phosphorylated by AMPK in vivo in several substrates, including protein phosphatase 1 regulatory subunit 12C (PPP1R12C) and p21-activated protein kinase (PAK2). AMPK-induced phosphorylation is necessary for PPP1R12C interaction with 14-3-3 and phosphorylation of myosin regulatory light chain. Both AMPK activity and PPP1R12C phosphorylation are increased in mitotic cells and are important for mitosis completion. These findings suggest that AMPK coordinates nutrient status with mitosis completion, which may be critical for the organism's response to low nutrients during development, or in adult stem and cancer cells.

A fluorescent reporter of AMPK activity and cellular energy stress.

AMP-activated protein kinase (AMPK) is activated when the AMP/ATP ratio in cells is elevated due to energy stress. Here, we describe a biosensor, AMPKAR, that exhibits enhanced fluorescence resonance energy transfer (FRET) in response to phosphorylation by AMPK, allowing spatiotemporal monitoring of AMPK activity in single cells. We show that this reporter responds to a variety of stimuli that are known to induce energy stress and that the response is dependent on AMPK α1 and α2 and on the upstream kinase LKB1. Interestingly, we found that AMPK activation is confined to the cytosol in response to energy stress but can be observed in both the cytosol and nucleus in response to calcium elevation. Finally, using this probe with U2OS cells in a microfluidic device, we observed a very high cell-to-cell variability in the amplitude and time course of AMPK activation and recovery in response to pulses of glucose deprivation.

Lysosome-related genes are regulated in the orbital fat of patients with graves' ophthalmopathy.

PURPOSE: The molecular mechanism involved in the hypertrophy of the orbital fat in patients with Graves' ophthalmopathy or thyroid eye disease (TED) remains unclear. Comparison of genome-wide expression profiles may help in the search for the gene sets involved in TED. METHODS: Twenty-five orbital adipose tissue specimens were obtained, from which the RNA was isolated. Four of the tissue specimens (from four individuals, two with TED and two control subjects) were subjected to cDNA microarray analysis. The data were analyzed by the gene set enrichment analysis (GSEA) to survey the biological pathways involved in the pathogenesis of TED. Messenger RNA levels of some top-ranked genes in GSEA-selected pathways are validated by quantitative PCR (QPCR). RESULTS: The expression of specific gene sets related to lytic vacuoles, lysosomes, and vacuoles were different between the specimens obtained from patients with TED and control subjects (P < 0.001). These three gene sets overlapped. For QPCR, four top-ranked genes were selected from these overlapping gene sets and another one that related to visual failure, using 21 independent samples of patients with TED (n = 15) and control subjects (n = 6). The results showed that ceroid-lipofuscinosis, neuronal 2, late infantile (CLN2; P = 0.044) and ceroid-lipofuscinosis, neuronal 3, juvenile (CLN3, which related to visual failure; P = 0.012) were significantly downregulated in the orbital fat of patients with TED. The expression of the beta subunit of hexosaminidase A (HEXB) was reduced as well, but the change did not reach statistical significance (P = 0.058). CONCLUSIONS: Lysosome-related genes, such as CLN2, CLN3, and HEXB, may be involved in the pathogenesis of adipose tissue hypertrophy in TED.

Sex-related differences between adiponectin and insulin resistance in schoolchildren.

OBJECTIVE: To study the effect of body composition and adiponectin on insulin resistance and beta-cell function in schoolchildren during puberty. RESEARCH DESIGN AND METHODS: Plasma adiponectin level and its relationships with insulin sensitivity and beta-cell function were analyzed in 500 randomly recruited nondiabetic Taiwanese schoolchildren (245 boys and 255 girls) aged 6-18 years in a national survey program for diabetes in 1999. Insulin resistance and beta-cell function were evaluated by homeostasis model assessment (HOMA). Plasma adiponectin concentrations were determined with radioimmunoassay. RESULTS: Plasma glucose levels remained stable, whereas insulin resistance increased with a compensatory rise in beta-cell function during this period. A transient drop of adiponectin level with a trough at 10-12 years was found in boys but not in girls. This pubertal drop of adiponectin levels in boys coincides with the sharp rise in testosterone concentration. A negative correlation between testosterone levels and adiponectin concentration was also noted in boys (r = -0.142, P = 0.032). Plasma adiponectin levels correlated inversely with relative body weight, fasting insulin concentrations, and insulin resistance index by HOMA in boys aged 15-18 years and in girls aged 11-14 years. No association was observed between adiponectin levels and beta-cell function by HOMA. CONCLUSIONS: There is a transient drop in the level of adiponectin during male puberty, correlated with the increase in testosterone level in boys. Plasma adiponectin levels were inversely correlated with obesity and insulin resistance in boys and girls during the pubertal period.

Allele-specific differential expression of a common adiponectin gene polymorphism related to obesity.

Adiponectin gene polymorphisms have recently been reported to be associated with obesity, insulin sensitivity, and the risk of type 2 diabetes. We examined a T94G polymorphism of the adiponectin gene in 245 ostensibly normal nondiabetic subjects. The G allele frequency was lower among subjects with higher BMI (> or =27) than in those with lower BMI. BMI was inversely correlated with the dose of G allele. Multivariate linear regression analyses showed that the adiponectin genotypes were significantly related to BMI after adjusting for age and gender. The dose of the G allele was associated with a reduction of approximately 1.12 kg/m(2) in BMI. We further found that the relative mRNA levels of G allele were consistently higher than those of T allele in the omental adipose tissue from 21 heterozygous subjects. Finally, we observed that the expression levels of adiponectin affected insulin-stimulated glucose uptake in differentiated 3T3-L1 adipocytes. In conclusion, the allele-specific differential expression of this common polymorphism could be responsible for its biological effects observed in this and the other studies.

Multinucleated giant cells in fine needle aspirates. Can they help differentiate papillary thyroid cancer from benign nodular goiter?

OBJECTIVE: To determine whether multinucleated giant cells (MGCs) in fine needle aspirates can help differentiate papillary thyroid carcinoma (PTC) from benign nodular goiter (BNG). STUDY DESIGN: Specimens from 100 cases of PTC and 100 cases of BNG with surgical and pathologic proof were randomly retrieved. The morphologic characteristics and frequency distribution of the maximal size and number of MGC nuclei were analyzed. A retrospective review of medical records was also undertaken. RESULTS: MGCs were twice as frequent (40%) in PTC than in BNG (26%) (odds ratio = 1.90). Most MGCs in BNG tended to be smaller and ovoid, with foamy cytoplasm, and had fewer nuclei. No MGC in BNG was > 116 microns in diameter or had > 27 nuclei. In contrast, MGCs in PTC were more diverse in size, shape, cytoplasm and number of nuclei. No significant association was found between the presence or nature of MGCs and disease extent in PTC. CONCLUSION: The presence of large MGCs with dense cytoplasm in a thyroid nodule without clinical evidence of thyroiditis should prompt careful exclusion of associated PTC.

Submassive liver necrosis in a hepatitis B carrier with Cushing's syndrome.

Reactivation of hepatitis B virus (HBV) replication in hepatitis B surface antigen (HBsAg)-positive patients is associated with immunosuppressive therapy. However, the interactions between endogenous glucocorticoid in Cushing's syndrome and HBV-related hepatitis remain unclear. Here, we describe the management of a 32-year-old male HBV carrier with Cushing's syndrome caused by an adrenal cortical adenoma, who developed severe hepatitis B. Repeated episodes of septicemia resulting from hypercortisolemia-related immunosuppression further compromised his hepatic condition. Adrenalectomy could not be performed due to coagulopathy. Lamivudine was not available at that time in Taiwan, and this patient died of hepatic failure and sepsis. At autopsy, his liver showed submassive necrosis with small regenerative nodules. The hepatocytes were positive for HBsAg (membrane and cytoplasmic) and hepatitis B core antigen (nuclear and cytoplasmic). Screening for HBsAg is of crucial significance for immunocompromised individuals. Once positive HBsAg is detected in immunosuppressed patients, liver function and viral status should be closely monitored to enable earlier diagnosis and prompt treatment with the newer nucleoside analogues that actively fight HBV.