AI model disgorgement: Methods and choices.

Over the past few years, machine learning models have significantly increased in size and complexity, especially in the area of generative AI such as large language models. These models require massive amounts of data and compute capacity to train, to the extent that concerns over the training data (such as protected or private content) cannot be practically addressed by retraining the model "from scratch" with the questionable data removed or altered. Furthermore, despite significant efforts and controls dedicated to ensuring that training corpora are properly curated and composed, the sheer volume required makes it infeasible to manually inspect each datum comprising a training corpus. One potential approach to training corpus data defects is model disgorgement, by which we broadly mean the elimination or reduction of not only any improperly used data, but also the effects of improperly used data on any component of an ML model. Model disgorgement techniques can be used to address a wide range of issues, such as reducing bias or toxicity, increasing fidelity, and ensuring responsible use of intellectual property. In this paper, we survey the landscape of model disgorgement methods and introduce a taxonomy of disgorgement techniques that are applicable to modern ML systems. In particular, we investigate the various meanings of "removing the effects" of data on the trained model in a way that does not require retraining from scratch.

Redundant Information Neural Estimation.

We introduce the Redundant Information Neural Estimator (RINE), a method that allows efficient estimation for the component of information about a target variable that is common to a set of sources, known as the "redundant information". We show that existing definitions of the redundant information can be recast in terms of an optimization over a family of functions. In contrast to previous information decompositions, which can only be evaluated for discrete variables over small alphabets, we show that optimizing over functions enables the approximation of the redundant information for high-dimensional and continuous predictors. We demonstrate this on high-dimensional image classification and motor-neuroscience tasks.

E46K-like α-synuclein mutants increase lipid interactions and disrupt membrane selectivity.

Parkinson's disease (PD) is one of the most common neurodegenerative disorders, and both genetic and histopathological evidence have implicated the ubiquitous presynaptic protein α-synuclein (αSyn) in its pathogenesis. Recent work has investigated how disrupting αSyn's interaction with membranes triggers trafficking defects, cellular stress, and apoptosis. Special interest has been devoted to a series of mutants exacerbating the effects of the E46K mutation (associated with autosomal dominant PD) through homologous Glu-to-Lys substitutions in αSyn's N-terminal region (i.e. E35K and E61K). Such E46K-like mutants have been shown to cause dopaminergic neuron loss and severe but L-DOPA-responsive motor defects in mouse overexpression models, presenting enormous translational potential for PD and other "synucleinopathies." In this work, using a variety of biophysical techniques, we characterize the molecular pathology of E46K-like αSyn mutants by studying their structure and membrane-binding and remodeling abilities. We find that, although a slight increase in the mutants' avidity for synaptic vesicle-like membranes can be detected, most of their deleterious effects are connected to their complete disruption of αSyn's curvature selectivity. Indiscriminate binding can shift αSyn's subcellular localization away from its physiological interactants at the synaptic bouton toward trafficking vesicles and organelles, as observed in E46K-like cellular and murine models, as well as in human pathology. In conclusion, our findings suggest that a loss of curvature selectivity, rather than increased membrane affinity, could be the critical dyshomeostasis in synucleinopathies.

Information Dropout: Learning Optimal Representations Through Noisy Computation.

The cross-entropy loss commonly used in deep learning is closely related to the defining properties of optimal representations, but does not enforce some of the key properties. We show that this can be solved by adding a regularization term, which is in turn related to injecting multiplicative noise in the activations of a Deep Neural Network, a special case of which is the common practice of dropout. We show that our regularized loss function can be efficiently minimized using Information Dropout, a generalization of dropout rooted in information theoretic principles that automatically adapts to the data and can better exploit architectures of limited capacity. When the task is the reconstruction of the input, we show that our loss function yields a Variational Autoencoder as a special case, thus providing a link between representation learning, information theory and variational inference. Finally, we prove that we can promote the creation of optimal disentangled representations simply by enforcing a factorized prior, a fact that has been observed empirically in recent work. Our experiments validate the theoretical intuitions behind our method, and we find that Information Dropout achieves a comparable or better generalization performance than binary dropout, especially on smaller models, since it can automatically adapt the noise to the structure of the network, as well as to the test sample.