How to evaluate uncertainty estimates in machine learning for regression?

As neural networks become more popular, the need for accompanying uncertainty estimates increases. There are currently two main approaches to test the quality of these estimates. Most methods output a density. They can be compared by evaluating their loglikelihood on a test set. Other methods output a prediction interval directly. These methods are often tested by examining the fraction of test points that fall inside the corresponding prediction intervals. Intuitively, both approaches seem logical. However, we demonstrate through both theoretical arguments and simulations that both ways of evaluating the quality of uncertainty estimates have serious flaws. Firstly, both approaches cannot disentangle the separate components that jointly create the predictive uncertainty, making it difficult to evaluate the quality of the estimates of these components. Specifically, the quality of a confidence interval cannot reliably be tested by estimating the performance of a prediction interval. Secondly, the loglikelihood does not allow a comparison between methods that output a prediction interval directly and methods that output a density. A better loglikelihood also does not necessarily guarantee better prediction intervals, which is what the methods are often used for in practice. Moreover, the current approach to test prediction intervals directly has additional flaws. We show why testing a prediction or confidence interval on a single test set is fundamentally flawed. At best, marginal coverage is measured, implicitly averaging out overconfident and underconfident predictions. A much more desirable property is pointwise coverage, requiring the correct coverage for each prediction. We demonstrate through practical examples that these effects can result in favouring a method, based on the predictive uncertainty, that has undesirable behaviour of the confidence or prediction intervals. Finally, we propose a simulation-based testing approach that addresses these problems while still allowing easy comparison between different methods. This approach can be used for the development of new uncertainty quantification methods.

Reply to "Comment on 'Nodal infection in Markovian susceptible-infected-susceptible and susceptible-infected-removed epidemics on networks are non-negatively correlated' ".

We emphasize that correlations between infection states in both the SIS and SIR model are always positive and that the title of the article "Nodal infection in Markovian susceptible-infected-susceptible and susceptible-infected-removed epidemics on networks are non-negatively correlated" [Phys. Rev. E 89, 052802 (2014)PLEEE81539-375510.1103/PhysRevE.89.052802] is correct. The history and motivation that led to the proof is placed in perspective.

Aboveground and Belowground Herbivores Synergistically Induce Volatile Organic Sulfur Compound Emissions from Shoots but Not from Roots.

Studies on aboveground (AG) plant organs have shown that volatile organic compound (VOC) emissions differ between simultaneous attack by herbivores and single herbivore attack. There is growing evidence that interactive effects of simultaneous herbivory also occur across the root-shoot interface. In our study, Brassica rapa roots were infested with root fly larvae (Delia radicum) and the shoots infested with Pieris brassicae, either singly or simultaneously, to study these root-shoot interactions. As an analytical platform, we used Proton Transfer Reaction Mass Spectrometry (PTR-MS) to investigate VOCs over a 3 day time period. Our set-up allowed us to monitor root and shoot emissions concurrently on the same plant. Focus was placed on the sulfur-containing compounds; methanethiol, dimethylsulfide (DMS), and dimethyldisulfide (DMDS), because these compounds previously have been shown to be biologically active in the interactions of Brassica plants, herbivores, parasitoids, and predators, yet have received relatively little attention. The shoots of plants simultaneously infested with AG and belowground (BG) herbivores emitted higher levels of sulfur-containing compounds than plants with a single herbivore species present. In contrast, the emission of sulfur VOCs from the plant roots increased as a consequence of root herbivory, independent of the presence of an AG herbivore. The onset of root emissions was more rapid after damage than the onset of shoot emissions. The shoots of double infested plants also emitted higher levels of methanol. Thus, interactive effects of root and shoot herbivores exhibit more strongly in the VOC emissions from the shoots than from the roots, implying the involvement of specific signaling interactions.

Tomato ACS4 is necessary for timely start of and progression through the climacteric phase of fruit ripening.

Climacteric fruit ripening, as it occurs in many fruit crops, depends on a rapid, autocatalytic increase in ethylene production. This agriculturally important process has been studied extensively, with tomato simultaneously acting both as a model species and target crop for modification. In tomato, the ethylene biosynthetic genes ACC SYNTHASE2 (ACS2) and ACS4 are highly expressed during fruit ripening, with a combined loss of both ACS2 and ACS4 activity preventing generation of the ethylene burst necessary for fruit ripening. However, the individual roles and importance of ACS2 and ACS4 have not been determined. In this study, we examined specifically the role of ACS4 by comparing the phenotype of an acs4 mutant firstly with that of the wild-type, and secondly with two novel ripening-inhibitor (rin) mutants. Ethylene production during ripening was significantly reduced in both acs4-1, and rin lines, with rin genotypes showing the weaker ethylene burst. Also i) the time between anthesis and the start of fruit ripening and ii) the time required to progress through ripening were significantly longer in acs4-1 than in the wild type, but shorter than in the strongest rin mutant. The delay in ripening was reflected in the lower expression of ripening-related transcripts during the mature green and light red ripening stages. Furthermore, expression of ACS2 and ACS4 was strongly dependent on a functional RIN gene, while ACS2 expression was largely independent of ACS4. Altogether, we show that ACS4 is necessary for normal progression of tomato fruit ripening and that mutation of this gene may provide a useful means for altering ripening traits.

The combined evidential value of autosomal and Y-chromosomal DNA profiles obtained from the same sample.

When a Y-chromosomal and a (partial) autosomal DNA profile are obtained from one crime sample, and both profiles match the suspect's profiles, we would like to know the combined evidential value. To calculate the likelihood ratio of observing the autosomal and Y-chromosomal DNA profiles combined, we need to know the conditional random match probability of the observed autosomal DNA profile, given the Y-chromosomal match. We examine this conditional probability in two ways: (1) with a database containing data of 2,085 men and (2) using a simulation model. We conclude that if the Y-chromosomal DNA profiles match, we can still regard the autosomal DNA profile as independent from the Y-chromosomal DNA profile if the matching person is not a descendant of the father of the donor of the (crime) sample. The evidential value can, in that case, be computed by multiplying the random match probabilities of the individual profiles.

Epidemics in networks with nodal self-infection and the epidemic threshold.

Since the Susceptible-Infected-Susceptible (SIS) epidemic threshold is not precisely defined in spite of its practical importance, the classical SIS epidemic process has been generalized to the ε-SIS model, where a node possesses a self-infection rate ε, in addition to a link infection rate ß and a curing rate δ. The exact Markov equations are derived, from which the steady state can be computed. The major advantage of the ε-SIS model is that its steady state is different from the absorbing (or overall-healthy state) and approximates, for a certain range of small ε > 0, the in reality observed phase transition, also called the "metastable" state, that is characterized by the epidemic threshold. The exact steady-state analysis for the complete graph illustrates the effect of small ε and the quality of the first-order mean-field approximation, the N-intertwined model, proposed earlier. Apart from duality principles, often used in the mathematical literature, we present an exact recursion relation for the Markov infinitesimal generator.