Shallow Univariate ReLU Networks as Splines: Initialization, Loss Surface, Hessian, and Gradient Flow Dynamics.

Understanding the learning dynamics and inductive bias of neural networks (NNs) is hindered by the opacity of the relationship between NN parameters and the function represented. Partially, this is due to symmetries inherent within the NN parameterization, allowing multiple different parameter settings to result in an identical output function, resulting in both an unclear relationship and redundant degrees of freedom. The NN parameterization is invariant under two symmetries: permutation of the neurons and a continuous family of transformations of the scale of weight and bias parameters. We propose taking a quotient with respect to the second symmetry group and reparametrizing ReLU NNs as continuous piecewise linear splines. Using this spline lens, we study learning dynamics in shallow univariate ReLU NNs, finding unexpected insights and explanations for several perplexing phenomena. We develop a surprisingly simple and transparent view of the structure of the loss surface, including its critical and fixed points, Hessian, and Hessian spectrum. We also show that standard weight initializations yield very flat initial functions, and that this flatness, together with overparametrization and the initial weight scale, is responsible for the strength and type of implicit regularization, consistent with previous work. Our implicit regularization results are complementary to recent work, showing that initialization scale critically controls implicit regularization via a kernel-based argument. Overall, removing the weight scale symmetry enables us to prove these results more simply and enables us to prove new results and gain new insights while offering a far more transparent and intuitive picture. Looking forward, our quotiented spline-based approach will extend naturally to the multivariate and deep settings, and alongside the kernel-based view, we believe it will play a foundational role in efforts to understand neural networks. Videos of learning dynamics using a spline-based visualization are available at http://shorturl.at/tFWZ2.

Strawberry notch homolog 2 is a novel inflammatory response factor predominantly but not exclusively expressed by astrocytes in the central nervous system.

Interleukin-6 (IL-6) participates in the host response to injury and infection in the central nervous system (CNS). We identified strawberry notch homolog 2 (Sbno2) as an IL-6-stimulated gene in murine astrocytes. Sbno2 is a mouse homolog of the sno gene in Drosophila but little is known about the regulation or function of the mammalian gene. Here we examined the regulation of the Sbno2 gene in astrocytes in vitro and in the murine CNS following systemic endotoxin administration. In murine and human cultured astrocytes, Sbno2 gene expression was significantly upregulated in a dose- and time-dependent fashion by hyper-IL-6 (IL-6 + soluble IL-6 receptor). The level of Sbno2 mRNA was also upregulated significantly in murine astrocytes by other glycoprotein130 cytokine-family members and the pro-inflammatory cytokines interleukin-1 beta and tumor necrosis factor alpha. These changes were reflected by corresponding alterations in the level of the SBNO2 protein. Inhibiting protein synthesis resulted in higher Sbno2 mRNA and did not abolish the upregulation of Sbno2 mRNA mediated by hyper-IL-6. Inhibition of transcription led to a rapid reduction in hyper-IL-6-induced Sbno2 mRNA in astrocytes suggesting that the Sbno2 mRNA is quite unstable. Following intra-peritoneal lipopolysaccharide injection in mice, Sbno2 mRNA levels in the brain were significantly increased. Cellular localization studies revealed that this increase in Sbno2 mRNA occurred predominantly in astrocytes and in the choroid plexus and in some microglia, endothelial cells, and neurons. These findings are consistent with SBNO2 functioning as an acute inflammatory response gene in astrocytes as well as other cells in the CNS.