Feature-wise scaling and shifting: Improving the generalization capability of neural networks through capturing independent information of features.

From the perspective of input features, information can be divided into independent information and correlation information. Current neural networks mainly concentrate on the capturing of correlation information through connection weight parameters supplemented by bias parameters. This paper introduces feature-wise scaling and shifting (FwSS) into neural networks for capturing independent information of features, and proposes a new neural network FwSSNet. In the network, a pair of scale and shift parameters is added before each input of each network layer, and bias is removed. The parameters are initialized as 1 and 0, respectively, and trained at separate learning rates, to guarantee the fully capturing of independence and correlation information. The learning rates of FwSS parameters depend on input data and the training speed ratios of adjacent FwSS and connection sublayers, meanwhile those of weight parameters remain unchanged as plain networks. Further, FwSS unifies the scaling and shifting operations in batch normalization (BN), and FwSSNet with BN is established through introducing a preprocessing layer. FwSS parameters except those in the last layer of the network can be simply trained at the same learning rate as weight parameters. Experiments show that FwSS is generally helpful in improving the generalization capability of both fully connected neural networks and deep convolutional neural networks, and FWSSNets achieve higher accuracies on UCI repository and CIFAR-10.

Low-degree term first in ResNet, its variants and the whole neural network family.

To explain the working mechanism of ResNet and its variants, this paper proposes a novel argument of shallow subnetwork first (SSF), essentially low-degree term first (LDTF), which also applies to the whole neural network family. A neural network with shortcut connections behaves as an ensemble of a number of subnetworks of differing depths. Among the subnetworks, the shallow subnetworks are trained firstly, having great effects on the performance of the neural network. The shallow subnetworks roughly correspond to low-degree polynomials, while the deep subnetworks are opposite. Based on Taylor expansion, SSF is consistent with LDTF. ResNet is in line with Taylor expansion: shallow subnetworks are trained firstly to keep low-degree terms, avoiding overfitting; deep subnetworks try to maintain high-degree terms, ensuring high description capacity. Experiments on ResNets and DenseNets show that shallow subnetworks are trained firstly and play important roles in the training of the networks. The experiments also reveal the reason why DenseNets outperform ResNets: The subnetworks playing vital roles in the training of the former are shallower than those in the training of the latter. Furthermore, LDTF can also be used to explain the working mechanism of other ResNet variants (SE-ResNets and SK-ResNets), and the common phenomena occurring in many neural networks.

Complexity and characteristic frequency studies in ECG signals of mice based on multiple scale factors.

Existing methods of physiological signal analysis based on nonlinear dynamic theories only examine the complexity difference of the signals under a single sampling frequency. We developed a technique to measure the multifractal characteristic parameter intimately associated with physiological activities through a frequency scale factor. This parameter is highly sensitive to physiological and pathological status. Mice received various drugs to imitate different physiological and pathological conditions, and the distributions of mass exponent spectrum curvature with scale factors from the electrocardiogram (ECG) signals of healthy and drug injected mice were determined. Next, we determined the characteristic frequency scope in which the signal was of the highest complexity and most sensitive to impaired cardiac function, and examined the relationships between heart rate, heartbeat dynamic complexity, and sensitive frequency scope of the ECG signal. We found that all animals exhibited a scale factor range in which the absolute magnitudes of ECG mass exponent spectrum curvature achieve the maximum, and this range (or frequency scope) is not changed with calculated data points or maximal coarse-grained scale factor. Further, the heart rate of mice was not necessarily associated with the nonlinear complexity of cardiac dynamics, but closely related to the most sensitive ECG frequency scope determined by characterization of this complex dynamic features for certain heartbeat conditions. Finally, we found that the health status of the hearts of mice was directly related to the heartbeat dynamic complexity, both of which were positively correlated within the scale factor around the extremum region of the multifractal parameter. With increasing heart rate, the sensitive frequency scope increased to a relatively high location. In conclusion, these data provide important theoretical and practical data for the early diagnosis of cardiac disorders.