Improving Artificial Neural Network Performance with Zero Initialization

doi:10.21203/rs.3.rs-4890533/v1

Download PDF

Article

Improving Artificial Neural Network Performance with Zero Initialization

https://doi.org/10.21203/rs.3.rs-4890533/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

This study explores zero initialization in artificial neural networks, mimicking synaptic resetting during sleep. Despite the common belief that zero initialization hinders learning by causing identical outputs, our approach diversifies outputs by initializing weights to zero and biases to random values. We evaluated models on Modified National Institute of Standards and Technology (MNIST), Canadian Institute for Advanced Research (CIFAR)-10, and CIFAR-100 datasets using multilayer perceptrons (MLPs), convolutional neural networks (CNNs), residual networks (ResNets), vision transformers (ViTs), and multilayer perceptron mixers (MLP-Mixers). Results showed mixed outcomes: while zero initialization can hinder learning in some cases, it can also match or surpass random initialization in others, especially in plain neural network configurations. Among contemporary deep learning models, MLP-Mixers with zero initialization matched the performance of fully randomly initialized counterparts, despite half of the learnable parameters being set to zero. This study challenges the conventional view that zero initialization inherently degrades neural network performance.

Physical sciences/Mathematics and computing/Computational science

Physical sciences/Mathematics and computing/Computer science

Artificial Neural Networks

Zero Initialization

Weight Initialization

Neural Network Performance

Inspired by the complex and efficient neural system of the brain, researchers have constructed artificial neural networks (ANNs) by mimicking biological neurons¹. These efforts have resulted in the emergence of deep learning, where artificial intelligence (AI) has exhibited remarkable performance in fields ranging from computer vision to natural language processing and reinforcement learning^2–5. Such improvements in performance can be attributed to the brain’s innately optimized architecture for intelligence. Thus, by faithfully mimicking the intricate dynamics of synapses, further advancements in AI are possible.

Sleep is an essential behavior in animals and plays a pivotal role in learning. It facilitates significant structural changes in the brain, particularly through synaptic plasticity within neural networks. The efficacy of learning mechanisms is considerably augmented by sleep⁶. Notably, synapses that are strengthened during wakeful periods undergo recalibration or resetting during sleep^7,8, leading to enhanced learning performance and readiness for subsequent learning phases⁶. This process involves a reduction in the size of the synapse that stores information, paralleled in AI by decreasing the absolute values of synaptic weights towards zero. Although AI systems do not require sleep, the concept of synaptic resetting during sleep offers a compelling model for improving AI performance.

The majority of existing research in academia focuses on developing techniques of initializing weights in various ways and for different purposes rather than initializing weights to zero. For example, studies have examined weight initialization for hypernetworks⁹, developed techniques that eliminate the need for warmup or layer normalization¹⁰, investigated re-initializing neural networks for performance gain¹¹, addressed parameter initialization for weight-normalized networks to avoid explosion and vanishment of information¹², and proved that certain conditions of weight initialization prevent training failures¹³. Accelerated training was achieved through a proposed framework for weight initialization¹⁴.

There is a paucity of prior research focusing on zero initialization. Historically, zero initialization has been associated with inferior performance due to its inability to extract diverse features^15,16. Despite this bias, research has explored its potential. For example, one study applied zero initialization in ANNs and adjusted the learning rate frequently to explore alternative training strategies¹⁷. Although this demonstrated that zero initialization could serve as an alternative method, the study lacked detailed information about bias initialization, did not specify the model structure, and did not utilize popular datasets. Furthermore, zero initialization has also been explored within ResNets^18–20. One study proposed initialization with zeros and ones in ResNets¹⁸, another addressed gradient exploding and vanishing through specific weight initialization strategies¹⁹, and a third modified the ResNets formula by incorporating a zero-initialized parameter²⁰. Additionally, research has shown that initializing the last batch normalization layer in each residual block to zero can enhance performance²¹, and zero-initialized residual addition has been found to improve results²². Other studies highlighted that zero initialization stabilizes the training of spiking neural networks with residual blocks²³, and found that stable learning occurred when their model was initialized to zero²⁴. Zero initialization in residual connection²⁵ has also been applied to attention-based transformers²⁰. Zero-initialized attention is effective in large language models²⁶. However, the limitations of these studies are notable. Firstly, there is scarce evidence that zero initialization can enhance performance in plain neural networks, as opposed to ResNets. Secondly, zero initialization has not been extensively studied as a primary focus, and detailed investigations into its effects are limited. For instance, the study in combined zero initialization with other methods¹⁹, making it difficult to isolate the contributions of zero initialization to performance improvements. Similarly, other reports only briefly noted that zero initialization stabilizes training, without providing a direct comparison to standard initialization methods^23,24.

In AI research, most ANNs are initialized with randomly generated weights before learning begins¹⁵, reflecting the natural state of pre-existing synapses in biological systems before they engage with specific problems. We hypothesized that synaptic resetting in the brain might be analogous to zero initialization in ANNs. Our study investigates how zero initialization at the layer level impacts performance. Initially, we established baseline models where weights and biases are uniformly distributed, following the default settings in PyTorch²⁷. These baselines were then compared to models initialized with zeros or specific constants and Xavier uniform initialization²⁸ at individual layers. We utilized datasets such as Modified National Institute of Standards and Technology (MNIST)²⁹, Canadian Institute for Advanced Research (CIFAR)-10³⁰, and CIFAR-100³⁰ to evaluate and compare the effects of zero initialization on learning, aiming to determine if it mirrors the beneficial aspects of synaptic resetting seen in biological systems.

We investigate the impact of zero initialization on ANNs by setting initial weight values to zero and comparing this approach with the conventional method of random weight initialization. Eq. (1) shows the backpropagation formula.

$$\:\frac{\partial\:Loss}{\partial\:Weight}=\left(An\:error\:from\:upper\:layers\right)\times\:\left(A\:result\:from\:lower\:layers\right)$$

The gradient of the weight with respect to the loss is calculated as the product of an error generated from the upper layers and an output value from the lower layers. Naturally, the output value from the lower layers is entirely independent of the weight values. For instance, whether the weight value is 0.03, − 0.1, 0, or an unimaginably large value, the output value remains the same. However, indirectly, the weight value affects the results of the error received from above, thereby participating in the weight gradient computation. Other reports claim that initializing weights to zero leads to poor learning, despite the zero value not seeming particularly problematic for learning based on the two parts (i.e., independent and indirect parts). A detailed investigation is needed on this matter.

Analysis of Configurations

When a layer is entirely initialized to zero, it directly affects both the layers above and below it. The left of Fig. 1(a) illustrates the relationship between a zero-initialized layer and the layer below it. For simplicity, the activation function is omitted. If the input values to this zero-initialized layer are diverse, the weights will inevitably differ after backpropagation¹, despite initially starting at zero. However, error information no longer propagates backward. During the next backpropagation, the weights of the upper layer already have values, allowing the backpropagation information to be passed backward.

Conversely, the right of Fig. 1(a) shows the relationship between a zero-initialized layer and the layer above it. Since all weights are zero, the input values to the upper layer are also zero, resulting in zero gradients for the weights of that layer during backpropagation, which halts learning in the upper layer. Nevertheless, backpropagation information is still transmitted backward. If the input values in the bottom remain diverse, the weights in the lower layer will get varied gradients, and be learned. In the next round of forward propagation, the input values for the upper layer will get diverse values, resulting in learning the weights of the upper layer.

Figure 1(a) supposes that there are randomly initialized layers above and below the zero-initialized layer. Under these conditions, learning proceeds without problems. However, if two or more layers initialized to zero exist in succession, there is a problem. In the left of Fig. 1(b), all gradients for the weights in the upper layer and the nodes acting as inputs to the upper layer become zero following backpropagation, inhibiting any transmission of backpropagation information and learning weights. Consequently, learning does not occur regardless of the number of iterations, except for the bias in the upper layer.

To address this issue, one effective strategy is to randomly initialize bias values, when weights are zero, as shown in the right of Fig. 1(b). This setup allows the upper layer to receive diverse input values through the biases of the lower layer, generating varied gradients during backpropagation. However, because there is no transmission of backpropagation information, the learnable parameters in the lower layers fail to learn. Because, in subsequent backward propagations, the weights in the upper layer have non-zero values, the gradients of the upper layer can be transmitted to the lower layer, eventually making non-zero values in the lower layer weights. This dynamic is further exemplified in Fig. 1(c) and 1(d), where layers initialized to zero develop non-zero weight values after several cycles of forward and backpropagation. We use a similar setting to the right of Fig. 1(b): Weights are initialized to zero and biases are randomly initialized for experiments.

In the condition of Fig. 1(a), and (c), learning may be thwarted by activation functions used. For example, many currently used activation functions, such as ReLU, output a gradient value of zero when they receive an input of zero. This results in no gradient being transmitted to the lower layer, thereby hindering proper learning progression.

Most modern learning algorithms, such as Adam³¹, do not use gradient values for weight updates; instead, Adam utilizes values that incorporate previous gradient information. Adam uses the three gradient values during initial backpropagation: 0, 1, and − 1. If the gradient calculated is zero, it remains zero; if not, it is set to 1 or − 1 based on its direction, regardless of its original magnitude. These gradients are then scaled by the learning rate for weight updates. Although the weights may initially appear uniform after this first update, the original gradient values are retained and used in subsequent updates, ensuring diversity in weight values over time.

In the case of residual connection²⁵, the calculations are generally similar to those depicted the ones in Fig. 1, with some notable differences. For example, if the residual connection is connected to the middle nodes in the right of Fig. 1(a), the gradient values of the weights in the upper layer will be diverse, allowing for immediate learning of these weights, unlike the scenario in the right of Fig. 1(a).

However, zero initialization has a problem in ResNets²⁵, where the batch normalization layer follows each layer operation. If the layer is initialized to zero, all input values to the batch normalization layer will be either zero or match the bias values. In the case where inputs are zero, the data distribution will have a mean and variance of zero. In the backpropagation of the batch normalization layer, the variance is used in the denominator, and a very small real number is inserted to prevent the denominator from becoming zero during learning. Here, the gradient values have very large values because the denominator is too small. Moreover, in forward propagation, people intend that the outputs of the batch normalization layer have a data distribution with a mean of zero and a variance of one. However, the variance of input values is zero, and it cannot be one. Therefore, while learning is technically possible, the layer operates in a way that is completely different from the intention, making this method not recommended.

In the case where the input values are the same as bias values, the output values from the batch normalization layer will still exhibit a mean and variance of zero. This occurs because batch normalization is performed over the channel, and when all values in a channel are identical, they produce a mean and variance of zero. According to an original paper³², this effect arises from the property of deleting the bias effect. Therefore, the setting such as the right of Fig. 1(b) proves ineffective. In the case of ResNets, where the batch normalization layer follows each layer operation immediately, zero initialization is applicable only for the fully connected layers used at the end of the network.

On the other hand, in layer normalization³³, biases affect the outputs of the normalization layer. If weights and biases are initialized to zero, a similar situation to the batch normalization occurs. However, if only weights are initialized to zero and biases are not, the input values have a data distribution with non-zero mean and variance. The outputs of the layer can have a data distribution with a mean of zero and a variance of one because the layer normalization operates differently, not normalizing over the channel. Since layer normalization is used in multilayer perceptron mixers (MLP-Mixers)⁴ and vision transformers (ViTs)³⁴, it is possible to initialize internal layers to zero and the last layer in these models.

Performance on Plain Neural Networks

Table 1 presents the results of multilayer perceptrons (MLPs)¹ and convolutional neural networks (CNNs)¹. In Table 1, “Weight” refers to the method used for weight initialization, and “Layer” specifies the layers where the indicated weight initialization method is applied. The term “All” in “Layer” indicates that all layers are initialized as described under “Weight.” “Zero all” means that both weights and biases across all layers are set to zero. “1” refers to the first layer, closest to the input layer. Unless specified as “All” or “Zero all,” layers not explicitly mentioned use weights initialized from a uniform distribution, which is PyTorch’s standard setting. For instance, under “Zero” in “Weight” and “1” in “Layer,” all layers except the first are initialized with values drawn from a uniform distribution. “Default” in “Weight” refers to PyTorch’s default initialization settings. “Xavier” in “Weight” denotes initialization using the Xavier uniform distribution. It should be noted that when weights are initialized to zero, biases are always used, except under the “Zero all” condition. The values following “Constant” in Table 1 represent initialization with that constant. The bold text in Table 1 indicates the highest value observed in each model, and all values are rounded for clarity.

Table 1

Results of MLPs and CNNs
	MNIST			CIFAR-10			CIFAR-100
Model	Weight	Layer	Mean	Weight	Layer	Mean	Weight	Layer	Mean
MLPs	Default	All	98.66	Default	All	53.49	Default	All	24.57
	Normal	All	95.28	Normal	All	40.67	Normal	All	12.10
	Xavier	All	98.43	Xavier	All	51.83	Xavier	All	25.32
	Constant 1	All	97.02	Constant 1	All	52.70	Constant 1	All	24.85
	Constant 2	All	96.49	Constant 0.1	All	53.71	Constant 0.1	All	25.34
	Constant 3	All	95.69	Constant − 0.1	All	53.43	Constant − 0.1	All	25.27
	Constant 5	All	94.87	Zero all	All	10.00	Zero all	All	1.00
	Constant 10	All	92.12	Zero	All	53.53	Zero	All	25.54
	Constant 0.5	All	97.50	Zero	1	53.21	Zero	1	24.65
	Constant 0.1	All	98.13	Zero	2	53.63	Zero	2	25.58
	Constant − 0.1	All	97.88	Zero	3	54.10	Zero	3	25.26
	Zero all	All	11.35	-			-
	Zero	All	85.12
	Zero	1	98.67
	Zero	2	98.60
	Zero	3	98.61
CNNs	Default	All	99.19	Default	All	71.61	Default	All	39.07
	Normal	All	97.18	Normal	All	48.03	Normal	All	8.40
	Xavier	All	99.14	Xavier	All	70.67	Xavier	All	36.88
	Constant 1	All	99.03	Constant 1	All	68.03	Constant 1	All	35.42
	Constant 0.1	All	99.13	Constant 0.1	All	67.64	Constant 0.1	All	35.30
	Constant − 0.1	All	98.75	Constant − 0.1	All	64.23	Constant − 0.1	All	25.34
	Zero all	All	11.35	Zero all	All	10.00	Zero all	All	1.00
	Zero	All	99.12	Zero	All	68.03	Zero	All	36.69
	Zero	1	99.25	Zero	1	70.90	Zero	1	39.38
	Zero	2	99.20	Zero	2	72.62	Zero	2	40.99
	Zero	3	99.25	Zero	3	72.96	Zero	3	42.01
	Zero	1,2	99.16	Zero	4	73.31	Zero	4	41.12
	Zero	2,3	99.22	Zero	1,2	66.74	Zero	1,2	36.36
	Zero	1,3	99.21	Zero	1,3	70.94	Zero	1,3	39.22
	-			Zero	1,4	71.24	Zero	1,4	37.45
				Zero	2,3	71.84	Zero	2,3	40.01
				Zero	2,4	73.57	Zero	2,4	40.68
				Zero	3,4	73.98	Zero	3,4	41.68
				Zero	1,2,3	68.76	Zero	1,2,3	36.58
				Zero	1,2,4	67.05	Zero	1,2,4	34.02
				Zero	1,3,4	73.24	Zero	1,3,4	39.44
				Zero	2,3,4	73.91	Zero	2,3,4	40.82
*Default: PyTorch's default initialization settings; Xavier: Xavier uniform distribution for initialization; Zero all: Both weights and biases across all layers are set to zero; Constant X: Initialization with a specific constant value, X.
**Bold: Highest accuracy achieved for each model configuration.

Results for MNIST in Table 1 show that uniform initialization typically outperforms other methods in MLPs and CNNs, with the exception of zero initialization. As expected, training under the “Zero all” condition does not occur. Performance deteriorates as constants increase. Interestingly, performance metrics are not symmetrical between “Constant 0.1” and “Constant − 0.1.” When all layers are initialized to zero, there is a notable decrease in performance: average accuracy drops to 85.12% for MLPs and 99.12% for CNNs. Performance appears to improve only when at least one layer undergoes random initialization.

For CIFAR-10, as shown in Table 1, the default setting in PyTorch yields the best performance for MLPs and CNNs, with the exception of zero initialization. Similar to earlier results, performance declines when all layers are initialized to zero. Additionally, there is variability and asymmetry in performance when initialized with “Constant 0.1” and “Constant − 0.1.”

This trend is further investigated with the CIFAR-100 dataset. The performance differences between “Constant − 0.1” and “Constant 0.1” continue across datasets, with the best outcomes observed in models incorporating zero-initialized layers.

First, Fig. 2 illustrates the variation of CNNs in weights of the last layer over time when it is initialized to zero, showing a progression from uniformity to diversity in weight values as epochs increase. One convolutional neural network is used to create each of the figures. Second, the accuracy trends of CNNs initialized with zero were investigated across epochs. Initializing the last layer to zero consistently outperformed uniform initialization across all epochs. This suggests that zero initialization in the final layers can have a significant positive impact on the learning process. Third, except for the MLP results on MNIST, zero initialization showed a statistically significant performance improvement. Finally, Fig. 2 illustrates a comparison among models with the last layer of CNNs initialized to constant, zero, and default settings. The bars represent the averages and the standard deviations. As shown in Fig. 2, models initialized with a constant value consistently outperform those with random initialization across all datasets, suggesting that constant values can enhance accuracy. Note that initializing to zero is considered a specific case of constant initialization.

Performance on Modern Architectures

Results for ResNets, ViTs, and MLP-Mixers on CIFAR-10 and CIFAR-100 datasets are shown in Fig. 3. In Fig. 3, “Below” refers to the closest linear layer to the input of the MLP module, while “Above” refers to the farthest linear layer from the input. For example, ViTs are set under the “Last, Above” condition in “Layer,” with seven zero-initialized layers. Specifically, the layers in all MLP modules and the last layer for classification are initialized to zero. Note that even if a layer is initialized to zero, biases are randomly initialized.

All data in Fig. 3 pass the normality test, allowing T-tests to be conducted with the options: unpaired, assuming Gaussian distribution, not assuming equal standard deviations, and two-tailed. Performance decreases or remains unchanged except in the case of MLP-Mixers under the “Zero, Last” condition in CIFAR-10 and ViTs in the “Zero, Last” condition in CIFAR-100. Consistently, using the “Below” condition tends to result in performance decreases. However, it is important to note three aspects: First, although there is a statistically significant decrease in performance, it is not drastic—for instance, models that typically achieve an average of 92.8% do not drop below 92.2% due to zero initialization (Fig. 3(a)). Secondly, many performance results are comparable to those achieved by models with random initialization. Thirdly, some results rather surpass those of randomly initialized models, suggesting that zero initialization could enhance performance in modern deep learning algorithms.

Table 2 details the ratio of randomly initialized parameters to total parameters in the models that achieved the highest average accuracy for each dataset. Notably, in the MNIST dataset, CNNs achieve the highest accuracy when the zero-initialized layer is either the first or third layer. So, Table 2 shows results when the third layer is zero-initialized. Remarkably, CNNs with approximately 20% of parameters randomly initialized tend to outperform others in the CIFAR-10 dataset. Additionally, it is noteworthy that even a small fraction of zero values, amounting to just 0.1% of total parameters, can significantly contribute to performance improvement. We particularly examine ResNets with the last layer initialized to zero and ViTs and MLP-Mixers under the “Last, Above” condition. It is noteworthy that the performances of MLP-Mixers with half of the parameters initialized to zero are maintained.

Table 2

The ratio of randomly initialized parameters to total parameters
Dataset	Model	Ratio	The total number of parameters	Model	Ratio	The total number of parameters
MNIST	MLPs	0.504	1,238,730	-
MNIST	CNNs	0.936	79,626	-
CIFAR-10	MLPs	0.999	8,157,010	ResNets	0.999	25,819,466
	CNNs	0.199	381,066	ViTs	0.696	19,774,606
	-			MLP-Mixers	0.502	17,110,026
CIFAR-100	MLPs	0.757	8,247,100	ResNets	0.992	26,003,876
	CNNs	0.377	473,316	ViTs	0.695	19,819,696
	-			MLP-Mixers	0.500	17,156,196

In summary, our comprehensive investigation across various datasets and models reveals that while zero initialization can occasionally lead to decreased performance, it enhances outcomes under certain conditions, such as when applied to the last layer in MLP-Mixers and ViTs. These findings challenge conventional beliefs about weight initialization in neural networks, indicating that zero initialization combined with other initialization methods can effectively improve the performance of modern deep learning architectures. It also challenges the belief that there should always be as many randomly initialized parameters as possible.

In this study, we have explored zero initialization in neural networks, challenging the traditional view that it impairs learning by initializing weights to zero and biases to random values to mimic synaptic resetting during sleep, allowing diverse outputs despite initial uniformity. Our findings show that zero initialization can sometimes match or even surpass random initialization, especially in plain network configurations. Notably, MLP-Mixers with zero initialization performed similarly to fully randomly initialized models, highlighting the potential of zero initialization as an effective strategy under certain conditions. An intriguing finding is the sustained performance of MLP-Mixers even when half of their parameters are initialized to zero. This result underscores the robustness of MLP-Mixers, suggesting that zero initialization does not inherently limit their capability to learn effectively. It demonstrates that strategic zero initialization, even when applied extensively within a model, can preserve performance comparable to traditional random initialization approaches.

In modern algorithms, performance consistently suffers under the “Below” condition (Fig. 3). The reason why the “Above” condition maintains performance, is possibly due to the use of residual connections²⁵. These connections can allow diverse inputs to be propagated through layers, even if outputs from zero-initialized layers do not vary significantly, mimicking the effects of random initialization.

In summary, this study explores the potential of zero initialization in neural networks, challenging the prevailing notion that it necessarily degrades performance. Our findings suggest that while zero initialization can hinder learning in some conditions, it can perform comparably or even better than random initialization in others, especially within plain neural network configurations. By incorporating residual connections and leveraging normalization techniques appropriately, modern models can maintain performance levels, highlighting the need for further detailed investigations into the conditions under which zero initialization can be advantageous. Our research opens new avenues for understanding weight initialization strategies, emphasizing that traditional assumptions about zero initialization may not always hold true. Future work should continue to explore the mathematical underpinnings and practical applications of zero initialization across various neural network architectures and learning algorithms.

Experimental Setup

As PyTorch is a widely-used deep learning framework that provides robust tools for building and training neural networks, we employed several versions of PyTorch (1.13.1 and 2.2.2), both utilizing the same weight initialization methods for the default setting of fully connected and convolutional layers. The algorithms tested include MLPs, CNNs, ViTs, and MLP-Mixers. These architectures were chosen for their varied structures and widespread use in deep learning research and applications. The datasets used in this study were the MNIST, CIFAR-10, and CIFAR-100.

Training Parameters

For plain neural networks, the number of epochs is 500 for MNIST. For CIFAR-10 and CIFAR-100, the epochs are set to 200. The batch size is 100. The learning rate is 1e-4 which is multiplied by 0.95 every 20 epochs. Each experiment is repeated 20 times to ensure the reliability and consistency of the results. Repetition helps in averaging out any anomalies and provides a robust estimate of the model performance. Table 3 outlines the parameters of the algorithms used in this study. For fully connected layers (FC), the numbers in parentheses indicate the number of nodes. For convolutional layers (CON), they specify the number of output channels, kernel length, and stride length in the horizontal and vertical directions. The numbers in the parentheses on the right of “Maxpool” denote kernel and stride lengths in the horizontal and vertical directions. For modern neural networks, we use the “Bottleneck” structure in ResNets. Here, we use PyTorch’s default settings for weight initialization, with a starting learning rate of 1e-3, multiplied by 0.95 every five epochs. The experiments are repeated 10 times. We incorporate data augmentation as described in previous research²⁵, enhancing the model’s ability to generalize from the training data. The number of epochs is 500 with a batch size of 100. Given that batch normalization layers³² negate the effects of biases in lower convolutional layers, all convolutional layers are configured without biases. Down-sampling is conducted by the first layer of Conv2, 3, 4, and 5, as noted in Table 3, with a stride of 2 following the previous research²⁵. It is important to note that biases are included when a layer is initialized to zero. This configuration is used consistently across all datasets, except for necessary adjustments in the output size of the last layer to fit each dataset. Zero initialization is applied to ViTs to assess their effectiveness on the latest algorithms and attention mechanisms. Unlike the original study, which utilized pretraining on a large dataset, our approach does not involve pretraining, resulting in performances that fall short of the original study. The patch size is 4, and Dropout³⁵ is not applied. The learning rate starts at 1e-4. MLP-Mixers are also tested with zero initialization. Like ViTs, Dropout is omitted, and the learning rate starts at 1e-4. Any parameters and settings not specifically mentioned follow those used for ResNets.

Table 3

The parameters of the algorithms
Parameters of MLPs and CNNs
Dataset	Model			Hidden layer			Output layer
MNIST	MLPs			FC (784)–FC (784)			FC (10)
MNIST	CNNs			CON (64,3,1)–Maxpool (3,3) –CON (128,3,1)–Maxpool (3,3)			FC (10)
CIFAR-10 or 100	MLPs			FC (2000)–FC (1000)			FC (10) or (100)
CIFAR-10 or 100	CNNs			CON (64,3,1)–Maxpool (2,2) –CON (128,3,1)–Maxpool (2,2) –CON (256,3,1)–Maxpool (2,2)			FC (10) or (100)
Parameters of ResNets
Layer name		50-layer
Conv1		3×3, 64, stride 1
Conv2		3×3 max pool, stride 2
Conv2		$\:\left[\begin{array}{c}1\times\:1,\:64\\\:3\times\:3,\:64\\\:1\times\:1,\:256\end{array}\right]\times\:4$
Conv3		$\:\left[\begin{array}{c}1\times\:1,\:128\\\:3\times\:3,\:128\\\:1\times\:1,\:512\end{array}\right]\times\:4$
Conv4		$\:\left[\begin{array}{c}1\times\:1,\:256\\\:3\times\:3,\:256\\\:1\times\:1,\:1024\end{array}\right]\times\:4$
Conv5		$\:\left[\begin{array}{c}1\times\:1,\:512\\\:3\times\:3,\:512\\\:1\times\:1,\:2048\end{array}\right]\times\:4$
		Average pool, 2048-d fully connected layer
Parameters of ViTs
Layers			Hidden size		MLP size	Heads
6			500		2000	10
Parameters of MLP-Mixers
Parameter					Value
Number of layers					8
Patch resolution P×P					4×4
Hidden size					512
Sequence length					64
MLP dimension D_C					2048
MLP dimension D_S					256

Initialization and Biases

The proposed method ensures that biases are uniformly initialized even when weights are initialized to zero. Uniform initialization of biases helps to maintain some diversity in the network’s initial situations, potentially aiding the learning process. If not specified otherwise, standard settings are applied, with parameters set according to PyTorch’s default configurations. For example, if not mentioned, the weights in each fully connected layer and convolutional layer are initialized to values drawn from a uniform distribution. The function that implements Kaiming uniform distribution-based initialization³⁶ is used in the code of PyTorch for weight initialization. However, parameters are cleverly set so that it is equal to general uniform distribution-based initialization. It is consistent with the content in the official document which describes the way for weight initialization.

Activation Functions and Loss Computation

The results of ReLU³⁷ were excluded from the experimental results due to its inferior performance compared to Leaky ReLU³⁸. All activation functions except in the last layer are Leaky ReLU, known for allowing a small, non-zero gradient when the unit is not active. Cross-entropy loss³⁹ is utilized for loss computation, a standard loss function for classification tasks, which measures the performance of a classification model whose output is a probability value between 0 and 1. The Adam optimizer is used for optimization, which combines the advantages of two other extensions of stochastic gradient descent. Adam is known for its efficiency in training deep neural networks.

Accuracy Calculation

Accuracies of models on test datasets were calculated at every epoch during training, and the best performance epochs were selected for comparison. This approach ensures that the models are evaluated based on their peak performance during training.

Statistical analysis

To assess the statistical significance of these findings, a T-test was utilized. The D’Agostino-Pearson test was first employed to check for normality. The default setting results for MLPs on CIFAR-100, which do not follow a normal distribution, were analyzed using the Mann-Whitney test. For other cases, T-tests were conducted with the following parameters: unpaired, assuming a Gaussian distribution, not assuming equal standard deviations, and two-tailed.

Author Contribution

J. Seo designed and performed the experiments, with W. Koh providing supervision. Both J. Seo and W. Koh prepared the figures and wrote the manuscript.

Acknowledgements

This work was supported by the Center for Cognition and Sociality (Grant No. IBS-R001-D2) and a Young Scientist Fellowship (Grant No. IBS-R001-Y1) from the Institute for Basic Science, South Korea.

LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. nature 521, 436–444 (2015).
Vaswani, A. et al. Attention is all you need. Advances in neural information processing systems 30 (2017).
Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
Tolstikhin, I. O. et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems 34, 24261–24272 (2021).
Silver, D. et al. Mastering the game of go without human knowledge. nature 550, 354–359 (2017).
Tononi, G. & Cirelli, C. Sleep and the price of plasticity: from synaptic and cellular homeostasis to memory consolidation and integration. Neuron 81, 12–34 (2014).
Kuhn, M. et al. Sleep recalibrates homeostatic and associative synaptic plasticity in the human cortex. Nature communications 7, 12455 (2016).
De Vivo, L. et al. Ultrastructural evidence for synaptic scaling across the wake/sleep cycle. Science 355, 507–510 (2017).
Chang, O., Flokas, L. & Lipson, H. Principled weight initialization for hypernetworks. arXiv preprint arXiv:2312.08399 (2023).
Huang, X. S., Perez, F., Ba, J. & Volkovs, M. in International Conference on Machine Learning. 4475–4483 (PMLR).
Zaidi, S. et al. in Proceedings on. 12–26 (PMLR).
Arpit, D., Campos, V. & Bengio, Y. How to initialize your network? robust initialization for weightnorm & resnets. Advances in Neural Information Processing Systems 32 (2019).
Hanin, B. & Rolnick, D. How to start training: The effect of initialization and architecture. Advances in neural information processing systems 31 (2018).
de Pater, I. & Mitici, M. A mathematical framework for improved weight initialization of neural networks using Lagrange multipliers. Neural Networks 166, 579–594 (2023).
Narkhede, M. V., Bartakke, P. P. & Sutaone, M. S. A review on weight initialization strategies for neural networks. Artificial intelligence review 55, 291–322 (2022).
Ghatak, A. & Ghatak, A. Initialization of network parameters. Deep Learning with R, 87–102 (2019).
Masood, S. & Chandra, P. in Proceedings of the CUBE International Information Technology Conference. 235–239.
Zhao, J., Schäfer, F. & Anandkumar, A. ZerO initialization: Initializing neural networks with only zeros and ones. arXiv preprint arXiv:2110.12661 (2021).
Zhang, H., Dauphin, Y. N. & Ma, T. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv:1901.09321 (2019).
Bachlechner, T., Majumder, B. P., Mao, H., Cottrell, G. & McAuley, J. in Uncertainty in Artificial Intelligence. 1352–1361 (PMLR).
Goyal, P. et al. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 (2017).
Dong, Q., Cao, C. & Fu, Y. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11358–11368.
Fang, W. et al. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems 34, 21056–21069 (2021).
Liu, H., Dai, Z., So, D. & Le, Q. V. Pay attention to mlps. Advances in Neural Information Processing Systems 34, 9204–9215 (2021).
He, K., Zhang, X., Ren, S. & Sun, J. in Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
Zhang, R. et al. in The Twelfth International Conference on Learning Representations.
Paszke, A. et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
Glorot, X. & Bengio, Y. in Proceedings of the thirteenth international conference on artificial intelligence and statistics. 249–256 (JMLR Workshop and Conference Proceedings).
LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 2278–2324 (1998).
Krizhevsky, A. & Hinton, G. Learning multiple layers of features from tiny images. (2009).
Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
Ioffe, S. & Szegedy, C. in International conference on machine learning. 448–456 (pmlr).
Ba, J. L., Kiros, J. R. & Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).
Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15, 1929–1958 (2014).
He, K., Zhang, X., Ren, S. & Sun, J. in Proceedings of the IEEE international conference on computer vision. 1026–1034.
Apicella, A., Donnarumma, F., Isgrò, F. & Prevete, R. A survey on modern trainable activation functions. Neural Networks 138, 14–32 (2021).
Maas, A. L., Hannun, A. Y. & Ng, A. Y. in Proc. icml. 3 (Atlanta, Georgia, USA).
Wang, Q., Ma, Y., Zhao, K. & Tian, Y. A comprehensive survey of loss functions in machine learning. Annals of Data Science, 1–26 (2020).

No competing interests reported.

Download PDF

Reviews received at journal
11 Sep, 2024
Reviews received at journal
11 Sep, 2024
Reviews received at journal
10 Sep, 2024
Reviewers agreed at journal
04 Sep, 2024
Reviewers agreed at journal
30 Aug, 2024
Reviewers agreed at journal
29 Aug, 2024
Reviewers agreed at journal
28 Aug, 2024
Reviewers agreed at journal
28 Aug, 2024
Reviewers agreed at journal
27 Aug, 2024
Reviewers agreed at journal
26 Aug, 2024
Reviews received at journal
26 Aug, 2024
Reviewers agreed at journal
26 Aug, 2024
Reviewers invited by journal
26 Aug, 2024
Editor assigned by journal
26 Aug, 2024
Editor invited by journal
22 Aug, 2024
Submission checks completed at journal
21 Aug, 2024
First submitted to journal
10 Aug, 2024

You are reading this latest preprint version

Improving Artificial Neural Network Performance with Zero Initialization

Status:

Version 1

Abstract

Figures

Introduction

Results

Analysis of Configurations

Performance on Plain Neural Networks

Performance on Modern Architectures

Discussion

Methods

Experimental Setup

Training Parameters

Initialization and Biases

Activation Functions and Loss Computation

Accuracy Calculation

Statistical analysis

Declarations

Author Contribution

Acknowledgements

References

Additional Declarations

Status:

Version 1