BN（Batch Normalization）

### Batch Normalization in Neural Networks Explained #### Definition and Purpose Batch normalization is a method utilized during the training of artificial neural networks that aims at improving the speed, performance, and stability of these models by normalizing the input layer through re-centering and rescaling[^2]. This process helps mitigate issues related to internal covariate shift, which refers to changes in the distribution of inputs to layers deep within the network as weights are updated. #### Implementation Details During each mini-batch update step, batch normalization computes mean μ_B and variance σ²_B across all instances present in the current batch B. Then, for every instance x^(i), normalized value y^(i) gets calculated according to: \[ y^{(i)}=\frac{x^{(i)}-\mu_{B}}{\sqrt{\sigma_{B}^{2}+\epsilon}} \] where ε represents a small constant added for numerical stability purposes. Afterward, two learnable parameters γ and β get introduced per feature dimension so that transformed output z^(i)=γ*y^(i)+β allows controlling scale and offset after normalization has been applied. This approach ensures activations remain well-behaved throughout forward propagation while also providing regularization benefits similar to dropout techniques described elsewhere[^3]. #### Benefits During Training Phase Applying batch norm leads to several advantages including faster convergence rates due to reduced vanishing/exploding gradient problems; enhanced model generalization capabilities thanks partly because this mechanism acts like another form of noise injection into learned representations; less sensitivity towards specific weight initialization schemes since optimal ranges become more forgiving post-normalization treatment. ```python import tensorflow as tf model = tf.keras.models.Sequential([ ... tf.keras.layers.BatchNormalization(), ... ]) ```