Neural Network Training Methods

🚀 Deep Learning Playlist by Nitish Singh: Lectures 21–30 In Lectures 21–30, I moved deeper into improving, stabilizing, and optimizing neural networks. 🔍 Key Learnings 21: Improving Neural Network Performance Explored the core parameters that influence model performance: • Hidden layers & neurons • Learning rate • Batch size • Activation functions • Epochs Also learned common challenges like insufficient data, vanishing gradients, overfitting, and slow training — and how optimization methods, transfer learning, and regularization help. 22: Early Stopping Understood overfitting and how early stopping prevents it by monitoring validation loss. Learned how to tune “patience” and other parameters, and how tracking training vs validation curves shows when the model begins to memorize rather than learn. 23: Normalization & Standardization Learned why scaling inputs (like Age vs Salary) is essential for stable learning. • Normalization → [0,1] range • Standardization → mean=0, std=1 Applied these techniques and saw faster convergence and improved model stability. 24–25: Dropout (Theory + Practice) Dropout = randomly turning off neurons during training to avoid overfitting. Saw its effect on: • Regression • Classification Learned how dropout rate p changes model behavior (low p → overfitting, high p → underfitting) and how CNNs/RNNs need different ratios. 26–27: Regularization (L1/L2) Understood why overfitting happens and how L1 & L2 regularization reduce model complexity by penalizing large weights. Implemented L1/L2 and compared performance with vs without regularization. Also explored data augmentation and simplifying architecture. 28: Activation Functions — Dying ReLU Studied the dying ReLU problem, where neurons permanently output zero and stop learning. Causes include: • High learning rate • Negative bias Learned fixes: • Lower LR • Add positive bias • Use Leaky ReLU / PReLU to keep gradients flowing. 29–30: Weight Initialization (What NOT to do → What to do) Covered why bad initialization causes vanishing/exploding gradients. ❌ Zero initialization ❌ Same-value initialization ❌ Very small/very large random values Then learned correct methods: ✔ Xavier Initialization (for sigmoid/tanh) ✔ He Initialization (for ReLU/Leaky ReLU) Understanding initialization made it clear why deep networks need proper variance to train efficiently. 💡 Core Takeaways 🔹 Proper scaling, regularization, and initialization are just as important as architecture. 🔹 Overfitting can be controlled through dropout, early stopping, and L2 regularization. 🔹 Weight initialization + activation function pairing dramatically impacts training stability. 🔹 A well-tuned neural network learns faster, generalizes better, and avoids vanishing/exploding gradients. ✨ Reflection These lectures strengthened my understanding of why neural networks behave the way they do — and how small design choices can make a big difference in performance.

Can we avoid using gradient descent to train physics-informed neural networks (PINNs)? PINNs are slow to train because we rely on gradient descent to optimise their highly nonlinear loss function. In our latest work, we only optimise the weights of the last layer of a PINN, which, for linear PDEs, turns the loss function into a least-squares problem, allowing us to use a linear solver to train the PINN. Essentially, we are finding unknown coefficients of randomised neural network basis functions. We supercharge this approach by 1) using domain decomposition, which adds a sparse structure to the least-squares problem, and 2) preconditioning the least-squares system with QR decomposition. Overall, we are able to reduce the training times of PINNs by up to 3+ orders of magnitude, whilst keeping or even improving accuracy. Want to learn more? I ran a 1-day workshop minimally reproducing this approach, on YouTube: https://lnkd.in/gNXFsce4 Paper: https://lnkd.in/efVqqReG Code: https://lnkd.in/eiHEkCMM This was a fantastic collaboration with: Jan Willem van Beek and Victorita Dolean Building on our previous work with: Samuel Anderson and Jennifer Pestana #SciML #PINNs #DomainDecomposition #LinearAlgebra #ExtremeLearningMachines

If you want to master neural networks, don’t start using PyTorch or TensorFlow directly. Start mastering the building blocks. I have begun to implement Neural Networks entirely from scratch. 26 videos have been already uploaded on Youtube and they have received a great response so far. (1) Coding a single neuron and a layer: https://lnkd.in/gfSRKuxt (2) The beauty of numpy and the dot product in coding neurons and layers: https://lnkd.in/grBjwTu4 (3) Coding multiple neural network layers: https://lnkd.in/gSwmEnZP (4) Implementing the Dense Layer class in Python: https://lnkd.in/gSEeZzTZ (5) Broadcasting and Array Summation in Python: https://lnkd.in/gt9u5hca (6) Coding Neural Network Activation Functions from scratch: https://lnkd.in/gxav-8-2 (7) Coding one neural network forward pass: https://lnkd.in/geyZAvAn (8) Coding the cross entropy loss in Python (from scratch): https://lnkd.in/gbgyQbJi (9) Introduction to Optimization in Neural Network training: https://lnkd.in/gU2ZyXNq (10) Partial Derivatives and Gradient in Neural Networks: https://lnkd.in/gmb4TgUC (11) Understand Chain Rule-The backbone of Neural Networks: https://lnkd.in/gpWqaB2s (12) Backpropagation from scratch on a single neuron: https://lnkd.in/gPXNvxwG (13) Backpropagation through an entire layer of neurons - from scratch: https://lnkd.in/gpTcyz3G (14) Role of matrices in backpropagation: https://lnkd.in/gME4Ey53 (15) Finding derivatives of inputs in backpropagation and why we need them: https://lnkd.in/gyDsmmkS (16) Coding Backpropagation building blocks in Python: https://lnkd.in/gA75tWfz (17) Backpropagation on the ReLU activation class: https://lnkd.in/gaTgYZGa (18) Implementing backpropagation on the cross entropy loss function: https://lnkd.in/gHdFwJBf (19) Combined backpropagation on softmax activation and cross entropy loss: https://lnkd.in/gNJMrCX3 (20) Build the entire backpropagation pipeline for neural networks | No PyTorch or Tensorflow| Only Numpy: https://lnkd.in/gqqmb8AN (21) Coding the entire neural network forward backward pass in Python: https://lnkd.in/grmHYnbn (22) Learning Rate Decay in Neural Network Optimization: https://lnkd.in/gCbciAu9 (23) Momentum in training neural networks: https://lnkd.in/gZwFz46b (24) Coding the ADAGRAD optimizer for Neural Network training: https://lnkd.in/gmyarquq (25) Coding the RMSProp Optimizer with Neural Network training: https://lnkd.in/gryU7Rsw (26) Coding the ADAM optimizer for neural networks: https://lnkd.in/grn2V7Yg I have spent a lot of time and effort in making these lectures. I show everything on a whiteboard and then show it through Python code. Nothing is assumed. Everything is spelled out. Here is a video of my 100 pages of handwritten notes on the topic "Building Neural Networks from Scratch"

Day 8/30 of LLMs/SLMs - Training LLMs at Scale Training LLMs isn’t just about brute force compute — it’s about engineering tricks that make the impossible possible. When you hear that models like GPT-4 or LLaMA were trained on hundreds of billions of tokens, it’s easy to imagine endless racks of GPUs chewing through data. But the truth is: without optimization techniques, even the largest clusters would run out of memory or grind to a halt. Four of the most important techniques are 𝐠𝐫𝐚𝐝𝐢𝐞𝐧𝐭 𝐜𝐡𝐞𝐜𝐤𝐩𝐨𝐢𝐧𝐭𝐢𝐧𝐠, 𝐦𝐢𝐱𝐞𝐝 𝐩𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧, 𝐙𝐞𝐑𝐎, 𝐚𝐧𝐝 𝐅𝐥𝐚𝐬𝐡𝐀𝐭𝐭𝐞𝐧𝐭𝐢𝐨𝐧. Gradient checkpointing trades memory for compute. Normally, backpropagation requires storing all intermediate activations. With checkpointing, you only store a subset, and recompute the rest on the backward pass. Docs - https://lnkd.in/gX4ub4pw Mixed precision training uses half-precision (FP16 or BF16) instead of full FP32 for most operations. The payoff is twofold: faster computation (modern GPUs are optimized for it) and reduced memory usage. For example, switching to BF16 can nearly double throughput while keeping numerical stability intact. Docs - https://lnkd.in/gftvMXRk ZeRO (Zero Redundancy Optimizer) from DeepSpeed takes a more radical approach: instead of replicating model states across all GPUs, it shards them. Gradients, optimizer states, and parameters are split across devices. This means no single GPU needs to hold the entire model. Tutorial - https://lnkd.in/gjave7V6 FlashAttention is a recent innovation that rethinks how attention is computed. The classic implementation wastes memory by materializing giant intermediate matrices. FlashAttention computes attention in chunks directly in GPU SRAM, reducing memory usage and increasing speed. On real-world workloads, this can mean 2–4× faster training without changing model quality. Docs - https://lnkd.in/gBQBzzgE Together, these techniques are why we can train models with billions of parameters on clusters of thousands of GPUs without blowing past memory and compute limits. Tune in tomorrow for more SLM/LLMs deep dives. -- 🚶➡️ To learn more about LLMs/SLMs, follow me - Karun! ♻️ Share so others can learn, and you can build your LinkedIn presence! (Img Src: https://lnkd.in/gCNwAZT7)

In a previous discussion, we explored the No-Free-Lunch Theorem, which tells us there is no universally best model for all problems. The key takeaway was that domain knowledge can guide us in inducing the right bias into our models, especially in fields like engineering and physics, where we often have extensive domain expertise but limited data. I proposed three strategies for tackling this challenge: 1. 𝗚𝗲𝘁 𝗠𝗼𝗿𝗲 𝗗𝗮𝘁𝗮: The straightforward approach, but not always feasible. 2. 𝗜𝗻𝗱𝘂𝗰𝗲 𝗯𝗶𝗮𝘀 𝗜𝗡𝗧𝗢 𝘁𝗵𝗲 𝗱𝗮𝘁𝗮: Leveraging domain knowledge to shape model behavior. 3. 𝗠𝗼𝗱𝗶𝗳𝘆 𝘁𝗵𝗲 𝗼𝗯𝗷𝗲𝗰𝘁𝗶𝘃𝗲 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻: Adjusting the model's training target to favor desired solutions. Physics-Informed Neural Networks (PINNs) are a perfect example of the third strategy: modifying the objective function to induce a bias. Let’s dive into how this works using the viral simple harmonic oscillator as an example. 𝗪𝗵𝘆 𝗠𝗼𝗱𝗶𝗳𝘆 𝘁𝗵𝗲 𝗢𝗯𝗷𝗲𝗰𝘁𝗶𝘃𝗲 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻? In the illustration, we see two models trained on the same problem: a vanilla neural network and a PINN. Both models aim to predict the behavior of a harmonic oscillator based on limited examples. During training, both models match the training data exactly. However the PINN has a modified objective function; not only does it minimize the MSE loss, but the network is also regularized with an additional objective to minimize the harmonic oscillation function based on derivatives of the output as additional constraints. That's why the model trains longer than the vanilla neural network. 𝗔 𝗗𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝗞𝗶𝗻𝗱 𝗼𝗳 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 While it might seem like the neural network and the PINN are solving the same problem, they are actually optimizing different objectives. The PINN doesn't just try to fit the data; it also ensures the model follows our bias of generating results that follow physical laws. This added constraint changes the optimization landscape, leading the model to a more physically consistent solution. The resulting predictions are far more accurate for cases where data is scarce or noisy. 𝗥𝗲𝗮𝗹-𝗪𝗼𝗿𝗹𝗱 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 𝗮𝗻𝗱 𝗟𝗶𝗺𝗶𝘁𝗮𝘁𝗶𝗼𝗻𝘀 In practice, I've never seen a PINN in production. However, the principle of regularizing models with additional constraints is widely used across various domains. For instance, adding penalty terms to enforce business preferences, smoothness, boundary conditions, or physical laws can significantly improve model generalization. The core idea remains the same: by modifying the objective function to include domain-specific constraints, we can induce a bias to guide our models toward better solutions, even when data is limited. It’s a strategy worth considering in any situation where domain knowledge provides insights into the desired behavior of a system. #DSwithSaul #PhysicsInformedNeuralNetworks #PINN

𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗮𝗹 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 refers to the ability to learn new data distributions without forgetting previous knowledge. 𝗧𝗵𝗶𝘀 𝗰𝗼𝗻𝗰𝗲𝗽𝘁 𝗮𝗽𝗽𝗹𝗶𝗲𝘀 𝘁𝗼 𝗯𝗼𝘁𝗵 𝗵𝘂𝗺𝗮𝗻𝘀 𝗮𝗻𝗱 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗺𝗼𝗱𝗲𝗹𝘀. For large language models (LLMs), continual learning is essential to keep models updated with current information, correct errors, and address biases without the need for complete retraining. While humans excel at continual learning, neural networks face a significant challenge: 𝗰𝗮𝘁𝗮𝘀𝘁𝗿𝗼𝗽𝗵𝗶𝗰 𝗳𝗼𝗿𝗴𝗲𝘁𝘁𝗶𝗻𝗴. When trained on new tasks, networks often forget previously learned information because the network parameters are adjusted to optimize for the new task, potentially disrupting the optimal configuration from earlier tasks. Early researchers, including Geoffrey Hinton, acknowledged that forgetting would happen during incremental learning, but they hypothesized that small parameter changes would cancel each other out. However, McCloskey and Cohen (1989) and Ratcliff (1990) demonstrated that catastrophic forgetting occurs even with minimal new data, especially when tasks are sequential and disjoint. Today, various techniques are employed to mitigate catastrophic forgetting, including: ▪️ 𝗥𝗲𝗽𝗹𝗮𝘆 𝗠𝗲𝘁𝗵𝗼𝗱𝘀: Experience replay, generative replay, and iCaRL ▪️ 𝗣𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿 𝗥𝗲𝗴𝘂𝗹𝗮𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Elastic Weight Consolidation (EWC), Synaptic Intelligence (SI), Memory Aware Synapses (MAS) ▪️ 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝗮𝗹 𝗥𝗲𝗴𝘂𝗹𝗮𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Knowledge Distillation, Learning without Forgetting (LwF) ▪️ 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻-𝗕𝗮𝘀𝗲𝗱 𝗔𝗽𝗽𝗿𝗼𝗮𝗰𝗵𝗲𝘀: Orthogonal Gradient Descent (OGD), Gradient Episodic Memory (GEM), Meta-learning ▪️ 𝗖𝗼𝗻𝘁𝗲𝘅𝘁-𝗗𝗲𝗽𝗲𝗻𝗱𝗲𝗻𝘁 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Memory networks, task-specific networks, modular networks, and contextual attention mechanisms Each approach balances memory, computation, and scalability in different ways. In practice, hybrid methods (e.g., combining replay with regularization) are often used to achieve better performance. 👀 Read more in the full paper linked in the comments 👇

No gradients. No backprop. Just projections — a fundamentally different, mathematically grounded approach to neural network training that scales. Joint work with Manish Krishan Lal, Stefanie Jegelka and Suvrit Sra. 📄 https://lnkd.in/eajcfeH3 Here’s how it works 🧵 • We reformulate training as a 𝗳𝗲𝗮𝘀𝗶𝗯𝗶𝗹𝗶𝘁𝘆 𝗽𝗿𝗼𝗯𝗹𝗲𝗺, not loss minimization. • Each neuron and data point add a constraint. • We then project onto the constraint sets, finding a point that satisfies all constraints = a trained model. Why this is cool: 1. projections are 𝗰𝗵𝗲𝗮𝗽; roughly the cost of a forward pass 2. they can be computed independently across neurons and data points -> 𝗶𝗻𝗵𝗲𝗿𝗲𝗻𝘁 𝗽𝗮𝗿𝗮𝗹𝗹𝗲𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 3. natural support for 𝗻𝗼𝗻-𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁𝗶𝗮𝗯𝗹𝗲 components and 𝗵𝗮𝗿𝗱 𝗰𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻𝘁𝘀 We built a whole framework for this: 𝗣𝗝𝗔𝗫 • Think autodiff for projections. • Built on JAX, it inherits hardware acceleration & JIT, with a familiar interface. • We trained MLPs, CNNs, and RNNs with PJAX. • 🔗 https://lnkd.in/ea4pc-SG Looking forward to the community's response! The approach has potential beyond standard training — particularly for tasks with non-differentiable components or local constraints, like 𝗽𝗿𝘂𝗻𝗶𝗻𝗴, 𝗾𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻, and 𝘀𝗽𝗮𝗿𝘀𝗲 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴.

🚀 𝐑𝐍𝐍 𝐬𝐞𝐪𝐮𝐞𝐧𝐭𝐢𝐚𝐥 𝐯𝐞𝐜𝐭𝐨𝐫 𝐭𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧𝐬 Recurrent Neural Networks (RNNs) are often wrapped in metaphors, but at their essence, they are simply 𝐬𝐞𝐪𝐮𝐞𝐧𝐭𝐢𝐚𝐥 𝐯𝐞𝐜𝐭𝐨𝐫 𝐭𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧𝐬. They take variable-length input and compress it into a single, fixed-length embedding that represents the entire sequence. 𝟏. 𝐓𝐇𝐄 𝐑𝐄𝐂𝐔𝐑𝐑𝐄𝐍𝐂𝐄 𝐄𝐍𝐆𝐈𝐍𝐄 The core of an RNN is a process that updates a "hidden state" at every step. 𝐓𝐡𝐞 𝐇𝐢𝐝𝐝𝐞𝐧 𝐒𝐭𝐚𝐭𝐞: Think of this as a local variable that holds a partial result. It is not a learned parameter; it is a vector calculated at each step. 𝐓𝐡𝐞 𝐈𝐧𝐩𝐮𝐭𝐬: An RNN processes one symbol at a time (like one character in a word). 𝐓𝐡𝐞 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧: The model uses two specific matrices—one to transition the previous "memory" and another to integrate the new input. 𝐓𝐡𝐞 𝐒𝐮𝐦𝐦𝐚𝐫𝐲: The final hidden state becomes the summary of everything seen from the start to the end of the sequence. 𝟐. 𝐁𝐀𝐂𝐊𝐏𝐑𝐎𝐏𝐀𝐆𝐀𝐓𝐈𝐎𝐍 𝐓𝐇𝐑𝐎𝐔𝐆𝐇 𝐓𝐈𝐌𝐄 Learning in an RNN means finding the optimal values for the internal matrices. 𝐑𝐞𝐮𝐬𝐚𝐛𝐥𝐞 𝐌𝐚𝐭𝐫𝐢𝐜𝐞𝐬: The same weights are used for every symbol in the sequence. This is what allows the model to handle sequences of any length. 𝐓𝐡𝐞 𝐆𝐫𝐚𝐝𝐢𝐞𝐧𝐭 𝐅𝐥𝐨𝐰: Because the matrices are reused, the errors are summed across all steps to update the shared weights. 𝐓𝐫𝐮𝐧𝐜𝐚𝐭𝐞𝐝 𝐔𝐩𝐝𝐚𝐭𝐞𝐬: For massive sequences like long documents, we stop the error calculation after a certain number of steps to save memory, though the hidden state continues to process the full history. 𝟑. 𝐌𝐈𝐍𝐈𝐁𝐀𝐓𝐂𝐇𝐈𝐍𝐆 & 𝐏𝐀𝐃𝐃𝐈𝐍𝐆 To speed up training, we process multiple records at once in a batch. 𝐁𝐚𝐭𝐜𝐡 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠: Instead of a single hidden vector, we use a matrix where each row represents the state for a different record in the batch. 𝐋𝐞𝐟𝐭 𝐏𝐚𝐝𝐝𝐢𝐧𝐠: When records have different lengths (like words), we add zero-vectors to the 𝐥𝐞𝐟𝐭. 𝐖𝐡𝐲? If you add empty data at the end, the final "summary" vector is diluted by useless information. Left padding ensures the final result is based on the most relevant, real input. 𝟒. 𝐊𝐄𝐘 𝐃𝐈𝐒𝐓𝐈𝐍𝐂𝐓𝐈𝐎𝐍𝐒 𝐇𝐢𝐝𝐝𝐞𝐧 𝐒𝐭𝐚𝐭𝐞: A transient local variable; the "running total" of the sequence. 𝐋𝐞𝐚𝐫𝐧𝐞𝐝 𝐌𝐚𝐭𝐫𝐢𝐜𝐞𝐬: The core parameters; initialized once and updated during training. 𝐈𝐭𝐞𝐫𝐚𝐭𝐨𝐫: The variable used to step through the input symbols. 𝐄𝐦𝐛𝐞𝐝𝐝𝐢𝐧𝐠: The final vector produced after the last symbol is processed. 🔥 𝐓𝐇𝐄 𝐁𝐎𝐓𝐓𝐎𝐌 𝐋𝐈𝐍𝐄: An RNN is a function that maps a variable sequence into a fixed-dimensional space. It is a "memory machine" where the architecture is defined by linear transformations and the intelligence is stored in weights that learn how to update that memory over time. #𝐑𝐍𝐍 #𝐃𝐞𝐞𝐩𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 #𝐌𝐚𝐜𝐡𝐢𝐧𝐞𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 #𝐀𝐈

🚀 Demystifying Gradients: From Theory to Practice in Training LLMs Gradients sit at the very heart of training neural networks—and especially LLMs. We all learn the basics early on: loss functions, backpropagation, optimizers, vanishing or exploding gradients. But when you’re actually fine-tuning large models—or training from scratch—the real question becomes: 👉 What do gradients actually look like in practice, and how do you work with them effectively? In my latest blog post, I take a deeper dive into gradients, covering: 🔹 The mathematical foundations behind gradient computation 🔹 How architectural choices—like residual connections and normalization—shape gradient flow 🔹 Practical techniques such as gradient clipping and monitoring to keep training stable One key takeaway: Good architectures set the stage, but active gradient inspection is what keeps large-scale training on track. Effective practitioners treat gradients as a first-class diagnostic signal—not an afterthought. In a way, training models mirrors life itself: a continuous hill-climbing journey toward better outcomes. Gradients define each step—small, local, but directionally meaningful. 📖 Read the full post here: 👉 https://lnkd.in/gKD4CThu Happy gradient ascending in 2026! 📈✨ #LLM #MachineLearning #DeepLearning #Training #Gradients