In Continual Learning settings, deep neural networks are prone to Catastrophic Forgetting.
Orthogonal Gradient Descent was proposed to tackle the challenge. However, no theoretical
guarantees have been proven yet. We present a theoretical framework to study Continual
Learning algorithms in the Neural Tangent Kernel regime. This framework comprises closed
form expression of the model through tasks and proxies for Transfer Learning,
generalisation and tasks similarity. In this framework, we prove that OGD is robust to …

Abstract In Continual Learning settings, deep neural networks are prone to Catastrophic
Forgetting. Orthogonal Gradient Descent was proposed to tackle the challenge. However, no
theoretical guarantees have been proven yet. We present a theoretical framework to study
Continual Learning algorithms in the Neural Tangent Kernel regime. This framework
comprises closed form expression of the model through tasks and proxies for Transfer
Learning, generalisation and tasks similarity. In this framework, we prove that OGD is robust …