Training Deep Networks
A 16-module path through training deep networks, in study order from foundations to the frontier.
Modules
- Gradient Descent for Deep Learning
- Stochastic and Mini-Batch Gradient Descent
- Momentum and Nesterov Accelerated Gradient
- AdaGrad and RMSProp
- Adam and Its Variants
- Learning-Rate Schedules
- Weight Decay and L2 Regularization
- Overfitting and Regularization Strategies
- Dropout
- Early Stopping
- Data Augmentation
- Batch Normalization
- Layer Normalization
- Debugging the Training Process
- Hyperparameter Tuning
- Transfer Learning and Fine-Tuning