Adam optimizer

The Adam optimizer, short for adaptive moment estimation, is a gradient descent method that gives every parameter its own step size, computed from running averages of recent gradients and their squares.

Adam keeps two exponential moving averages per parameter: a first moment for the mean of past gradients and a second moment for the mean of their squares. Both start at zero, so a bias correction rescales them over the early steps. Each update then divides the corrected first moment by the square root of the corrected second moment, which damps steps along noisy or steep directions.

Below, plain gradient descent and Adam descend a valley that is steep in one parameter and shallow in the other.

Kingma and Ba introduced Adam in 2014, and it became the standard choice for training deep neural networks because it converges quickly with little tuning. Typical defaults are a learning rate of 0.001 with decay rates of 0.9 and 0.999 for the two moments. The tradeoff is memory, since Adam stores two extra values per weight.

Large language model training runs mostly use AdamW, a variant that applies weight decay directly to the weights instead of routing it through the loss function. That separation generalizes better.

Related Resources

Tutorial

Practical Text Classification With Python and Keras

Learn about Python text classification with Keras. Work your way from a bag-of-words model with logistic regression to more advanced methods leading to convolutional neural networks. See why word embeddings are useful and how you can use pretrained word embeddings. Use hyperparameter optimization to squeeze more performance out of your model.

advanced data-science machine-learning

For additional information on related topics, take a look at the following resources:

• Stochastic Gradient Descent Algorithm With Python and NumPy (Tutorial)
• Python AI: How to Build a Neural Network & Make Predictions (Tutorial)
• Building a Neural Network & Making Predictions With Python AI (Course)
• PyTorch vs TensorFlow for Your Python Deep Learning Project (Tutorial)
• Python Deep Learning: PyTorch vs Tensorflow (Course)
• Scientific Python: Using SciPy for Optimization (Tutorial)
• Learn Text Classification With Python and Keras (Course)
• Python AI: How to Build a Neural Network & Make Predictions (Quiz)