stochastic gradient descent (SGD)

Stochastic gradient descent (SGD) is a gradient descent method that updates a model’s parameters from the gradient of the loss on a single training example, or a small batch of them, rather than on the full dataset.

Each step draws a fresh sample from shuffled training data, so the gradient it computes is a noisy estimate of the full-dataset gradient. An update then costs the same regardless of dataset size, so a model improves many times per pass over the data instead of once. That noise also helps the search slip past saddle points and shallow minima where a full-batch step stalls.

Sweep the batch size below to watch that trade play out on a least-squares fit of 256 noisy points: every setting spends the same six passes over the data, so full batch buys six clean updates that stall short of the minimum, while single-example SGD buys 1,536 jittery ones that reach it.

Practical runs use mini-batches of tens to thousands of examples, which steadies the estimate and maps onto GPU parallelism. A learning-rate schedule shrinks the step size over time, and momentum carries a running average of past gradients forward to damp oscillation. Nesterov momentum measures that gradient one step ahead, at the point the update is already heading toward.

Robbins and Monro described the underlying stochastic approximation procedure in 1951. Plain SGD is still the baseline optimizer behind scikit-learn’s SGDClassifier and PyTorch’s torch.optim.SGD, though training runs for deep neural networks and large language models usually turn to adaptive variants such as Adam.

Related Resources

Tutorial

Stochastic Gradient Descent Algorithm With Python and NumPy

In this tutorial, you'll learn what the stochastic gradient descent algorithm is, how it works, and how to implement it with Python and NumPy.

advanced algorithms machine-learning numpy

For additional information on related topics, take a look at the following resources:

• Python AI: How to Build a Neural Network & Make Predictions (Tutorial)
• Building a Neural Network & Making Predictions With Python AI (Course)
• PyTorch vs TensorFlow for Your Python Deep Learning Project (Tutorial)
• Python Deep Learning: PyTorch vs Tensorflow (Course)
• Python AI: How to Build a Neural Network & Make Predictions (Quiz)