gradient descent
Gradient descent is a first-order iterative optimization method that updates parameters in the direction of the negative gradient to reduce a differentiable objective function.
In practice, it applies a learning rate (step size) to each update and repeats the procedure until a stopping criterion is met. The criterion may include a maximum number of steps or minimal change in the loss function.
Common variants include:
- Batch gradient descent: using the full dataset
- Stochastic gradient descent: updating from individual examples
- Mini-batch methods: updating from small batches
Practical implementations rarely tune the step size with line search, which is too costly on large datasets. Instead, they rely on learning-rate schedules and adaptive optimizers such as Adam, which combine momentum with per-parameter scaling.
Gradient descent forms the basis of training for many machine learning models, including deep neural networks, where it’s used to minimize loss functions by iteratively refining model parameters.
Related Resources
Tutorial
Stochastic Gradient Descent Algorithm With Python and NumPy
In this tutorial, you'll learn what the stochastic gradient descent algorithm is, how it works, and how to implement it with Python and NumPy.
For additional information on related topics, take a look at the following resources:
By Leodanis Pozo Ramos • Updated Aug. 23, 2026
