gradient descent

Gradient descent is a first-order iterative optimization method that updates parameters in the direction of the negative gradient to reduce a differentiable objective function.

In practice, it applies a learning rate (step size) to each update and repeats the procedure until a stopping criterion is met. The criterion may include a maximum number of steps or minimal change in the loss function.

Common variants include:

• Batch gradient descent: using the full dataset
• Dtochastic gradient descent: updating from individual examples
• Mini-batch methods: updating from small batches

Practical implementations often use learning-rate schedules or line search techniques to adjust the step size over time.

Gradient descent forms the basis of training for many machine learning models, including deep neural networks, where it’s used to minimize loss functions by iteratively refining model parameters.