Gradient Descent

Gradient descent is the fundamental optimization engine behind nearly all modern machine learning. Training a neural network means finding the right values for millions or billions of parameters. Gradient descent provides a systematic way to search for those values by repeatedly making small improvements in the direction of lower error.

The algorithm works by calculating the gradient of the loss function - a measurement of how wrong the model's current predictions are - with respect to each parameter. The gradient tells you which direction to move each parameter to reduce the loss most quickly. By repeatedly taking steps proportional to the negative gradient, the model gradually 'descends' toward a minimum of the loss function.

There are several variants of gradient descent. Batch gradient descent calculates the gradient over the entire training dataset, which is accurate but slow for large datasets. Stochastic gradient descent (SGD) calculates the gradient on one random sample at a time, which is noisy but fast. Mini-batch gradient descent strikes a balance, calculating gradients on small batches of data. Most modern deep learning training uses a variant of mini-batch gradient descent with adaptive learning rates, such as the Adam optimizer.

The learning rate is the most critical hyperparameter in gradient descent. A learning rate that is too large causes the model to overshoot the minimum and oscillate without converging. A rate that is too small makes training painfully slow. Finding the right learning rate is part of the art and science of training machine learning models effectively.

Gradient descent can get stuck in local minima - valleys in the loss landscape that are lower than their immediate surroundings but not the global lowest point. In practice, the high-dimensional loss landscapes of large neural networks have many such valleys, but researchers have found that most local minima in deep networks are 'good enough' to produce high-quality models.