Batch Size

Batch size sits at the center of a fundamental tradeoff in model training. In theory, the ideal parameter update would use the gradient computed over the entire training dataset, giving a perfectly accurate signal of how to improve the model. In practice, this is computationally prohibitive for large datasets. Batch size is the practical compromise: process a subset of examples, compute the gradient over that subset, and update the model's parameters based on that approximate gradient.

Different batch size regimes have distinct characteristics. Stochastic Gradient Descent (SGD) uses a batch size of one, updating parameters after every single example. This is computationally fast but produces noisy, high-variance updates that can make the loss fluctuate erratically. Large batches, sometimes called mini-batches, produce smoother, more accurate gradient estimates but require more memory to store the intermediate activations needed for backpropagation. The sweet spot for most practical training runs is somewhere in between, often in the range of 32 to 512 examples, depending on model size, hardware, and task.

Batch size has a nuanced relationship with learning rate that practitioners must manage carefully. Using a larger batch size generally requires scaling the learning rate upward to maintain similar training dynamics, a relationship sometimes called linear scaling. Failing to adjust the learning rate when changing batch size is a common cause of training instability or degraded final model performance. This is one reason why scaling training to many GPUs, which naturally increases effective batch size through data parallelism, requires careful attention to the full set of training hyperparameters.

Large batch sizes have also been associated with models that overfit more and generalize less well on held-out data, a phenomenon that has been studied extensively in the deep learning literature. Smaller batches introduce noise into the training process that, counterintuitively, can act as a regularizer, helping the model find flatter minima in the loss landscape that generalize better. Understanding how batch size, learning rate, epochs, and regularization interact is a core skill for ML engineers running serious training experiments.