What is Activation Function?

Activation functions are what give neural networks their expressive power. A neuron without an activation function computes a weighted sum of its inputs plus a bias: a purely linear operation. Stack a thousand of these linear layers and you still get a linear function. Real-world data, images, text, audio, complex patterns in any domain, is emphatically not linear. Activation functions introduce the non-linearity that allows neural networks to approximate arbitrarily complex functions.

Several activation functions have become standard through the history of deep learning. The sigmoid function squashes inputs to a range between 0 and 1, making it historically popular for binary classification outputs. The hyperbolic tangent (tanh) squashes to -1 to 1, centering the output distribution. ReLU (Rectified Linear Unit), which outputs the input directly if positive and zero otherwise, became dominant because it is computationally simple and avoids the vanishing gradient problem that plagued sigmoid and tanh in deep networks. Variants of ReLU including Leaky ReLU, ELU, and GELU are widely used in modern architectures, with GELU being the standard in transformer-based language models.

The choice of activation function affects both training dynamics and final model performance. A poorly chosen activation can cause neurons to 'die,' always outputting zero and contributing nothing to learning, or gradients to vanish or explode during backpropagation, making training unstable or impossibly slow. Modern deep learning frameworks default to well-validated activation functions for standard architectures, so practitioners rarely need to choose from scratch, but understanding what activation functions do and why they matter is foundational for debugging training problems and designing novel architectures.

Activation functions also play a role outside the hidden layers of a network. The output layer activation function is chosen to match the task: softmax for multi-class classification (producing a probability distribution over classes), sigmoid for binary classification (producing a probability between 0 and 1), and no activation (linear output) for regression tasks where the model should output an unconstrained numerical value. The loss function is then selected to complement the output activation, forming a mathematically consistent training objective.