Activation Function

A function that follows the linear function in a neuron, to introduce nonlinearity.

---

If there existed no activation function, there would be no point in having individual neurons, as the entire network would become a single, big linear equation.

With Activation Function

\(w_2 \cdot \text{activation}(w_1 x + b_1) + b_2\)

Without Activation Function

\(w_2 \cdot (w_1 x + b_1) + b_2 = w_2 w_1 x + w_2 b_1 + b_2\)