Activation Function

A function that follows the linear function in a neuron, to introduce nonlinearity.

If there existed no activation function, there would be no point in having individual neurons, as the entire network would become a single, big linear equation.

With Activation Function

$w_{2} \cdot activation (w_{1} x + b_{1}) + b_{2}$

Without Activation Function

$w_{2} \cdot (w_{1} x + b_{1}) + b_{2} = w_{2} w_{1} x + w_{2} b_{1} + b_{2}$