Analysis of common activation functions

sigmoid and ReLU

$$sigmoid(x)=\frac{1}{1+e^{-x}}$$

$sigmoid$ The problem with activating functions is that as the input approaches $\pm \infty$ when , The gradient will quickly become 0, When gradient returns , Shallow parameters cannot be effectively updated .

$$ReLU(x)=max(0,x)$$
$ReLU$ stay x>0 when , Gradient constant 1, There will be no gradient vanishing . stay x<0 when , The gradient of 0, No reverse transmission , Can be similar $dropout$ Introduce more nonlinearity . After adding the model , The stability and effect of training are better than $sigmoid$.

ReLU and ReLU6

$$ReLU6(x)=min(6,max(0,x))$$
Limit $ReLU$ The maximum output of does not exceed 6, It can enhance the small model on the end , Robustness in low precision reasoning .

sigmoid and hard sigmoid

$$hard\_sigmoid(x)=(ReLU(x)+3)/6$$
Approximatable $sigmoid$ function , Less computation .

swish and hard swish

$$swish(x)=x\cdot sigmoid(\beta x)$$
$$hard\_swish(x)=x\cdot (ReLU6(x)+3)/6$$

$swish$ Medium $sigmoid$ operation , The calculation amount on the end is too heavy , So use $hard\_sigmoid$ To approximate .$h\_swish$ The activation operation is in $mobilenetv3$ Is used in .