Author:LYY

Activation Functions

Sigmoid

Transfer logit $z$ to a probability by mapping $z$ to $(0, 1)$ .

Definition: $P = \frac{1}{1 + e x p ( - z )}$ Derivative:

\frac{\partial P}{\partial z} = - (1 + e x p (- z))^{- 2} \cdot e x p (- z) \cdot (- 1) = \frac{e x p ( - z )}{( 1 + e x p ( - z ) ) ^{2}} = \frac{1}{1 + e x p ( - z )} \cdot \frac{e x p ( - z )}{1 + e x p ( - z )} = P \cdot (1 - P) > 0 {P \in (0, 1)}

It means $z ↑, p ↑$ .

Transfer logits $z_{i}$ to probabilities by normalizing sequence $z_{i}$ to $(0, 1)$ , where the summation is equal to 1.

Definition:

P_{k} = \frac{e x p ( z _{k} )}{\sum _{i} e x p ( z _{i} )}

Derivative:

\frac{\partial P _{k}}{\partial z _{k}} = \frac{e x p ( z _{k} ) \sum _{i} e x p ( z _{i} ) - e x p ( z _{k} ) ^{2}}{( \sum _{i} e x p ( z _{i} ) ) ^{2}} = \frac{e x p ( z _{k} )}{\sum _{i} e x p ( z _{i} )} \cdot \frac{\sum _{i} e x p ( z _{i} ) - e x p ( z _{k} )}{\sum _{i} e x p ( z _{i} )} = P_{k} \cdot (1 - P_{k}) > 0 {P_{i} \in (0, 1)}

It means $z_{k} ↑, p_{k} ↑$ .

\frac{\partial P _{k}}{\partial z _{j}} = \frac{- e x p ( z _{k} ) \cdot e x p ( z _{j} )}{( \sum _{i} e x p ( z _{i} ) ) ^{2}} = - P_{k} \cdot P_{j} < 0 {P_{i} \in (0, 1)}

It means $z_{j} ↑, p_{k} ↓$ .

Introduction:

Definition: Assume have probability $p$ to win,

$odds = win rate = \frac{p}{1 - p}$ We usually use $l o g (o dd s)$ , because it has better symmetry property.

Relationship between probability and $l o g (o dd s)$ : sigmoid $σ$

p = \frac{o dd s}{1 + o dd s} = \frac{e x p ( l o g ( o dd s ))}{1 + e x p (( l o g ( o dd s )))} = \frac{1}{1 + e x p ( - l o g ( o dd s ))} = σ (l o g (o dd s))

Example for machine learning: 1.Logistic Regression: 对数线性模型

Probabiliy(Postive Class) = P = f (x) = \frac{1}{1 + e ^{- z}}, z = w x + b

Easy to find

z = l o g (\frac{p}{1 - p})

2.XgBoost for Classification

L oss = L (y_{i}, p_{i}) = - (y_{i} l o g (p_{i}) + (1 - y_{i}) l o g (1 - p_{i})) = - (y_{i} l o g (\frac{p _{i}}{1 - p i}) + l o g (1 - p_{i})) = - (y_{i} l o g (o dd s) - l o g (1 + \frac{p _{i}}{1 - p _{i}})) = - (y_{i} l o g (o dd s) - l o g (1 + o dd s)) = - y_{i} l o g (o dd s) + l o g (1 + o dd s)

Derivative

g_{i} = \frac{\partial L}{\partial l o g ( o dd s _{i} )} = - y_{i} + \frac{e ^{l o g (o dd s_{i})}}{1 + e ^{l o g (o dd s_{i})}} = - y_{i} + \frac{o dd s _{i}}{1 + o dd s _{i}} = - y_{i} + p_{i}

Second Order Derivative

h_{i} = \frac{\partial ^{2} L}{\partial ^{2} l o g ( o dd s _{i} )} = \frac{\partial g _{i}}{\partial l o g ( o dd s _{i} )} = σ (l o g (o dd s_{i})) (1 - σ (l o g (o dd s_{i}))) = p_{i} (1 - p_{i})