Calculus

Info

• Acctually, there is not so much math that we need to do. However it’s gonna make it easier to understand if we understand how to calculate the basic things i.e. derivatives / gradients.
• The reason that modern nueral network is powerful is that it utilizes matrix caluculation with python(enven though python is not so efficient in some way). So here we do some matrix calculus.

Simple NN

---
config:
  layout: dagre
  look: handDrawn
---
flowchart LR
 subgraph subGraph0["Input Layer"]
    direction LR
        x1(("x₁"))
        x2(("x₂"))
        x3(("x₃"))
        x4(("x₄"))
        x5(("x₅"))
        xb[("b")]
  end
 subgraph subGraph1["Hidden Layer"]
    direction LR
        h1(("h₁"))
        h2(("h₂"))
        h3(("h₃"))
  end
    subGraph0 --> subGraph1
    subGraph1 --> s(("s"))
    style subGraph0 fill:#C8E6C9
    style subGraph1 fill:#E1BEE7
    style s fill:#BBDEFB

• input layer ⇒ hidden layer ⇒ output layer
• $\sigma$ is the activation function, which is usually ReLU or sigmoid.

$$\begin{array}{rl}\mathbf{x}& =[x_1,x_2,...,x_n{]}^T\\ & \mathbf{z}=\mathbf{W}\mathbf{x}+\mathbf{b}\\ \mathbf{h}& =\sigma (\mathbf{z})\\ s& ={\mathbf{u}}^{\mathbf{T}}\mathbf{h}\end{array}\text{\hspace{1em}(1)}$$

Note

• So for this simple neural network, we can use the above formula to calculate the output $s$. The input layer is the input vector $\mathbf{x}$, the hidden layer is the hidden vector $\mathbf{h}$, and the output layer is the output scalar $s$.
• And thoughout the network, we need to calculate the gradient of the loss function with respect to each parameter, which is $\frac{\partial L}{\partial W},\frac{\partial L}{\partial b},\frac{\partial L}{\partial u}$. For simplicity, we can just use $s$ to represent the gradient of the loss function with respect to each parameter.

Tools

1. Jacobian Matrix

$$\frac{\partial \mathbf{f}}{\partial \mathbf{x}}=\left[\begin{array}{ccc}\frac{f_1}{x_1}& \cdots & \frac{f_1}{x_n}\\ ⋮& \ddots & ⋮\\ \frac{f_m}{x_1}& \cdots & \frac{f_m}{x_n}\end{array}\right]\text{\hspace{1em}(2)}$$

2. Chain Rule

$$\frac{\partial f}{\partial x}=\frac{\partial f}{\partial y}\frac{\partial y}{\partial x}\text{\hspace{1em}(3)}$$

Calculation

Going back to Simple NN and using Tools, we have:

1. What is $\frac{\partial \mathbf{h}}{\partial \mathbf{z}}$? Because $\sigma (\mathbf{z})$ is element-wise, which means $h_i=\sigma (z_i)$

$$\begin{array}{rl}{\left(\frac{\partial \mathbf{h}}{\partial \mathbf{z}}\right)}_{i,j}& =\frac{\partial h_i}{\partial z_j}=\frac{\partial \sigma (z_i)}{\partial z_j}\\ & =\{\begin{array}{cc}{\sigma }^{{}^{\prime }}(z_i)& \text{ if i=j}\\ 0& \text{ if otherwise}\end{array}\end{array}\text{\hspace{1em}(4)}$$ $$\frac{\partial \mathbf{h}}{\partial \mathbf{z}}=\left[\begin{array}{ccc}{\sigma }^{{}^{\prime }}(z_1)& \cdots & 0\\ ⋮& \ddots & ⋮\\ 0& \cdots & {\sigma }^{{}^{\prime }}(z_n)\end{array}\right]=diag({\sigma }^{{}^{\prime }}(\mathbf{z}))\text{\hspace{1em}(5)}$$

2. Other Jacobians

$$\begin{array}{rl}& \frac{\partial }{\partial \mathbf{x}}(\mathbf{Wx}+\mathbf{b})=\mathbf{W}\\ & \frac{\partial }{\partial \mathbf{W}}(\mathbf{Wx}+\mathbf{b})={\mathbf{x}}^{\mathbf{T}}\\ & \frac{\partial }{\partial \mathbf{b}}(\mathbf{Wx}+\mathbf{b})=\mathbf{I}\\ & \frac{\partial }{\partial \mathbf{u}}({\mathbf{u}}^{\mathbf{T}}\mathbf{h})={\mathbf{h}}^{\mathbf{T}}\end{array}\text{\hspace{1em}(6)}$$

For example:

1. What is $\frac{\partial s}{\partial \mathbf{b}}$?

$$\begin{array}{rl}\frac{\partial s}{\partial \mathbf{b}}& =\frac{\partial s}{\partial \mathbf{h}}\frac{\partial \mathbf{h}}{\partial \mathbf{z}}\frac{\partial \mathbf{z}}{\partial \mathbf{b}}\\ & ={\mathbf{u}}^{\mathbf{T}}diag({\sigma }^{{}^{\prime }}(\mathbf{z}))\mathbf{I}\\ & ={\mathbf{u}}^{\mathbf{T}}\odot {\sigma }^{{}^{\prime }}(\mathbf{z})\\ & =\delta \end{array}\text{\hspace{1em}(7)}$$

2. What is $\frac{\partial s}{\partial \mathbf{W}}$?

$$\begin{array}{rl}\frac{\partial s}{\partial \mathbf{W}}& =\underset{\text{same as before}}{\underbrace{\frac{\partial s}{\partial \mathbf{h}}\frac{\partial \mathbf{h}}{\partial \mathbf{z}}}}\frac{\partial \mathbf{z}}{\partial \mathbf{W}}\\ & ={\mathbf{u}}^{\mathbf{T}}diag({\sigma }^{{}^{\prime }}(\mathbf{z})){\mathbf{x}}^{\mathbf{T}}\\ & ={\delta }^T{\mathbf{x}}^{\mathbf{T}}\in {\mathbb{R}}^{m\times n}\end{array}\text{\hspace{1em}(8)}$$

Shape of derivatives

• Jacobian Form: the result should be a row vector. For easy calculation of chain rule.
• Shape Convention: the result should be a column vector. For easy SGD when training.

1. Use Jacobian form as much as possible, reshape to follow the shape convention at the end.
2. Always follow the shape convention.

Backpropagation

Backpropagation is almost the most important part of DL. There are two steps:

1. Using upstream derivatives and local derivatives to calculate downstream derivatives.
2. Re-using these derivatives to update the parameters.

• Calculate all gradients at once just like using $\sigma$

How to calculate the gradients in NN frame?

1. Forwardprogate and record the intermediate values
2. Backpropagate and calculate the gradients
   • Load basic explicit formula for each step’s gradient
   • Calculate the upsteam gradient
   • Then calculate the local gradient
   • Calculate the downsteam gradient

Reference

CS224N Lecture 3 PPT