Assign 4 Transformer

Content

In Assignment 3, we will explore the reason why using multihead self-attention and the positional encoding and the little but powerful pre-trained model minGPT

Attention exploration

For a simple single-head self-attention, we can use the following formula to calculate the attention score:

$$\begin{array}{rl}c& =\sum _{i=1}^n{v}_i{\alpha }_i\\ {\alpha }_i& =\frac{exp(k_i^{\top })q}{{\sum }_{j=1}^{n}exp(k_j^{\top })q}\end{array}\text{\hspace{1em}(1)}$$

Single head

Suppose that all the keys are orthogonal, i.e., $k_i^{\top }{k}_j=0$ for all $i,j$.

• If $q$ is very closely aligned to $k_i$, then the attention score will be very high. The output vector $c$ will be very similar to $v_i$.

Think that we have a query vector $q$ incorporating information from two vectors $v_a$ and $v_b$ with corresponding keys $k_a$ and $k_b$:

$$q=s\cdot \frac{k_a+k_b}{\Vert k_a+k_b\Vert }\text{\hspace{1em}(2)}$$

Where $s$ is a large positive scalar. By doing this combination, we can achive a value $c\approx v_a+v_b$.

• However, if the key vector, let’s say $k_a$ has large veriance. And when $k_a$ is a vector with great magnitude, the query vector $q$ will be very close to $k_a$, and the attention score will be very high. The output vector $c$ will be very similar to $v_a$.

Muilti-head

Suppose we have two heads, so we can use the following formula to calculate the attention score combined the two values as before:

$$\begin{array}{rl}{q}_1& =s\cdot \frac{{\mu }_a}{\Vert {\mu }_a\Vert }\\ q_2& =s\cdot \frac{{\mu }_b}{\Vert {\mu }_b\Vert }\end{array}\text{\hspace{1em}(3)}$$

• In this case, even ${\mu }_a$ sometimes can reach a very high value, the only impact is that $c_a$ will be focusing on $v_a$. The output vector $c$ will still be very similar to ${\mu }_a+{\mu }_b$.

Positional embedding

Note

• Permuting the input will not change the attention score without positional embedding. Basically, when computing the attention score, if the input is permuted, the attention score will be permuted as same way.
• $H_{perm}=PH$

This will be problematic because the meaning of a sentence often depends on the order of the words. For example, “The cat sat on the mat” has a different meaning than ”On the mat sat the cat”. If the model cannot distinguish between these two sequences due to its permutation invariance, it may fail to capture important syntactic and semantic information necessary for understanding and generating coherent text.

Fixed method

$$\begin{array}{rl}{\Phi }_{(t,2i)}& =\sin \left(t/10000^{2i/d}\right)\\ {\Phi }_{(t,2i+1)}& =\cos \left(t/10000^{2i/d}\right)\end{array}\text{\hspace{1em}(4)}$$

Where $t\in 0,1,\dots ,T-1$ and $i\in 0,1,\dots ,d/2-1$

Learnable method

In some cases, we can just set the positional encoding as a learnable parameter.

Rotary Positional Encoding

$$RoPE(x_t^{(1)},x_t^{(2)},t)=\left(\begin{array}{cc}cost\theta & -sint\theta \\ sint\theta & cost\theta \end{array}\right)\left(\begin{array}{c}{x}_t^{(1)}\\ x_t^{(2)}\end{array}\right)\text{\hspace{1em}(5)}$$

Where $t$ is the position index and $\theta$ is the angle: $\theta =10000^{-2(i-1)/d}$

$$\begin{array}{r}\left(\begin{array}{ccccccc}\cos t{\theta }_1& -\sin t{\theta }_1& 0& 0& \cdots & 0& 0\\ \sin t{\theta }_1& \cos t{\theta }_1& 0& 0& \cdots & 0& 0\\ 0& 0& \cos t{\theta }_2& -\sin t{\theta }_2& \cdots & 0& 0\\ 0& 0& \sin t{\theta }_2& \cos t{\theta }_2& \cdots & 0& 0\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& 0& \cdots & \cos t{\theta }_{d/2}& -\sin t{\theta }_{d/2}\\ 0& 0& 0& 0& \cdots & \sin t{\theta }_{d/2}& \cos t{\theta }_{d/2}\end{array}\right)\left(\begin{array}{c}{x}_t^{(1)}\\ x_t^{(2)}\\ x_t^{(3)}\\ x_t^{(4)}\\ ⋮\\ x_t^{(d-1)}\\ x_t^{(d)}\end{array}\right)\end{array}\text{\hspace{1em}(6)}$$ $$\begin{array}{r}\left(\begin{array}{c}\cos t{\theta }_1+i\sin t{\theta }_1\\ \cos t{\theta }_2+i\sin t{\theta }_2\\ ⋮\\ \cos t{\theta }_{d/2}+i\sin t{\theta }_{d/2}\end{array}\right)\odot \left(\begin{array}{c}{x}_t^{(1)}+i{x}_t^{(2)}\\ x_t^{(3)}+i{x}_t^{(4)}\\ ⋮\\ x_t^{(d-1)}+i{x}_t^{(d)}\end{array}\right)\end{array}\text{\hspace{1em}(7)}$$

The above Equa.(6) and Equa.(7) is equivalent. And this positional embedding is a relative methods.

Language pre-trained model

In this part, we have several tasks to do. All the details are in the Code part.

1. Implemnent finetuning and make predictions;
2. Define a span corruption function;
3. Pretrain, finetune and make predictions;
4. Implement RoPE function;
5. Pretrain, finetune and make predictions;

Results

Type	Accuracy
Baseline	5.00 %
Vallina(no pretraining)	1.40 %
Vallina(pretraining)	23.20 %
RoPE	38.60 %

Note

1. Tasks today shall be done today.

Code

Assign 4 Completion