Recurrent Neural Networks cheatsheet

Star5,479

By Afshine Amidi and Shervine Amidi

https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks

Overview
Architecture of a traditional RNNRecurrent neural networks, also known as RNNs,
are a class of neural networks that allow previous outputs to be used as inputs while
having hidden states. They are typically as follows:

For each timestep

t, the activation

a^{< t >}

a
<t>

and the output

y^{< t >}
y
<t>

are expressed as follows:

\boxed{a^{< t >}=g_1(W_{aa}a^{< t-1 >}+W_{ax}x^{< t >}+b_a)}\quad\

textrm{and}\quad\boxed{y^{< t >}=g_2(W_{ya}a^{< t >}+b_y)}

a
<t>

=g
1

(W
aa

a
<t−1>

+W
ax

x
<t>

+b
a

)andy
<t>
=g
2

(W
ya

a
<t>

+b
y

where
W_{ax}, W_{aa}, W_{ya}, b_a, b_y
W
ax
,W
aa
,W
ya
,b
a
,b
y
are coefficients that are shared temporally and
g_1, g_2
g
1
,g
2
activation functions.
The pros and cons of a typical RNN architecture are summed up in the table below:
Advantages Drawbacks

• Possibility of processing input of any • Computation being slow

length
• Difficulty of accessing information
• Model size not increasing with size from a long time ago
of input
• Cannot consider any future input for
• Computation takes into account the current state
historical information

• Weights are shared across time

Applications of RNNsRNN models are mostly used in the fields of natural language
processing and speech recognition. The different applications are summed up in the
table below:

Type of Illustration Example

RNN
One-to-one Traditional
neural network
T_x=T_y=
1

T
x

=T
y

=1

One-to- Music
many generation

T_x=1,
T_y>1

T
x

=1,T
y

>1
Many-to-one Sentiment
classification
T_x>1,
T_y=1

T
x

>1,T
y

=1

Many-to- Name entity

many recognition

T_x=T_y

T
x

=T
y
 Many-to- Machine
many translation

T_x\neq
T_y

T
x

=T
y

Loss functionIn the case of a recurrent neural network, the loss function

\mathcal{L}

L of all time steps is defined based on the loss at every time step as follows:
\boxed{\mathcal{L}(\widehat{y},y)=\sum_{t=1}^{T_y}\mathcal{L}(\
widehat{y}^{< t >},y^{< t >})}

L(

,y)=
t=1

∑
T

y
 L(

<t>

,y
<t>

Backpropagation through timeBackpropagation is done at each point in time. At

timestep

T, the derivative of the loss

\mathcal{L}

L with respect to weight matrix

W is expressed as follows:
\boxed{\frac{\partial \mathcal{L}^{(T)}}{\partial W}=\sum_{t=1}^T\left.\
frac{\partial\mathcal{L}^{(T)}}{\partial W}\right|_{(t)}}

∂W

∂L
(T)

=
t=1
 ∑
T

∂W

∂L
(T)

(t)

Handling long term dependencies

Commonly used activation functionsThe most common activation functions used in
RNN modules are described below:

Sigmoid Tanh RELU

\displaystyle g(z)=\ \displaystyle g(z)=\ \displaystyle g(z)=\

frac{1}{1+e^{-z}} frac{e^{z}-e^{-z}} max(0,z)
{e^{z}+e^{-z}}
g(z)= g(z)=max(0,z)
1+e g(z)=
−z e
1 z

+e
−z

e
z

−e
−z
Vanishing/exploding gradientThe vanishing and exploding gradient phenomena are
often encountered in the context of RNNs. The reason why they happen is that it is
difficult to capture long term dependencies because of multiplicative gradient that can
be exponentially decreasing/increasing with respect to the number of layers.

Gradient clippingIt is a technique used to cope with the exploding gradient problem
sometimes encountered when performing backpropagation. By capping the maximum
value for the gradient, this phenomenon is controlled in practice.
Types of gatesIn order to remedy the vanishing gradient problem, specific gates are
used in some types of RNNs and usually have a well-defined purpose. They are usually
noted

\Gamma

Γ and are equal to:

\boxed{\Gamma=\sigma(Wx^{< t >}+Ua^{< t-1 >}+b)}

Γ=σ(Wx
<t>

+Ua
<t−1>

+b)

where

W, U, b

W,U,b are coefficients specific to the gate and

\sigma
σ is the sigmoid function. The main ones are summed up in the table below:

Type of gate Role Used in

Update gate How much past should matter GRU, LSTM

now?
\Gamma_u

Γ
u

Relevance gate Drop previous information? GRU, LSTM

\Gamma_r

Γ
r

Forget gate Erase a cell or not? LSTM

\Gamma_f

Γ
f

Output gate How much to reveal of a cell? LSTM

\Gamma_o

Γ
o
GRU/LSTMGated Recurrent Unit (GRU) and Long Short-Term Memory units (LSTM)
deal with the vanishing gradient problem encountered by traditional RNNs, with LSTM
being a generalization of GRU. Below is a table summing up the characterizing
equations of each architecture:

Characteri Gated Recurrent Unit (GRU) Long Short-Term Memory

zation (LSTM)

\ \textrm{tanh}(W_c[\ \textrm{tanh}(W_c[\
tilde{c}^{ Gamma_r\star a^{< t-1 Gamma_r\star a^{< t-1
< t >} >},x^{< t >}]+b_c) >},x^{< t >}]+b_c)

c tanh(W tanh(W

~ c c

<t>
[Γ [Γ
r r

⋆a ⋆a
<t−1> <t−1>

,x ,x
<t> <t>

]+b ]+b
c c

) )

c^{< t >} \Gamma_u\star\tilde{c}^{< t \Gamma_u\star\tilde{c}^{< t

>}+(1-\Gamma_u)\star c^{< >}+\Gamma_f\star c^{< t-1
c
<t> t-1 >} >}

Γ Γ
u u

⋆ ⋆

c c

~ ~
<t> <t>

+(1−Γ +Γ
u f

)⋆c ⋆c
<t−1> <t−1>

a^{< t >} c^{< t >} \Gamma_o\star c^{< t >}

a c Γ
<t> <t> o

⋆c
<t>
 Dependenc
ies

Remark: the sign

\star

⋆ denotes the element-wise multiplication between two vectors.

Variants of RNNsThe table below sums up the other commonly used RNN
architectures:

Bidirectional (BRNN) Deep (DRNN)

Learning word representation
In this section, we note

V
V the vocabulary and

|V|

∣V∣ its size.

Motivation and notations

Representation techniquesThe two main ways of representing words are summed up
in the table below:

1-hot representation Word embedding

• Noted • Noted

o_w e_w

o e
 w w

• Naive approach, no similarity • Takes into account words similarity

information

Embedding matrixFor a given word

w, the embedding matrix

E is a matrix that maps its 1-hot representation

o_w

o
w

to its embedding

e_w

e
w

as follows:

\boxed{e_w=Eo_w}

e
w
=Eo
w

Remark: learning the embedding matrix can be done using target/context likelihood
models.

Word embeddings
Word2vecWord2vec is a framework aimed at learning word embeddings by estimating
the likelihood that a given word is surrounded by other words. Popular models include
skip-gram, negative sampling and CBOW.

Skip-gramThe skip-gram word2vec model is a supervised learning task that learns

word embeddings by assessing the likelihood of any given target word

t happening with a context word

c. By noting

\theta_t

θ
t

a parameter associated with

t, the probability

P(t|c)

P(t∣c) is given by:

\boxed{P(t|c)=\frac{\exp(\theta_t^Te_c)}{\displaystyle\sum_{j=1}^{|V|}\exp(\
theta_j^Te_c)}}

P(t∣c)=
j=1

∑
∣V∣

exp(θ
j

e
c

exp(θ
t

T
e
c

Remark: summing over the whole vocabulary in the denominator of the softmax part
makes this model computationally expensive. CBOW is another word2vec model using
the surrounding words to predict a given word.

Negative samplingIt is a set of binary classifiers using logistic regressions that aim at
assessing how a given context and a given target words are likely to appear
simultaneously, with the models being trained on sets of

k negative examples and 1 positive example. Given a context word

c and a target word

t, the prediction is expressed by:

\boxed{P(y=1|c,t)=\sigma(\theta_t^Te_c)}

P(y=1∣c,t)=σ(θ
t

e
c
 )

Remark: this method is less computationally expensive than the skip-gram model.

GloVeThe GloVe model, short for global vectors for word representation, is a word
embedding technique that uses a co-occurence matrix

X where each

X_{i,j}

X
i,j

denotes the number of times that a target

i occurred with a context

j. Its cost function

J is as follows:
\boxed{J(\theta)=\frac{1}{2}\sum_{i,j=1}^{|V|}f(X_{ij})(\
theta_i^Te_j+b_i+b_j'-\log(X_{ij}))^2}

J(θ)=

1
 i,j=1

∑
∣V∣

f(X
ij

)(θ
i

e
j

+b
i

+b
j

−log(X
ij

))
2
where

f is a weighting function such that

X_{i,j}=0\Longrightarrow f(X_{i,j})=0

X
i,j

=0⟹f(X

i,j

)=0.
Given the symmetry that

e and

\theta

θ play in this model, the final word embedding

e_w^{(\textrm{final})}

e
w

(final)

is given by:

\boxed{e_w^{(\textrm{final})}=\frac{e_w+\theta_w}{2}}
 e
w

(final)

e
w

+θ
w

Remark: the individual components of the learned word embeddings are not necessarily
interpretable.

Comparing words
Cosine similarityThe cosine similarity between words

w_1

w
1

and

w_2

w
2

is expressed as follows:

\boxed{\textrm{similarity}=\frac{w_1\cdot w_2}{||w_1||\textrm{ }||w_2||}=\

cos(\theta)}

similarity=

∣∣w
1

∣∣ ∣∣w
2

∣∣

w
1

⋅w
2

=cos(θ)

Remark:

\theta

θ is the angle between words

w_1
w
1

and

w_2

w
2

t-SNE

t-SNE (

t-distributed Stochastic Neighbor Embedding) is a technique aimed at reducing high-

dimensional embeddings into a lower dimensional space. In practice, it is commonly
used to visualize word vectors in the 2D space.
Language model
OverviewA language model aims at estimating the probability of a sentence

P(y)

P(y).

n-gram modelThis model is a naive approach aiming at quantifying the probability that
an expression appears in a corpus by counting its number of appearance in the training
data.

PerplexityLanguage models are commonly assessed using the perplexity metric, also
known as PP, which can be interpreted as the inverse probability of the dataset
normalized by the number of words

T. The perplexity is such that the lower, the better and is defined as follows:
\boxed{\textrm{PP}=\prod_{t=1}^T\left(\frac{1}{\sum_{j=1}^{|V|}y_j^{(t)}\
cdot \widehat{y}_j^{(t)}}\right)^{\frac{1}{T}}}
 PP=
t=1

∏
T

∑
j=1

∣V∣

y
j

(t)

(t)

)
T

Remark: PP is commonly used in

 t

t-SNE.

Machine translation
OverviewA machine translation model is similar to a language model except it has an
encoder network placed before. For this reason, it is sometimes referred as a
conditional language model.

The goal is to find a sentence

y such that:
\boxed{y=\underset{y^{< 1 >}, ..., y^{< T_y >}}{\textrm{arg max}}P(y^{< 1
>},...,y^{< T_y >}|x)}

y=
y

<1>

,...,y

<T

>

arg max

P(y
<1>

,...,y
<T

>

∣x)

Beam searchIt is a heuristic search algorithm used in machine translation and speech
recognition to find the likeliest sentence

y given an input

x.
• Step 1: Find top

B likely words

y^{< 1 >}

y
<1>

• Step 2: Compute conditional probabilities

y^{< k >}|x,y^{< 1 >},...,y^{< k-1 >}

y
<k>

∣x,y
<1>

,...,y
<k−1>

• Step 3: Keep top

B combinations

x,y^{< 1>},...,y^{< k >}

x,y
<1>

,...,y
<k>

Remark: if the beam width is set to 1, then this is equivalent to a naive greedy search.

Beam widthThe beam width

B is a parameter for beam search. Large values of

B

B yield to better result but with slower performance and increased memory. Small
values of

B lead to worse results but is less computationally intensive. A standard value for

B is around 10.

Length normalizationIn order to improve numerical stability, beam search is usually

applied on the following normalized objective, often called the normalized log-likelihood
objective, defined as:

\boxed{\textrm{Objective } = \frac{1}{T_y^\alpha}\sum_{t=1}^{T_y}\log\
Big[p(y^{< t >}|x,y^{< 1 >}, ..., y^{< t-1 >})\Big]}

Objective =

T
y

t=1

∑
T

log[p(y
<t>

∣x,y
<1>

,...,y
<t−1>

)]

Remark: the parameter

\alpha

α can be seen as a softener, and its value is usually between 0.5 and 1.

Error analysisWhen obtaining a predicted translation

\widehat{y}

that is bad, one can wonder why we did not get a good translation

y^*

y
∗

by performing the following error analysis:

Case P(y^*|x)>P(\ P(y^*|x)\leqslant P(\

widehat{y}|x) widehat{y}|x)

P(y P(y
 ∗ ∗

∣x)>P( ∣x)⩽P(

y y

∣x) ∣x)

Root cause Beam search faulty RNN faulty

Remedies Increase beam width • Try different architecture

• Regularize

• Get more data

Bleu scoreThe bilingual evaluation understudy (bleu) score quantifies how good a
machine translation is by computing a similarity score based on

n-gram precision. It is defined as follows:

\boxed{\textrm{bleu score}=\exp\left(\frac{1}{n}\sum_{k=1}^np_k\right)}

bleu score=exp(

k=1

∑
n

p
k

where
p_n
p
n
is the bleu score on
n
n-gram only defined as follows:

p_n=\frac{\displaystyle\sum_{\textrm{n-gram}\in\widehat{y}}\
textrm{count}_{\textrm{clip}}(\textrm{n-gram})}{\displaystyle\sum_{\
textrm{n-gram}\in\widehat{y}}\textrm{count}(\textrm{n-gram})}

p
n

=
n-gram∈

count(n-gram)
n-gram∈

∑
 count
clip

(n-gram)

Remark: a brevity penalty may be applied to short predicted translations to prevent an

artificially inflated bleu score.

Attention
Attention modelThis model allows an RNN to pay attention to specific parts of the input
that is considered as being important, which improves the performance of the resulting
model in practice. By noting

\alpha^{< t, t'>}

α
<t,t

>

the amount of attention that the output

y^{< t >}

y
<t>

should pay to the activation

a^{< t' >}

a
<t
′

>

and

c^{< t >}

c
<t>

the context at time

t, we have:
\boxed{c^{< t >}=\sum_{t'}\alpha^{< t, t' >}a^{< t' >}}\quad\textrm{with}\
quad\sum_{t'}\alpha^{< t,t' >}=1

c
<t>

=
t

α
<t,t

>

a
<t
′

>

with
t

α
<t,t

>

=1

Remark: the attention scores are commonly used in image captioning and machine
translation.
Attention weightThe amount of attention that the output

y^{< t >}

y
<t>

should pay to the activation

a^{< t' >}

a
<t

′
>

is given by

\alpha^{< t,t' >}

α
<t,t

>

computed as follows:

\boxed{\alpha^{< t,t' >}=\frac{\exp(e^{< t,t' >})}{\displaystyle\

sum_{t''=1}^{T_x}\exp(e^{< t,t'' >})}}

α
<t,t

>

=
t

′′

=1

∑
T

exp(e
<t,t
′′

>

exp(e
<t,t

>

Remark: computation complexity is quadratic with respect to

T_x

T
x