Version
: 2.02
Sequence 06
Données séquentielles et/ou temporelles.
Réseaux de Neurones Récurrents (RNN)
https://fidle.cnrs.fr
SAISON
2122
Le Réseau des informaticiens
du SillonAlpin
SIMaP
Cette session va être enregistrée.
Retrouvez-nous sur notre chaine YouTube
This session will be recorded.
Find us on our YouTube channel
https://fidle.cnrs.fr/youtube
SAISON
2122
Version : 2.02
Sequence 06
Données séquentielles et/ou temporelles.
Réseaux de Neurones Récurrents (RNN)
https://fidle.cnrs.fr
SAISON
2122
Le Réseau des informaticiens
du SillonAlpin
SIMaP
Before getting serious
Available ressources on : https://fidle.cnrs.fr
Slides Gitlab Datasets Vidéos
For doctoral students, we can make attestations :
● Remember to use Zoom , rather than YouTube
● Log in with your academic email address
● For any question :
[email protected] Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)
https://creativecommons.org/licenses/by-nc-nd/4.0/
Resources
You can also subscribe to :
Fidle information list http://fidle.cnrs.fr/listeinfo
To receive all FIDLE informations
[email protected] https://listes.services.cnrs.fr/wws/info/devlog
List of ESR* developers
[email protected] https://listes.math.cnrs.fr/wws/info/calcul
List of ESR* « calcul » group
https://fidle.cnrs.fr
(*) ESR : Enseignement Supérieur et Recherche, french universities and public academic research organizations
Roadmap
SAISON
2122 Full description on https://fidle.cnrs.fr/programme 6
37
Roadmap
6.1 Sequences data
➔ Recurrent Neural Network
➔ LSTM and GRU
6 6.2 Example 1 : Ladybug1
➔ Prediction of a virtual trajectory
Sequences data
RNN
6.3 Example 2 : SYNOP1/3
➔ Weather prediction at 3h and 12h
Roadmap
6.1 Sequences data
➔ Recurrent Neural Network
➔ LSTM and GRU
6 6.2 Example 1 : Ladybug1
➔ Prediction of a virtual trajectory
Sequences data
RNN
6.3 Example 2 : SYNOP1/3
➔ Weather prediction at 3h and 12h
What if the world was just one big sequence ?
...
9
37
From classic to recurrent neuron
Classical neuron.
10
37
From classic to recurrent neuron Where :
wy : is a scalar
Wx : is a vector
b : is a scalar
y : is a scalar
Classical neuron.
Recurrent neuron.
11
37
Simple reccurent neuron Where :
wy : is a scalar
Wx : is a vector
b : is a scalar
y : is a scalar
Recurrent neuron.
12
37
Simple reccurent neuron Where :
wy : is a scalar
Wx : is a vector
b : is a scalar
y : is a scalar
Recurrent neuron.
13
37
Recurrent Layer / Cell
Where :
Wy : is a tensor
Wx : is a tensor
b : is a vector
Y : is a vector
Recurrent neuron. Recurrent Layer / Cell
14
37
Recurrent Layer / Cell Where :
Wy : is a tensor
Wx : is a tensor
b : is a vector
Y : is a vector
Recurrent Layer / Cell
15
37
Recurrent Layer / Cell Where :
Wy : is a tensor
Wx : is a tensor
b : is a vector
Y : is a vector
Recurrent Layer / Cell
16
37
Recurrent Layer / Cell Wx shape is : (nb units, x size)
Wy shape is : (nb units, nb units)
y shape is : (nb units,)
Recurrent Layer / Cell 17
37
Simple RNN limits...
But….
... ...
Slow convergence, In short...
Short memory, it doesn't
Vanishing / exploding gradients work !
18
37
Long Short-Term Memory (LSTM)
Long term
state
Short term
state
Long short-term memory (LSTM)1
Gated recurrent unit (GRU)2
19
1
Sepp Hochreiter, Jürgen Schmidhuber, (1997) [LSTM] 2
Kyunghyun Cho et al, (2014) [GRU] 37
Long Short-Term Memory (LSTM)
inputs = tf.random.normal([32, 20, 8])
lstm = tf.keras.layers.LSTM(16)
output = lstm(inputs)
Serie to vector
Inputs shape is : (32, 20, 8)
Output shape is : (32, 16)
lstm = tf.keras.layers.LSTM(18, return_sequences=True, return_state=True)
output, memory_state, carry_state = lstm(inputs)
Serie to serie
Output shape : (32, 20, 18)
Memory state : (32, 18)
Carry state : (32, 18)
More about :
https://keras.io/api/layers/recurrent_layers/lstm/ 20
https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM 37
Reccurent Neural Network (RNN)
Serie to serie Serie to vector
Example : Time serie prediction Example : Sentiment analysis
Vector to serie Encoder-decoder
Example : Image annotation Example : Language Translation
21
37
How to predict a sequence ?
Input X = [ X1, X2, …, Xn ]
Sequence :
Known sequence : [ X1, X2, …, Xn, Xn+1 ]
Expected Y = [ Xn+1 ]
output :
Serie to vector :
The objective will be to train our
RNN network to predict the n+1
vector of our input sequence :
22
37
Preparation of sequence data
The distribution of data between
trains and test series must be
comparable.
23
37
Preparation of sequence data
Can the past explain the
future ?
24
37
Preparation of sequence data
Note that a data
generator should be
very useful !
25
37
Roadmap
6.1 Sequences data
➔ Recurrent Neural Network
➔ LSTM and GRU
6 6.2 Example 1 : Ladybug1
➔ Prediction of a virtual trajectory
Sequences data
RNN
6.3 Example 2 : SYNOP1/3
➔ Weather prediction at 3h and 12h
Time series with RNN
Notebook : [LADYB1]
Objective :
Prediction of a 2D ladybug trajectory with a RNN
Dataset :
Generated trajectory
27
37
LADYB1
<60 s
train
test
Import Generation Compile
and init of a dataset and train
START
Step 1 Step 2 Step 3 Step 4 Step 5
END
Objectives :
Parameters Create a model Try a
Prediction sequence_len = 20 trajectory
of a 2D predict_len = 5 prediction
ladybug* scale
train_prop
=
=
1
.8
trajectory batch_size
epochs
=
=
32
5
with a RNN
28
*Disclaimer : No ladybugs were harmed during the making of this notebook. 37
Roadmap
6.1 Sequences data
➔ Recurrent Neural Network
➔ LSTM and GRU
6 6.2 Example 1 : Ladybug1
➔ Prediction of a virtual trajectory
Sequences data
RNN
6.3 Example 2 : SYNOP1/3
➔ Weather prediction at 3h and 12h
Time series with RNN
Notebook : [SYNOP1-3]
Objective :
Guess what the weather will be like !
Dataset :
SYNOP meteorological data*.
Data from LYS airport for the period 2010-2020
* Observational data from international surface observation messages (SYNOP) circulating on
the World Meteorological Organization's (WMO) Global Telecommunication System (GTS). 30
https://public.opendatasoft.com/explore/dataset/donnees-synop-essentielles-omm/information/?sort=date 37
Time series with RNN
Notebook : [SYNOP1-3]
Episode 1 : Data analysis and creation of a usable dataset
Episode 2 : Training session and first predictions
Episode 3: Attempt to predict in the longer term
31
37
SYNOP1 | SYNOP2 | SYNOP3
< 3 s !
SYNOP Pandas
Import Retrieve the About our
and init dataset enhanced dataset
START
Step 1 Step 2 Step 3 Step 4 Step 5 Step 6
END
Objectives : Prepare
Parameters Save it
Data analysis dataset
(No need to change)
and Enhanced ./data
preparation Pandas Enhanced
of a usuable
dataset
32
37
SYNOP1 | SYNOP2 | SYNOP3
scale = 1
train_prop = .8 ± 1’50s
sequence_len = 16
batch_size = 32
epochs = 10
Import
and init Create a model Predict
START
Step 1 Step 2 Step 3 Step 4 Step 5
END
Read and Compile
Objectives : prepare dataset and train
First weather
prediction, train
using LSTM. ./data
test
Attempt at 3h
33
37
SYNOP1 | SYNOP2 | SYNOP3
Iterations = 4
scale = 1
train_prop = .8 < 3s !
sequence_len = 16
batch_size = 32
epochs = 10
Import
and init Predict
START
Step 1 Step 2 Step 5
END
Read and
Objectives : prepare dataset
First weather
prediction, train
using LSTM. ./data
test
Attempt at 12h
34
37
Little things and concepts to keep in mind
– Can the past explain the future ?
– Understand the data, again and again !
– Beware of overfiting
– Remember that Pandas is good for you !
– The json files are cool, too
– Preparing your data can cost 70% of the work
– Think about data generators
– Matplotlib are also very good for you !
– There is a lot of sequential data
– Not everything can uses GPUs...
35
37
Next, on Fidle :
Jeudi 27 janvier,
Épisode 7 :
7 5b
"Attention Is All You Need",
quand les Transformers changent la donne
«Attention is Nouvelle evolution des modeles
All You Need» Fine tuning
Transformers Cas d'usages : BERT, GPT
Durée : 2h00
Next on Fidle :
6 Jeudi 27 janvier,
Séquence 6 :
Sequence data "Attention Is All You Need",
RNN quand les Transformers changent la donne !
Merci !
To be continued...
[email protected]
FIDLE https://fidle.cnrs.fr
Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) 37
https://fidle.cnrs.fr/youtube https://creativecommons.org/licenses/by-nc-nd/4.0/
37
