Practical Deep Learning for NLP
Maarten Versteegh
NLP Research Engineer
Overview
● Deep Learning Recap
● Text classification:
– Convnet with word embeddings
● Sentiment analysis:
– ResNet
● Tips and tricks
What is this deep learning thing again?
Output
Activation
Error
Hidden
Input
Rectified Linear Units
Backpropagation involves repeated multiplication with derivative of activation function
→ Problem if result is always smaller than 1!
Text Classification
Traditional approach: BOW + TFIDF
“The car might also need a front end alignment”
"alignment" (0.323) "also need" (0.343)
"also" (0.137) "car might" (0.358)
"car" (0.110) "end alignment" (0.358)
"end" (0.182) "front end" (0.296)
"front" (0.167) "might also" (0.358)
"might" (0.178) "need front" (0.358)
"need" (0.157) "the car" (0.161)
"the" (0.053)
20 newsgroups performance
F1-Score*
BOW+TFIDF+SVM Some number
(*) Scores removed
Deep Learning 1: Replace Classifier
Output
Hidden x 256
Hidden x 512
BOW
x 1000
Features
from keras.layers import Input, Dense
from keras.models import Model
input_layer = Input(shape=(1000,))
fc_1 = Dense(512, activation='relu')(input_layer)
fc_2 = Dense(256, activation='relu')(fc_1)
output_layer = Dense(10, activation='softmax')(fc_2)
model = Model(input=input_layer, output=output_layer)
model.compile(optimizer='rmsprop',
loss='categorical_crossentropy',
metrics=['accuracy'])
model.fit(bow, newsgroups.target)
predictions = model.predict(features).argmax(axis=1)
20 newsgroups performance
F1-Score*
BOW+TFIDF+SVM Some number
BOW+TFIDF+SVD+ 2-layer NN Some slightly higher number
(*) Scores removed
What about the deep learning promise?
Convolutional Networks
Source: Andrej Karpathy
Pooling layer
Source: Andrej Karpathy
Convolutional networks
Source: Y. Kim (2014) Convolutional Networks for Sentence Classification
Word embedding
from keras.layers import Embedding
# embedding_matrix: ndarray(vocab_size, embedding_dim)
input_layer = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
layer = Embedding(
embedding_matrix.shape[0],
embedding_matrix.shape[1],
weights=[embedding_matrix],
input_length=max_sequence_length,
trainable=False
)(input_layer)
from keras.layer import Convolution1D, MaxPooling1D,
BatchNormalization, Activation
layer = Embedding(...)(input_layer)
layer = Convolution1D(
128, # number of filters
5, # filter size
activation='relu',
)(layer)
layer = MaxPooling1D(5)(layer)
Performance
F1-Score*
BOW+TFIDF+SVM Some number
CBOW+TFIDF+SVD+NN Some slightly higher number
ConvNet (3 layers) Quite a bit higher now
ConvNet (6 layers) Look mom, even higher!
(*) Scores removed
Sentiment Analysis
Data Set
Facebook posts from media organizations:
– CNN, MSNBC, NYTimes, The Guardian, Buzzfeed,
Breitbart, Politico, The Wall Street Journal, Washington
Post, Baltimore Sun
Measure sentiment as “reactions”
Title Org Like Love Wow Haha Sad Angry
Poll: Clinton up big on Trump in Virginia CNN 4176 601 17 211 11 83
It's a fact: Trump has tiny hands. Will Guardian 595 17 17 225 2 8
this be the one that sinks him?
Donald Trump Explains His Obama- NYTimes 2059 32 284 1214 80 2167
Founded-ISIS Claim as ‘Sarcasm’
Can hipsters stomach the unpalatable Guardian 3655 0 396 44 773 69
truth about avocado toast?
Tim Kaine skewers Donald Trump's MSNBC 1094 111 6 12 2 26
military policy
Top 5 Most Antisemitic Things Hillary Breitbart 1067 7 134 35 22 372
Clinton Has Done
17 Hilarious Tweets About Donald Buzzfeed 11390 375 16 4121 4 5
Trump Explaining Movies
Go deeper: ResNet
Convolutional Layers with shortcuts
He et al. Deep Residual Learning
for Image Recognition
Go deeper: ResNet
input_layer = ...
layer = Convolution1D(128, 5, activation='linear')
(input_layer)
layer = BatchNormalization()(layer)
layer = Activation('relu')(layer)
layer = Convolution1D(128, 5, activation='linear')(layer)
layer = BatchNormalization()(layer)
layer = Activation('relu')(layer)
block_output = merge([layer, input_layer], mode='sum')
block_output = Activation('relu')(block_output)
%
Dense
The Guardian Dense
(1-of-K)
News Org
MaxPooling
ResNet Block
… Conv (128) x 10
ResNet Block
It's a fact: Trump has tiny hands.
(EMBEDDING_DIM=300)
Title + Message
Cherry-picked predicted response
distribution*
Sentence Org Love Haha Wow Sad Angry
Trump wins the election Guardian 3% 9% 7% 32% 49%
Trump wins the election Breitbart 58% 30% 8% 1% 3%
*Your mileage may vary. By a lot. I
mean it.
Tips and Tricks
Initialization
● Break symmetry:
– Never ever initialize all your weights to
the same value
● Let initialization depend on activation
function:
– ReLU/PReLU → He Normal
– sigmoid/tanh → Glorot Normal
Choose an adaptive optimizer
Choose an adaptive optimizer
Source: Alec Radford
Choose the right model size
● Start small and keep adding layers
– Check if test error keeps going down
● Cross-validate over the number of units
● You want to be able to overfit
Y. Bengio (2012) Practical
recommendations for gradient-based
training of deep architectures
Don't be scared of overfitting
● If your model can't overfit, it also can't learn enough
● So, check that your model can overfit:
– If not, make it bigger
– If so, get more date and/or regularize
Source: wikipedia
Regularization
● Norm penalties on hidden layer weights, never
on first and last
● Dropout
● Early stopping
Size of data set
● Just get more data already
● Augment data:
– Textual replacements
– Word vector perturbation
– Noise Contrastive Estimation
● Semi-supervised learning:
– Adapt word embeddings to your domain
Monitor your model
Training loss:
– Does the model converge?
– Is the learning rate too low or too high?
Training loss and learning rate
Source: Andrej Karpathy
Monitor your model
Training and validation accuracy
– Is there a large gap?
– Does the training accuracy increase
while the validation accuracy
decreases?
Training and validation accuracy
Source: Andrej Karpathy
Monitor your model
● Ratio of weights to updates
● Distribution of activations and gradients
(per layer)
Hyperparameter optimization
After network architecture, continue with:
– Regularization strength
– Initial learning rate
– Optimization strategy (and LR decay
schedule)
Friends don't let friends do a full grid search!
Hyperparameter optimization
Friends don't let friends do a full grid search!
– Use a smart strategy like Bayesian
optimization or Particle Swarm Optimization
(Spearmint, SMAC, Hyperopt, Optunity)
– Even random search often beats grid search
Keep up to date: arxiv-sanity.com
We are hiring!
DevOps & Front-end
NLP engineers
Full-stack Python engineers
www.textkernel.com/jobs
Questions?
Source: http://visualqa.org/
