Sign in Get started

Follow 556K Followers · Editors' Picks Features Deep Dives Grow Contribute About
 Photo by Liam Charmer on Unsplash

A Guide to Image Captioning

How Deep Learning helps in captioning
 Abhijit Roy Dec 9, 2020 · 10 min read

Image Captioning is the process of generating a textual description for

given images. It has been a very important and fundamental task in the
Deep Learning domain. Image captioning has a huge amount of application.
NVIDIA is using image captioning technologies to create an application to
help people who have low or no eyesight.

Several approaches have been made to solve the task. One of the most
notable work has been put forward by Andrej Karpathy, Director of AI, Tesla
in his Ph.D. at Standford. In this article, we will be talking about the most
used and well-known approaches proposed as a solution to this problem.
We will also be looking at a python demo example on the Flickr Dataset in
Python.

So, let’s start.

Image captioning can be regarded as an end-to-end Sequence to Sequence

problem, as it converts images, which is regarded as a sequence of pixels to
a sequence of words. For this purpose, we need to process both the
language or statements and the images. For the Language part, we use
recurrent Neural Networks and for the Image part, we use Convolutional
Neural Networks to obtain the feature vectors respectively.

Now, How does the idea work?

Say, we as humans are seeing a scene as given below.

 Photo by Isabela Kronemberger on Unsplash: FIg 1

If we are told to describe it, maybe we will describe it as: “A puppy on a blue
towel” or “ A brown dog playing with a green ball”. So, how are we doing
this? While forming the description, we are seeing the image but at the
same time, we are looking to create a meaningful sequence of words. The
first part is handled by CNNs and the second is handled by RNNs.

If we can obtain a suitable dataset with images and their corresponding

human descriptions, we can train networks to automatically caption
images. FLICKR 8K, FLICKR 30K, and MS-COCO are some most used
datasets for the purpose.

Now, one issue we might have overlooked here. We have seen that we can
describe the above images in several ways. So, how do we evaluate our
model? For a sequence to sequence problems, like summarization, language
translations, or captioning we use a Metrics called the BLEU score.
BLEU stands for Bilingual Evaluation Understudy. It is a metric for
evaluating a generated sentence to a reference sentence. The perfect match
is 1.0 and a perfect mismatch is 0.0. You can study more about the BLEU
score from this awesome blog post.

We have seen that we need to create a multimodal neural network that uses
feature vectors obtained using both RNN and CNN, so consequently, we will
have two inputs. One is the image we need to describe, a feed to the CNN,
and the second is the words in the text sequence produced till now as a
sequence as the input to the RNN.

We are dealing with two types of information, a language one and another
image one. So, the question arises how or in what order should we
introduce the pieces of information into our model? Elaborately
speaking, we need a language RNN model as we want to generate a
word sequence, so, when should we introduce the image data vectors
in the language model. A paper by Marc Tanti and Albert Gatt, Institute
of Linguistics and Language Technology, University of Malta covered a
comparison study, of all the approaches. Let’s look into the approaches.

Types of Architectures
There are two basic types of architectures:
 Source

The first architecture is called Injecting Architecture and the second one is
called Merging architecture. FF shows feed-forward networks.

In the Injecting Architecture, the image data is introduced along with the
language data, and the image and language data mixture is represented
together. The RNN trains on the mixture. So, at every step of training, the
RNN uses the mixture of both pieces of information to predict the next
word, and consequently, the RNN finetunes image information as well
during training.

In the Merging Architecture, the image data is not introduced in the RNN
network. So, the image and the language information are encoded
separately and introduced together in a feed-forward network, creating a
multimodal layer architecture.

Overall in general, there are 4 approaches to this problem as described by

the paper. The injecting methodology has three variants.
 Source

Init-Inject: Normally in the case of an RNN we use an initial state vector

which is set to a zero vector of the given dimension. In the case of Init-
inject, we obtain the Image feature vector using the CNN network, as the
same size of the RNN hidden state vector and pass the Image vector as the
initial state of the RNN.

Pre-Inject: In the case of the Pre-inject architecture, we pass the image

vector as the 1st word. For RNN, at each step, we need to send the words in
encoded vector versions at each time step as a sequence. So, for the first
time step, we send the image vector.

Par-Inject: In this case, at every step, we merge the word vector and the
image vector into a similar-sized embedding space and pass it to be trained
by the RNNs.

In all these cases, the image feature vectors are generated by architectures
like Inception Networks and complex Resnet structures. One thing common
in all the Inject architectures is, the hidden state of the RNN is effected by
the image vector. This is the point of difference between the inject and
merge architectures.

According to tested works, pre-inject works better than par-inject. Several

studies have also proven that merging architectures works better than
injecting architectures for some cases. Most studies use Bidirectional RNNs
and LSTMs for better results. Pre-trained embeddings like Glove and
word2vec are used in the studies.

Now, having seen the possible kinds of architectures, let’s look at some most
famous architectures proposed and how they work.

Andrej Karpathy’s Proposed Architecture

First, let’s talk about the architecture introduced by Andrej Karpathy in his
Ph.D. I think it is literally the most talked-about approach. I personally like
his approach because it is so similar to the way we as humans will describe a
scene.

Karpathy’s work mostly contributed to two areas:

Agenda 1

Suppose, we as humans are describing the scene, given above (Fig 1). So,
we first try to recognize the objects in the image, like the ball, the dog, and
the towel. Then we create the description, which obviously has the words
representing those objects. Karpathy focussed on the same logic. He
pointed out that every continuous segment of words in the description
corresponds to a particular position in the image, but obviously to the
machine the positions are unknown. So, his work used these mappings to
create a description generation model. Now, if we think of it, we as humans
also create chunks of words seeing some important parts of the image
having objects and then move to form the sentence. The approach given by
the paper is similar.
 Source

The model proposed created a multimodal embedding space using the two
modalities to find the alignments between the segments of the sentences
and their corresponding represented areas in the image. RCNN or Regional
Convolutional Neural Networks were used in the model to detect the object
region and a CNN trained on the ImageNet dataset having 200 classes was
used to recognize the objects. For language modeling, the paper suggests
the use of BRNN or Bidirectional Neural Networks as it best represents the
inter-modal relationships among the n-grams of the sentences. A 300d
word2vec embedding is used for obtaining the word vectors.

The paper established an image-sentence score to obtain a correlation

between the image and the sentence. It is a function of the individual word
and image region score. If the latter is high, then the words are said to have
high support for the image.
 SOurce

The above diagram expresses the approach. The RCNN basically creates a
bounding box, so if we regard it as the i-th region of the image, it’s
confidence is matched with every t-th word in the description. So, for every
region and word pair, the dot product Vt.Si between the i-th region and t-th
word is a measure of similarity.

Source

The formula defines the total sentence score.

Agenda 2:

The second part deals with the development of a Multimodal Recurrent

Neural Network for generating captions.

Source
The above image shows the proposed model. The model takes in the image
pixels I and sequence of input word vectors (x1, x2….., xn) and calculates
the sequence of hidden states (h1,h2……., hn) to give the sequence of
outputs (y1,y2,….., yn). The image feature vectors are passed only once as
the initial hidden state. So, the next hidden state is calculated from the
image vector I, the previously hidden state ht-1, and the current input xt.

The current output yt is produced by using a softmax layer on the given

hidden state activation function.

The above set of equations govern the model.

The above discussed were the contributions of Andrej Karpathy’s paper. I

will be providing the references for all the material used in this post below.

Next, let’s talk about two of the other most commonly used architectures.

Google’s Architecture
The architecture was proposed in a paper titled “Show and Tell: A Neural
Image Caption Generator” by Google in 2k15.

The architecture by Google uses LSTMs instead of plain RNN architecture.

 Source

The above image shows the architecture. The other parts of the functioning
are similar to the functions of the model introduced by Karpathy. The image
feature vector ‘I’ is inserted into the LSTM sequence at t=-1, only once.
After that from t=0, the sequence of word vectors are input. The final
output word of each step is the word with maximum probability obtained
from the activation function of the hidden state at that particular step. The
governing equations are given below.

Source

For obtaining the best prediction for the captions, the Architecture uses a
Beam Search method. The method considers k best sentences as
candidates at each time step t. At every time step t+1, k most probable
words are taken and with k sentences and k words, k² probable sentences
arise, from them again k best probable sentences are chosen for the
candidates of time step t+1.

Microsoft’s Architecture
Microsoft proposed its architecture in the paper titled “Rich Image
Captioning in the Wild”. The architecture was taken one step further from
the two architectures discussed above. The architecture is inspired by
machine translation’s encoder-decoder framework architecture. The paper
stated as most of the image captioning architecture provides generic
content without identifying entities like famous landmarks, celebrities, etc,
their model will take into consideration all these factors separately.

Microsoft architecture uses deep Resnet implementation in the

convolutional networks, and detect if the image is of some famous celebrity
or landmark. If identified, that serves as an input feature to the model along
with the image vector itself.

The main component of the model includes

1. A deep Resnet based model for image feature extraction

2. A language model for caption candidate generation and ranking

3. An entity recognition for landmark and celebrities

4. A classifier to estimate the confidence score.

The last point is another modification by Microsoft. Often during

captioning, the image becomes too hard for generating a caption. In such
situations, the confidence score mechanism helps to determine how much
can we depend on the generated captions.
 Source

The above image gives the visualization of Microsoft's pipeline.

A Practical Implementation on Flickr 8K dataset

The Flickr 8k dataset has 8091 images and for each image, there are 5
descriptions. The dataset can be found at the University of Illinois site.

I have used merging architecture and I have created my own convolutional

network though normally the image feature extractions are done using pre-
trained CNN architectures using transfer learning.
The above is the architecture I used. As you can see I have not trained the
RNN on the image features, so it is a merging architecture.

I have created my own tokenizer mapping by the name “word_map.json” for

the RNN training and used Glove-50d for the embedding layer.

The code can be found here.

I hope this helps.

References
Comparison of Architectures: https://arxiv.org/pdf/1703.09137.pdf

Andrej Karpathy’s Architecture:

https://cs.stanford.edu/people/karpathy/cvpr2015.pdf

Google’s Architecture: https://arxiv.org/pdf/1411.4555.pdf

Microsoft’s Architecture: https://www.microsoft.com/en-us/research/wp-

content/uploads/2016/06/ImageCaptionInWild-1.pdf

More on Image Captioning:

RNN’s Applications:https://arxiv.org/pdf/1708.02043.pdf

Yahoo’s Approach:
https://papers.nips.cc/paper/2019/file/680390c55bbd9ce416d1d69a9ab
4760d-Paper.pdf

Attention implementation in captioning:

https://arxiv.org/pdf/1502.03044.pdf

Sign up for The Variable

By Towards Data Science

Every Thursday, the Variable delivers the very best of Towards Data Science: from
hands-on tutorials and cutting-edge research to original features you don't want to
miss. Take a look.

You'll need to sign in or create an account to receive this

Get this newsletter
newsletter.

Image Captioning

About Write Help Legal