Word embeddings are numerical representations of words that enable machine learning models to process and understand human language. Traditional embedding techniques such as Word2Vec and GloVe assign a single fixed vector to each word, regardless of the context in which it appears. As a result, words with multiple meanings are represented by the same embedding, limiting their ability to capture contextual information.
- ELMo generates different vector representations for the same word based on its surrounding context.
- Combines information from multiple layers of a bidirectional language model.
Word2Vec vs ELMo vs BERT / RoBERTa
| Feature | Word2Vec | ELMo | BERT / RoBERTa |
|---|---|---|---|
| Embedding Type | Generates a single fixed vector for each word. | Generates context-dependent embeddings using the entire sentence. | Generates deep contextual embeddings using Transformer encoders. |
| Context Awareness | Same embedding regardless of context. | Embedding changes based on surrounding words. | Captures context from the complete sentence using self-attention. |
| Architecture | Shallow neural network (CBOW or Skip-gram). | Bidirectional LSTM language model (biLM). | Transformer-based encoder architecture. |
| Input Representation | Word-level tokens only. | Character-based input with contextual word representations. | Subword tokens using WordPiece or Byte-Pair Encoding (BPE). |
| Handling Multiple Meanings | Cannot distinguish different meanings of the same word. | Assigns different embeddings based on context. | Effectively captures multiple meanings through contextual encoding. |
| Training Objective | Predicts surrounding words or the target word. | Learns forward and backward language modeling. | Uses masked language modeling (and next sentence prediction in BERT). |
| Fine-tuning | Typically used as fixed pretrained embeddings. | Can be integrated and optionally fine-tuned for downstream tasks. | Specifically designed for end-to-end fine-tuning on downstream tasks. |
| Model Complexity | Low | Moderate | High |
Working of ELMo
Step 1: Character-based Word Representation
- Instead of directly assigning a fixed vector to each word, ELMo first represents every word as a sequence of characters.
- A Character Convolutional Neural Network (Character CNN) extracts features such as prefixes, suffixes, and word morphology to create an initial word representation.
- This enables ELMo to effectively handle rare and out-of-vocabulary words.
Step 2: Bidirectional Language Modelling
The character-based representations are then passed through a Bidirectional Language Model (biLM), which consists of two LSTM networks working in opposite directions, as illustrated in the figure.
- The Forward LSTM reads the sentence from left to right and learns the context by predicting the next word.
- The Backward LSTM reads the sentence from right to left and learns the context by predicting the previous word.
Step 3: Multi-layer Contextual Embeddings
- ELMo does not rely only on the final LSTM output.
- Instead, it combines representations from multiple layers of the bidirectional language model, including the character-based embedding layer and the hidden states from each BiLSTM layer.
- Lower layers generally capture syntactic information, while higher layers capture semantic meaning.
- A weighted combination of these layers forms the final ELMo embedding.
Step 4: Integration with Downstream Tasks
- The generated ELMo embeddings are used as input features for downstream NLP models such as text classifiers, named entity recognition systems, question answering models, and sentiment analysis models.
- During training, the downstream model learns how much importance to assign to each ELMo layer, while the pretrained language model can either remain fixed or be fine-tuned depending on the application.
Implementation of ELMo Embeddings
In this implementation, we use TensorFlow and TensorFlow Hub to load the pretrained ELMo model and generate contextualized word embeddings for input sentences.
Step 1: Install Required Libraries
Install TensorFlow and TensorFlow Hub, which provide the pretrained ELMo model and the APIs required to generate contextual embeddings.
pip install tensorflow tensorflow_hub
Step 2: Import Libraries and Load ELMo
- Import TensorFlow for tensor operations and TensorFlow Hub for loading the pretrained ELMo model.
tensorflowis imported as tf to create tensors and execute the model.tensorflow_hubis imported as hub to download and load the pretrained ELMo model.
import tensorflow as tf
import tensorflow_hub as hub
Step 3: Load the Pretrained ELMo Model
- Load the pretrained ELMo model directly from TensorFlow Hub.
hub.load()downloads and loads the pretrained ELMo model.- The loaded model generates 1024-dimensional contextual embeddings for every word in the input sentence.
elmo = hub.load("https://tfhub.dev/google/elmo/3")
Step 4: Define a Function to Generate ELMo Embeddings
- Create a function that accepts one or more sentences and returns their contextual word embeddings.
tf.constant()converts the sentences into TensorFlow tensors.elmo.signatures["default"]passes the sentences through the pretrained model.["elmo"]extracts the contextual embedding tensor produced by the model.
def get_elmo_embeddings(sentences):
embeddings = elmo.signatures["default"](
tf.constant(sentences)
)["elmo"]
return embeddings
Step 5: Create Sample Sentences
- Create two sentences containing the word bank used in different contexts.
- The word bank refers to a financial institution in the first sentence and the edge of a river in the second.
- ELMo generates different embeddings for the word based on its surrounding context.
sentences = [
"The bank approved the loan.",
"He sat on the bank of the river."
]
Step 6: Generate and Display the Embeddings
The output tensor has three dimensions:
- 2 → Number of input sentences.
- 7 → Maximum number of words (after padding) among the input sentences.
- 1024 → Size of the contextual embedding generated for each word.
- Although the word "bank" appears in both sentences, ELMo produces different embedding vectors because it considers the surrounding words while generating the representation.
embeddings = get_elmo_embeddings(sentences)
print("Embedding Shape:", embeddings.shape)
Output:
Embedding Shape: (2, 8, 1024)
You can download the complete code form here.
Applications
- Sentiment Analysis: Identifies the sentiment of words based on their context, improving opinion classification.
- Named Entity Recognition (NER): Distinguishes entities such as people, organizations, and locations using contextual information.
- Question Answering: Produces more accurate answers by understanding the meaning of words within the given passage.
- Machine Translation: Improves translation quality by selecting the contextually appropriate meaning of ambiguous words.
- Text Classification: Enhances document and intent classification by generating richer contextual representations.
- Information Retrieval: Improves search relevance by matching documents based on contextual meaning rather than exact keywords.
Advantages
- Generates different embeddings for the same word based on its surrounding text.
- Correctly distinguishes words with multiple meanings in different contexts.
- Learns representations from characters, making it effective for rare and unseen words.
- Combines information from multiple BiLSTM layers to learn both grammatical and semantic features.
- Can be added as pretrained embeddings to existing NLP models with minimal architectural changes.
Limitations
- BiLSTM layers process tokens sequentially, making training and inference slower than Transformer models.
- Requires more memory and processing time than static embedding methods such as Word2Vec.
- LSTMs may struggle to capture dependencies across very long sentences.
- The 1024-dimensional embeddings increase storage requirements and model size.
- Modern Transformer-based models like BERT and RoBERTa generally achieve better performance on most NLP benchmarks.