Tokenization Using Spacy

Before we get into tokenization, let’s first take a look at what spaCy is. spaCy is a popular library used in Natural Language Processing (NLP). It’s an object-oriented library that helps with processing and analyzing text. We can use spaCy to clean and prepare text, break it into sentences and words and even extract useful information from the text using its various tools and functions. This makes spaCy a great tool for tasks like tokenization, part-of-speech tagging and named entity recognition.

What is Tokenization?

Tokenization is the process of splitting a text or a sentence into segments, which are called tokens. These tokens can be individual words, phrases, or characters depending on the tokenization method used. It is the first step of text preprocessing and is used as input for subsequent processes like text classification, lemmatization and part-of-speech tagging. This step is essential for converting unstructured text into a structured format that can be processed further for tasks such as sentiment analysis, named entity recognition and translation.

Example of Tokenization

This is the sentence: “I love natural language processing!”

After tokenization: [“I”, “love”, “natural”, “language”, “processing”, “!”]

Each token here represents a word or punctuation mark, making it easier for algorithms to process and analyze the text.

Implementation of Tokenization using Spacy Library

import spacy

# Creating blank language object then
# tokenizing words of the sentence
nlp = spacy.blank("en")

doc = nlp("GeeksforGeeks is a one stop\
learning destination for geeks.")

for token in doc:
    print(token)

Output:

GeeksforGeeks
is
a
one
stop
learning
destination
for
geeks
.

We can also add functionality in tokens by adding other modules in the pipeline using spacy.load().

nlp = spacy.load("en_core_web_sm")

nlp.pipe_names

Output:

['tok2vec', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer', 'ner']

Here is an example to show what other functionalities can be enhanced by adding modules to the pipeline.

import spacy

# loading modules to the pipeline.
nlp = spacy.load("en_core_web_sm")

# Initialising doc with a sentence.
doc = nlp("If you want to be an excellent programmer \
, be consistent to practice daily on GFG.")

# Using properties of token i.e. Part of Speech and Lemmatization
for token in doc:
    print(token, " | ",
          spacy.explain(token.pos_),
          " | ", token.lemma_)

Output:

If  |  subordinating conjunction  |  if
you  |  pronoun  |  you
want  |  verb  |  want
to  |  particle  |  to
be  |  auxiliary  |  be
an  |  determiner  |  an
excellent  |  adjective  |  excellent
programmer  |  noun  |  programmer
,  |  punctuation  |  ,
be  |  auxiliary  |  be
consistent  |  adjective  |  consistent
to  |  particle  |  to
practice  |  verb  |  practice
daily  |  adverb  |  daily
on  |  adposition  |  on
GFG  |  proper noun  |  GFG
.  |  punctuation  |  .

In the example above, we utilized part-of-speech (POS) tagging and lemmatization through the spaCy NLP modules. This allowed us to obtain the POS for each word and convert each token to its base form through lemmatization. Prior to loading the NLP model with “en_core_web_sm”, we would not have had access to this functionality. The en_core_web_sm model is essential as it provides the necessary linguistic features, such as tokenization, POS tagging and lemmatization, enabling these advanced NLP capabilities.

Read More:

• Sentiment Analysis using VADER
• Text Generation using Recurrent Long Short Term Memory Network
• Text Preprocessing in Python