Python - Tokenize text using Enchant

Python – Chunking text using Enchant

Last Updated : 26 May, 2020

Enchant is a module in Python which is used to check the spelling of a word, gives suggestions to correct words. Also, gives antonym and synonym of words. It checks whether a word exists in dictionary or not.

Enchant also provides the enchant.tokenize module to tokenize text. Tokenizing involves splitting words from the body of the text. But at times not all the words are required to be tokenized. Suppose we have and HTML file, on tokenization all the tags will also be included. Usually the HTML tags do not contribute to the content of the article, therefore there is a need to tokenize excluding them.

Currently the only implemented chunker is HTMLChunker. A chunker for LaTeX documents is in the works.

# import the required modules 
from enchant.tokenize import get_tokenizer 
from enchant.tokenize import HTMLChunker 
  
# the text to be tokenized 
text = "<div> <h1> Geeks for Geeks </h1> <br> </div>"
  
# getting tokenizer class 
tokenizer = get_tokenizer("en_US") 
  
# printing tokens without chunking 
print("Printing tokens without chunking:") 
token_list = [] 
for words in tokenizer(text): 
    token_list.append(words) 
print(token_list) 
  
  
# getting tokenizer class with chunk 
tokenizer_chunk = get_tokenizer("en_US", chunkers = (HTMLChunker, )) 
  
# printing tokens after chunking 
print("\nPrinting tokens after chunking:") 
token_list_chunk = [] 
for words in tokenizer_chunk(text): 
    token_list_chunk.append(words) 
print(token_list_chunk) 

Output :

Printing tokens without chunking:
[(‘div’, 1), (‘h’, 7), (‘Geeks’, 11), (‘for’, 17), (‘Geeks’, 21), (‘h’, 29), (‘br’, 34), (‘div’, 40)]

Printing tokens after chunking:
[(‘Geeks’, 11), (‘for’, 17), (‘Geeks’, 21)]

Python - Tokenize text using Enchant

Y

Yash_R

Improve

Article Tags :

Practice Tags :

python

Similar Reads

Python - Tokenize text using Enchant

Enchant is a module in Python which is used to check the spelling of a word, gives suggestions to correct words. Also, gives antonym and synonym of words. It checks whether a word exists in dictionary or not. Enchant also provides the enchant.tokenize module to tokenize text. Tokenizing involves spl

Python - Spelling checker using Enchant

Enchant is a module in Python, which is used to check the spelling of a word, gives suggestions to correct words. Also, gives antonym and synonym of words. It checks whether a word exists in the dictionary or not. Enchant can also be used to check the spelling of words. The check() method returns Tr

Python - Filtering text using Enchant

Enchant is a module in Python which is used to check the spelling of a word, gives suggestions to correct words. Also, gives antonym and synonym of words. It checks whether a word exists in dictionary or not. Enchant also provides the enchant.tokenize module to tokenize text. Tokenizing involves spl

Extracting text from HTML file using Python

Extracting text from an HTML file is a common task in web scraping and data extraction. Python provides powerful libraries such as BeautifulSoup that make this task straightforward. In this article we will explore the process of extracting text from an HTML file using Python. Use the below command t

enchant.get_enchant_version() in Python

Enchant is a module in python which is used to check the spelling of a word, gives suggestions to correct words. Also, gives antonym and synonym of words. It checks whether a word exists in dictionary or not. enchant.get_enchant_version() enchant.get_enchant_version() is an inbuilt method of enchant

Wand text() function in Python

Text can also be added using wand.drawing object. text() function is used to add text in the Drawing object. It takes x and y coordinates and string that we want to write on (x, y) position. Syntax : wand.drawing.text(x, y, body) Parameters : ParameterInput TypeDescriptionxnumbers.Integralthe baseli

Python - Find the Levenshtein distance using Enchant

Levenshtein distance between two strings is defined as the minimum number of characters needed to insert, delete or replace in a given string string1 to transform it to another string string2. Examples : Input : string1 = "geek", string2 = "gesek" Output : 1 Explanation : We can convert string1 into

enchant.request_dict() in Python

Enchant is a module in python which is used to check the spelling of a word, gives suggestions to correct words. Also, gives antonym and synonym of words. It checks whether a word exists in dictionary or not. enchant.request_dict() enchant.request_dict() is an inbuilt method of enchant module. It is

Story Generator App Using Python

In the realm of programming and natural language processing, there's an opportunity to create engaging and creative applications that spark the imagination. One such project is the development of a Story Generator App using Python. In this article, we'll embark on a journey to understand how to buil

Clean Web Scraping Data Using clean-text in Python

If you like to play with API's or like to scrape data from various websites, you must've come around random annoying text, numbers, keywords that come around with data. Sometimes it can be really complicating and frustrating to clean scraped data to obtain the actual data that we want. In this artic