Transform Text Features to Numerical Features with CatBoost

Handling text and category data is essential to machine learning to create correct prediction models. Yandex's gradient boosting library, CatBoost, performs very well. It provides sophisticated methods to convert text characteristics into numerical ones and supports categorical features natively, both of which may greatly enhance model performance. This article will focus on how to transform text features into numerical features using CatBoost, enhancing the model's predictive power.

Text Processing in CatBoost

Text features in CatBoost are used to build new numeric features. These features are essential for tasks involving natural language processing (NLP), where raw text data needs to be converted into a format that machine learning models can understand and process effectively.

There are many processes involved in CatBoost's text processing:

• Tokenization: The process of dividing text into relevant tokens.
• Embedding: Changing tokens into vectors of numbers.
• Aggregation: Creating fixed-length numerical characteristics by summing these vectors.

Handling Text Features in CatBoost

When dealing with text features, it is crucial to ensure that the order of columns in the training and test datasets matches. This can be managed by using the Pool method in CatBoost, where columns can be added by name.

Example of Using Text Features:

model.fit(x_train, y_train, text_features=['text'])

For prediction, ensure the text features are correctly specified:

preds_class = model.predict(X_test)

Steps to Transform Text Features to Numerical Features

1. Loading and Storing Text Features

Text features are loaded into CatBoost similarly to other feature types. They can be specified in the column descriptions file or directly in the Python package using the text_features parameter.

2. Preprocessing Text Features

CatBoost uses dictionaries and tokenizers to preprocess text features. The dictionaries define how text data is converted into tokens, while tokenizers break down the text into these tokens.

Example of a Dictionary:

dictionaries = [{
    'dictionaryId': 'Unigram',
    'max_dictionary_size': '50000',
    'gram_count': '1',
}, {
    'dictionaryId': 'Bigram',
    'max_dictionary_size': '50000',
    'gram_count': '2',
}]

Example of a Tokenizer:

tokenizers = [{
    'tokenizerId': 'Space',
    'delimiter': ' ',
}]

3. Calculating New Features

Feature calculators (feature calcers) are used to generate new numeric features from the preprocessed text data. These calculators can include methods like Bag of Words (BoW), Naive Bayes, and others.

Example of Feature Calcers:

feature_calcers = [
    'BoW:top_tokens_count=1000',
    'NaiveBayes',
]

4. Training the Model

Once the text features are preprocessed and new numeric features are calculated, they are passed to the regular CatBoost training algorithm.

Text Features to Numerical Features using CatBoost : Implementation

Step 1: Install CatBoost and Import CatBoost

Ensure you have CatBoost installed:

!pip install catboost

Importing CatBoost

from catboost import CatBoostClassifier, Pool
import pandas as pd

Step 2: Prepare Dataset

We'll illustrate the procedure using an example dataset. Here, categorical characteristics like "City" and "Weather" are present in the dataset:

data = {
    'City': ['New York', 'London', 'Tokyo', 'New York', 'Tokyo'],
    'Weather': ['Sunny', 'Rainy', 'Sunny', 'Snowy', 'Rainy'],
    'Label': [1, 0, 1, 0, 0]
}
df = pd.DataFrame(data)

Step 3: Define Features and Target

Determine the target variable and its characteristics:

X = df[['City', 'Weather']]
y = df['Label']

Step 4: Initialize and Train the Model

Establish categorical characteristics and set the CatBoostClassifier's initialization, To manage the data and indicate which characteristics are categorical, create a Pool object as follows:

categorical_features = ['City', 'Weather']
model = CatBoostClassifier(iterations=100, depth=3, learning_rate=0.1, loss_function='Logloss')
train_pool = Pool(data=X, label=y, cat_features=categorical_features)
model.fit(train_pool)

Step 5: View Transformed Features

During training, CatBoost internally modifies the category characteristics. You may access the feature importances in order to examine the altered features:

importances = model.get_feature_importance(train_pool, prettified=True)
print(importances)

Output:

  Feature Id  Importances
0       City    82.857487
1    Weather    17.142513

Conclusion

Transforming text features into numerical features in CatBoost involves preprocessing text data using dictionaries and tokenizers, calculating new numeric features with feature calcers, and then training the model. This process enhances the model's ability to handle text data effectively, making CatBoost a robust tool for NLP tasks. By following the steps outlined in this article, you can leverage CatBoost's capabilities to transform and utilize text features in your machine learning models, improving their predictive performance.