Sign Language Recognition System using TensorFlow in Python

Sign language is a important mode of communication for individuals with hearing impairments. Building an automated system to recognize sign language can significantly improve accessibility and inclusivity.

In this article we will develop a Sign Language Recognition System using TensorFlow and Convolutional Neural Networks (CNNs) .

Step 1: Importing Libraries

In the first step we will import all the necessary libraries. Python libraries simplify data handling and machine learning tasks.

import string
import pandas as pd
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from tensorflow.keras.preprocessing.image import ImageDataGenerator

Step 2: Load the Dataset

The dataset is available as two CSV files, sign_mnist_train.csv and sign_mnist_test.csv. Each row in the CSV file is a training sample with the 0th index having the labels from 0-25 and the rest of the row containing the 784-pixel values of a 28 x 28 image. Each pixel value will be in the range [0, 255].

You can download the dataset from here.

df = pd.read_csv('/content/sign_mnist_train.csv')
df.head()

Output:

First five rows of the dataset

Step 3: Data Preprocessing

The dataset has been provided in two files one is for training and the other one is for testing. We will load this data and then one hot encode the labels considering the fact we are not building the classifier for ‘J’ and ‘Z’ alphabet.

• load_data(path): loads and processes the dataset. It reshapes the image data into 28×28 and applies one-hot encoding to the labels.

def load_data(path):
    df = pd.read_csv('/content/sign_mnist_train.csv')
    y = np.array([label if label < 9
                  else label-1 for label in df['label']])
    df = df.drop('label', axis=1)
    x = np.array([df.iloc[i].to_numpy().reshape((28, 28))
                  for i in range(len(df))]).astype(float)
    x = np.expand_dims(x, axis=3)
    y = pd.get_dummies(y).values

    return x, y

X_train, Y_train = load_data('/content/sign_mnist_train.csv')
X_test, Y_test = load_data('/content/sign_mnist_test.csv')
X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.2, random_state=42)

Now let’s check the shape of the training and the testing data.

print(X_train.shape, Y_train.shape)
print(X_test.shape, Y_test.shape)

Output:

(21964, 28, 28, 1) (21964, 24)
(27455 28, 28, 1) (27455, 24)

Step 4: Data Visualization

In this section, we will try to visualize images for signs of some of the alphabets which have been provided to us to build the classifier for each class.

class_names = list(string.ascii_lowercase[:26].replace(
    'j', '').replace('z', ''))

plt.figure(figsize=(10, 10))
for i in range(10):
    plt.subplot(5, 5, i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(X_train[i].squeeze(), cmap=plt.cm.binary)
    plt.xlabel(class_names[np.argmax(Y_train, axis=1)[i]])
plt.tight_layout()
plt.show()

Output:

Visualized images of the signs corresponding to various alphabet classes.

Step 5: Model Development

From this step onward, we use TensorFlow to build our CNN model. The Keras framework of TensorFlow provides all the functionalities needed to define the architecture of a Convolutional Neural Network and train it on the data.

We will implement a sequential model containing the following parts:

• Three Convolutional Layers followed by MaxPooling layers.
• Flatten Layer to flatten the output of the convolutional layers.
• Fully Connected Layers followed by the output layer.
• BatchNormalization Layers to enable stable and fast training.
• Dropout Layer to avoid overfitting.

The final output layer will have 24 units corresponding to the 24 alphabet classes.

model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(filters=32,
                           kernel_size=(3, 3),
                           activation='relu',
                           input_shape=(28, 28, 1)),
    tf.keras.layers.MaxPooling2D(2, 2),

    tf.keras.layers.Conv2D(filters=64,
                           kernel_size=(3, 3),
                           activation='relu'),
    tf.keras.layers.MaxPooling2D(2, 2),

    tf.keras.layers.Flatten(),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(24, activation='softmax')
])
model.summary()

Output:

When compiling the model, we provide three essential parameters:

• Optimizer: The method used to optimize the cost function (e.g., gradient descent).
• Loss Function: The function used to evaluate the model’s performance.
• Metrics: Metrics used to evaluate the model during training and testing.

model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

Now we will train the model using the training data and validate it using the validation data, utilizing model.fit() function.

history = model.fit(
    X_train, Y_train, 
    validation_data=(X_val, Y_val), epochs=10, batch_size=32, verbose=1)

Output:

Step 6: Model Evaluation

After training the model, we will visualize the training and validation accuracy as well as the loss for each epoch. This helps us analyze how well the model is performing.

history_df = pd.DataFrame(history.history)
history_df.loc[:,['loss','val_loss']].plot()
history_df.loc[:,['accuracy','val_accuracy']].plot()
plt.show()

Output:

The training loss keeps decreasing which means the model is learning well. However, the validation loss is fluctuating a little suggesting the model might be overfitting slightly. The training accuracy is very high, close to 100% while validation accuracy also improves but is slightly lower. This means the model performs well on training data but may need fine-tuning to improve generalization.

Finally, let’s evaluate the performance of the model on the test dataset using model.evaluate() function.

loss, accuracy = model.evaluate(X_test, Y_test, verbose=1)
print(f'Test Accuracy: {accuracy * 100:.2f}%')

Output:

The model achieves 99% accuracy on the test set, which is impressive for a simple CNN model.

By using just a simple CNN model we are able to achieve an accuracy of 99% which is really great. This shows that this technology is certainly going to help us build some amazing applications which can be proved a really great tool for people with some special needs.