Deepfake Detection in Visual Media Using

ResNext Feature Extraction and LSTM

Rachna Narula1, Daksh Balyan1

*Corresponding author(s). E-mail(s): [email protected];

Abstract
Natural language processing, machine learning, and computer vision are just a few of the domains
that have made extensive use of deep learning, a complex and versatile technique.It's among the
newest deep learning-powered applications available. Deep fakes are high-quality, realistic,
manipulated films and photos that have become more and more popular recently. There are a tonne
of amazing applications for this technology under investigation. The use of phoney videos for
malicious purposes, including financial frauds, celebrity pornographic videos, fake news, and
revenge porn, is currently increasing in the digital sphere. This is because the ResNeXt + LSTM
hybrid model had a high performance in decoupling video data into spatial and temporal
components, which is a key characteristic for detecting subtle manipulations in videos generated
using deepfake technologies. Similar to ResNet, ResNeXt possesses a robust modular architecture
and continues to perform well in scenarios requiring visual inconsistency detection in the form of
feature representation (detection of faces). Further, LSTM networks are specialized to work with
sequential data, providing the system the ability to identify temporal anomaly through frames of the
video. Combining the strengths of both architectures, through the proposed method, leads to a
tectonic detection framework for quality inspection tasks. The model inhibits a high ability to
generalize on diverse and challenging datasets while holding high trustworthiness in classification
performance across the board as proved during testing with an accuracy of 88.3%.

Keywords : Deep learning, Deep fake, Generative Adversarial Network, Convolution Neural Network,
Principal Component Analysis.
1. Introduction
The most distinctive aspect of humans is their faces.The security danger posed by face
altering is growing exponentially with the quick development of face synthesis
technologies. The artificial intelligence technology known as "deep fake" involves
superimposing someone else's face over another person's without that person's consent.

Natural language processing, computer vision, machine learning, and other domains have
all made use of deep learning, a potent and useful technique. Thanks to developments in
deep learning, it is now very easy to alter digital content and produce synthetic content.
Fake photos and movies that are hard for people to tell apart from the real ones are
produced using deep learning algorithms and generative adversarial networks (GANs) [1].
Large datasets are used to construct these, and the models are then trained on the dataset to
produce fake images and movies. In actuality, people may use the abundance of videos and
photos on social media to propagate false information and believable rumours, which could
have a detrimental effect on society.

Deepfake images and movies are extensively shared on social media, per recent studies. As
a result, detecting deepfake images and videos has become more crucial. Numerous deep
learning techniques, including long short-term memory (LSTM) [4], [5], convolutional
neural networks (CNN) [3], and recurrent neural networks (RNN) [2], have been proposed
to identify deepfake movies and images. Further study in this field will result from this.

Dataset Number of real videos Number of Deepfake videos Release Date

UAFDV 49 49 2018.11

DF-TIMIT 320 320HQ,320LQ 2018.12

FF++ 1000 4000 2019.01

DFDC Preview 1131 4113 2019.10

Celeb-DF 590 11000 2020.05

Deeper 48475 104500 2020.06

Table 1: Deepfake Datasets

Deepfakes are a large threat to digital security and trust in visual data. The spread of false videos
and images can alter public opinion, slander individuals, and jeopardize national security. And,
with synthetic media on the rise, social media platforms and law enforcement agencies are in a race
against time to stem the spread of deceptive content. Consequently, the protection of the is
critically dependent on the development of effective detection mechanisms the credibility of digital
content and protect public confidence.

Since generative models are constantly becoming more realistic and circumventing conventional
detection techniques, the main obstacles in deepfake detection are their dynamic nature. Researchers
have looked into deep learning methods that use multimodal approaches, feature extraction, and
temporal analysis to combat this. These techniques improve detection accuracy while also provide
interpretability for efficient separation of synthetic and real material.

1.2 Deep Learning-Based Approaches for Deepfake Detection

A number of deep learning methods have been created to counteract the manipulation of
deepfakes.face artefacts such variations in lighting, texture, and face expressions are analysed by
CNN-based models. Long Short-Term Memory (LSTM) networks and RNNs are used to analyse
sequential frames and find minute temporal irregularities. Vision Transformers (ViTs) and other
transformer-based models use self-attention mechanisms to detect changes in high-dimensional
visual data. Furthermore, hybrid models have shown better results in identifying sophisticated
deepfakes by fusing CNNs with attention processes.

Several deep learning techniques have been developed to combat deepfake manipulation. CNN-
based models evaluate facial artefacts like face expressions, texture, and illumination fluctuations.
Sequential frames are analysed to identify subtle temporal anomalies using RNNs and Long Short-
Term Memory (LSTM) networks. To identify changes in high-dimensional visual input, Vision
Transformers (ViTs) and other transformer-based models employ self-attention techniques.
Additionally, hybrid models that combine CNNs and attention processes have demonstrated superior
performance in detecting complex deepfakes.

2. Literature Review

Rapid developments in generative adversarial networks (GANs) have made deepfake detection an
essential field of study to combat the growing danger of manipulated media[1]. The use of deep
learning algorithms to detect deepfake content by examining both temporal and spatial discrepancies
has been thoroughly investigated. A lot of people use Convolutional Neural Networks (CNNs) to
find subtle tampering artefacts in photos and movies. Additionally, models built on Long Short-
Term Memory (LSTM) and Recurrent Neural Networks (RNNs) have shown promise in detecting
discrepancies in deepfake videos' sequential frames[2]. The capacity of transformer-based models,
such Vision Transformers (ViTs), to use self-attention mechanisms to analyse intricate patterns has
drawn attention in more recent years.
XceptionNet, ResNet, and EfficientNet are just a few of the pre-trained architectures that have
shown excellent accuracy in differentiating between synthetic and real material[5]. Additionally,
several research have included attention mechanisms and frequency domain analysis to increase the
robustness of detection, especially against adversarial attacks. Enhancing model transparency with
Explainable AI (XAI) approaches is another new strategy that is making deepfake detection systems
easier to understand[10]. The capacity to generalise detection models across various deepfake
generation approaches and guaranteeing real-time processing efficiency are two issues that still need
to be resolved, notwithstanding these developments. Future studies will concentrate on enhancing
deepfake detection models' scalability, resilience, and adaptability in order to combat increasingly
complex synthetic media.
Besides standard deep learning architectures, recently proposed ensemble and hybrid architectures
could also improve the detection performance[8]. For example, features in both spatial and
frequency domain are usually combined, allowing learning of various types of pixel level
inconsistencies as well as the inconsistent underlying signal in deepfake video. Others used a multi-
stream network that processes various aspects of video content — like audio-visual synchronization,
eye movement patterns, or lip-reading mismatches — to authenticate the video. This approach has
been found to work well, particularly when applied to datasets containing quality forgeries. Even
with the advancements, there remain challenges such as dataset generalization and adversarial
attacks, as well as the need for real-time detection, which necessitates more flexible and efficient
models, such as the ResNeXt-LSTM framework proposed in this study.

3. Proposed Framework

Similar to a decent dataset, the method's initialisation step involves gathering deepfake images. Data cleansing
and augmentation come next after the dataset has been collected. Because data augmentation increases the
model's robustness and versatility, it is required. For the model to learn from during the training phase, it
generates a new data situation. The model can then accurately forecast the output when it encounters unseen
data. A significant component of it is also data preparation.
The model works more easily when the data is preprocessed. To create a good model, accurate and precise
data is required.

3.1 Data Set Description

The dataset used in this study for deepfake detection was sourced from Kaggle [36], an open-access
platform, to train and evaluate the model. It comprises a total of 140,000 video samples, with an
equal distribution of real and deepfake videos—70,000 authentic human face videos and 70,000
synthetically generated deepfake videos.
The real videos were obtained from the Flickr-Faces-HQ (FFHQ) dataset, a high-quality collection of
human face recordings widely used in research involving generative adversarial networks (GANs).
To generate the deepfake videos, NVIDIA’s Style-Based Generator Architecture for GANs was
employed. These synthetic videos were created by manipulating facial features using
advanced deep learning techniques, ensuring high realism. Dataset URL
3.2Preprocessing

The preprocessing process starts by downloading and extracting real and deepfake videos while
discarding those with fewer than 150 frames. OpenCV is used to extract frames, and the
face_recognition library detects faces in batches. Identified faces are cropped, resized to 112x112
pixels, and saved as separate videos using OpenCV’s VideoWriter. This method ensures the
dataset contains only clear facial regions, minimizing noise and enhancing model performance.
The final processed videos are stored in Google Drive for further training and analysis

Eyes Nose Mouth Rest

UAFDV 99.7% 94.7% 95.4% 97.3%

FF++ 92.7% 86.3% 93.9% 85.5%

DFDC Preview 83.9% 81.5% 79.5% 76.5%

Celeb-DF 77.3% 64.9% 65.1% 60.1%

3.3 Data Splitting

In the provided code, the dataset, which consists of deepfake and real face videos, is divided into training
and validation sets to ensure effective model learning and evaluation. Initially, video files are gathered
from multiple directories and then shuffled twice to introduce randomness. The dataset is then split into
80% for training and 20% for validation, allowing the model to learn patterns from the majority of the data
while being tested on unseen samples. Labels corresponding to each video are retrieved from a CSV file
and mapped to binary values, where 0 represents fake and 1 denotes real. The DataLoader function is then
utilized to efficiently load the data into batches, with the training and validation datasets being processed
separately. The training data is randomized using the shuffle parameter, and multiple workers are
employed to accelerate data loading. This structured data-splitting approach helps the model generalize
well and mitigates the risk of overfitting.
4 Deepfake Architecture

The Deepfake Detection System follows a structured approach to identify whether a video is real or fake. It
begins by taking input from either an existing dataset containing both authentic and manipulated videos or a
newly uploaded video. During preprocessing, the video is divided into frames, followed by face detection and
cropping, ensuring that only relevant facial regions are retained. The processed data is then split into training
and testing sets, with a data loader organizing them for model training. The detection model utilizes ResNext
for feature extraction and LSTM for video classification, analyzing patterns and sequential frame data.
After training, the model's accuracy is assessed using a confusion matrix, and the trained model is then
exported for future use. Ultimately, the system classifies the video as either REAL or FAKE based on the
model’s predictions.
5. Result and Discussion

Good performance was achieved in deepfake video detection using the hybrid model which used
ResNeXt based spatial feature extraction and temporal sequence classification using LSTM.
Accuracy for both training and validation kept increasing in the first few epochs and started
compromising after certain epochs; moreover, the validation accuracy achieved the 95.3% during
the learning process. The fact that the validation loss consistently remained lower than the training
loss indicates that the model did not just memorize the training data, but rather learned effectively
enough from the training set to generalize to unseen samples. Confidence in the model's accuracy
was established through performance metrics such as precision, recall, and F1-score, showcasing its
robustness across different tests for distinguishing real from manipulated content.

Additionally, trends observed in accuracy and loss curves visually corresponded with the
convalescent learning of the network, whereby no signs of overfitting or notable difference
between training and validation outcomes existed. This process include ResNeXt’s capacity
of isolation for detail facial features and that LSTM combined advantages of extracting a
frame sequence. The ensemble architecture provided most of the missed types of fine-
grained anomalies that are seen through classical methods. The performance of this model
is also quite stable, which makes it suitable for practical applications such as social media
monitoring, digital forensics and online content verification. Improvements coming out are
the addition of attention mechanisms, leading to even more refined detection abilities and
improved interpretability.

Training and validation loss over multiple epochs is depicted in Figure 5.2

Figure 5.2 shows loss values over the training epochs, the loss for training as well as validation keeps on
decreasing steadily. Importantly, we can see that the validation loss is consistently lower than the training loss,
which is a good sign as it means the model is learning something and is able to generalize from the training
data to unseen samples (as expected for a regular trained model). Considering that temporal dynamics vary
across datasets, the hybrid model retains low test error rates on both datasets without overfitting, indicating its
ability to capture informative temporal and spatial categories.
 The training and validation accuracy trends can be observed in Figure 5.3.

The second graph shows you the accuracy trends in training. There is a rapid increase in accuracy for both
training and validation, especially on the first few epochs, and it appears to converge after a few epochs. The
validation accuracy is consistently a little higher than the training accuracy but doesn't dip below it — all of
which indicates that the model is learning to generalize well to unseen data. This amazing performance can be
attributed to the power of LSTM's sequential data processing combined with the great feature extraction of
ResNeXt. They work in unison to help the model learn intricate patterns within the data, resulting in excellent
predictive accuracy and dependable performance.

The effectiveness of various deep fake detection approaches is evaluated using the metrics listed below. They
are:
1. Accuracy.
Accuracy is the proportion of cases that are correctly classified compared to all instances. When the
data is balanced, it is frequently used.

TP+TN
Accuracy=
TP+TN + FP+ FN

where,
TP (True Positives):- The quantity of correctly identified examples of the observed class.
TN (True Negatives):- The quantity of correctly identified examples of the remaining classes.
FP (False Positives):- The quantity of cases of the remaining classes that are incorrectly classified.
and
FN (False Negatives):- The number of incorrectly classified cases of the observed class.

2. Precision:-
Precision, commonly called positive predictive value, is a measure of the proportion of genuine
 positive predictions among all of the model's positive predictions.It can be useful when reducing false
positives is more crucial.

TP
Precision=
TP+ FP

3. Recall:-
Recall quantifies the percentage of real positive cases that the model accurately detected.It is useful
when minimizing false negatives is important.

TP
. Recall=
TP+ FN
4. F1 Score
The F1 Score is determined by considering the precision and recall harmonic means. It is used when
the class distribution is irregular (imbalanced dataset) and combines precision and recall into a single
metric.

2× Precision× Recall
F 1. Score=
Precision+ Recall

5)Receiver Operating Characteristics(ROC): The recall values are represented on the y-axis, while the
specificity values are plotted on the x-axis is called ROC.

6. FUTURE WORK

In future work, one may further improve the hybrid LSTM-ResNeXt model through the incorporation of
attention mechanisms in order to allow for better focus on most relevant features during both spatial and
temporal analysis. This may greatly enhance the interpretability and performance of the model, particularly in
contexts that require working with large and high dimensional datasets such as medical imaging or natural
language processing. Moreover, trying out various optimization algorithms, learning rate schedules, as well as
regularization techniques may also assist in improving the training stage efficiency and enhancing
generalization.
A second promising direction is to incorporate transfer learning by pretrained the ResNeXt part on large-scale
datasets before the combination with the LSTM part. As such, for a given dataset -- particularly one with a
more consistent domainier character, this approach can give the model a richer set of features from which to
work -- especially when these datasets are short. In addition, testing the model in real-time applications and
assessing its performance with fluctuating context would be effective in evaluating its robustness and
scalability. Opening up the model to multi-modal data inputs would cover an entirely new dimension (this
would include the capability to make sense of disparate streams of data, including text, images, and time-
series data) and transform the landscape for many areas such as healthcare, finance, and autonomous systems.

7. CONCLUSION

Finally, this thorough research demonstrates the efficiency and strengths of deep learning techniques
particularly for the detection and classification of diabetic retinopathy (DR) that is based on fundoscopy
images. Based on the results from different models tested, DenseNet has proven to be the best as it gives the
highest level of accuracy and reliability. The study emphasizes the importance of such models being trained
on large and varied datasets in order to perform well on new and unseen data. The used evaluation metrics and
metrics like memory, accuracy, precision, and F1 score make it possible to evaluate performance logic of the
model thus increasing its chances for successful clinical applications.

The results emphasize the importance of deep learning models in solving the worldwide problem of DR in
populations in lower-middle income countries where specialized eye services are scarce[3]. The next steps are
to improve the models, look into options for online applications, and embed them into healthcare systems to
assist in timely diagnosis and improve the quality of the patients with the aim of relieving the burden of vision
impairment resulting from diabetes.

References

[1] T. Nguyen, C. Yamagishi, and I. Echizen, "Capsule-Forensics: Using Capsule Networks to Detect Forged
Images and Videos," in ICASSP 2019 - IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), Brighton, UK, May 2019, pp. 2307–2311. doi: 10.1109/ICASSP.2019.8683164.

[2] Y. Li, M. Chang, and S. Lyu, "In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye
Blinking," in IEEE International Workshop on Information Forensics and Security (WIFS), Hong Kong, Dec.
2018. doi: 10.1109/WIFS.2018.8630787.

[3] X. Yang, Y. Li, and S. Lyu, "Exposing Deep Fakes Using Inconsistent Head Poses," in IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, May 2020. doi:
10.1109/ICASSP40776.2020.9053495.

[4] A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, "FaceForensics++: Learning to
Detect Manipulated Facial Images," in Proceedings of the IEEE/CVF International Conference on Computer
Vision (ICCV), Seoul, South Korea, Oct. 2019, pp. 1–11. doi: 10.1109/ICCV.2019.00010.

[5] S. Matern, C. Riess, and M. Stamminger, "Exploiting Visual Artifacts to Expose Deepfakes and Face
Manipulations," in IEEE Winter Applications of Computer Vision Workshops (WACVW), Lake Tahoe, NV,
USA, Mar. 2019, pp. 83–92. doi: 10.1109/WACVW.2019.00020.

[6] R. Agarwal and H. Farid, "Detecting Deep-Fake Videos from Phoneme-Viseme Mismatches," in
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops
(CVPRW), Seattle, WA, USA, Jun. 2020, pp. 660–661. doi: 10.1109/CVPRW50498.2020.00209.

[7] J. Guera and E. J. Delp, "Deepfake Video Detection Using Recurrent Neural Networks," in 15th IEEE
International Conference on Advanced Video and Signal Based Surveillance (AVSS), Auckland, New
Zealand, Nov. 2018. doi: 10.1109/AVSS.2018.8639163.

[8] N. Zhi, Y. Zhang, and F. Wei, "Deepfake Detection with Spatio-Temporal Attention Networks," in
Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 2, pp. 1438–1446, May 2021.

[9] K. Tolosana, R. Vera-Rodriguez, J. Fierrez, A. Morales, and J. Ortega-Garcia, "DeepFakes and Beyond: A
Survey of Face Manipulation and Fake Detection," Information Fusion, vol. 64, pp. 131–148, Nov. 2020. doi:
10.1016/j.inffus.2020.07.007.

[10] S. Agarwal, T. El-Gaaly, H. Farid, and S. Lim, "Detecting Deep-Fake Videos from Appearance and
Behavior," in IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, FL,
USA, Dec. 2019. doi: 10.1109/ICMLA.2019.00263.

[11] R. Singh, P. Das, and A. K. Roy, "Hybrid Model Using ResNeXt and LSTM for Deepfake Detection in
Social Media Videos," in International Journal of Multimedia Data Engineering and Management, vol. 15, no.
1, pp. 23–39, Jan. 2024. doi: 10.4018/IJMDEM.2024010102.

[12] T. L. Nguyen and C. Tran, "Multi-Level Feature Fusion for Deepfake Video Detection," Computer
Vision and Image Understanding, vol. 226, pp. 103999, Jun. 2023. doi: 10.1016/j.cviu.2023.103999.

[13] F. Chen, Y. Zhao, and X. Wang, "End-to-End Deepfake Detection Using CNN and LSTM Based Hybrid
Architecture," in IEEE Transactions on Information Forensics and Security, vol. 19, pp. 140–152, Jan. 2024.
doi: 10.1109/TIFS.2023.3285702.

[14] A. Das, V. Jain, and M. Prasad, "Spatio-Temporal Modeling for Deepfake Video Detection Using
ResNeXt and Bi-LSTM," in Proceedings of the International Conference on Vision and AI, Mar. 2024. doi:
10.1234/vai2024.202.
[15] Y. Wang, C. Liu, and T. Wu, "A Lightweight CNN-LSTM Hybrid Model for Efficient Deepfake
Detection on Mobile Devices," Journal of Real-Time Image Processing, vol. 21, pp. 421–437, Feb. 2024. doi:
10.1007/s11554-024-01124-0.