Proceedings of the International Conference on Sustainable Computing and Smart Systems (ICSCSS 2023)

IEEE Xplore Part Number: CFP23DJ3-ART; ISBN: 979-8-3503-3360-2

XLNet Transfer Learning Model for Sentimental

Analysis
2023 International Conference on Sustainable Computing and Smart Systems (ICSCSS) | 979-8-3503-3360-2/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICSCSS57650.2023.10169445

C.R.Dhivyaa K.Nithya G.Sendooran

Assistant Professor Assistant Professor PG Scholar
Computer Science and Engineering Computer Science and Engineering Computer Technology-PG
Kongu Engineering College Kongu Engineering College Kongu Engineering College
Erode,India Erode,India Erode,India
[email protected] [email protected] [email protected]

R.Sudhakar K.Sathis Kumar S.Santhoshkumar

Assistant Professor Assistant Professor grade II Research Scholar
Computer Science and Engineering Computer Science and Engineering Computer Science and Engineering
Nandha College of Technology Bannari Amman Institute of Technology Kongu Engineering College
Erode,India Sathyamangalam, India Erode,India
[email protected] [email protected] [email protected]

Abstract— In natural language processing, an important approaches to sentiment analysis relied on handcrafted
objective is to perform sentiment analysis, which involves features and machine learning techniques like SVM and
categorizing textual content based on whether it expresses a Naive Bayes classifiers. However, these methods were
positive, negative, or neutral sentiment. Sentiment analysis limited by the quality and quantity of labeled data and the
systems face challenges such as ambiguity, subjectivity, complexity of the relationships between words and contexts.
contextual understanding, and domain adaptation. These
challenges make accurately determining sentiment in text a In machine learning-based sentiment analysis, a labeled
complex task. To address these challenges, the proposed dataset is used to train a model to classify new text into one
objective for sentiment analysis on movie review datasets is to of the three sentiment categories. The input features used for
develop a transfer learning-based XLNet model. The the model can include word frequency, word embeddings, or
utilization of transformer-based models has resulted in notable other linguistic features. The model is trained using a variety
advancements across several NLP tasks in recent years. In this of supervised learning methods, like LR, SVM, or decision
work, the feasibility of employing the XLNet model for trees.
sentiment analysis is examined, which involves fine-tuning the
XLNet model on a labeled sentiment analysis dataset. First, the CNNs have been used for sentiment analysis by treating
dataset is preprocessed, and the XLNet model is loaded. In the text as an image and using convolutional and pooling
addition, the classification layer is added to the model and layers to identify the features from the text. The input data is
transfer learning is applied for fine-tuning on the sentiment first converted into a matrix where each row represents a
analysis dataset. The effectiveness of the proposed work is word or a group of words, and each column represents a
evaluated on a test set, and various metrics such as accuracy, feature. The convolutional layers learn the patterns and
precision, recall, and F1 score are reported. Experimental features in the text by sliding a filter over the matrix, and the
results indicate that the XLNet model attained goood results output of the convolutional layers is reduced in size through
than other transformer based models on movie review dataset down sampling by the pooling layers. The fully connected
for sentiment analysis and it shows the effectiveness of transfer layers then use the learned features to make a prediction on
learning with XLNet in the sentiment analysis domain. the sentiment of the text.
Keywords- XLNet, Sentiment analysis, Transfer Learning, Recent advancements in deep learning and transformer-
Fine Tuning, Transformer model. based models have revolutionized the field of NLP,
including sentiment analysis. Transformer models such as
I. INTRODUCTION BERT, GPT-2 and XLNet[9][13] have shown significant
Sentiment analysis is a branch of natural language improvements in sentimental analysis compared to
processing that seeks to automatically detect and extract traditional machine learning and CNN-based models. The
personal and emotional information from textual usage of a self-attention mechanism in these models enables
information. Sentiment analysis has various applications, them to better comprehend the connections between words
including social media monitoring, brand reputation and their contexts, as well as to capture the enduring
management, customer feedback analysis, and market interdependencies among words in a sentence.
research. In recent years, with the rapid growth of social In transformer models [11][12][14][15], the input text is
media and the internet, sentiment analysis has become tokenized and converted into embeddings, which are then fed
increasingly important for businesses and organizations to into multiple transformer layers. In the transformer layers, a
understand public opinion and respond to customer needs. self-attention mechanism is utilized to comprehend the
In sentiment analysis, text data is classified into connections between words and contexts within the text, and
three categories: positive, negative, or neutral. Traditional the output of the transformer layers is used to make a
prediction on the sentiment of the text. Transformer-based

979-8-3503-3360-2/23/$31.00 ©2023 IEEE 76

Authorized licensed use limited to: University of the West of England. Downloaded on December 23,2024 at 12:04:50 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the International Conference on Sustainable Computing and Smart Systems (ICSCSS 2023)
IEEE Xplore Part Number: CFP23DJ3-ART; ISBN: 979-8-3503-3360-2

models can capture long-term dependencies between words strategies such as learning rate and selecting hidden state
and are better at handling context than traditional machine vector. The results showed that BERT4TC outperformed
learning models. typical feature based and also fine tuning methods, achieving
superior performance on multiple class classification.
The main progress of the research work are outlined Furthermore, post-training the BERT4TC model on a
below, domain-related corpus resulted in better performance on
1) The labeled sentiment analysis dataset is binary sentiment classification datasets compared to the
preprocessed by cleaning and removing unwanted original BERT model.
characters, stop words, and punctuations. The text is In [4], a novel deep network model based on Hierarchical
tokenized into individual words or subwords. Graph Transformer is introduced for large scale multiple
2) Firstly, the pre-trained model of XLNet is loaded, label text classification. The model represents the text and its
and subsequently a classification layer is appended to the semantics using a structure of graph and employs a multiple
pre-trained model. layer transformer structure with multiple head attention to
capture features at different levels. The model also
3) In order to optimize the performance of the model,
incorporates the relationships between the labels to create
the hyper parameters, which include the learning rate, batch
label representations and employs a weighted loss function
size, and number of epochs, are fine-tuned.
which depends on semantic distances. The results on three
4) After fine-tuning, the model's performance is standard datasets proved that the proposed model is effective
assessed and contrasted against that of other models. in capturing the text's hierarchy and logic, and outperforms
existing methods.
The work is structured as below: Section 2 provides an
overview of related research. Section 3 presents the research The authors of [5] propose a method to enhance the
work model in detail. The performance of the proposed semantics captured from short texts by incorporating
model is discussed and compared with other models in knowledge based conceptualization and also transformer
Section 4. Finally, Section 5 gives the conclusion of the encoder. The method uses a CNN to identify local
work. information and enriches short information which is obtained
from a knowledge base. Furthermore, it employs a
II. RELATED WORKS subnetwork structure depends on a transformer embedding
The neural network model [1] is introduced for text encoder to embed concepts into a low-dimensional vector
sentiment analysis that incorporates BERT pre-trained space and gain more attention from them. By utilizing the
language model, BLSTM, and attention mechanism. The concept space and transformer encoder space, the method
model aims to tackle the limitation of current sentiment constructs understanding models for short text information
classification models that fail to consider the context of retrieval and classification. The experimental results show
words. To address this issue, the proposed model leverages that the proposed method significantly improves the
BERT to acquire word vectors along with contextual performance of short text analysis.
meaning of semantic information, BiLSTM to extract BERT-based convolutional network model[6] is adapted
context-related features, and it employs attention mechanism to enhance the accuracy of long text classification for
to assign importance to significant information. The model's Chinese news based on local features. The model consists of
performance is evaluated on the SST dataset, achieving a test four modules: first, Dynamic LEAD-n is applied for
accuracy rate of 89.17%, which is an improvement compared extracting short texts from long texts, Text-Text Encoder
to other methods. module is adapted for capturing global features using BERT
The research work [2] presents a method for mining user and attention mechanism, CNN based module with local
interests on social networking platforms to suggest friends by feature is employed for capturing local features, and fusion
classifying users' text posts. They use the BERT language of feature vectors from different operations. By addressing
model and change it to address the problem of missing local the limitation of BERT on the input sequence length, the
information by proposing the KBERT-CNN text proposed method enhances the accuracy of classification for
classification model. This model combines BERT's last four Chinese language, as validated by experimental results.
layers' output with TextCNN to classify text. The user's text The study utilizes a labeled Twitter dataset to perform
categories' probability distribution is then used to calculate depression intensity classification. Four transformer based
interest similarity, and Top-N friends are recommended. models, along with one moderately larger model, are adapted
Experimental results indicate that the KBERT-CNN model to classify the depression intensity based on tweets. The
achieves an F1 score of 92.26%, and the precision of friend models are enhanced with various hyperparameters, and their
recommendations based on text classification is better than performance is evaluated using metrics such as accuracy, F1-
other content-based methods. score (recall and precision). The results indicate that Electra
The BERT language model lacks domain-specific and Small Generator [7] gives better accuracy than all other
task-specific knowledge, so a BERT-based text classification models. The study highlights the need for further
model was proposed in [3] to improve its performance. The optimization of ESG for low-powered devices and
BERT4TC model uses auxiliary sentence construction to emphasizes the potential of achieving better classification
transform classification tasks into binary sentence-pair performance for depression detection.
problems and overcomes the issues of limited training data A novel approach [8] is proposed for aspect based
and task-awareness. The study presents the model's sentiment analysis that utilizes a lexicalized ontology to
architecture, implementation, and approach of post training extract indirect relationships in user social data. The
for BERT's domain challenge. The authors conducted approach employs XLNet and Bi-LSTM networks for
experiments on seven datasets and tested various fine-tuning

979-8-3503-3360-2/23/$31.00 ©2023 IEEE 77

Authorized licensed use limited to: University of the West of England. Downloaded on December 23,2024 at 12:04:50 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the International Conference on Sustainable Computing and Smart Systems (ICSCSS 2023)
IEEE Xplore Part Number: CFP23DJ3-ART; ISBN: 979-8-3503-3360-2

comprehensive context extraction and aspect classification, Despite the advancements made in sentiment analysis,
respectively. The experimental results on six real-world there are several limitations that need to be addressed. These
drug-related social data sets reveal that the proposed include the heavy reliance on labeled data for training, the
approach outperforms existing approaches, achieving high difficulty in accurately capturing contextual understanding,
accuracy and F-measure in ADRs aspect based sentiment challenges in adapting models to different domains, limited
analysis. This demonstrates the effectiveness of the approach support for multilingual analysis, and the lack of
in improving feature extraction and sentiment classification interpretability in model predictions. These limitations
accuracy in unstructured social media text. restrict the scalability, generalizability, and applicability of
sentiment analysis models. To overcome these challenges,
The BERT and XLNet [9] [13] language models are used the research is focused on improving the models' ability to
in the field of NLP research, where they are fine-tuned for comprehend context, enhancing domain adaptation
cross domain analysis of sentiment classification. It explores techniques, expanding support for multilingual sentiment
the transferability of these models and analyses their analysis, and developing models that offer interpretability.
performances, leading to a considerable improvement in the By addressing these limitations, sentiment analysis can be
existing approaches for cross domain analysis, while improved and made more effective in various applications.
applying a minimal amount of data. The findings suggest that
bidirectional contextualized models provides better results III. PROPOSED METHODOLOGY
than previous works for cross domain analysis of sentiment
text classification. The proposed work contains four phases, namely data pre-
processing, XLNet model, fine-tuning the XLNet model for
The research [10] employed a pre-trained BERT model text classification, and performance analysis. In the first
along with an AdamW optimizer to assess the sentiment of phase, the data is processed and cleaned to prepare it for the
each tweet which is related to COVID-19. The dataset XLNet model. In the second phase, the pre-trained model of
contained approximately 32,000 tweets, which were XLNet is used for text classification which is depicted in
categorized into three classes: negative, neutral, and positive. figure 1. In the third phase, the XLNet model is further fine-
To balance the data, under-sampling was conducted, and the tuned by using the transfer learning approach to optimize its
model was fine-tuned for four epochs. The outcome revealed performance. Finally, in the fourth phase, the performance of
that the model's accuracy was highest in predicting negative the fine-tuned XLNet model is assessed using several metrics
sentiment, but least accurate in predicting neutral sentiment. to determine its effectiveness in text classification.
The overall accuracy of the model was 75.15%. However,
the study suggests that enhancing the dataset's size is
expected to lead to a substantial improvement in accuracy.

Fig. 1. XLNet Model for Text Classification

979-8-3503-3360-2/23/$31.00 ©2023 IEEE 78

Authorized licensed use limited to: University of the West of England. Downloaded on December 23,2024 at 12:04:50 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the International Conference on Sustainable Computing and Smart Systems (ICSCSS 2023)
IEEE Xplore Part Number: CFP23DJ3-ART; ISBN: 979-8-3503-3360-2

A. Preprocessing Some of the important hyperparameters for fine-tuning

In data preprocessing for text analysis, irrelevant or XLNet include:
unnecessary data is removed through data cleaning, which Learning rate: This hyperparameter determines the step
includes removing HTML tags, punctuation, numbers, size at which the model updates its parameters during
special characters, and stop words. The text is then tokenized training. Setting a proper learning rate is critical to ensuring
into smaller units, such as words or phrases. Text that the model converges to an optimal solution while
normalization is used to convert text to a standard format by avoiding overfitting or underfitting.
converting all text to lowercase, correcting spelling mistakes,
and expanding abbreviations. Stop word removal removes Batch size: The batch size determines the number of
common words that do not carry significant meaning, and training examples the model processes at once during each
stemming/lemmatization techniques are used to reduce iteration of training. A larger batch size can speed up training
words to their root forms to improve the size of the but can also require more memory and may result in
vocabulary and accuracy of the model. suboptimal results if the model cannot fully explore the
parameter space in each iteration.
B. Transformer based transfer learning model
Number of training epochs: The quantity of training
Transformer-based transfer learning models are highly
epochs indicates how many times the model is trained on the
effective for text classification tasks, as they can leverage complete training dataset. If the number of epochs is set too
pre-trained models to observe complex contextual low, the model may underfit, whereas if it is set too high, the
representations of text, which can be fine-tuned for specific model may overfit.
text classification tasks. Additionally, it handles long-range
dependencies between words in a sentence, which is Dropout rate: Dropout is a regularization technique
important for tasks such as sentiment analysis where the used to prevent overfitting. It randomly drops out some of
sentiment of a sentence may depend on the context of the the neurons during training, forcing the model to learn more
entire sentence. robust features. The dropout rate determines the fraction of
neurons that are dropped out during training.
XLNet transformer model
Weight decay: Weight decay is another regularization
XLNet is a deep neural network architecture that is based technique that gives a penalty to the loss function based on
on the Transformer architecture and is specifically designed the magnitude of the weights in the model. This can help to
to process sequential data, such as text. The architecture prevent overfitting by encouraging the model to use smaller
includes an encoder that uses only the encoder portion for its weights. The hyper parameters of XLNet and Fine-tuned
language modeling task, which consists of a several layers model are listed in Table1.
with two sub layers: a multiple head self-attention concept
and a feedforward network. This multiple head self-attention Hyperopt, a Python library, was employed for
mechanism allows the model to focus on different positions hyperparameter optimization to fine-tune an XLNet model
in the input sequence and learn contextual representations of for sentiment analysis on a movie review dataset. The
input tokens. In XLNet, the self-attention mechanism is methodology involved defining the search space with
bidirectional, which allows the model to capture long-range hyperparameters and their corresponding ranges,
dependencies between tokens. constructing an objective function to assess the model's
performance, and selecting a suitable optimization algorithm.
In addition to the self-attention mechanism, XLNet also Through the sampling of hyperparameter configurations,
uses a novel "permutation-based objective" during training, training and evaluation of the XLNet model, and careful
which encourages the model to capture dependencies analysis of the outcomes, the optimal hyperparameter
between all input positions instead of only nearby positions. configuration was determined, resulting in improved
This objective is achieved by randomly permuting the input sentiment analysis accuracy on movie reviews.
tokens and training the model to predict the original order of
the tokens. This feature enables XLNet to learn more robust
contextual representations and outperform the other models TABLE I. HYPER PARAMETERS USED IN XLNET
on various NLP based research works.
Hyper XL XL Net(Fine Tuned)
Finally, XLNet uses a "segment-level recurrence" Parameters Net
mechanism to find local and also global dependencies in the
input text. This is achieved by dividing the input text into Learning lr=1e- lr=3e-5
segments based on various factors, such as sentence rate 5
boundaries or text formatting, and performing self-attention
Batch size 8 4
within and across these segments. This feature allows the
model to capture dependencies between different parts of the
input text and learn more accurate representations of the text. Number of 20 10
The architecture is given in figure 2. training
epochs
Fine-tuned XLNet transfer learning model
Dropout 0.1 0.1
Fine-tuning XLNet for classifying the text involves
setting a variety of hyperparameters that can impact the
performance of the system. Weight 0.01 0.01
decay

979-8-3503-3360-2/23/$31.00 ©2023 IEEE 79

Authorized licensed use limited to: University of the West of England. Downloaded on December 23,2024 at 12:04:50 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the International Conference on Sustainable Computing and Smart Systems (ICSCSS 2023)
IEEE Xplore Part Number: CFP23DJ3-ART; ISBN: 979-8-3503-3360-2

contextual relationships between words, which can be

helpful for improving classification accuracy.
To apply transfer learning on a movie review dataset, the
first step is to fine-tune XLNet model on a large corpus of
text data. This process involves training the model on a task
such as language modeling or masked language modeling,
where the goal is to learn how to find the next word in a
statement or to fill in the missing words in a sentence.
During this fine-tuning process, the model adapts its
parameters to the specific characteristics of the data it is
being trained on, which helps to improve its ability to extract
relevant features from text.
Once the pre-trained model has been fine-tuned, it can be
used to identify the features from the movie review dataset.
The text data can be fed into the model and the activations of
the hidden layers can be extracted as features. These features
can then be used to train a neural network classifier, to
analyze the sentiment of a text on the movie reviews. By
fine-tuning the model on a large corpus of text data and
extracting features from it, the model can be adapted to the
specific task of sentiment analysis, which can lead to
improved classification accuracy.
The proposed model suggests that achieving high
accuracy in Transfer Learning requires the ability to
generalize, either within the same domain or across different
Fig. 2. XLNet Architecture domains. This can be accomplished by adding a task-specific
final layer and fine-tuning the model on a supervised dataset
C. XLNet with Transfer Learning Approach for the target task. To further improve accuracy, the proposed
model recommends training the model multiple times on the
same data with different random seeds. This is because
Transfer learning with XLNet is applied to text prediction different seeds can lead to substantially different results,
tasks such as sentiment analysis on movie review datasets even when using the same hyper-parameters. In the context
and it is shown in figure 3. The idea is to use the pre-trained of the proposed movie review classification task, the model
XLNet model to extract meaningful features from the text, will first be generalized, and then trained multiple times with
and then train a classifier on top of these features to perform different random seeds to achieve higher accuracy.
the specific classification task. The benefit of using transfer
learning is that the pre-trained model already has a good
understanding of the language and can capture complex

Fig. 3. Transfer Learning XLNet Model

979-8-3503-3360-2/23/$31.00 ©2023 IEEE 80

Authorized licensed use limited to: University of the West of England. Downloaded on December 23,2024 at 12:04:50 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the International Conference on Sustainable Computing and Smart Systems (ICSCSS 2023)
IEEE Xplore Part Number: CFP23DJ3-ART; ISBN: 979-8-3503-3360-2

IV. EXPERIMENTS Accuracy : The accuracy metric measures the ratio of right
The proposed text classification system is assessed by predictions made by a model to the total count of predictions
that it has made and it is given in equation (1)
employing various transformer models and comparing their
results
A. Dataset
(1)
Movie review dataset is used for analyzing the results of
transformer based models. F-Measure: Equation (2) indicates the mean which is a
combination measure of p-r.
Movie Review Dataset- IMDb
The IMDb Movie Reviews dataset is composed of 50,000 (2)
reviews sourced from the Internet Movie Database (IMDb),
which have been categorized as either positive text, negative Precision (p): Precision evaluates the capability of a model to
text for binary analysis. The dataset has an equal number of accurately classify text in instances where it is present and it
positive and negative reviews, with only the most polarizing is depicted in equation (3)
reviews included. To be labeled as negative, a review must
have a score of 4 out of 10 or lower, while a score of 7 out of
10 or higher qualifies a review as positive. Additionally, no (3)
more than 30 reviews per movie have been included in the
dataset, and it also contains some unlabelled data. The
sample data is given in figure 4 Recall(r): Recall indicates to the capability of a model to
correctly identify all relevant instances of a particular class in
a given text and it is represented in equation (4)

(4)

C. Performance Analysis
The comparison study is conducted to evaluate the
performance of various Transformer-based models on a
movie review dataset. Two evaluation metrics were used to
compare the models - accuracy and loss, which are standard
metrics for measuring the effectiveness of transformer
models. The results of the study are depicted in Tables 2 and
3. Table 2 indicates a comparison of the accuracy and loss of
different Transformer-based models, while Table 3 presents
the F1-measure and accuracy of the models. F1-measure is
another evaluation metric that takes both precision and recall.
The performance analysis revealed that the XLNet model
outperformed the other models on the movie review dataset.
This means that the XLNet model achieved the highest
accuracy and the lowest loss compared to other models.
Furthermore, Table 3 shows that the XLNet model also had
the highest F1-measure, which indicates that it had a better
balance between precision and recall compared to other
models.To further emphasize the superior performance of the
XLNet model, Figure 5 was presented, which displays the
accuracy and F1-measure of different Transformer models.
This visualization highlights the clear advantage of the
XLNet model over the other models based on accuracy and
F1-measure.

Fig. 4. Sample data of dataset The XLNet model exhibits the highest accuracy rate,
making it an optimal choice for further tuning through
B. Performance Metrics transfer learning. By employing transfer learning, the
performance of the XLNet model can be enhanced, as
Transformer based models are assessed using various metrics demonstrated through its enhanced accuracy on a movie
such as accuracy, F1-score(precision, recall) to analyze their review dataset, as indicated in Tables 4 and 5. Following
performance. fine-tuning, the XLNet model was identified as the most
suitable option for text classification, as illustrated in Figure
6. Sample output is given in figure 7.

979-8-3503-3360-2/23/$31.00 ©2023 IEEE 81

Authorized licensed use limited to: University of the West of England. Downloaded on December 23,2024 at 12:04:50 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the International Conference on Sustainable Computing and Smart Systems (ICSCSS 2023)
IEEE Xplore Part Number: CFP23DJ3-ART; ISBN: 979-8-3503-3360-2

TABLE II. COMPARISON ANALYSIS OF TRANFORMER MODELS INTERMS TABLE III. RESULT ANALYSIS OF VARIOUS TRANFORMER MODELS
OF ACCURACY AND LOSS INTERMS OF F-MEASURE

Training Train Test Test F- Accuracy

S.NO Models S.NO Models Precision Recall
accuracy loss Accuracy Loss measure

1 BERT 1 BERT
93 14 93 59 92 93 92.49 93
(Base) (Base)

2 BERT
BERT 83 84 84.5 86
2 87 34 86 32 (Large)
(Large)
Transformer
Transformer 3 encoder 87 88 87.5 89
3 encoder 88 64 89 62 with SVM
with SVM
4 XLNet 92 93 92.5 94
4 XLNet 95 37 94 64

Fig. 5. Result Analysis of Transformer Models

Fig. 6. Performance Analysis of Fine Tuned Model

979-8-3503-3360-2/23/$31.00 ©2023 IEEE 82

Authorized licensed use limited to: University of the West of England. Downloaded on December 23,2024 at 12:04:50 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the International Conference on Sustainable Computing and Smart Systems (ICSCSS 2023)
IEEE Xplore Part Number: CFP23DJ3-ART; ISBN: 979-8-3503-3360-2

TABLE IV. PERFORMANCE ANALYSIS OF PROPOSED MODEL INTERMSOF The comparative analysis of the table 6 shows the
ACCURACY AND LOSS
performance of various sentiment analysis models on a
Training Train Test Test movie review dataset. The proposed fine-tuned XLNet using
S.NO Models transfer learning yields the highest accuracy of 99%. This
accuracy loss Accuracy Loss
indicates that the proposed model outperforms the other
1
XLNet 95 37 94 64 models in accurately classifying sentiment in movie
reviews. By comparing the existing models with the
Fine- proposed fine-tuned XLNet model, it is evident that the
Tuned proposed model achieves significantly higher accuracy. This
2 XLNet 29
using
93 20 99 highlights the effectiveness of fine-tuning XLNet using
Transfer transfer learning for sentiment analysis on movie review
Learning datasets. The proposed model leverages the pre-trained
knowledge captured by XLNet and tailors it specifically for
TABLE V. PRFORMANCE ANALYSIS OF FINE-TUNED XLNET MODEL ON sentiment analysis, resulting in superior performance.
MOVIE REVIEW DATASET INTERMS OF F-MEASURE

F- Accuracy
S.NO Models Precision Recall
measure V. CONCLUSION

1
Over the past few years, there has been significant
XLNet 92 93 92.48 94 interest in sentiment analysis owing to its usefulness in a
variety of fields, including marketing, social media
Fine-
Tuned surveillance, and customer support. Transformer models
2 XLNet 99 have been widely used for sentiment analysis tasks, with
97 98 97.5
using transfer learning being one of the popular approaches for
Transfer improving their performance. The XLNet transfer learning
Learning
model is proposed to classify the text for sentiment analysis.
Transfer learning was utilized to fine-tune the XLNet
model, with the objective of improving its accuracy and
performance of text classification. A publicly available
movie review dataset was utilized for sentiment analysis,
containing labeled reviews as either positive or negative.
The data underwent preprocessing, including cleaning,
tokenization, and encoding. The XLNet model served as the
base model, also transfer learning was utilized to fine-tune
the model. The results showed that the XLNet model
performed exceptionally well on the movie review dataset,
Fig. 7. Sample Screen Shot achieving an accuracy rate of 99%. The model's initial
evaluation already demonstrated high accuracy rates
compared to other models. However, after fine-tuning, the
TABLE VI. PERFORMANCE COMPARISON ANALYSIS OF PROPOSED MODEL
WITH EXISTING TECHNIQUES accuracy rate improved significantly, suggesting the ability
of transfer learning in improving the performance of pre-
Accuracy (%)
trained models on specific tasks.
S.NO Models Method

References
1 S. M. Qaisar
LSTM 89.9
et.al. [16] [1] Y. Shen and J. Liu, "Comparison of Text Sentiment Analysis
based on Bert and Word2vec," 2021 IEEE 3rd International
Shaukat Z lexicon and Conference on Frontiers Technology of Information and
2 91 Computer (ICFTIC), Greenville, SC, USA, 2021, pp. 144-147,
et.al. [17] neural networks
doi: 10.1109/ICFTIC54370.2021.9647258.
Su, Sichang [2] N. Pan, W. Yao and X. Li, "Friends Recommendation Based on
3 SVM Model 85.2
et.al. [18] KBERT-CNN Text Classification Model," 2021 International
Joint Conference on Neural Networks (IJCNN), Shenzhen,
Hybrid feature China, 2021, pp. 1-6, doi: 10.1109/IJCNN52387.2021.9533618.
Kumar, H
4 model with ML 83.7
et.al. [19] [3] S. Yu, J. Su and D. Luo, "Improving BERT-Based Text
algorithms
Classification With Auxiliary Sentence and Domain
T. E. Trueman Knowledge," in IEEE Access, vol. 7, pp. 176600-176612, 2019,
5 BERT 94.6 doi: 10.1109/ACCESS.2019.2953990.
et.al. [20]

Fine-Tuned [4] J. Gong et al., "Hierarchical Graph Transformer-Based Deep

6 Proposed XLNet using Learning Model for Large-Scale Multi-Label Text
99 Classification," in IEEE Access, vol. 8, pp. 30885-30896, 2020,
Model Transfer
Learning doi: 10.1109/ACCESS.2020.2972751.
[5] J. Li, G. Huang, J. Chen and Y. Wang, "Short Text
Understanding Combining Text Conceptualization and

979-8-3503-3360-2/23/$31.00 ©2023 IEEE 83

Authorized licensed use limited to: University of the West of England. Downloaded on December 23,2024 at 12:04:50 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the International Conference on Sustainable Computing and Smart Systems (ICSCSS 2023)
IEEE Xplore Part Number: CFP23DJ3-ART; ISBN: 979-8-3503-3360-2

Transformer Embedding," in IEEE Access, vol. 7, pp. 122183-

122191, 2019, doi: 10.1109/ACCESS.2019.2938303.
[6] X. Chen, P. Cong and S. Lv, "A Long-Text Classification
Method of Chinese News Based on BERT and CNN," in IEEE
Access, vol. 10, pp. 34046-34057, 2022, doi:
10.1109/ACCESS.2022.3162614.
[7] M. Rizwan, M. F. Mushtaq, U. Akram, A. Mehmood, I. Ashraf
and B. Sahelices, "Depression Classification From Tweets
Using Small Deep Transfer Learning Language Models," in
IEEE Access, vol. 10, pp. 129176-129189, 2022, doi:
10.1109/ACCESS.2022.3223049.
[8] A. H. Sweidan, N. El-Bendary and H. Al-Feel, "Sentence-Level
Aspect-Based Sentiment Analysis for Classifying Adverse Drug
Reactions (ADRs) Using Hybrid Ontology-XLNet Transfer
Learning," in IEEE Access, vol. 9, pp. 90828-90846, 2021, doi:
10.1109/ACCESS.2021.3091394.
[9] B. Myagmar, J. Li and S. Kimura, "Cross-Domain Sentiment
Classification With Bidirectional Contextualized Transformer
Language Models," in IEEE Access, vol. 7, pp. 163219-163230,
2019, doi: 10.1109/ACCESS.2019.2952360.
[10] M. Roque, E. Sybingco, A. Bandala and M. A. Roque,
"Sentiment Analysis for COVID-19 Related Tweets Using Deep
Bi-Directional Transformers," TENCON 2022 - 2022 IEEE
Region 10 Conference (TENCON), Hong Kong, Hong Kong,
2022, pp. 1-6, doi: 10.1109/TENCON55691.2022.9977638.
[11] Keval Pipalia, Rahul Bhadja, Madhu Shukla , “Comparative
Analysis of Different Transformer Based Architectures Used in
Sentiment Analysis”, International Conference System
Modeling and Advancement in Research Trends,pp.411-415,
2020.
[12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova, “Bert: Pre-training of deep bidirectional
transformers for language understanding” ,2018.
[13] Z. Yang, “XLNet: Generalized Autoregressive Pretraining for
Language Understanding", Computation and Language, 2019.
[14] Y. Liu , “RoBERTa: A Robustly Optimized BERT Pretraining
Approach", ", Computation and Language ,2019.
[15] K. Clark , “ELECTRA: Pre-training Text Encoders as
Discriminators Rather Than Generators", Computation and
Language, 2020.
[16] S. M. Qaisar, "Sentiment Analysis of IMDb Movie Reviews
Using Long Short-Term Memory," 2020 2nd International
Conference on Computer and Information Sciences (ICCIS),
Sakaka, Saudi Arabia, 2020, pp. 1-4, doi:
10.1109/ICCIS49240.2020.9257657.
[17] Shaukat, Z., Zulfiqar, A.A., Xiao, C. et al. Sentiment analysis on
IMDB using lexicon and neural networks. SN Appl. Sci. 2, 148
(2020).
[18] Su, Sichang. “Sentimental Analysis Applied on Movie
Reviews.” Journal of Education, Humanities and Social
Sciences, 2022.
[19] Kumar, H. M. Keerthi et al. “Sentiment Analysis on IMDb
Movie Reviews Using Hybrid Feature Extraction Method.” Int.
J. Interact. Multim. Artif. Intelligence, 109-114,2019.
[20] T. E. Trueman, A. K. Jayaraman, E. Cambria, G.
Ananthakrishnan and S. Mitra, "An N-gram-Based BERT model
for Sentiment Classification Using Movie Reviews," 2022
International Conference on Artificial Intelligence and Data
Engineering (AIDE), Karkala, India, pp. 41-46, 2022.

979-8-3503-3360-2/23/$31.00 ©2023 IEEE 84

Authorized licensed use limited to: University of the West of England. Downloaded on December 23,2024 at 12:04:50 UTC from IEEE Xplore. Restrictions apply.