2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII)

Fine-Grained Emotion Prediction by Modeling

Emotion Definitions
2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII) | 978-1-6654-0019-0/21/$31.00 ©2021 IEEE | DOI: 10.1109/ACII52823.2021.9597436

Gargi Singh∗ , Dhanajit Brahma† , Piyush Rai‡ , Ashutosh Modi§

CSE Department, Indian Institute of Technology Kanpur (IIT-K), Kanpur 208016, India
Email: ∗ [email protected], † [email protected], ‡ [email protected], § [email protected]

Text Label(s)
Abstract—In this paper, we propose a new framework for It’s better to say a moment like that could truly
fine-grained emotion prediction in the text through emotion ignite her love for the game rather than love
definition modeling. Our approach involves a multi-task learning putting a bit of a damper on it.
framework that models definitions of emotions as an auxiliary I did hear that [NAME] is friends with [NAME],
task while being trained on the primary task of emotion predic- I think on the SC sub. This was probably like confusion
tion. We model definitions using masked language modeling and a month ago. Could be.
class definition prediction tasks. Our models outperform existing Sometimes life actually hands you lemons.
approval,
state-of-the-art for fine-grained emotion dataset GoEmotions. We We’re just lucky that we have a proverbial phrase
realization
further show that this trained model can be used for transfer that gives us an idea of what we can do with them.
learning on other benchmark datasets in emotion prediction with Does nobody notice that this is a doctored photo..??
Like really bad clone stamp.. I mean it could be disapproval
varying emotion label sets, domains, and sizes. The proposed beautiful but this is not a real photo.
models outperform the baselines on transfer learning experiments I think the fan base is mostly past that at this point.
demonstrating the generalization capability of the models. Almost everyone has MASSIVE problems with neutral
Index Terms—Fine-grained Emotion Prediction, Multi-task some of the decisions Nintendo makes.
Learning, Transformers, Transfer Learning TABLE I
E XAMPLES FROM G O E MOTIONS DATASET

I. I NTRODUCTION

Recent advancements in Artificial Intelligence (AI) have Formally, given some text, the task of emotion recognition
made machines/computers an integral part of our lives, e.g., is to predict explicit as well as implicit emotions. Emotions
personal digital assistants. These AI based technologies are are represented either as a categorical label (e.g., sad, happy,
getting better at understanding what humans express explicitly surprise) or as continuous values (in the form of valence and
via language (both speech and text). However, humans use arousal). In this paper, we use the former for representing
language not just to convey information but also to express emotions. Recognizing emotions is challenging as these are
their inner feelings and mental states. The majority of the often implicit in the text, and the inherent complexity of
time, humans express emotions in language implicitly. AI emotions makes it difficult to model them computationally (see
pioneer Marvin Minsky in his book “The Emotion Machine” Table I).
[1], has explained the importance of emotions and how it is A number of approaches have been proposed for emotion
not that different from the underlying process of thinking in recognition in text [2]: keyword-based approaches [13], rule-
humans. To develop truly intelligent systems that understand based approaches [14], classical learning-based approaches
and assist humans, we would like these machines to recognize [15], deep learning approaches [16], and hybrid approaches
and understand implicit emotions in the interactions. Humans [17]. However, all the proposed approaches follow a two-
do emotion recognition effortlessly; however, for machines, step method. The first step involves the extraction of emotion-
it is not a trivial activity. Intending to develop emotionally centric features. Subsequently, in the second step, these fea-
intelligent machines, the fundamental problems of emotion tures are used for identifying emotions via a machine learning-
recognition [2]–[5] and generation [6] are active areas of based model. With advances in deep learning, state-of-the-art
research in the AI community. (SOTA) results have been obtained in almost all major areas,
Humans express emotions via multiple modalities like video and the task of emotion recognition is not an exception to this.
(facial expression), speech (intonations and prosody), and text. Recently, a new type of neural attention-based deep archi-
However, with the growth of the internet, and social media, tecture referred to as Transformer [18] has been proposed.
at a global scale, text is one of the more prominent modes Transformers excel at handling long-term dependencies in
of communication and interaction. With the motivation to texts. Since the introduction of the initial transformer model,
perform automatic analysis of emotions expressed in the text, several advanced architectures have been proposed: BERT
in this paper, we focus on the task of emotion recognition [19], XLNet [20], GPT [21], RoBERTa [22], ALBERT [23].
in text. Emotion recognition in text has wide applications Typically, transformers (and their variants) are pre-trained on
[7], e.g., marketing [8], advertising [9], political science [10], huge language corpora (e.g., web corpus, Wikipedia), and
human-computer interaction [11], conversational AI [12]. these pre-trained models can be fine-tuned (adapted) for usage

978-1-6654-0019-0/21/$31.00 ©2021 IEEE

Authorized licensed use limited to: LIVERPOOL JOHN MOORES UNIVERSITY. Downloaded on May 19,2023 at 09:17:40 UTC from IEEE Xplore. Restrictions apply.
in novel tasks via transfer learning. Pre-training and fine- discrete labels. The former is derived from the circumplex
tuning strategy has shown SOTA results on various NLP model proposed by [34]; the model states that the space of
tasks like natural language inference, machine comprehension, emotions is continuous, and the emotion experienced can be
machine translation, as well as text-based emotion recognition represented in three dimensions: valence, arousal, and domi-
task [24]. nance. Based on the circumplex model, various datasets have
This paper proposes a new transformer-based framework been created and annotated with three-dimensional continuous
for fine-grained emotion classification that leverages semantic values, e.g., EmoBank [35] and fb-valence-arousal [36]. There
knowledge of the emotion classes. We use BERT as the base is no single agreed set of discrete emotion labels, giving rise
model and attempt to model the semantic meaning of emotion to various emotion label sets. The most common ones are
classes through their definitions while training the model for Ekman’s six emotions (anger, disgust, fear, joy, sadness, and
emotion classification. We employ a multi-tasking framework surprise) [37] and Plutchik’s eight emotions (joy, trust, fear,
with proportional sampling between emotion classification and surprise, sadness, disgust, anger, and anticipation) [38]. Recent
definition modeling. We experiment with three setups for developments have shown that emotions in other modalities
definition modeling: 1. Class Definition Prediction (CDP) 2. may be more fine-grained such as 24 in brief vocalization
Masked Language Modeling (MLM) 3. both Class Definition [39], 28 in facial expression [40], 12 in speech prosody [41].
Prediction and Masked Language Modeling(CDP+MLM). Our A similar fine-grained emotion label set (27 emotions and
model gives an overall improvement in F1 score in all the three neutral) for text has been proposed by [25]. The authors have
setups on the fine-grained GoEmotions dataset [25], while the created the largest manually annotated dataset (referred to as
best score is obtained from the setup of CDP. We further GoEmotions) of emotions in English (details in Section III).
test (via transfer learning) on benchmark datasets with smaller In this work, we propose new models for emotion recognition
emotion label sets from different domains to check the general- based on the GoEmotions dataset. [42] have proposed a
ization capability of the proposed model. We use ISEAR [26], Unified Dataset for recognizing emotions in text; they combine
EmoInt [27] and Emotion-stimulus [28]. We observe that our different emotion datasets into one with a common annotation
models outperform the existing baselines [25] in the transfer schema.
learning experiments, especially for smaller training data sizes. Transformers have dominated recent progress in the NLP
In summary, we make the following contributions: community. [18] introduced attention-based transformer archi-
• We propose a new framework for emotion prediction that tecture for machine translation which was later adapted to
takes into account the definitions of the emotion classes build powerful NLP models independently/jointly using en-
• We obtain the state-of-the-art result for the fine-grained coder or/and decoder components of the proposed architecture.
emotion labels prediction task [19] presented BERT, which outperformed existing baselines
• We perform transfer learning experiments on other on eleven NLP tasks. The BERT model is based on the encoder
datasets from different domains and label sets and out- component of transformers. It was pretrained on English
perform the existing SOTA models for these Wikipedia, and BooksCorpus [43] using next sentence predic-
We release the code for model implementations and ex- tion (NSP) and masked language modeling (MLM) tasks. It
periments via GitHub: https://github.com/Exploration-Lab/ was later fine-tuned with additional layers as needed to obtain
FineGrained-Emotion-Prediciton-Using-Definitions. The fol- state-of-the-art results on downstream NLP tasks. In recent
lowing section reviews some of the prominent literature in the years, there have been various attempts in emotion prediction
area of emotion recognition. Section III describes the proposed that have used BERT as the base model or in ensemble with
model framework and the training setup. Section IV provides a some other approach. [44] investigated the problem of emotion
note on experiments, followed by results in Section V. Section classification in code-switched documents using BERT. The
VI discusses the observations, improvements, and future work. task of emotion classification and constructing emotion lexicon
We conclude the paper in section VII. was combined by [45] that improved emotion classification
results on the Twitter dataset. The top three models at the
II. R ELATED W ORK recent EmotionX 2019 [46] challenge for emotion recognition
Emotion recognition is an actively researched problem, with in text are BERT-based models. The baseline on GoEmotions
experts working on it from different fronts in artificial intelli- is also on a BERT model with an added classifier layer fine-
gence and cognitive science communities [29]. The text-based tuned on the dataset.
challenge in emotion recognition has gained popularity in view In the general text classification task using a transformer-
of its practical and extensive applications. [30] evaluated cus- based model, an additional layer of a classifier is added to
tomer satisfaction by comparing the linguistic characteristics the model, and it is fine-tuned for the task at hand. Here, the
of emotional expressions of positive and negative attitudes. semantics of the individual class and its relation with others is
[31] use Twitter data to predict mortality from Atherosclerotic not leveraged. There have been some efforts made in this direc-
Heart Disease (AHD). There has also been research on the use tion lately. [47] formalized the task of sentence classification
of emotions to predict personality traits [32], [33]. as a question answering task where the description of category
There has been extensive research on representing emotions. label and text were concatenated as input for a binary output
These broadly fall into two categories: continuous values and of yes or no depending on whether the description was correct.

Authorized licensed use limited to: LIVERPOOL JOHN MOORES UNIVERSITY. Downloaded on May 19,2023 at 09:17:40 UTC from IEEE Xplore. Restrictions apply.
This process required the presence of descriptions, and thus, Text
This is honestly the cherry on
top of the cake with this story.
hard attention was used, even during inference. [48] included That's absolutely hilarious!
Definition
four kinds of semantic knowledge (word embeddings, class Finding something funny or
being entertained.
descriptions, class hierarchy, and a general knowledge graph) Label
IsDefinition
Text
in their framework to facilitate zero-shot text classification. This is honestly the
cherry on top of the cake
[49] explicitly modeled label embeddings and their mutual with this story. That's
absolutely hilarious!
correlations in emotion inference for ROC stories [50]. Related Emotion Label
amusement
Text
to emotion recognition is the task of Aspect-based sentiment This is honestly the cherry on
top of the cake with this story.
That's absolutely hilarious!
analysis (ABSA) which entails the identification of opinion Definition
Sadness or displeasure
polarity towards a specific aspect in a given comment. [51] caused by the nonfulfillment of
one’s hopes or expectations.
Label
transformed this into a sentence pair classification task with IsNotDefinition

the concatenation of instance text to an auxiliary sentence

constructed with respect to aspect. Fig. 1. Examples from auxiliary dataset. For each instance in the primary
dataset, two instances are constructed for auxiliary dataset; in the first, the text
is concatenated with correct emotion’s definition and is labeled “IsDefinition”
III. M ETHODOLOGY whereas in the second, the text is concatenated with definition of incorrect
emotion label and is labeled “IsNotDefinition”.
Dataset: In this paper, we use the GoEmotions dataset. Intro-
duced by [25], it is the largest manually annotated corpus of
58k English Reddit comments created via crowdsourcing. The model captures the task-specific information in the sentence.
text in the corpus is labeled with fine-grained emotion labels This is further used for emotion prediction.
consisting of 27 emotions and neutral: admiration, amuse- Auxiliary Task Definition - Class Definition Prediction
ment, anger, annoyance, approval, caring, confusion, curiosity, (CDP): BERT model is trained using the Next Sentence
desire, disappointment, disapproval, disgust, embarrassment, Prediction (NSP) and Masked Language Model (MLM) tasks.
excitement, fear, gratitude, grief, joy, love, nervousness, op- In the NSP task, given a pair of sentences, the BERT
timism, pride, realization, relief, remorse, sadness, surprise, model predicts if the second sentence semantically follows
neutral. It is a multi-labeled dataset with 83% examples the first or not. NSP forces the model to learn the seman-
marked with one label, 15% with two labels, 2% with three tic correspondences between sentences. Taking inspiration
labels, and 0.2% with four or more labels. The dataset is from this, we posit that if a text instance is appended with
split into train, validation (dev), and test set in 80 − 10 − 10 an emotion label definition and this combination is passed
proportion, respectively. Due to space constraints, we do not through BERT, then the model would learn the relationship
go into details of the annotation scheme and crowdsourcing between emotions expressed in the sentence and the corre-
experiments conducted for labeling emotions in GoEmotions; sponding appended emotion label definition. We propose an
we refer the reader to the GoEmotions paper. auxiliary task: given a sequence with a sentence x followed
In our approach, we attempt to model emotion labels’ by an emotion definition c, the task is to predict if the
definitions semantics while training the model to predict appended definition depicts the emotions expressed in the
emotions in a sentence so as to teach the model the descriptive text. Formally, we pass the following sequence through BERT
meaning of the emotion classes and their relation to the given model: {[CLS], wx1 , wx2 , ..., wxn , [SEP ], wc1 , wc2 , ..., wcm }
instance sentence. We take inspiration from language modeling where n is the length of the text and m is the length of the
techniques used to pre-train BERT. With this goal in mind, definition. Here, [SEP ] is a special token that helps the model
we propose a multi-task learning setup, with a primary task of distinguish between the sentence and the emotion definition.
emotion prediction and different auxiliary tasks for learning For example, consider the sentence, “For art? I think it’s kind
the definitions. of silly, but it’s also fun. Let people live :P”, we append this
Primary Task Definition: Formally, given a sentence x with the definition of anger “A strong feeling of displeasure or
composed of tokens {w1 , w2 , ..., wn } where n is the length antagonism”. This combination is passed to the BERT model,
of x, the task is to predict the emotion(s) (from a pre-defined and in this case, it predicts the “IsNotDefinition” label (see
set of L discrete emotion labels: e1 , e2 , ..., eL ) expressed in section III), since the text is not expressing anger.
the sentence. For example: “For art? I think it’s kind of silly, Auxiliary Task Definition - Masked Language Model
but it’s also fun. Let people live :P” would be classified (MLM): In the MLM task, given a sentence, some of the
into the category “joy”. Note that a sentence may express tokens are randomly removed from the sentence and replaced
more than one emotion, in which case it will be annotated with a special “[M ASK]” token. This modified sentence
with multiple labels. We are using a transformer-based model is passed as input to the BERT model, and the task is
for emotion prediction; in accordance with that, the sentence to predict the [M ASK] tokens. This forces the model to
is prepended with a special token [CLS] before passing understand the semantics of the text to help predict the masked
it to the transformer model. Hence, the sentence becomes: tokens. We adapt the MLM task to our setting. Given a
{[CLS], w1 , w2 , ..., wn }. Intuitively, the vector representation sequence with a sentence x followed by the definition of
corresponding to [CLS] token position at the output of the one of the emotions depicted in the text c composed of

Authorized licensed use limited to: LIVERPOOL JOHN MOORES UNIVERSITY. Downloaded on May 19,2023 at 09:17:40 UTC from IEEE Xplore. Restrictions apply.
 Classification Classification Classification
CDP CDP
Mask LM Mask LM Mask LM Mask LM

C T1 ... Tn S T’1 ... T’m C T1 ... Tn S T’1 ... T’m C T1 ... Tn S T’1 ... T’m

BERT BERT BERT

[CLS] Tok1 ... Tokn [SEP] Tok’1 ... Tok’m [CLS] Tok1 ... Tokn [SEP] Tok’1 ... Tok’m [CLS] Tok1 ... Tokn [SEP] Tok’1 ... Tok’m

(masked tokens of text) (masked tokens of definition) (masked tokens of text) (masked tokens of definition)

Text Definition Text Definition Text Definition

Fig. 2. Model Architecture. In setup 1, only CDP is employed. In setup 2, only MLM is employed. In setup 3, both MLM and CDP are used. Green
components correspond to the primary task of emotion prediction only. Red components belong to the auxiliary task only. Yellow components are common
between both primary and auxiliary tasks.

tokens {[CLS], wx1 , wx2 , ..., wxn , [SEP ], wc1 , wc2 , ..., wcm }, for prediction directly via the [CLS] token. The probability of
random tokens are masked by a special token [M ASK], where each emotion label being present in the given text is predicted
n is the length of the text and m is the length of the definition. by the model. The labels that cross a threshold value (0.3 in our
The masked tokens are predicted from the vocabulary of the case) are predicted as the emotion for the sentence. Primary
model. For example, given masked input sequence “[CLS] For and auxiliary tasks are not trained equally; a proportional
art? I think it’s kind of silly, but it’s also [MASK]. Let people sampling method is employed to sample between the two
live :P [SEP] A feeling of pleasure and [MASK]”, correct tasks. During each iteration, we sample from a Bernoulli
predictions would be “fun” and “happiness” respectively. distribution: Bernoulli(p), to decide between primary and
auxiliary tasks to train. Here, the probability p corresponds
Auxiliary Dataset: For training the auxiliary tasks, we cre-
to the selection of the primary task. We experiment with three
ate an auxiliary dataset. The primary dataset, GoEmotions,
different auxiliary tasks (Figure 2): noitemsep,topsep=0pt
consists of sentences annotated with emotion labels. For each
instance in the dataset, two instances are created in the 1) CDP: The pooled output from the BERT-base model is
auxiliary dataset, one labeled ‘IsDefinition’ and the other input to a dense linear classification layer followed by
‘IsNotDefinition’ (Figure 1). The ‘IsDefinition’ labeled in- computation of cross-entropy loss.
stance contains instance text and definition corresponding to 2) MLM: Only the positive (‘IsDefinition’ labeled) in-
correct emotion as the second sentence in the sequence. The stances of the auxiliary dataset are used for this setup.
‘IsNotDefinition’ labeled instance, on the other hand, contains The input sentence-pair tokens are masked by following
instance text and definition for an emotion not included in the same strategy used by BERT. A softmax activation
the corresponding label as the second sentence. We use the function is used over the tokens in vocabulary for
definitions of emotion classes provided by GoEmotions paper predicting the masked tokens using the hidden vectors
authors [25] to the raters during the annotation process. For in output from BERT. This is also followed by cross-
multi-labeled instances, each emotion label in the annotation entropy loss.
is considered a different instance. 3) CDP+MLM: Model is trained on both CDP and MLM
tasks in this setup, and all the instances of the auxiliary
Model Architecture: We propose a multi-task neural architec- dataset are included in the training. A combined loss
ture. The primary task is emotion prediction, and the auxiliary from both CDP and MLM is used.
task includes CDP or MLM, as described previously. BERT-
base model is the backbone for both the primary and auxiliary IV. E XPERIMENTS
tasks, i.e., the layers of BERT are shared between the two Different Transformer Architectures Experiments: We ex-
tasks, primary and auxiliary, with hard parameter sharing (see periment with different transformer architectures: BERT, XL-
Figure 2). The pooled output from BERT is passed through a Net, RoBERTa, ALBERT, to compare their performances as
dropout layer followed by a linear dense classification layer for base models for emotion classification on GoEmotions dataset.
the primary task. Since it is a multi-label classification prob- We keep the same hyperparameters as in [25], except for an
lem, sigmoid-based binary cross-entropy is used to compute increase in the number of training epochs to 10. A classifier
the loss between the model predictions and class label values. layer is added to all the models, followed by binary cross-
During inference, the text is directly (without the definition) entropy loss. Binary classification on all categories is used to
passed to the trained BERT model (primary task), and it is used accommodate multi-labeled instances.

Authorized licensed use limited to: LIVERPOOL JOHN MOORES UNIVERSITY. Downloaded on May 19,2023 at 09:17:40 UTC from IEEE Xplore. Restrictions apply.
Fine-Grained Experiments: We use the same hyperparam- Transfer Learning Experiments: Figure 3 shows the plot for
eters as in the previous experiment. We experiment with results on different datasets for various train dataset sizes. We
proportional sampling during training with the following find that GoEmotions-trained BERT+MLM initialized models
range of values for Bernoulli distribution probability p ∈ consistently outperform the existing baseline of GoEmotions-
{0.1, 0.2, 0.3, ..., 0.9}, where p corresponds to the probability trained BERT on all the datasets. For smaller dataset sizes,
of training the primary task during an iteration. These ex- models employing transfer learning from GoEmotions give
periments are carried on all the three auxiliary tasks setups better results as compared to just pretrained BERT, but the
considered for the proposed framework as explained in Section difference decreases with increment in training data size.
III:
1) CDP only VI. D ISCUSSION AND E RROR A NALYSIS
2) MLM only
BERT+CDP gives the best results outperforming the earlier
3) both CDP and MLM
baselines in [25], but BERT+CDP+MLM also has a com-
Transfer Learning-Based Experiments: In transfer learning- petitive performance; BERT+MLM does not perform as well
based experiments, we test the adaptability and generalization but still performs better than the baseline BERT model. In
capability of the proposed model to other domains and emo- the results for individual categories, we notice an increment
tion label sets. We use the best performing model(s) trained in the scores for categories with fewer examples (embar-
on GoEmotions dataset obtained in previous experiments to rassment, grief, nervousness, pride, relief ) in training for
initialize training on other datasets, thereby transferring the BERT+CDP+MLM setup, which shows the relevance of our
knowledge learned from GoEmotions to a target domain. We model with unbalanced training data. A majority of the other
compare the performance of our model(s) to vanilla BERT emotion categories’ scores also improve. We observe that the
and BERT trained on GoEmotions proposed by [25] following GoEmotions trained BERT+MLM transferred models perform
the same parameters and experimental setup. Emotion corpora consistently better than the baseline in the transfer learning
from the Unified Dataset [42] are selected for the experiments; experiment. There are variations across the three datasets,
in particular, we experiment on benchmark datasets ISEAR which might be due to their diverse nature in terms of the
[26], EmoInt [27], and Emotion-Stimulus [28] owing to their source domain, emotion labels, and balance across classes for
diversity in source domains and emotion label sets. During training examples.
training on each dataset, train data size is varied in the range
Emotion classification is one of the more challenging tasks
(100, 200, 500, 1000, 80% of total), the remaining data is
in text classification not just because of the abstract nature
taken for the test set. The above is repeated for ten random
of emotions but also because of the possible subjectivity in
splits for each set size.
interpretation. Unlike a number of NLP tasks, problems in
V. R ESULTS the affective domain cannot always be determined through
Standard Precision, Recall, and F1-Score metrics are used linguistic cues alone. We notice this issue in our error analysis,
for evaluating various models. where we find that some predictions of the trained model,
Different Transformer Architectures: The results on various though incorrect as per annotations, could be reasonable from
transformer architectures are shown in Table II. The baseline a different perspective. We present examples of some of these
model is a BERT model as proposed for GoEmotions dataset instances in Table V. These instances, among many others,
[25]. All the transformer models are trained for 10 epochs. though judged as erroneous predictions, make sense. This
However, the baseline model was trained for only 4 epochs calls for developing methods for addressing the subjectivity
by [25]. As evident from the results, training for more epochs of emotion labels and annotation uncertainty due to it [52],
improves the model performance. We also observe that the [53]; we plan to explore this in the future work.
F1-scores for RoBERTa and ALBERT are similar to BERT,
Text Label(s) Prediction(s)
whereas the XLNet model performs slightly worse. Im really sorry about your situation :(
love,
Fine-Grained Experiments: We experimented with various Although I love the names Sapphira, sadness
remorse
proportional sampling probabilities. The results for different Cirilla, and Scarlett!
Kings fan here, good luck to you guys! excitement,
probabilities are shown in III. We found that a proportion Will be an interesting game to watch!
excitement
optimism
sampling probability of 0.5 for primary tasks works best in all annoyance,
Boomers ruined the world neutral
three setups of the framework. The best-performing models of disappointment
Now I’m wondering if [NAME] drinks,
each setup outperform the existing baseline, and BERT+CDP and if he’s ever been inebriated during surprise
curiosity,
provides the best results overall. Table IV provides the perfor- surprise
one of his deals.
mance of the best model on each of the emotion categories. I totally thought the same thing! I was
neutral realization
like, oh honey nooooo!
We find that our BERT+CDP+MLM model framework im-
proves the performance significantly among most categories. TABLE V
E XAMPLES OF MISMATCH BETWEEN THE PREDICTIONS BY THE MODEL
In particular, the improvement for the following categories AND AVERAGE PREDICTIONS BY ANNOTATORS . T HIS IS PRIMARILY DUE
(with fewer training data examples (< 500)) are significant: TO THE SUBJECTIVE NATURE OF EMOTIONS .
embarrassment, grief, nervousness, pride, relief.

Authorized licensed use limited to: LIVERPOOL JOHN MOORES UNIVERSITY. Downloaded on May 19,2023 at 09:17:40 UTC from IEEE Xplore. Restrictions apply.
 Model Dev-Precision Dev-Recall Dev-F1 Test-Precision Test-Recall Test-F1
Baseline - - - 40 63 46
BERT 53.84 50.13 50.28 48.86 53.11 50.17
XLNet 49.3 52.89 50.27 47.09 51.92 48.5
RoBERTa 51.82 53.76 50.87 51.82 52.94 51.21
ALBERT 54.56 50.12 50.03 56.98 48.89 50.12
BERT+CDP 53.57 53.21 52.07 54.66 53.8 52.34
BERT+MLM 52.04 53.13 51.57 52.42 52.67 51.25
BERT+CDP+MLM 53.29 54.12 52.02 53.42 54.37 51.96
TABLE II
P ERFORMANCE (F1-S CORE ( IN %)) OF BASELINE , TRANSFORMERS AND PROPOSED MODELS ON DEV AND TEST SPLIT OF G O E MOTIONS

Sampling Probability (p) BERT+CDP BERT+MLM BERT+CDP+MLM

Dev Test Dev Test Dev Test
0.1 44.57 44.05 43.1 42.74 45.33 44.98
0.2 47.24 46.18 48.35 48.19 48.71 48.3
0.3 47.61 47.38 49.73 49.11 51.49 50.36
0.4 51.65 51.07 51.1 50.07 51.33 50.87
0.5 52.07 52.34 51.57 51.25 52.02 51.96
0.6 50.98 52.32 49.92 50.27 51.32 49.72
0.7 52.13 51.2 49.57 50.6 50.76 50.09
0.8 51.32 51.73 49.5 49.51 49.88 49.43
0.9 50.35 50.79 50.34 49.77 49.85 51.26

TABLE III
C OMPARISON OF F1 SCORES ( IN %) OF DIFFERENT PROPORTION SAMPLINGS FOR BERT+CDP, BERT+MLM, BERT+CDP+MLM ON DEV AND TEST
SPLIT OF G O E MOTIONS

Emotion Baseline BERT BERT+CDP BERT+MLM BERT+CDP+MLM

admiration 65 66.85 66.54 67.42 68.56
amusement 80 80.27 81.26 82.47 81.8
anger 47 49.64 48.93 49.65 50
annoyance 34 33.43 36.43 36.26 39.51
approval 36 35.63 39.8 41.75 39.12
caring 39 43.48 41.09 43.26 40.92
confusion 37 44.32 39.48 46.88 41.62
curiosity 54 51.77 55.84 56.24 55.62
desire 49 47.8 48.89 52.29 47.8
disappointment 28 27.91 30.34 33.33 30.09
disapproval 39 41.26 39.66 41.37 36.49
disgust 45 43.82 50.86 48 46.15
embarrassment 43 43.59 50.85 50 50.79
excitement 34 43.95 39.2 41.95 41.15
fear 60 65.9 65.91 62.5 64.68
gratitude 86 89.79 91.47 91.49 91.73
grief 0 25 50 0 50
joy 51 60.57 57.8 61.86 58.22
love 78 79.09 77.69 79.84 78.49
nervousness 35 37.5 43.9 45.45 44.44
optimism 51 55.47 55.61 54.84 56.58
pride 36 41.67 34.78 45.45 43.48
realization 21 21.25 25 24.1 25.58
relief 15 33.33 50 40 40
remorse 66 64.12 67.18 67.14 64.75
sadness 49 55.93 56.86 53.09 47.94
surprise 50 54.98 54.05 53.1 53.92
neutral 68 66.49 66.17 65.33 65.4
macro-average 46 50.17 52.34 51.25 51.96
std 19 16.95 15.6 18.14 15.47

TABLE IV
C OMPARISON OF F1 SCORES ( IN %) OF EACH EMOTION CATEGORY FROM THE G O E MOTIONS DATASET. T HE HIGHLIGHTED CATEGORIES HAVE LESS
TRAINING EXAMPLES IN THE G O E MOTIONS DATASET, HOWEVER THE PROPOSED MODEL SHOWS PERFORMANCE IMPROVEMENTS .

Authorized licensed use limited to: LIVERPOOL JOHN MOORES UNIVERSITY. Downloaded on May 19,2023 at 09:17:40 UTC from IEEE Xplore. Restrictions apply.
Fig. 3. Transfer learning experiments on ISEAR, EmoInt and Emotion-Stimulus datasets with differing initialisation of base model BERT for finetuning.
VanillaBERT is pretrained BERT, BERT refers to pretrained BERT finetuned on GoEmotions. The other three variants (BERT+CDP, BERT+MLM,
BERT+CDP+MLM) refer to the different frameworks finetuned on GoEmotions using definitions.

VII. C ONCLUSION [7] A. Seyeditabari, N. Tabari, and W. Zadrozny, “Emotion detection in text:
a review,” 2018.
In this paper, we addressed the task of fine-grained emotion [8] R. Bagozzi, M. Gopinath, and P. Nyer, “The role of emotions in
prediction. The key idea is to use emotion label definitions marketing,” Journal of the Academy of Marketing Science, vol. 27, pp.
184–206, 04 1999.
for prediction. We employed language modeling tasks inspired [9] G. Qiu, X. He, F. Zhang, Y. Shi, J. Bu, and C. Chen, “Dasa:
from work by [19] to model definitions of emotion classes as Dissatisfaction-oriented advertising based on sentiment analysis,” Expert
an auxiliary task to inform the model of the affective meaning Systems with Applications, vol. 37, no. 9, pp. 6182–6191, 2010.
of the emotion conveyed in a sentence. We tested our model [10] J. Druckman and R. Mcdermott, “Emotion and the framing of risky
choice,” Political Behavior, vol. 30, pp. 297–321, 09 2008.
with the two auxiliary tasks: class definition prediction (CDP) [11] R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias,
and masked language modeling (MLM), in both isolation and W. Fellenz, and J. Taylor, “Emotion recognition in human-computer
combined format, thereby giving us three different architec- interaction,” Signal Processing Magazine, IEEE, vol. 18, pp. 32 – 80,
02 2001.
tures and experimental setups with BERT transformer as the [12] P. Colombo, W. Witon, A. Modi, J. Kennedy, and M. Kapadia, “Affect-
base model and primary task emotion classification. We obtain driven dialog generation,” in Proceedings of the 2019 Conference of
the state-of-the-art result for fine-grained labels of the GoE- the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, Volume 1 (Long and Short
motions dataset consisting of 27 emotions and neutral. Lastly, Papers). Minneapolis, Minnesota: Association for Computational
the proposed state-of-the-art model also transfers knowledge Linguistics, Jun. 2019, pp. 3734–3743.
well and outperforms the existing baseline of transfer learning [13] J. Tao, “Context based emotion detection from text input,” in Eighth
International Conference on Spoken Language Processing, 2004.
to other datasets. Fine-grained emotions can be represented as [14] S. Y. M. Lee, Y. Chen, and C.-R. Huang, “A text-driven rule-based
taxonomy, and this could be used for learning the relationships system for emotion cause detection,” in Proceedings of the NAACL
between different emotion categories. This could, in turn, be HLT 2010 Workshop on Computational Approaches to Analysis and
Generation of Emotion in Text. Los Angeles, CA: Association for
useful for the general task of emotion recognition. In the Computational Linguistics, Jun. 2010, pp. 45–53. [Online]. Available:
future, we plan to pursue this line of research. https://www.aclweb.org/anthology/W10-0206
[15] C. O. Alm, D. Roth, and R. Sproat, “Emotions from text:
R EFERENCES Machine learning for text-based emotion prediction,” in Proceedings
of Human Language Technology Conference and Conference on
[1] M. Minsky, The emotion machine: Commonsense thinking, artificial Empirical Methods in Natural Language Processing. Vancouver,
intelligence, and the future of the human mind. Simon and Schuster, British Columbia, Canada: Association for Computational Linguistics,
2007. Oct. 2005, pp. 579–586. [Online]. Available: https://www.aclweb.org/
[2] N. Alswaidan and M. E. B. Menai, “A survey of state-of-the-art anthology/H05-1073
approaches for emotion recognition in text,” Knowledge and Information [16] H. Meisheri and L. Dey, “TCS research at SemEval-2018 task 1:
Systems, vol. 62, pp. 2937–2987, 2020. Learning robust representations using multi-attention architecture,”
[3] A. Kołakowska, A. Landowska, M. Szwoch, W. Szwoch, and M. R. in Proceedings of The 12th International Workshop on Semantic
Wrobel, “Emotion recognition and its applications,” in Human-Computer Evaluation. New Orleans, Louisiana: Association for Computational
Systems Interaction: Backgrounds and Applications 3. Springer, 2014, Linguistics, Jun. 2018, pp. 291–299. [Online]. Available: https:
pp. 51–62. //www.aclweb.org/anthology/S18-1043
[4] W. Witon, P. Colombo, A. Modi, and M. Kapadia, “Disney at iest [17] S. Gievska, K. Koroveshovski, and T. Chavdarova, “A hybrid approach
2018: Predicting emotions using an ensemble,” in Proceedings of the for emotion detection in support of affective interaction,” 2014 IEEE
9th Workshop on Computational Approaches to Subjectivity, Sentiment International Conference on Data Mining Workshop, pp. 352–359, 2014.
and Social Media Analysis, 2018, pp. 248–253. [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
[5] A. Singh, S. Hingane, S. Wani, and A. Modi, “An end-to-end network for L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017.
emotion-cause pair extraction,” in Proceedings of the Eleventh Workshop [19] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training
on Computational Approaches to Subjectivity, Sentiment and Social of deep bidirectional transformers for language understanding,” in
Media Analysis. Online: Association for Computational Linguistics, Proceedings of the 2019 Conference of the North American Chapter
Apr. 2021, pp. 84–91. of the Association for Computational Linguistics: Human Language
[6] T. Goswamy, I. Singh, A. Barkati, and A. Modi, “Adapting a language Technologies, Volume 1 (Long and Short Papers). Minneapolis,
model for controlled affective text generation,” in Proceedings of the Minnesota: Association for Computational Linguistics, Jun. 2019,
28th International Conference on Computational Linguistics, 2020, pp. pp. 4171–4186. [Online]. Available: https://www.aclweb.org/anthology/
2787–2801. N19-1423

Authorized licensed use limited to: LIVERPOOL JOHN MOORES UNIVERSITY. Downloaded on May 19,2023 at 09:17:40 UTC from IEEE Xplore. Restrictions apply.
[20] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, [40] A. Cowen and D. Keltner, “What the face displays: Mapping 28
“Xlnet: Generalized autoregressive pretraining for language understand- emotions conveyed by naturalistic expression,” American Psychologist,
ing,” 2020. vol. 75, 06 2019.
[21] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., [41] A. Cowen, P. Laukka, H. Elfenbein, R. Liu, and D. Keltner, “The
“Language models are unsupervised multitask learners,” OpenAI blog, primacy of categories in the recognition of 12 emotions in speech
vol. 1, no. 8, p. 9, 2019. prosody across two cultures,” Nature Human Behaviour, vol. 3, p. 1,
[22] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, 04 2019.
L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert [42] L.-A.-M. Bostan and R. Klinger, “An analysis of annotated corpora for
pretraining approach,” 2019. emotion classification in text,” in Proceedings of the 27th International
[23] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, Conference on Computational Linguistics. Santa Fe, New Mexico,
“Albert: A lite bert for self-supervised learning of language representa- USA: Association for Computational Linguistics, Aug. 2018, pp. 2104–
tions,” 2020. 2119. [Online]. Available: https://www.aclweb.org/anthology/C18-1179
[24] F. A. Acheampong, H. Nunoo-Mensah, and W. Chen, “Transformer [43] Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba,
models for text-based emotion detection: a review of bert-based ap- and S. Fidler, “Aligning books and movies: Towards story-like visual
proaches,” Artificial Intelligence Review, pp. 1–41, 2021. explanations by watching movies and reading books,” in 2015 IEEE
[25] D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and International Conference on Computer Vision (ICCV), 2015, pp. 19–27.
S. Ravi, “Goemotions: A dataset of fine-grained emotions,” 2020. [44] T. Tang, X. Tang, and T. Yuan, “Fine-tuning bert for multi-label
[26] K. Scherer and H. Wallbott, “Evidence for universality and cultural vari- sentiment analysis in unbalanced code-switching text,” IEEE Access,
ation of differential emotion response patterning.” Journal of personality vol. 8, pp. 193 248–193 256, 2020.
and social psychology, vol. 66 2, pp. 310–28, 1994. [45] D. Zhou, S. Wu, Q. Wang, J. Xie, Z. Tu, and M. Li, “Emotion
classification by jointly learning to lexiconize and classify,” in
[27] S. Mohammad and F. Bravo-Marquez, “WASSA-2017 shared task
Proceedings of the 28th International Conference on Computational
on emotion intensity,” in Proceedings of the 8th Workshop on
Linguistics. Barcelona, Spain (Online): International Committee on
Computational Approaches to Subjectivity, Sentiment and Social Media
Computational Linguistics, Dec. 2020, pp. 3235–3245. [Online].
Analysis. Copenhagen, Denmark: Association for Computational
Available: https://www.aclweb.org/anthology/2020.coling-main.288
Linguistics, Sep. 2017, pp. 34–49. [Online]. Available: https:
[46] B. Shmueli and L.-W. Ku, “Socialnlp emotionx 2019 challenge
//www.aclweb.org/anthology/W17-5205
overview: Predicting emotions in spoken dialogues and chats,” 2019.
[28] D. Ghazi, D. Inkpen, and S. Szpakowicz, “Detecting emotion stimuli [47] D. Chai, W. Wu, Q. Han, F. Wu, and J. Li, “Description based text
in emotion-bearing sentences,” in Computational Linguistics and Intel- classification with reinforcement learning,” 2020.
ligent Text Processing, A. Gelbukh, Ed. Cham: Springer International [48] J. Zhang, P. Lertvittayakumjorn, and Y. Guo, “Integrating semantic
Publishing, 2015, pp. 152–165. knowledge to tackle zero-shot text classification,” 2019.
[29] R. W. Picard, Affective Computing. Cambridge, MA, USA: MIT Press, [49] R. Gaonkar, H. Kwon, M. Bastan, N. Balasubramanian, and
1997. N. Chambers, “Modeling label semantics for predicting emotional
[30] F. Ren and C. Quan, “Linguistic-based emotion analysis and recognition reactions,” in Proceedings of the 58th Annual Meeting of the
for measuring consumer satisfaction: An application of affective com- Association for Computational Linguistics. Online: Association for
puting,” Information Technology and Management, vol. 13, 12 2012. Computational Linguistics, Jul. 2020, pp. 4687–4692. [Online].
[31] J. Eichstaedt, H. Schwartz, M. Kern, G. Park, D. Labarthe, R. Merchant, Available: https://www.aclweb.org/anthology/2020.acl-main.426
S. Jha, M. Agrawal, L. Dziurzynski, M. Sap, C. Weeg, E. Larson, [50] N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra,
L. Ungar, and M. Seligman, “Psychological language on twitter predicts L. Vanderwende, P. Kohli, and J. Allen, “A corpus and cloze
county-level heart disease mortality,” Psychological Science, vol. 26, evaluation for deeper understanding of commonsense stories,” in
no. 2, pp. 159–169, Feb. 2015, funding Information: This work was Proceedings of the 2016 Conference of the North American
supported by the Robert Wood Johnson Foundation’s Pioneer Portfolio, Chapter of the Association for Computational Linguistics: Human
through Exploring Concepts of Positive Health Grant 63597 (to M. E. Language Technologies. San Diego, California: Association for
P. Seligman), and by a grant from the Templeton Religion Trust. Computational Linguistics, Jun. 2016, pp. 839–849. [Online]. Available:
[32] N. Majumder, S. Poria, A. Gelbukh, and E. Cambria, “Deep learning- https://www.aclweb.org/anthology/N16-1098
based document modeling for personality detection from text,” IEEE [51] C. Sun, L. Huang, and X. Qiu, “Utilizing BERT for aspect-
Intelligent Systems, vol. 32, no. 2, pp. 74–79, 2017. based sentiment analysis via constructing auxiliary sentence,” in
[33] S. M. Mohammad and S. Kiritchenko, “Using nuances of emotion to Proceedings of the 2019 Conference of the North American Chapter
identify personality,” 2013. of the Association for Computational Linguistics: Human Language
[34] J. Russell, “A circumplex model of affect,” Journal of Personality and Technologies, Volume 1 (Long and Short Papers). Minneapolis,
Social Psychology, vol. 39, pp. 1161–1178, 12 1980. Minnesota: Association for Computational Linguistics, Jun. 2019,
[35] S. Buechel and U. Hahn, “EmoBank: Studying the impact of pp. 380–385. [Online]. Available: https://www.aclweb.org/anthology/
annotation perspective and representation format on dimensional N19-1035
emotion analysis,” in Proceedings of the 15th Conference of the [52] Z. Zhang, J. Han, E. Coutinho, and B. Schuller, “Dynamic difficulty
European Chapter of the Association for Computational Linguistics: awareness training for continuous emotion prediction,” IEEE Transac-
Volume 2, Short Papers. Valencia, Spain: Association for Computational tions on Multimedia, vol. 21, no. 5, pp. 1289–1301, 2018.
Linguistics, Apr. 2017, pp. 578–585. [Online]. Available: https: [53] G. Rizos and B. Schuller, “Modelling sample informativeness for deep
//www.aclweb.org/anthology/E17-2092 affective computing,” in ICASSP 2019-2019 IEEE International Con-
[36] D. Preoţiuc-Pietro, H. A. Schwartz, G. Park, J. Eichstaedt, M. Kern, ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE,
L. Ungar, and E. Shulman, “Modelling valence and arousal in 2019, pp. 3482–3486.
Facebook posts,” in Proceedings of the 7th Workshop on Computational
Approaches to Subjectivity, Sentiment and Social Media Analysis. San
Diego, California: Association for Computational Linguistics, Jun.
2016, pp. 9–15. [Online]. Available: https://www.aclweb.org/anthology/
W16-0404
[37] P. Ekman, “An argument for basic emotions,” Cognition and Emotion,
vol. 6, pp. 169–200, 1992.
[38] R. PLUTCHIK, “Chapter 1 - a general psychoevolutionary theory of
emotion,” in Theories of Emotion, R. Plutchik and H. Kellerman,
Eds. Academic Press, 1980, pp. 3–33. [Online]. Available: https:
//www.sciencedirect.com/science/article/pii/B9780125587013500077
[39] A. Cowen, H. Elfenbein, P. Laukka, and D. Keltner, “Mapping 24 emo-
tions conveyed by brief human vocalization,” American Psychologist, 12
2018.

Authorized licensed use limited to: LIVERPOOL JOHN MOORES UNIVERSITY. Downloaded on May 19,2023 at 09:17:40 UTC from IEEE Xplore. Restrictions apply.