2023 5th International Conference on Advances in Computing, Communication Control and Networking (ICAC3N)

Movie & Song Recommender System Based on

Emotion Detection using Machine Learning
2023 5th International Conference on Advances in Computing, Communication Control and Networking (ICAC3N) | 979-8-3503-3086-1/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICAC3N60023.2023.10541832

Dr. Sanjeev Kumar Singh Shruti Gupta

Department of Information Technology Department of Information Technology
Galgotias College of Engineering and Technology Galgotias College of Engineering and Technology
Greater Noida, Uttar Pradesh, India Greater Noida, Uttar Pradesh, India
[email protected] [email protected]

Shivam Prajapati Rahul Tewari

Department of Information Technology Department of Information Technology
Galgotias College of Engineering and Technology Galgotias College of Engineering and Technology
Greater Noida, Uttar Pradesh, India Greater Noida, Uttar Pradesh, India
[email protected] [email protected]

Abstract—Emotions play a great role in human lives. Our mind state of mind and proposes the item that matches current mood
unconsciously notices the emotions of people we meet every day. It of the user. This technology is designed to enhance the user's
interprets the signals people emit through these emotions. It also entertainment experience by providing personalized
tries to determine how to respond to and deal with people around recommendations that cater to their emotional needs [2]. By
us. Facial expressions can reveal emotions like happiness, anger, suggesting movies and songs that matches state of mind of user,
sadness, surprise, fear, excitement, desire, contempt, disgust, this system assists user to relax, relieve stress, improve their
confusion and many more. This paper proposes an emotion-based mood, and overall enhance their emotional well-being.
movie and song recommender system that utilizes deep learning
techniques for user's emotion detection. It also provides Information filtering system like recommender system
recommendation of movies and songs that match user’s emotional predicts user interests and does recommendations as per that. It
state at that time. collects information about the choices of the user and provide
recommendations as per system’s knowledge domain. Our
The emotion detection based movie and song recommender research paper is focused on developing a machine learning
system is a revolutionary system that aims to come up with based recommendation engine that will implement combining
customized suggestions formed on basis of user’s emotional state context-based rule mining techniques and user based
at that time. It uses advanced techniques of artificial intelligence
collaborative filtering (CF) engine. The system works on facial
for emotion detection of user. It then suggests movies and songs
expressions of the active user and knowledge base of in-active
that align with their mood. Also, it can help users find latest items
that they may not have found by themselves. It leads to a more
system users for examining and evaluating detected user’s
enjoyable and fulfilling entertainment experience. Additionally, emotions. It then recommends movies and songs to the active
the system has the potential to improve mental health outcomes by users. It also implement Pearson's correlation for evaluating
providing users with content that can positively impact their resemblance measurement between choices of the user.
emotional state. Overall, the movie and song recommender system Exploring the evolution and execution of emotion-based
using emotion detection is a valuable tool for anyone seeking movie and song recommender system is the aim of this research
personalized entertainment recommendations. Our research work
paper. We reviewed the existing literature on emotion detection
focused on using facial expressions for movies and song
and how it has been used in various applications including
recommendation to the user. Depending on the expression user
reveals at particular moment we can best recommend movies that entertainment. We also provided an overview of the
suit his current mood. recommender system architecture and the algorithms used for
emotion detection based movie and song recommendation.
Keywords—movie and song recommender system; movie
recommender system; song recommender system; recommender II. LITERATURE SURVEY
system; emotions detection; collaborative filtering; cosine similarity. Research on recommender system has suggested various
approaches to recommend users the items of their choice. One
I. INTRODUCTION of the approaches says that feedback of user is not that good in
Movie and Song Recommender System based on emotion identifying likes and dislikes of the users so we should use
detection proposes songs and movies depending on emotional Percentage of View [1]. With this approach the recommender
state of user at that time. It examines user’s face to detect system can effectively recommend movies matching the movies
emotions, and other physiological signals to find out current that have a higher percentage of views. What if some movie is

ISBN: 979-8-3503-3086-1/23/$31.00©2023 IEEE 690

Authorized licensed use limited to: Zhejiang University. Downloaded on August 25,2024 at 07:08:13 UTC from IEEE Xplore. Restrictions apply.
 2023 5th International Conference on Advances in Computing, Communication Control and Networking (ICAC3N)

new and doesn’t have that many views that the recommender help in human face expression detection with an average
system can pick it as a recommendation? precision of 85% [10].
In such cases we can use Facial expressions because these Sentiments has big impact on processing information, and
help in enhancing the user experience and promoting mental decision making in real-world scenarios. Hence Sentiment
well-being. We can save users time in searching individual Recognition if performed on an Intelligent System that are real-
movies if we use emotions to recommend movies [2]. Neural time based can help validate the accuracy through transfer
Networks are more accurate than other approaches. Also, if we learning. It can perform functionalities like detection of face
combine recommender systems of both types i.e., content-based and classification of sentiments. It provides a list of probable
filtering (which uses features of one item to recommend other predicted labels in real-time from a webcam feed in one mixed
items that has similar features i.e., based on user’s past likes for step has achieved 76.62 % accuracy [11].
something) and collaborative filtering (which considers what
features all users that like something has in common and One more approach says music recommender systems
provide the recommendations accordingly) offers more power works best if music is grouped according to user’s engagement
for recommender system. User based collaborative filtering has [12]. FUSION MUSIC that combines music affective space and
some certain challenges that increases depending on users and fusion-based music recommendation method which will create
items count. So, we can use collaborative filtering that is an a partial affective space that has the potential to recommend
item-based one to recommend items, it is very dynamic too and more serendipitous [13].
also it can learn from the users’ feedback [3]. Convolution neural network (CNN) to detect emotions and
Hybrid recommender system is the integration of artificial neural network (ANN) for classification of songs can
collaborative and content-based recommender systems, uses achieve 84% accuracy [14]. By identifying semantic
rating score and consensus of the top users [4]. If we compare characteristics of a playlist, Latent Dirichlet Allocation and
Item-based and User-based Collaborative filtering we can find Case-Based Reasoning reuses the most similar past ones, for
out User-based collaborative filtering has better performance recommending relevant playlist continuations [15].
and results [5]. Sentiment analysis-based hybrid movie III. METHODOLOGY
recommendation is very effective in recommendation. It uses
preference of user as well as ratings of the movies [6] and The methodology as shown in Fig. 1. begins with capturing
focuses on user assessments rather than user reviews to face images in real time through camera or direct input of image
recommend the movies. or video files followed by preprocessing of these images, then
emotions extraction from these images, subjective selection,
Content/Collaborative hybrid approach, uses movie/song classification of the facial images based on facial expression
ratings to perform collaborative filtering and also for capturing and finally recommending movies or songs based on the user's
movie content information it uses user tags and/or keywords of
classified emotions.
the movies that are available in IMDB. Previous research
results showed that information that keywords can provide
helps build a movie recommender system that are better than
other approaches that uses neighborhood concepts and also
better than other complex approaches [7].
Rule mining can be used to satisfy an individual's
requirement. Rule mining performed on rating matrix also gives
perfect recommendation. Correlation rule mining yields F1
Score that are even better than normal rule mining [8].
Recommender systems based on collaborative filtering can be
of two types. It is either model-based recommender systems or
memory-based recommender systems. The recommender
systems based on memory are comparatively accurate but in
comparison with recommender systems based on model, it is
very less scalable [9]. Hybrid movie recommender system
retains the accuracy provided by collaborative filtering
approach based on memory and also retains scalability feature
of collaborative filtering technique based on model. This Fig. 1. Block Diagram – Movie & Song Recommender System Based on
approach is fast and compact, and also it outperforms the Emotion Detection
classical one.
Our research work in Movie & Song Recommender System
The human face is not very simple in understanding and is divided into two major parts. First part detects emotions and
identifying human face expressions. It is easy to identify second part uses that emotions to recommend Movies and
expressions of particular person’s image. But very difficult to Songs as shown below:
automated recognition system. Histogram of Oriented
Gradients (HOG), followed by characterization by SVM can (i) Facial Emotion Detection
(ii) Movie and Song Recommendation System.

691

Authorized licensed use limited to: Zhejiang University. Downloaded on August 25,2024 at 07:08:13 UTC from IEEE Xplore. Restrictions apply.
 2023 5th International Conference on Advances in Computing, Communication Control and Networking (ICAC3N)

Facial Emotion Detection: for computing the resemblance among them. In the context of a
movie and song recommender system, the measure of cosine
Emotion detection explained in Fig. 2. begins with
capturing input images as real time key frames through camera similarity is utilized for computing the resemblance between
or custom input of images or video files. The converted key the preference of current user and other users.
frames are passed to the Haar Cascade classifier for facial
expression detection. Haar cascades are algorithm for the object
detection that are very popular and are utilized in computer
vision and image processing. This object detection algorithm Textual Emotion
was developed by Viola and Jones and has since been widely
used in various applications, including face detection,
pedestrian detection, and gesture recognition. These classifiers
are designed to detect specific patterns in an image, such as the Dominant Emotion
presence of edges, lines, or rectangular shapes that make up a
face. To detect an object using Haar cascades, the algorithm
first divides the image into small rectangular sub regions and
applies the pre-trained classifiers to each of these sub regions.
Map emotion with genre or
The algorithm then combines the results of these classifiers to description keys
determine whether the object is present in the image or not.
They are fast and efficient, making them suitable for real-time
applications such as video surveillance and augmented reality.
The algorithm recognizes the facial expression and outputs the
expressions. This data is used by the recommender system for Cosine Recommended
movies and
providing movie recommendations based on emotions. Similarity
songs

Movies Dataset Song Dataset

Fig. 3. Pipeline for Recommender system

The recommender system explained in Fig. 3. works by

creating a vector representation of each user's movie and song
preferences, which are then compared to other users'
preferences using cosine similarity. The system then suggests
movies and songs based on the similarity between preference of
current user and other users who have similar tastes. One of the
major advantages of a movies and song recommender system
based on cosine similarity is that it can recommend movies and
songs that the user may not have otherwise discovered.
Additionally, the system can adjust to the modifications in the
preferences of user over time, as the cosine similarity measure
Fig. 2. Pipeline of Facial Emotion Detection is calculated based on the preferences of user at current time.
Movies and Song Recommendation System: IV. SIMILARITY TECHNIQUES
The next step is recommendation part. This part of the 1. Cosine Similarity: Similarity measure between two non-
project starts with the output provided by the emotion detection zero vectors of an inner product space are called cosine
part. We assumed the most dominant expression and map it similarity. Cosine Similarity used in fields like text mining,
with the respective genre or description variable. Cosine recommendation systems, and image recognition. Cosine
similarity is used for finding movies or songs similar to a given similarity technique calculates the cos theta angle between two
genre. Recommender system based on cosine similarity is a vectors in n-dimensional space. The value ranges between -1 to
form of collaborative filtering algorithm. The measure of cosine 1. The value 1 means that the two vectors are perfectly similar
similarity calculates the cosine of the angle amidst two vectors

692

Authorized licensed use limited to: Zhejiang University. Downloaded on August 25,2024 at 07:08:13 UTC from IEEE Xplore. Restrictions apply.
 2023 5th International Conference on Advances in Computing, Communication Control and Networking (ICAC3N)

and value -1 between two vectors signifies two vectors are any of the emotions like Anger, Disgust, Fear, Happy, Neutral,
totally different. Sad, etc.
cosine similarity (X, Y) = (X. Y) / (||X|| ||Y||) …(i) The Recommender System then comes into picture. Movie
and song recommender basically outputs the most relevant
2. Correlation: Statistical measure indicating the degree to movies and songs on the basis of similarity measures followed
which two variables are linearly related is called correlation. by exploratory data analysis (EDA). Genre is used as principal
Correlation is usually denoted by the symbol "r" and can range component for recommendation. Every emotion is mapped with
between -1 to +1. Value -1 signifies correlation as a perfect its similar genre and whole recommendation and accuracy is
negative one while no correlation is when it is 0, and correlation based on genre column. Hybrid (Collaborative + Demographic)
is perfect positive when value is +1. filtering is used for better recommendation. Training of model
for movie and song dataset is performed using Support Vector
Correlation coefficient (r) = (nΣXY - ΣXΣY) / sqrt((nΣX^2 Machine (SVM) and correlation matrix.
- (ΣX)^2) * (nΣY^2 - (ΣY)^2)) …(ii) Display and Testing is the final step. Here most relevant
movies and songs according to facial emotion detected is
3. Euclidean distance: Distance between two points in a n-
returned in data frame format. Recommendation results can be
dimensional space is Euclidean Distance. It is calculated as the tested against testing dataset to better ensure accuracy of overall
square root of the sum of the squared differences between model. Top 10 songs and movies will be displayed in column
corresponding elements in two vectors. wise manner. Along with that table of movies and songs will be
d(X, Y) = sqrt((x1 - y1)^2 + (x2 - y2)^2 + ... + (xn - yn)^2) generated based on recommendations.
…(iii) VI. CONCLUSION
4. Manhattan distance: Metric that measures distance between After conducting detailed research on emotion detection-
two points in a N dimensional vector space is called Manhattan based movie and song recommender systems, it can be
Distance. In other terms we can say the sum of lengths of the culminated that these systems have huge capability to
projections of the line segment between the points onto the revolutionize the entertainment industry. Movie and song
coordinate axes. In simple terms, sum of the absolute difference recommender systems can provide customized
between the measures in all dimensions of two points. recommendations for examining emotional behavior and
preferences of user by using algorithms of machine learning.
d(X, Y) = |x1 - y1| + |x2 - y2| + ... + |xn - yn| …(iv) The research exhibits movie and song recommender systems
V. IMPLEMENTATION based on emotion detection can significantly improve user
satisfaction, engagement, and retention. Moreover, these
Data Acquisition is the first step in the process. This research systems have the capability to increase revenue for streaming
used primarily visual data in three formats i.e., real time data, platforms by driving user engagement and reducing attrition
video and images. For image processing, we have used the user rates.
provided image dataset in png, jpg or jpeg format and for video Cosine similarity with collaborative filtering is the best
processing, user can provide any video in mp4 format only and approach that we used and implemented using the cosine and
finally for real time processing of user’s face expressions we the Pearson correlation as the distance function. Also, we
used a camera connected to the same device. performed emotion detection from facial expressions using a
The next step is Data Preprocessing. The stored data pre- very time efficient Haar cascades Algorithm. Further we used
processed using inbuilt python libraries. Video and real time CNN Model based content filtering and performed matrix
data is converted into key frames and then annotation along factorization. Finally, we used supervised learning methods
with labeling is done using Viola-Jones algorithm. The movie used for classification, regression and outliers’ detection.
and song dataset are pre-processed using average and missing In order to ensure proper accuracy and success of the
values followed by exploratory data analysis and visualization. emotion-based movie and song recommender systems, there are
some challenges that are need be addressed. These include the
Then comes Facial Emotion Detection in which the need for accurate emotion detection algorithms, user privacy
annotated image key frames are now compared with Haar- concerns, and the potential for algorithmic bias. Therefore, the
cascade dataset on the basis of different types of similarity potential benefits of emotion-based movie and song
measure like correlation, Pearson and Manhattan distance recommender systems outweigh certain challenges. With
matrix. The training model is built on the top of Haar-cascade further development and improvement, these systems could
pre-trained model. The emotion which occurs in maximum become a key differentiator for streaming platforms in the
probability is detected and returned as final output. highly competitive entertainment industry.
Machine Translation of emotions step works on emotions VII. FUTURE SCOPE
detected by emotion detection system and test it before Now-a-days, users have multiple options to choose from
transforming against textual data. These textual data are stored such as online shopping, watching movies, etc. and businesses
in dictionary format and being provided as input to can lose their customers in the case that they are not able to meet
recommender system. These machine translations may have

693

Authorized licensed use limited to: Zhejiang University. Downloaded on August 25,2024 at 07:08:13 UTC from IEEE Xplore. Restrictions apply.
 2023 5th International Conference on Advances in Computing, Communication Control and Networking (ICAC3N)

their requirements in an effective way. As users feel it difficult Knowledge Based Engineering and Innovation (KBEI), Tehran, Iran,
2019, pp. 656-660, doi: 10.1109/KBEI.2019.8734976.
to spend hours searching for some items. This is the reason
[2] S. Chauhan, R. Mangrola and D. Viji, "Analysis of Intelligent movie
companies need some ways of arranging the content for the recommender system from facial expression," 2021 5th International
customers, as searching through millions of products over the Conference on Computing Methodologies and Communication
web is just impossible. Recommender systems are basically (ICCMC), Erode, India, 2021, pp. 1454-1461, doi:
designed to recommend things according to the interest of the 10.1109/ICCMC51019.2021.9418421.
customers, it can also help them discover new products or [3] M. K. Kharita, A. Kumar and P. Singh, "Item-Based Collaborative
Filtering in Movie Recommendation in Real time," 2018 First
media that suits their area of interest. International Conference on Secure Cyber Computing and
Communication (ICSCCC), Jalandhar, India, 2018, pp. 340-342, doi:
A movie and song recommender system based on emotion 10.1109/ICSCCC.2018.8703362.
detection technology can have several potential future scopes. [4] A. N. Varma and K. Petluri, "Movie Recommender System using critic
Here are a few possibilities given below: consensus," 2021 International Conference on Advances in Computing,
Communication, and Control (ICAC3), Mumbai, India, 2021, pp. 1-4,
I. Personalization: As emotion detection technology doi: 10.1109/ICAC353642.2021.9697196.
becomes more advanced, movie and song [5] N. Shrivastava and S. Gupta, "Analysis on Item-Based and User-Based
recommender systems can use it to personalize Collaborative Filtering for Movie Recommendation System," 2021 5th
recommendations based on detected facial emotions. International Conference on Electrical, Electronics, Communication,
Computer Technologies and Optimization Techniques (ICEECCOT),
For example, if someone is feeling sad or frustrated, Mysuru, India, 2021, pp. 654-656, doi:
the system can recommend movies and songs that can 10.1109/ICEECCOT52851.2021.9707955.
be used to uplift their mood. [6] R. Lavanya and B. Bharathi, "Systematic analysis of Movie
II. Enhanced user experience: Emotion detection Recommendation System through Sentiment Analysis," 2021
International Conference on Artificial Intelligence and Smart Systems
technology can help to creates better engagement and (ICAIS), Coimbatore, India, 2021, pp. 614-620, doi:
impressive user experience. For example, a 10.1109/ICAIS50930.2021.9395854.
recommender system could suggest a playlist having [7] A. Stanescu, S. Nagar and D. Caragea, "A Hybrid Recommender System:
content of songs or movies that match the user's User Profiling from Keywords and Ratings," 2013 IEEE/WIC/ACM
current mood and display appropriate movies or songs. International Joint Conferences on Web Intelligence (WI) and Intelligent
Agent Technologies (IAT), Atlanta, GA, USA, 2013, pp. 73-80, doi:
III. New revenue streams: As emotion detection 10.1109/WI-IAT.2013.11.
technology improves, there may be opportunities to [8] T. Anwar and V. Uma, "MRec-CRM: Movie Recommendation based on
cast this data by contracting with advertisers. For Collaborative Filtering and Rule Mining Approach," 2019 International
example, a movie or song recommender system could Conference on Smart Structures and Systems (ICSSS), Chennai, India,
suggest products or services that match the user's 2019, pp. 1-5, doi: 10.1109/ICSSS.2019.8882864.
current emotional state. [9] M. Y. H. Al-Shamri and K. K. Bharadwaj, "A Compact User Model for
Hybrid Movie Recommender System," International Conference on
IV. Improved mental health: A movie and song Computational Intelligence and Multimedia Applications (ICCIMA
recommender system based on emotion detection 2007), Sivakasi, India, 2007, pp. 519-524, doi:
technology can help people manage their mental 10.1109/ICCIMA.2007.15.
health. For example, if someone is feeling anxious, the [10] C. Jain, K. Sawant, M. Rehman and R. Kumar, "Emotion Detection and
Characterization using Facial Features," 2018 3rd International
system can suggest tranquilizing movies and songs Conference and Workshops on Recent Advances and Innovations in
that can help them calm down and relax. Engineering (ICRAIE), Jaipur, India, 2018, pp. 1-6, doi:
V. Unification with other technologies: Emotion 10.1109/ICRAIE.2018.8710406.
detection technology can be integrated with other [11] S. Pandey, S. Handoo and Yogesh, "Facial Emotion Recognition using
technologies, such as virtual reality and augmented Deep Learning," 2022 International Mobile and Embedded Technology
Conference (MECON), Noida, India, 2022, pp. 248-252, doi:
reality, to create new practical insights. For example, 10.1109/MECON53876.2022.9752189.
a virtual reality headset could recommend movies and [12] M. D. Daniyal and V. Balu, "Api-Based Music Recommender System,"
songs that matches with the user's current state of mind 2022 11th International Conference on System Modeling & Advancement
and display appropriate movies or songs that creates in Research Trends (SMART), Moradabad, India, 2022, pp. 1104-1108,
doi: 10.1109/SMART55829.2022.10047230.
better experience.
[13] T. Saito, E. Sato-Shimokawara and L. -H. Chen, "Fusion-based Music
Overall, a movie and song recommender system using Recommender System Using Music Affective Space based on
emotion detection technology has a lot of capability for Serendipity," 2022 International Conference on Technologies and
Applications of Artificial Intelligence (TAAI), Tainan, Taiwan, 2022, pp.
improving user experiences and generating new revenue 171-176, doi: 10.1109/TAAI57707.2022.00039.
streams. As the technology continues to improve, we can expect
[14] K. Patel and R. K. Gupta, "Song Playlist Generator System Based on
to see more exciting and innovating applications in the next Facial Expression and Song Mood," 2021 International Conference on
generation. Artificial Intelligence and Machine Vision (AIMV), Gandhinagar, India,
2021, pp. 1-6, doi: 10.1109/AIMV53313.2021.9670976.
[15] A. Gatzioura, J. Vinagre, A. M. Jorge and M. Sànchez-Marrè, "A Hybrid
Recommender System for Improving Automatic Playlist Continuation,"
REFERENCES in IEEE Transactions on Knowledge and Data Engineering, vol. 33, no.
[1] R. E. Nakhli, H. Moradi and M. A. Sadeghi, "Movie Recommender 5, pp. 1819-1830, 1 May 2021, doi: 10.1109/TKDE.2019.2952099.
System Based on Percentage of View," 2019 5th Conference on

694

Authorized licensed use limited to: Zhejiang University. Downloaded on August 25,2024 at 07:08:13 UTC from IEEE Xplore. Restrictions apply.