Contents

Acknowledgements viii

List of Figures xiii

List of Tables xiv

Abbreviations xv

1 Introduction 1
1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.1 Primary Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.2 Secondary Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.3 Research Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 Literature Review 6
2.1 Evolution of Music Recommendation Systems . . . . . . . . . . . . . . . . . 6
2.1.1 Traditional Approaches: Collaborative and Content-Based Filtering 6
2.1.2 Integration of Deep Learning: Hybrid Models . . . . . . . . . . . . . 7
2.2 Emotion Recognition in Music Recommendation . . . . . . . . . . . . . . . 8
2.3 Real-Time Emotion Detection: Mediapipe Face Mesh . . . . . . . . . . . . . 9
2.4 Retrieval-Augmented Generation (RAG) in Recommendation Systems . . . 10
2.5 Multimodal Recommender Systems . . . . . . . . . . . . . . . . . . . . . . . 11
2.6 Proposed System: Integration of Emotion Recognition and RAG-LLM . . . 12
2.7 Challenges and Future Directions . . . . . . . . . . . . . . . . . . . . . . . . 13

3 Proposed System Architecture 17

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3 Detailed Architecture Components . . . . . . . . . . . . . . . . . . . . . . . 18
3.3.1 RAG-LLM Based Recommender System . . . . . . . . . . . . . . . . 18

ix
Contents x

3.3.1.1 Data Ingestion and Processing Pipeline . . . . . . . . . . . 19

3.3.1.2 Query Construction and Context Fusion . . . . . . . . . . . 19
3.3.1.3 Retrieval Mechanism . . . . . . . . . . . . . . . . . . . . . 20
3.3.1.4 RAG Processing and Recommendation Generation . . . . . 21
3.3.1.5 Feedback Integration and Continuous Learning . . . . . . . 22
3.3.2 Deep Learning Model For Facial Emotion Recognition . . . . . . . . 23
3.3.2.1 Input Processing and Facial Feature Extraction . . . . . . 23
3.3.2.2 Hierarchical Convolutional Architecture . . . . . . . . . . . 24
3.3.2.3 Classification Head Design . . . . . . . . . . . . . . . . . . 25
3.3.2.4 Training and Optimization Strategy . . . . . . . . . . . . . 25
3.3.2.5 Real-time Performance Optimization . . . . . . . . . . . . 26
3.3.3 Database Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.3.1 Vector Database Implementation . . . . . . . . . . . . . . . 27
3.3.3.2 Document Database Structure . . . . . . . . . . . . . . . . 28
3.3.3.3 Database Integration and Synchronization . . . . . . . . . 30
3.3.3.4 Scalability and Performance Considerations . . . . . . . . . 31
3.4 System Integration and Workflow . . . . . . . . . . . . . . . . . . . . . . . . 31
3.4.1 Real-time Emotion Processing Pipeline . . . . . . . . . . . . . . . . 32
3.4.2 Recommendation Generation Process . . . . . . . . . . . . . . . . . 32
3.4.3 Feedback Processing and System Learning . . . . . . . . . . . . . . . 33
3.5 Deployment Architecture and System Requirements . . . . . . . . . . . . . 33
3.5.1 Deployment Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.5.2 System Requirements and Specifications . . . . . . . . . . . . . . . . 34

4 Methodology 36
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.2 RAG Mathematical Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.2.1 Vector Representations . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.2.1.1 Document Embeddings . . . . . . . . . . . . . . . . . . . . 38
4.2.1.2 Query Embeddings . . . . . . . . . . . . . . . . . . . . . . 38
4.2.2 Document Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.2.2.1 Similarity Calculation . . . . . . . . . . . . . . . . . . . . . 39
4.2.2.2 Top-K Retrieval . . . . . . . . . . . . . . . . . . . . . . . . 39
4.2.3 Generation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.2.3.1 Context Formation . . . . . . . . . . . . . . . . . . . . . . 39
4.2.3.2 Generation Probability . . . . . . . . . . . . . . . . . . . . 39
4.2.4 Training and Optimization . . . . . . . . . . . . . . . . . . . . . . . 40
4.2.4.1 Retriever Loss . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.2.4.2 Generator Loss . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.2.4.3 Combined Loss . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.2.5 Scoring for Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.2.5.1 Dense Retrieval Score . . . . . . . . . . . . . . . . . . . . . 41
4.2.5.2 Sparse Retrieval Score (Optional BM25) . . . . . . . . . . 41
4.2.5.3 Hybrid Score . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Contents xi

4.2.6 Model Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.2.6.1 Retriever Parameters . . . . . . . . . . . . . . . . . . . . . 41
4.2.6.2 Generator Parameters . . . . . . . . . . . . . . . . . . . . . 41
4.2.6.3 Parameter Update . . . . . . . . . . . . . . . . . . . . . . . 42
4.3 Deep Learning Model for Facial Emotion Recognition . . . . . . . . . . . . 42
4.3.1 Dataset Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.3.2 Preprocessing Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.3.3 Deep Convolutional Neural Network Architecture . . . . . . . . . . . 44
4.3.3.1 Convolutional Block Design . . . . . . . . . . . . . . . . . . 44
4.3.4 Real-Time Processing Pipeline . . . . . . . . . . . . . . . . . . . . . 45
4.3.4.1 Adaptive Frame Rate Processing . . . . . . . . . . . . . . . 46
4.3.4.2 GPU Acceleration and Memory Management . . . . . . . . 46
4.3.5 Addressing Real-world Challenges . . . . . . . . . . . . . . . . . . . 46
4.3.6 Training Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.3.7 Integration with Music Recommendation Engine . . . . . . . . . . . 47
4.3.8 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.3.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5 Implementation 49
5.1 Facial Emotion Recognition Module . . . . . . . . . . . . . . . . . . . . . . 49
5.1.1 Real-Time Landmark Detection . . . . . . . . . . . . . . . . . . . . . 50
5.2 Deep Convolutional Neural Network (DCNN) for Emotion Classification . . 50
5.2.1 Architecture Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.2.2 Regularization and Training . . . . . . . . . . . . . . . . . . . . . . . 51
5.3 RAG-LLM Based Recommendation Engine . . . . . . . . . . . . . . . . . . 51
5.3.1 Model Inference and Prompt Engineering . . . . . . . . . . . . . . . 51
5.3.2 Vector Storage and Similarity Search . . . . . . . . . . . . . . . . . . 52
5.4 Development and Testing Interface . . . . . . . . . . . . . . . . . . . . . . . 52
5.4.1 User Interaction Features . . . . . . . . . . . . . . . . . . . . . . . . 52
5.5 User Interface Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.6 Database Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.6.1 Custom Indexing Strategies . . . . . . . . . . . . . . . . . . . . . . . 55
5.6.2 Caching Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.6.3 Optimizing Index Structures . . . . . . . . . . . . . . . . . . . . . . 55
5.6.4 Scalability and Performance . . . . . . . . . . . . . . . . . . . . . . . 56
5.7 Management of User Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.7.1 Optimized Schemas for User Data . . . . . . . . . . . . . . . . . . . 56
5.7.2 Flexible Querying Capabilities . . . . . . . . . . . . . . . . . . . . . 57
5.7.3 Scalability and Performance . . . . . . . . . . . . . . . . . . . . . . . 57
5.7.4 Data Security and Privacy . . . . . . . . . . . . . . . . . . . . . . . . 57
5.8 System Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.8.1 Error Handling and Logging . . . . . . . . . . . . . . . . . . . . . . . 58
5.9 Future Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.9.1 Development Environment . . . . . . . . . . . . . . . . . . . . . . . 59
Contents xii

5.9.2 Regular Testing and Validation . . . . . . . . . . . . . . . . . . . . . 59

5.10 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

6 Testing and Results 61

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
6.2 Overview of the Emotion-Aware Music Recommendation System . . . . . . 61
6.2.1 Hybrid Recommendation System . . . . . . . . . . . . . . . . . . . . 62
6.3 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
6.4 Findings from the Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 63
6.4.1 Facial Emotion Recognition System . . . . . . . . . . . . . . . . . . 63
6.4.2 RAG-LLM Based Recommendation System . . . . . . . . . . . . . . 64
6.4.3 Key Performance Insights . . . . . . . . . . . . . . . . . . . . . . . . 66
6.4.3.1 Why RAG Outperforms . . . . . . . . . . . . . . . . . . . . 66
6.4.4 Limitations of the Hybrid System . . . . . . . . . . . . . . . . . . . . 67
6.5 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

7 Conclusion & Future Scope 72

7.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
7.1.1 Key Performance Insights . . . . . . . . . . . . . . . . . . . . . . . . 73
7.1.1.1 Key Findings . . . . . . . . . . . . . . . . . . . . . . . . . . 73
7.1.1.2 Why RAG Outperforms . . . . . . . . . . . . . . . . . . . . 74
7.1.2 Implications for Future Research . . . . . . . . . . . . . . . . . . . . 75
7.2 Future Scope of Emotion-Aware Music Recommendation System . . . . . . 76
7.2.1 Multi-Modal Emotion Recognition . . . . . . . . . . . . . . . . . . . 76
7.2.2 Enhanced Personalization and User Profiling . . . . . . . . . . . . . 77
7.2.3 Broader Applications Beyond Music . . . . . . . . . . . . . . . . . . 78
7.2.4 Integration with Emerging Technologies . . . . . . . . . . . . . . . . 78
7.2.5 Addressing Ethical and Privacy Concerns . . . . . . . . . . . . . . . 79
7.2.6 Continuous Improvement and Adaptation . . . . . . . . . . . . . . . 80
7.2.7 Cultural Adaptation and Globalization . . . . . . . . . . . . . . . . . 80
7.2.8 Scalability and Performance Optimization . . . . . . . . . . . . . . . 81
7.2.9 Partnerships and Collaborations . . . . . . . . . . . . . . . . . . . . 81
7.2.10 User Education and Awareness . . . . . . . . . . . . . . . . . . . . . 82
7.2.11 Addressing Limitations and Challenges . . . . . . . . . . . . . . . . 82
7.2.12 Fostering Innovation and Creativity . . . . . . . . . . . . . . . . . . 83

Bibliography 85
List of Figures

3.1 High-level architecture of the proposed RAG-LLM based emotion-aware

music recommendation system. . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.1 RAG Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.2 Count of Emotions in the Dataset . . . . . . . . . . . . . . . . . . . . . . . 44
4.3 Architecture of the Deep Convolutional Neural Network for facial emotion
recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

5.1 API Testing Interface using Gradio . . . . . . . . . . . . . . . . . . . . . . . 52

5.2 User Interface using Streamlit . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.3 API Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.4 User Interface using Kivy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

6.2 DCNN model accuracy and Loss . . . . . . . . . . . . . . . . . . . . . . . . 64

6.3 Comparison of Recall between RAG-LLM and hybrid recommendation sys-
tems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
6.4 Comparison of Precision between RAG-LLM and hybrid recommendation
systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.5 Comparison of Cold Start Performance between RAG-LLM and hybrid rec-
ommendation systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.6 Comparison of User Satisfaction rate between RAG-LLM and hybrid rec-
ommendation systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6.7 Comparison of NDCG between RAG-LLM and hybrid recommendation sys-
tems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6.8 Relevance vs Diversity Tradeoff between RAG-LLM and hybrid recommen-
dation systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6.9 Comparison of Execution Time between RAG-LLM and hybrid recommen-
dation systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
6.10 Comparison of Query Complexity between RAG-LLM and hybrid recom-
mendation systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
6.11 Comparison of overall performance metrics between RAG-LLM and hybrid
recommendation systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

xiii
List of Tables

2.1 Summary of Literature Review on Music Recommendation Systems . . . . 14

6.1 Summary of performance metrics for RAG-LLM vs. Hybrid recommenda-

tion systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

xiv
Abbreviations

AUJ Amity University Jharkhand

CSE Computer Science Engineering

NTCC Non-Teaching Credit Courses

RAG Retrieval-Augmented Generation

LLM Large Language Model

DCNN Deep Convolutional Neural Network

AI Artificial Intelligence

XAI Explainable Artificial Intelligence

AR Augmented Reality

VR Virtual Reality

NDCG Normalized Discounted Cumulative Gain

xv
Chapter 1

Introduction

Music has been valued always as a tool that has an impact on human feelings, state of
mind, even their health. Being in the digital era where we have access to millions of
songs in different platforms the problem is not about acquiring music content but about
finding out which content is of relevance to the current mood, and disposition of the
users. The seemingly complex modern recommendation algorithms used in conventional
music recommender systems lack the capability to model the adaptive characteristics of
human emotions and their correlation with musical tastes. Thanks to the onset of artificial
intelligence and specifically machine learning, the ways of human and computer interac-
tion with digital content significantly changed, however, today’s music recommendation
systems remain globally based on passive data, namely previous consumption behaviour,
given ratings and age/gender/interest data. This approach, while practical, results in a
large division between the user’s immediate emotional demands and what is suggested to
them. The introduction of the real-time emotion recognition technology into the current
recommendation systems offers a chance of closing this gap and exploiting the difference
to develop a more intimate and sensitive music listening pattern. This study present an
approach involving deep learning facial emotion recognition system with a RAG-LLM rec-
ommendation system. It is thus possible for the system to keep track of the users’ emotional
state by analysing their facial expressions while at the same time keeping a high level of
consistency with the previously provided musical selections. We utilize modern technolo-
gies such as MediaPipe Face Mesh for facial landmarks detection, Convolutional Neural
Networks (CNN) classifiers for the identification of emotions, and the RAG architecture for
the contextual recommendation of music. The integration of these two approaches forms a
fresh model that eliminates some of the challenges of basic recommendation systems while
opening up new opportunities in targeted emotion-based content delivery.
1
Chapter 1. Introduction 2

1.1 Problem Statement

The current landscape of music recommendation systems faces several significant chal-
lenges that limit their effectiveness in providing truly personalized user experiences:

1. Cold Start Problem:

• New users often experience low levels of personalization because there are few
applicable parameters available.
• Unlike modern recommendation systems, traditional ones require extensive in-
teraction records to offer suitable recommendations.
• As a result, initial suggestions frequently miss the mark in reflecting user pref-
erences.
• This paper also notes that user engagement tends to be lower during the early
stages of system usage.

2. Static Nature of Traditional Systems:

• Existing systems rely heavily on past experiences and straightforward evalua-

tions.
• They do not adapt to consider the users’ current emotional state.
• Recommendations remain static, regardless of the user’s context.
• There is a failure to address users’ needs in real-time.

3. Limited Contextual Understanding:

• Today’s systems primarily focus on genre preferences and listening habits.

• There is a lack of assessment regarding emotional knowledge in the recommen-
dation process.
• They fail to recognize the connection between mood and selected songs.
• They also struggle to meet situational listening requirements.

4. User Engagement and Satisfaction:

• Generic recommendations lead to decreased user engagement.

• There is minimal emotional connection with the recommended content.
• Users often leave the page quickly due to irrelevant suggestions.
Chapter 1. Introduction 3

• Sustaining user engagement over time remains a challenge.

5. Feedback Loop Limitations:

• Dependence on direct feedback systems

• Slow response to evolving user preferences
• Inability to effectively gather implicit feedback
• Lack of integration of contextual cues

6. Privacy and Data Collection:

• Conventional systems necessitate significant historical data gathering

• Concerns about the privacy of user data storage and usage
• Challenges in delivering personalization while ensuring privacy
• Reliance on continuous user tracking

1.2 Objectives

The main goals of this research project are aimed at tackling the challenges mentioned
earlier and pushing forward the development of emotion-aware music recommendation
systems:

1.2.1 Primary Objectives

1. Real-time Emotion Recognition System Development:

• Utilize MediaPipe Face Mesh for precise facial landmark detection

• Create and train a CNN model for effective emotion classification
• Enable real-time processing for ongoing emotion tracking
• Ensure accurate identification of seven different emotional states

2. Advanced RAG-LLM Recommendation Engine Implementation:

• Build a sophisticated vector database for storing music metadata

• Develop context-sensitive prompt engineering methods
• Set up efficient retrieval systems for selecting relevant music
Chapter 1. Introduction 4

• Design adaptive recommendation algorithms that align emotional context with

user preferences

3. Emotion-Aware Features Integration:

• Facilitate smooth communication between emotion recognition and recommen-

dation components
• Create real-time adaptation methods for dynamic content delivery
• Establish feedback loops for ongoing system enhancement
• Implement algorithms for learning user preferences

1.2.2 Secondary Objectives

1. System Optimization:

• Reduce latency in real-time processing

• Optimize resource use for effective operation
• Improve scalability for large-scale implementation
• Apply efficient data management techniques

2. User Experience Improvement:

• Design user-friendly interfaces for system interaction

• Ensure smooth transitions between recommendation phases
• Develop clear feedback mechanisms
• Create engaging visual representations of emotion recognition outcomes

3. Privacy and Security:

• Implement secure procedures for handling data

• Safeguard user privacy in emotion detection
• Develop methods for anonymous data collection
• Create secure storage solutions for user preferences
Chapter 1. Introduction 5

1.2.3 Research Objectives

1. Advancement of Knowledge:

• Enhance understanding of the relationship between emotion and music

• Investigate innovative approaches to context-aware recommendation systems
• Examine the effectiveness of real-time emotion detection in delivering content
• Analyze how emotional awareness influences user satisfaction

2. Technical Innovation:

• Create new strategies for emotion-aware recommendation systems

• Develop fresh methodologies for processing emotions in real-time
• Improve the integration of large language models in recommendation systems
• Establish new frameworks for delivering context-aware content

Successfully achieving these objectives will lead to a groundbreaking system that overcomes
the limitations of existing music recommendation systems while introducing new features
for emotion-aware content delivery. This research aims to showcase the practicality and
effectiveness of merging real-time emotion recognition with advanced recommendation
systems, potentially paving the way for personalized content delivery in various fields.

By accomplishing these goals, this research project aspires to develop a more intuitive,
responsive, and personalized music recommendation system that enhances user experience
while pushing the boundaries of emotion-aware computing and content recommendation
systems.
Chapter 2

Literature Review

2.1 Evolution of Music Recommendation Systems

2.1.1 Traditional Approaches: Collaborative and Content-Based Filter-

ing

Music recommendation technologies use primarily two established approaches which in-
clude collaborative filtering and content-based filtering methods. Users get music recom-
mendations from collaborative filtering by tracking how otherusers construct patterns with
their music choices for recommendations to users with matching preferences. Two method
categories operate in recommendations through collaborative filtering: memory-based ap-
proaches compute user-item and item-item similarities directly and model-based methods
use machine learning to extract hidden factors explaining user preferences [schedl2018].

Music recommendation frameworks which use content-based filtering evaluate music char-
acteristics to find songs compatible with user choices. The profiling process within these
systems retrieves item characteristics from data signals, metadata, and written descrip-
tions [pazzani2007].

Both content and collaborative filtering methods have important disadvantages despite
their common application. The collaborative filtering method encounters two major hur-
dles with insufficient data for new users and items known as ”cold start” [adomavicius2005].
Content-based recommendation systems have a limitation since they produce results that
mirror previous user preferences which reduces discovery opportunities [mcnee2006]. These

6
Chapter 2. Literature Review 7

recommendation systems fail to adjust their suggestions for users as they experience con-
textual changes such as mood variation and situational adaptations thus providing appro-
priate recommendations for the present time [schedl2018].

The initial solution to addressing these issues resulted in hybrid systems which integrated
collaborative and content-based recommendation methods. The study by Burke (2002) es-
tablished three categories for hybrid recommendation systems as weighted hybrids, switch-
ing hybrids, and feature combination hybrids [burke2002]. These techniques showed better
performance results in different assessment factors leading to advanced integration meth-
ods.

2.1.2 Integration of Deep Learning: Hybrid Models

Deep learning has transformed music recommendation systems through hybrid models
which merge collaborative methods with content delivery systems by using neural net-
works. The new architecture uses automatic learning to handle diverse combinations and
interrelationships among data elements that traditional approaches struggle to achieve
[covington2016].

The use of Convolutional Neural Networks (CNNs) extracts advanced features from audio
information to increase the system understanding beyond simple metadata capabilities.
Liu et al. (2017) implemented CNNs to evaluate spectrogram analysis for music emotion
classification through temporally and spectrally sensitive information processing [liu2017].

Experts have made progress by connecting CNNs to RNNs to assist music analyses of
temporal patterns. The research team of Choi et al. (2016) presented Convolutional
Recurrent Neural Networks (CRNNs) dedicated to music classification by merging CNN
features with RNN sequential modeling abilities [choi2016]. Studies have proved that this
combination of architectures delivers better performance when used for music tagging
which points to its availability for recommendation systems.

The use of deep learning techniques expanded its scope from content analysis toward
the implementation of collaborative filtering methods. The research team of He et al.
(2017) developed Neural Collaborative Filtering (NCF) by substituting traditional matrix
factorization inner products with neural components that capture data-driven arbitrary
functions [he2017]. NCF achieved superior performance than traditional algorithms to
solve benchmark datasets which proved deep learning works effectively in collaborative
filtering applications.
Chapter 2. Literature Review 8

2.2 Emotion Recognition in Music Recommendation

The personalization of music recommendations has become directly dependent on user

emotional understanding for optimization of service quality. According to juslin2013 the
music-emotion connection presents multiple dimensions since music expresses emotional
content while generating emotional responses from listeners [juslin2013].

Systematic recognition of emotions operates through multiple interfaces that read facial
signals together with speech patterns and physiological patterns in order to identify user
emotions. The different detection systems possess separate strengths and difficulties. The
analysis of facial expressions provides researchers with widespread opportunities but en-
counters performance limitations from cultural diversity as well as natural variations be-
tween individuals [jack2012]. The recognition of emotions through speech proves useful
for voice-command systems yet faces constraints because of different language patterns
[el2011].

Scientists from Tzirakis et al. (2017) developed an end-to-end process which employed
deep neural networks to merge auditory data with visual information and achieve precise
emotional state identification [tzirakis2017]. The system first used CNNs to obtain features
from both face images and audio spectrograms before applying RNNs to analyze temporal
patterns. Multi-modality data combination within their system enhanced performance
results against single-modal recognition approaches.

Malik et al. (2017) analyzed the application of stacked CNNs and RNNs in music emo-
tion recognition where audio characteristics were mapped onto a valence-arousal two-
dimensional space [malik2017]. The innovative approach by Malik et al. (2017) demon-
strated a new high benchmark on the MediaEval2015 dataset thus proving the exceptional
ability of deep learning to identify subtle emotions in music.

The research by Yang et al. (2018) introduced an emotional song recommendation system
which utilized face emotion reading technology for user feeling detection leading to suitable
song recommendations [yang2018]. The system applied Ekman’s model to assign six basic
emotions which it matched to specific music characteristics. The system achieved better
user satisfaction than systems lacking emotional state consideration especially when users
exhibited major preference changes based on their mood.
Chapter 2. Literature Review 9

2.3 Real-Time Emotion Detection: Mediapipe Face Mesh

Modern emotion detection in real-time functions through the use of Mediapipe Face Mesh
that delivers precise facial landmark recognition. Google created Mediapipe Face Mesh
which operates on devices to precisely detect 468 facial landmarks so users can analyze
facial expressions and movements [kartynnik2019]. The mobile-based system functions in
real-time thus becoming relevant for emotion-based music applications.

Mediapipe Face Mesh operates through stages which commence by detecting faces and
subsequently identifies facial landmarks which get tracked across consecutive frames. The
solution reaches high accuracy levels while operating at high speed which ensures smooth
performance for real-time frameworks [lugaresi2019].

The analysis of facial expressions through systems helps determine user emotions which
allows them to deliver personalized music suggestions according to the current emotional
state. The researchers at Li et al. (2020) implemented Mediapipe Face Mesh to detect
emotions by deriving geometric characteristics from facial points that were trained through
a neural network [li2020]. Their method performed similarly to complex models in terms
of accuracy and operated at real-time velocity.

The research of Kulkarni et al. (2021) merged Face Mesh from Mediapipe into a music
recommendation system which modified playlist ideas through emotion detection capabil-
ities [kulkarni2021]. Through their system they extracted facial action units from facial
landmarks which a pre-trained classifier converted into emotional states. Research par-
ticipants expressed strong positive feedback regarding emotional recommendation systems
because these suggestions proved beneficial during emotionally-charged conditions.

Facial analysis systems require privacy to be their primary concern. All processing takes
place directly on a device through Mediapipe Face Mesh thus preventing the transmission
of facial images to remote servers [google2020]. The design decision adheres to present-day
privacy-protecting AI techniques which enable users to obtain emotion-aware suggestions
without exposing their biometric information.
Chapter 2. Literature Review 10

2.4 Retrieval-Augmented Generation (RAG) in Recommen-

dation Systems

RAG introduces a new approach which connects retrieval functionality with generative
models to enhance recommendation relevance and personalization capabilities. RAG
emerged through Lewis et al. (2020) work for question-answer solutions but it now serves
multiple domains such as recommending systems (lewis2020).

The standard configuration of Recommendation Systems using RAG consists of three core
components made up of a retriever followed by a knowledge base and finishing with a
generator. Through the retriever component the system extracts needed items or infor-
mation points from its knowledge base using user queries or contextual understanding
[zamani2022].

RAG frameworks utilize LLMs as their primary generator component due to these models’
capacity to understand natural language and support multiple context needs. The research
by Hou et al. (2022) introduced a recommendation platform based on RAG with LLM
components that produces adaptive suggestions and natural language explanations for
each recommendation [hou2022].

Such an approach enables music systems to understand users’ present situation and his-
tory of preferred content which results in improved recommendations that deliver better
satisfaction. The authors Aggarwal et al. (2021) introduced a music recommendation sys-
tem which retrieves song data besides artist and style metadata from queries and provides
personalized song suggestions along with specific explanations on appeal factors [aggar-
wal2021].

Increasing application accuracy in RAG-based recommendation requires better integration

of emotional context capabilities. Wang et al. (2023) developed RecoRAG as a framework
to deliver recommendations that adapt to users requirements [wang2023]. The RecoRAG
framework which combines emotional context in retrieval and generation processes pre-
sented better performance than standard methods for recommendations based on users
needs.
Chapter 2. Literature Review 11

2.5 Multimodal Recommender Systems

Multimodal recommender systems represent a major field development because they merge
visual and auditory and textual data sources to create an extensive user preference un-
derstanding. Through multiple inputs multimodal systems surpass unimodal systems by
extracting joint aspects from user behaviors and items which creates a foundation for
better and more reliable recommendations [baltrusaitis2019].

Multiple information types consisting of audio features, lyrics, social tags, album artwork
and contextual data serve as sources for music recommendation systems. The researchers
at Oramas et al. (2017) built a multimodal deep learning architecture combining audio
signals together with text descriptions as well as visual information to recommend music
[oramas2017]. The research evaluation showed that integrating multiple kinds of informa-
tion produced better recommendation precision than single-data methods especially for
rare items because sparse data exists in individual information sources.

Context-aware music recommendation extends multimodal techniques by using content

and collaborative information together with situational elements that include time, loca-
tion, activity and social environment. The framework introduced by Schedl et al. (2014)
demonstrated that tailoring recommendations based on detected environmental conditions
leads to better user satisfaction [schedl2014].

The integration of ongoing emotional detection methods with listener history allows mul-
timodal systems to readjust their music suggestions for individual users based on their
present sentiment and situation. The analysis system of Yang et al. (2020) featured
emotion-aware music recommendations powered by collaborative filtering but adapted
through facial expression detection for better emotional state understanding [yang2020].

Multi-modal recommendation systems experienced additional development through the

growth of cross-modal learning techniques. Circular Cross-Modal Music Recommenda-
tion sets forth a framework by Xiao et al. (2022) to synchronize representation data
across audio together with lyrical and visual media [xiao2022]. Shared embeddings in
their framework detect musical content correlations which results in better multimodal
information integration for recommendation purposes.
Chapter 2. Literature Review 12

2.6 Proposed System: Integration of Emotion Recognition

and RAG-LLM

The system proposal combines CNN-based emotion recognition and Mediapipe Face Mesh
with Retrieval-Augmented Generation (RAG) that utilizes Large Language Models (LLMs)
as its framework. The system unites modern technology elements from three domains in-
cluding visual recognition for emotion analysis and retrieval of related music content with
artificial generation functionality for customized music suggestions.

The proposed system delivers its design through multiple components which work together
as one system. The emotion recognition system employs Face Mesh from Mediapipe to
detect facial markings that a CNN processor analyzes against emotion databases for user
emotional identification. Recommendations enter the pipeline from the recommendation
module that combines RAG architecture with an LLM to access knowledge base music
metadata which generates individualized recommendations.

Using real-time user emotional data and contextual signals the integration system creates
personalized music recommendations which respond to users’ current emotional states.
The system differs from regular approaches because it welcomes current emotional data
instead of historic preferences and applies LLMs for detailed suggestions and provides
explanations through natural language.

This adaptive recommendation system covers different applications that include main-
taining emotional alignment through musical suggestions for current moods while also
managing emotions toward specific desired states [saarikallio2011]. Such integrated sys-
tems underwent initial assessment which yielded optimistic findings. According to Chen
et al. (2022) emotion-aware music recommendation methods yield better user satisfaction
results by 27% [chen2022].

This method aims to improve recommendation systems by fixing their existing weaknesses
to provide better user satisfaction and engagement through context-sensitive interactions
which demonstrate empathy. A system that adapts in real-time to user emotions creates a
more interactive user interface which may establish better bonds between users and their
recommended content.
Chapter 2. Literature Review 13

2.7 Challenges and Future Directions

The combination of emotion recognition technology with RAG-LLM for music recommen-
dation holds substantial promise but multiple barriers continue to appear. The real-time
operation poses technical difficulties mainly because of advanced model deployment re-
quirements on consumer devices. The optimization process seeks to achieve the right
balance between complex models and fast computation speeds [howard2019].

Privacy and security concerns are paramount when dealing with facial data and personal
preferences. While Mediapipe’s on-device processing mitigates some risks, comprehen-
sive safeguards must be implemented throughout the system, including secure storage of
preference data and transparent user controls [voigt2017].

Music preferences demonstrate complex relationships with human emotions that vary dif-
ferently among individual people. General emotional relationships exist between facial
expressions and musical preferences yet personal opinions express themselves distinctively
because of cultural heritage and musical skill and life experiences [eerola2013]. Future re-
search should focus on creating adapted emotion-music mapping systems that understand
each user individually.

The future of this research will explore multi-signal emotion detection using facial ex-
pressions together with voice patterns and physical measurement data for better accuracy
[tzirakis2017] as well as specialized music for personal therapeutic use [saarikallio2011] with
systems that offer visibility into their algorithms for improved trust and usage [zhang2020].

The adaptability of cross-cultural environments produces dual effects from cultural dif-
ferences between emotional expressions because music interpretation becomes more com-
plicated [jack2012]. The expansion of culturally sensitive emotion recognition techniques
alongside music recommendation systems would allow worldwide use of these systems while
accepting different cultural interpretations of emotions and music meanings.
Chapter 2. Literature Review 14

Table 2.1: Summary of Literature Review on Music Recommendation Systems

Section Key Points References

Evolution of Mu- Two primary approaches: [Schedl et al., 2018; Adomavi-
sic Recommenda- Collaborative Filtering and cius et al., 2005; Burke, 2002]
tion Systems Content-Based Filtering. Col-
laborative filtering relies on
user patterns; content-based
filtering evaluates music char-
acteristics. Hybrid systems
combine both methods to
overcome limitations like cold
start and lack of diversity.
Integration of Deep learning enhances [Covington et al., 2016; Liu et
Deep Learning recommendation systems al., 2017; Choi et al., 2016; He
by merging collaborative et al., 2017]
and content-based methods.
CNNs extract advanced fea-
tures from audio; CRNNs
combine CNNs with RNNs
for better classification. Neu-
ral Collaborative Filtering
(NCF) improves performance
over traditional methods.
Emotion Recogni- Personalization relies on [Juslin et al., 2013; Tzirakis et
tion in Music Rec- understanding user emotions. al., 2017; Malik et al., 2017;
ommendation Various detection systems Yang et al., 2018]
(facial, speech, physiolog-
ical) have strengths and
limitations. Multi-modal
approaches improve emotion
recognition accuracy.
Chapter 2. Literature Review 15

Section Key Points References

Real-Time Emo- Mediapipe Face Mesh de- [Kartynnik et al., 2019; Li
tion Detection: tects 468 facial landmarks et al., 2020; Kulkarni et al.,
Mediapipe Face for real-time emotion analy- 2021]
Mesh sis. High accuracy and speed
make it suitable for emotion-
based music applications. Pri-
vacy concerns addressed by
on-device processing.
Retrieval- RAG combines retrieval [Lewis et al., 2020; Hou et al.,
Augmented Gen- and generative models for 2022; Wang et al., 2023]
eration (RAG) in enhanced recommendations.
Recommendation Utilizes LLMs for natural
Systems language understanding and
context adaptation. Emo-
tional context integration
improves recommendation
accuracy.
Multimodal Merge visual, auditory, and [Baltrusaitis et al., 2019; Ora-
Recommender textual data for better user mas et al., 2017; Yang et al.,
Systems understanding. Context- 2020]
aware recommendations
improve user satisfaction.
Cross-modal learning tech-
niques enhance integration of
different data types.
Proposed Sys- Combines CNN-based emo- [Chen et al., 2022; Saarikallio
tem: Integration tion recognition with RAG et al., 2011]
of Emotion and LLMs for personalized
Recognition and music suggestions. Real-time
RAG-LLM emotional data enhances user
engagement. Initial assess-
ments show improved user
satisfaction.
Chapter 2. Literature Review 16

Section Key Points References

Challenges and Technical challenges in real- [Howard et al., 2019; Voigt et
Future Directions time operation and model de- al., 2017; Eerola et al., 2013]
ployment. Privacy and se-
curity concerns regarding fa-
cial data. Need for culturally
sensitive emotion recognition
techniques.
Chapter 3

Proposed System Architecture

The architecture of a proposed system resembles a very recent amalgamation of several

advanced technologies including deep learning, information retrieval and database man-
agement systems. This framework, in its entirety, is intended to be used in service of
building a music recommendation platform of sophisticated information, powerful individ-
ualization, and particularly rich emotions, with high accuracy and minimal latency. The
system creates a new paradigm of personalized music discovery experience by seamlessly
marrying the real time emotion recognition capabilities with the state of the art music
recommendation algorithms and made it robust, scalable and flexible with a database in-
frastructure. As a result, the modular and extensible architecture is capable of achieving
superior performance, maintainability and also leaves plenty of room for future enhance-
ments and feature extensions. It was painstakingly designed with features that provide
amazingly personalized musical suggestions that are in real time adapted to the emotional
state of the users to deliver a extremely responsive and organic user experience which
evolves with the users preferences and emotional patterns.

3.1 Introduction

Traditional recommendation systems sometimes fail to handle the emotional aspect of

music appreciation in the fast changing terrain of digital music consumption. This ar-
chitectural design puts emotional context front and center in the recommendation pro-
cess, so revolutionizing the process. The system seeks to close the gap between technical
capabilities and human emotional experiences in music discovery by means of advanced
deep learning techniques, sophisticated information retrieval methodologies, and optimized
17
Chapter 3. Proposed System Architecture 18

database management strategies. The next parts offer a thorough analysis of every archi-
tectural element, clarifying their particular functions and combined activities inside the
whole system structure.

3.2 System Overview

Before we examine the details, it’s crucial to grasp how the whole system functions together
and the connections between its parts. This system includes three main components that
collaborate to suggest music based on emotions:

1. RAG-LLM Based Recommender System: This serves as the system’s main

processing unit, interpreting various signals to recommend music that matches your
preferences.

2. Deep Learning Model for Facial Emotion Recognition: This component in-
terprets facial expressions to accurately determine your emotional state.

3. Hybrid Database Architecture: This acts as the storage system, effectively or-
ganizing and managing all music and user data for easy retrieval.

These elements communicate through specific interfaces and data-sharing methods, ensur-
ing smooth operation. The design allows for the separate improvement and expansion of
each part without disrupting the overall system. In the sections that follow, we will explore
each component in detail, describing their design, how they operate, and the technology
involved.

3.3 Detailed Architecture Components

3.3.1 RAG-LLM Based Recommender System

Our recommendation system, powered by RAG-LLM (Retrieval-Augmented Generation

Large Language Model), is the advanced center of our setup. It marks a big improvement
over typical recommendation methods. This system is smart and combines the strong
thinking abilities of large language models with quick ways to gather information. Because
of this, it can make music suggestions that are very good at understanding the situation
and creating an emotional connection with users.
Chapter 3. Proposed System Architecture 19

3.3.1.1 Data Ingestion and Processing Pipeline

This system is built to efficiently collect and process detailed information. It focuses on
several key areas:

• Acoustic Features: This includes sound details like how fast or slow the music is
(tempo), patterns in rhythm, how the music is arranged (harmonic structures), the
instruments used, the range of loudness and softness (dynamic range), different sound
frequencies, and how the music feels in terms of sound texture (timbral qualities).

• Genre Classifications: It identifies types of music, including main genres and

also secondary and third-level genres. It considers music that mixes different genres
together.

• Emotional Attributes: These are the feelings or moods created by the music.
They are based on scientific studies and include how happy or sad the music makes
you feel (valence), how lively it is (arousal), and how strong the emotions are.

• Contextual Information: Background details like when the song was released, its
cultural importance, details about the artist, what the lyrics talk about, and where
the music comes from.

All this information is processed using advanced computer tools designed to deeply under-
stand music. These tools create complex data structures known as ”embeddings,” which
can have many parts and help in recognizing music elements, emotional content, and
listener preferences. The system uses high-level methods to make these data structures
effective at picking up on differences.

These embeddings and detailed information are stored in the Qdrant vector database.
This database allows for quick searches and organization of the entire music library. It
uses multiple ways to sort metadata, which helps in fast searching and filtering when
recommending music.

3.3.1.2 Query Construction and Context Fusion

The recommendation process begins by creating complex queries that mix different con-
textual elements, each with its own level of influence. These queries skillfully combine
several key areas:
Chapter 3. Proposed System Architecture 20

• Real-time Emotional State Data: This involves ongoing checks of your emotions
using facial recognition technology. It assesses both primary and secondary emotions,
providing confidence scores and looking at how stable these emotions are over time.

• User Preference Profiles: These profiles give a detailed look at your music pref-
erences. They include your ratings, feedback you might give without directly saying
anything, patterns in what you like at a detailed level, and how your preferences
change in different situations.

• Historical Listening Patterns: This examines your listening habits over time. It
considers the times you prefer to listen to music, the order in which you select songs,
the characteristics of your listening sessions, and how you reacted to recommenda-
tions in the past.

• Situational Context: These are the environmental factors like the time of day,
day of the week, seasonal changes, what you’re doing at the moment (if known), and
who you’re with. Each of these can influence your music choices.

The system uses advanced methods to balance these factors according to how relevant and
reliable they are for your current situation. It employs adaptive algorithms that adjust
the importance of the factors by looking at past performance data and how appropriate
each factor is in the current context. The system also uses a smart time-based approach
to ensure recent interactions and emotions have a bigger impact, while still valuing the
established long-term preferences you’ve developed over time.

3.3.1.3 Retrieval Mechanism

In the retrieval phase, the system makes use of Qdrant’s fast search to find music similar
to what you enjoy. It employs special search methods and checks how closely the music
matches your preferences to find the best options based on what you are requesting. Here’s
how it works:

• Multi-vector Querying: The system analyzes many factors at once, such as your
mood and music preferences, to give recommendations that fit what you feel or seek.

• Temporal Pattern Recognition: It observes how your music likes and feelings
change over time. This helps the system suggest music that matches the mood of
your entire listening session, adapting to your needs as they evolve.
Chapter 3. Proposed System Architecture 25

• 3×3 Convolutional Kernels: Keeps this size for efficient detail capture without
adding too much computational burden.

• Final Max Pooling Operation: Squeezes essential features into small packages
ready for classification.

• 50% Dropout Rate: The highest dropout level here ensures the network remains
capable of generalizing to new data effectively.

Throughout the network, dropout rates increase from 40% to 50%. This strategy prevents
the model from memorizing training data too precisely, ensuring it can understand varied
faces and expressions. This gradual increase matches the deeper specialization of features
and helps avoid overfitting as the layers go deeper into the network architecture.

3.3.2.3 Classification Head Design

The classification head processes the extracted spatial features through a sophisticated
sequence that begins with flattening operations followed by passage through a dense layer
comprising 128 neurons. This dense layer, also implementing ELU activation, functions
as a high-level feature integrator, consolidating the spatial and channel information accu-
mulated by the convolutional layers into a concise and discriminative representation. The
classification head includes:

• 60% Dropout Rate: An aggressive final-stage regularization measure that en-

sures the classification decisions rely on truly robust features rather than memorized
patterns.

• Softmax Output Layer: A carefully calibrated output mechanism that generates

well-calibrated probability distributions across six distinct emotion classes: neutral,
happy, sad, surprise, fear, and anger.

3.3.2.4 Training and Optimization Strategy

The network’s training process is designed to be smart and effective, ensuring high accuracy
and the ability to work well with different kinds of data. Here’s an overview of the steps
involved:
Chapter 3. Proposed System Architecture 31

• Asynchronous Update Propagation: This is a fast system that makes sure

changes in one database are quickly reflected in the other.

There are regular processes to keep everything in sync, with a focus on keeping data
accurate between the vector and document databases. These processes have strategies
to solve problems during conflicts, like simultaneous updates or temporary system splits,
ensuring the service keeps running smoothly with minimal disruption.

3.3.3.4 Scalability and Performance Considerations

The hybrid database system is designed to meet the fast needs of detecting emotions and
dealing with the complex needs of recommendation systems. Here’s how it handles growth
and speed:

• Horizontal Scaling Support: The database can grow easily in both storage space
and processing power. This means it can handle more users and larger music libraries
without slowing down.

• Read/Write Operation Segregation: Reading and writing actions are separated

because in recommendation systems, people read data much more often than they
write it.

• Geographically Distributed Replication: The system operates in multiple re-

gions. This reduces wait times for users who are far away and makes the system
more robust against local problems.

• Resource Isolation: Resources are dedicated to specific tasks to ensure that heavy
data analysis does not slow down the real-time recommendations.

This database structure includes tools to monitor how the system is performing, which
helps it manage capacity and optimize itself to maintain speed. It also has automated
alerts to find problems before they affect users, allowing quick fixes to keep everything
running smoothly.

3.4 System Integration and Workflow

Bringing together three key parts—the RAG-LLM recommender system, the emotion
recognition model, and the hybrid database—creates a smart and connected system. This
Chapter 3. Proposed System Architecture 32

setup gives you personalized music recommendations that fit your mood. It is both efficient
and accurate. The process of how the system works involves:

3.4.1 Real-time Emotion Processing Pipeline

The emotion processing starts by capturing video frames continuously at a rate of 15-30
frames per second, which depends on the computer’s power. First, these frames go through
a preparation step. Then, they are analyzed by a facial landmark detection system that
identifies and tracks 468 important points on the face, which are vital for recognizing
emotions.

Once the facial data is ready, it’s processed by the emotion recognition DCNN. This system
analyzes the data and provides probability results for six emotion categories. These initial
results are smoothed over time and checked to ensure they give reliable and stable emotion
readings that aren’t easily disturbed by quick expression changes or detection errors.

Finally, the emotion results, along with their confidence levels and stability, are sent
immediately to the recommendation system. At the same time, they are logged in an
emotion history collection for future study and to enhance the system’s performance.

3.4.2 Recommendation Generation Process

When the system gets new updates on emotions, it starts working on suggesting music.
It does this by creating a detailed plan, which includes how the user is feeling now, what
music they usually like, the songs they recently listened to, and other important details.

The plan helps find similar music from the Qdrant database, which stores music infor-
mation. The system also applies extra filters to ensure the music matches the user’s
preferences and current situation. The first list of music choices is adjusted to include
more variety and some new songs, ensuring that the recommendations aren’t repetitive or
boring.

This improved list then goes through another process using the RAG component. This
part makes use of a big language model to create the final list of music recommendations.
This list is tailored to match the user’s feelings, ensuring that the music fits well together
and has a good flow. The final set of recommendations is shown to the user and saved in
the system for later analysis to see how well it performs.
Chapter 3. Proposed System Architecture 33

3.4.3 Feedback Processing and System Learning

When people use the recommendation system, it collects two types of feedback. The first
type is clear feedback such as ratings, skips, and likes. The second type is indirect signals,
like how often something is completed, how frequently it is replayed, or how long a person
listens. This information helps the system check how well its suggestions perform and
improve future recommendations.

The system examines this feedback right away to make quick tweaks to recommendations.
It also gathers the feedback over time in larger sets to identify bigger patterns and trends
regarding user preferences. The insights gained from this process are used to update how
user preferences are profiled, adjust the parameters for generating content suggestions,
and refine the ranking methods for recommendations.

The system is designed to be always learning and getting better. It continuously improves
the relevance and emotional connection of its recommendations by frequently upgrading
its components based on the data it collects from how users interact.

3.5 Deployment Architecture and System Requirements

3.5.1 Deployment Strategy

The system is designed to be set up in different ways to suit various needs. Here are the
options:

• Cloud-native Implementation: This uses microservices, which are small, inde-

pendent parts of the system. These are installed on cloud platforms and can auto-
matically handle more users and processing needs.

• Hybrid Edge-Cloud Processing: This splits tasks between edge devices and the
cloud. Edge devices handle tasks that need to be done quickly, like detecting emo-
tions, while the cloud handles complex tasks, such as processing recommendations.
This ensures everything works efficiently and keeps costs down.

• On-premises Enterprise Deployment: This involves setting up the whole system

within a company’s own infrastructure. It’s ideal for when privacy is very important
or when the system needs to connect with other specific tools.
Chapter 3. Proposed System Architecture 34

The setup includes features to monitor the system continuously, logging activities, and
sending alerts if issues arise. It provides real-time data about performance and reliability.
If any part of the system fails, it can automatically fix itself with little service interruption,
maintaining smooth operation.

3.5.2 System Requirements and Specifications

The reference implementation of the architecture has the following system requirements:

For Cloud Deployment:

• Compute: 8+ vCPUs for recommendation processing, 4+ vCPUs for emotion recog-

nition

• Memory: 16+ GB RAM for recommendation system, 8+ GB RAM for emotion

recognition

• Storage: 500+ GB for music database and embedding storage

• Network: Low-latency connectivity between components, 50+ Mbps external band-

width

For Edge Devices (Emotion Recognition):

• Modern CPU with 4+ cores or dedicated mobile GPU

• 4+ GB RAM

• Camera capable of 720p+ video at 15+ fps

• Stable network connection of 5+ Mbps

Software Dependencies:

• TensorFlow 2.8+ or PyTorch 1.10+ for deep learning components

• MongoDB 5.0+ for document storage

• Qdrant 0.11.0+ for vector database functionality

• FastAPI or similar for API services

• Redis for caching and message queuing

Chapter 3. Proposed System Architecture 35

Figure 3.1: High-level architecture of the proposed RAG-LLM based emotion-aware

music recommendation system.
Chapter 4

Methodology

4.1 Introduction

The method of which the built emotion-aware music recommendation system relies is a
complete and complex methodology creating a delicate context of combining real-time
facial emotion recognition with the most modern techniques of music recommendation.
However, the facial emotion recognition pipeline is where this innovative approach started,
as a meticulous video input from the user’s camera is captured and analytically processed
using the MediaPipe Face Mesh technology. Using this technology, it can identify 468
different facial landmarks, which play a critical function in understanding the user’s emo-
tional state. Then, the landmarks are processed and fed into a Deep Convolutional Neural
Network (DCNN) that has been extensively trained on very large sets of emotional ex-
pressions.

In this DCNN architecture, there are multiple convolutional blocks, each having a growing
number of filter depths of 64, 128, and 256. Thus, this layered approach makes the network
responsive to more and more facial expression features. These blocks use Exponential
Linear Unit (ELU) activation functions to aid the learning and adaptability of the model
and combine batch normalization techniques to stabilize and speed up the training process
to improve the accuracy of emotion classification. This real-time emotion detection system
is continuously watching, detecting, and categorizing the user emotions into six distinct
categories: happiness, sadness, surprise, fear, anger, and neutrality. It provides a dynamic
input stream that gives the recommendation engine a rich and nuanced understanding of
the user’s emotional landscape, allowing for music suggestions that resonate intimately
with the user’s emotions.
36
Chapter 4. Methodology 37

Rather than the actual recommendation system, it is based on a very advanced architec-
ture referred to as RAG-LLM (Retrieval-Augmented Generation Large Language Model).
By combining useful information retrieval methods that lead to practical success with
contextual understanding and personalization, the system is able to deliver highly per-
sonalized music suggestions that are both relevant and emotionally connected to the user.
First, it starts with creating a detailed music database, which converts the metadata and
characteristics of each song into high-dimensional vectors using advanced sentence trans-
formers. The embeddings contain a wealth of information, including audio features of each
track, the genre, tempo, and emotional valence, among others. This data is then stored in
a rich set of data in a Qdrant vector database, which is specifically designed for efficient
similarity searches, ensuring the system can rapidly find music that matches the user’s
musical and emotional preferences.

For systems that generate recommendations, there’s one aspect that creates complexity:
the queries that need to be generated and fired. These queries must take into account the
user’s current emotional state, their historical listening patterns, and various contextual
elements such as the time of day or user activity. To conduct similarity searches in order to
retrieve relevant music contexts, the RAG component is an important part of the system.
The prompts crafted around these contexts are designed to process them in such a way
that the resulting recommendations not only match the user’s emotional context but are
also diverse and personally relevant. The dual approach of retrieving and generating allows
the system to maintain a balance between familiarity and novelty in the user’s suggestions,
ensuring that the user remains engaged and satisfied.

A hybrid system of database architecture is employed in the system to improve both

retrieval performance and user data management. The music embeddings, along with as-
sociated metadata, are primarily stored in the Qdrant vector database as the main storage
solution, providing efficient music retrieval by similarity using advanced indexing methods.
This capability is essential for real-time operations, as it guarantees that users will receive
timely music suggestions based on their emotional state. MongoDB is also used to man-
age user-specific data, including emotional history, interaction patterns, and additional
feedback on previous recommendations. This user-centric data management strategy is
equally important, enabling the system to learn and adapt to the users’ preferences over
time and enhancing personalization for each user.

These components are integrated into a strong and dynamic framework that can respond
instantly to the user’s emotional state, providing personalized music recommendations that
evolve as the user’s preferences change. This system not only improves music discovery
Chapter 4. Methodology 38

but also enables users to have a more intense, if not more emotional, experience with the
music they listen to. The system is designed to improve continuously by analyzing user
interactions and emotional responses, making constant improvements, and delivering mu-
sic that impacts the user’s emotional state in a consistent manner, creating a lasting effect
on the user’s listening experience. Among all the technological approaches to music rec-
ommendation, this system presents an innovative way to close the gap between technology
and human emotion, resulting in a solution that is both meaningful and impactful.

4.2 RAG Mathematical Model

4.2.1 Vector Representations

4.2.1.1 Document Embeddings

Given a document collection D = {d1 , d2 , d3 , ..., dn }, we define:

ϕ : D → Rd (4.1)

where:

• ϕ is an embedding function mapping each document to a d-dimensional space.

• d is the dimension of the embedding space.

4.2.1.2 Query Embeddings

For a query q:
ψ : Q → Rd (4.2)

where Q represents the query space, and ψ maps each query to the same d-dimensional
space.
Chapter 4. Methodology 39

4.2.2 Document Retrieval

4.2.2.1 Similarity Calculation

The similarity between a query q and a document d is computed using cosine similarity:

ψ(q) · ϕ(d)
sim(q, d) = cos(ψ(q), ϕ(d)) = (4.3)
||ψ(q)|| · ||ϕ(d)||

4.2.2.2 Top-K Retrieval

The top k most relevant documents for a query q are retrieved as:

R(q) = argmaxd∈D [sim(q, d)]1:k (4.4)

where:

• R(q) denotes the top k elements.

• k is the number of documents to be retrieved.

4.2.3 Generation Process

4.2.3.1 Context Formation

The context C(q) for generation is created by concatenating the query q with the top k
retrieved documents R(q):
C(q) = [q; R(q)] (4.5)

where ; denotes concatenation.

4.2.3.2 Generation Probability

The probability of generating output y given query q is:

X
P (y | q) = P (y | z, q)P (z | q) (4.6)
z
Chapter 4. Methodology 40

where:

• y is the generated output.

• z denotes retrieved documents.

• P (y | z, q) is the probability of generating y given document z and query q.

• P (z | q) represents the retrieval probability.

4.2.4 Training and Optimization

4.2.4.1 Retriever Loss

The retriever loss encourages retrieval of relevant documents:

Lret = − log(P (z ∗ | q)) (4.7)

where z ∗ represents the relevant documents.

4.2.4.2 Generator Loss

The generator loss measures the likelihood of producing the target output y ∗ :

Lgen = − log(P (y ∗ | z, q)) (4.8)

where y ∗ is the target output.

4.2.4.3 Combined Loss

The total loss combines the retriever and generator losses:

L = αLret + βLgen (4.9)

where:

• α and β are weighting parameters, with α + β = 1.

Chapter 4. Methodology 41

4.2.5 Scoring for Retrieval

4.2.5.1 Dense Retrieval Score

The dense retrieval score is based on similarity:

Sdense (q, d) = sim(ψ(q), ϕ(d)) (4.10)

4.2.5.2 Sparse Retrieval Score (Optional BM25)

An optional sparse retrieval score, such as BM25, can be included:

Ssparse (q, d) = BM25(q, d) (4.11)

4.2.5.3 Hybrid Score

A combined retrieval score, blending dense and sparse scores, is defined as:

S(q, d) = λSdense (q, d) + (1 − λ)Ssparse (q, d) (4.12)

where λ ∈ [0, 1] is an interpolation weight.

4.2.6 Model Parameters

4.2.6.1 Retriever Parameters

The retriever parameters include the query and document embedding functions:

θret = {ψ, ϕ} (4.13)

4.2.6.2 Generator Parameters

The parameters of the language model for generation are denoted by:

θgen (4.14)
Chapter 4. Methodology 42

4.2.6.3 Parameter Update

The parameters are updated via gradient descent on the combined loss:

θ′ = θ − η∇θ L (4.15)

where:

• η is the learning rate.

• ∇θ L represents the gradient of the loss with respect to model parameters.

Figure 4.1: RAG Architecture

4.3 Deep Learning Model for Facial Emotion Recognition

A detailed deep learning scheme, combining both facial landmark detection and real-time
emotion classification has been used in the facial emotion recognition component of our
system. However, this exotic approach addresses the challenging problem of instantly iden-
tifying and categorizing human emotions accurately, while having robust generalization to
different environmental conditions and varying users. First applications such as emotion
Chapter 4. Methodology 43

aware music recommendation systems require to know users in real time and knowing user
sentiment is key to the experience.

4.3.1 Dataset Overview

The OAHEGA (Open Access Human Emotional Gesture Analysis) dataset at the core of
our emotion recognition is a dataset of about 30,000 facial expression images. The Main
Emotion of this dataset is six main emotion, happy, sad, angry, surprised, fear, neutral and
additional expression of Six Emotion expressed as “ahegao.” Features of the OAHEGA
dataset that make it so strong from diversity are the broad range of ages and ethnicities,
as well as genders, as well a lot of variation in lighting conditions and camera angles. This
diversity is very important to train a reliable emotion recognition model and generalize
well to other populations and settings.

The images in the OAHEGA dataset are each meticulously labeled with the emotion tags
on which the experts tag validate: data and ensure the accuracy and reliability of the
training data. Additionally, there are temporal sequences in the dataset that contain the
transitions of expressions to micro expressions, which are subtle facial expressions that
represent the correspondence of different emotional states. As they are particularly useful
for real-time emotion recognition tasks, where the model should be able to interpret more
than just static expressions, but also understand emotional changes over time, this tem-
poral information is of great value.

4.3.2 Preprocessing Pipeline

MediaPipe Face Mesh is a powerful tool that we use initially in our preprocessing pipeline
for the implementation of which 468 precise facial landmarks are identified from the incom-
ing video stream. Facial geometry and expression representation and these landmarks are
critical map reference points. To deal with variations in head pose and facial orientation,
the identified landmarks are normalized in multiple steps. It is essential to normalize this
in order to help if the facial expression cannot be interpreted correctly in relation to the
instigator.

The next step is to convert the captured frames to grayscale and make them of resolution
100x100 pixels. Among other things, this includes contrast enhancement and intensity
Chapter 4. Methodology 44

Figure 4.2: Count of Emotions in the Dataset

normalization, mandatory operations to have a consistent input quality. So we enforce the

uniform size and quality of all input images to make the model robust between different
lightings and camera setup. We also use different data augmentation techniques on the
training phase such as rotation, scaling, and flipping to make model generalize better and
robust against real world determinators.

4.3.3 Deep Convolutional Neural Network Architecture

The heart of our emotion recognition system is a Deep Convolutional Neural Network
(DCNN) architecture specifically designed for real-time processing without compromising
classification accuracy. The network progressively extracts emotional features through
three main convolutional blocks, each designed to capture increasingly complex and ab-
stract representations of facial expressions.

4.3.3.1 Convolutional Block Design

First Block: There is the first block which first consists of two stages of convolutional
layers, each of 64 filters of 5 x 5 kernels. By this configuration, the network is capable to
Chapter 4. Methodology 45

catch low level features such as edges and textures. Especially on the dying neuron prob-
lem, Exponential Linear Unit (ELU) activation functions can also be useful in overcoming
it and smoothing gradient flow on training.

Second Block: In the second block it has 128 filters with 3x3 kernels which helps to
have more feature extraction ability. This block captures a bit more intricate patterns and
relationship with facial feature, so as to figure out the small emotional cues.

Final block: The final block adopted 256 filters to grab the high level emotional features.
This block is important for the model to determine complex emotional states and transi-
tions. Dropout layers with increased rates (0.4 to 0.5) were introduced in order to prevent
overfitting together with batch normalization to stabilize training in each convolutional
block.

Figure 4.3: Architecture of the Deep Convolutional Neural Network for facial emotion
recognition

4.3.4 Real-Time Processing Pipeline

We design the real time processing pipeline for using different optimization strategies
that enable accurate and efficient emotion classification. Predictions are stabilized across
consecutive frames by temporal smoothing algorithms, reducing the effect of noise, and
data fluctuations. In addition, we perform confidence thresholding to reject uncertain
classifications, leaving only those with very strong discrimination score.
Chapter 4. Methodology 46

4.3.4.1 Adaptive Frame Rate Processing

Adaptive frame rate processing balances recognition accuracy with load, and the system
includes adaptive frame rate processing. It adapts itself automatically: The processing
rate is changed on the basis of the available system resources and the particular needs
of the processing for an application. With frame rate optimization, with high levels of
accuracy, the system is still efficient even in resource constrained environments.

4.3.4.2 GPU Acceleration and Memory Management

Real-time ability to recognize emotion is exploited when feasible with GPU acceleration to
boost processing speed, to the point where the model can accomplish this task with little
latency. Memory management techniques that are advanced are used in order to ensure
stability of long term operation and prevent memory leaks and make effective resource
utilization.

4.3.5 Addressing Real-world Challenges

The design of our approach is very meticulous to allow us to effectively deal with the
complexities of real world situations. In order to overcome variations in illumination,
the system uses sophisticated preprocessing methods to perform illumination invariance.
Furthermore, head pose compensation algorithms are incorporated to ensure the model can
understand the facial expression under various viewing angle. In dynamic environments
where users are not necessarily facing the camera directly, this capability is especially
useful.

In order to further strengthen the robustness of our emotion recognition system, we have
some strategies to handle occlusions. These strategies will ensure reliable performance
under these conditions in which temporary obscuration of some parts of the face occurs,
for instance when a user wears glasses or a hair partially covers the forehead. By incorpo-
rating these techniques, we can keep the accuracy high levels which is a must have for the
applications that require continuous emotion monitoring.

In addition, algorithms are also applied to smooth expression transitions in order to pro-
vide fluid emotion classification outcomes. This is important, as it is often important
that users’ emotional states are gradually changing for applications such as music recom-
mendation systems. Consistent and accurate emotion classification by the system enables
Chapter 4. Methodology 47

music recommendations to be made in real time, thereby improving the user experience
in general.

4.3.6 Training Process

Our methodical and broad training process of our emotion recognition model inculcates
strong performance on different scenarios. To counteract the possibility of class imbalance
in the dataset we implement the use of balanced mini batch sampling. Such a technique
makes sure the model sees the representative distribution of emotions during training to
achieve high classification accuracy.

We further utilize learning rate scheduling strategy with warm up time to improve con-
vergence during training. To a certain extent, this approach alleviates the model from
being overly sensitive to the optimal learning rate, ensuring stability and performance.
The model is validated by cross validation techniques to see if it will generalize to other
subsets then unseen data.

The training process also includes early stopping mechanisms in order to prevent from
overfitting. Monitoring that validation performance does not degrade will keep training
until the generalization capability of a model stays the same or improves. To have a reli-
able emotion recognition system, this is exactly this careful attention toward the training
process.

4.3.7 Integration with Music Recommendation Engine

Using these methodological elements, we build a highly reliable facial emotion recognition
system, which is crucial in being an input source for our music recommendation engine.
Its real time emotion classification ability and the quality of its adaptation to different
environments, makes the system highly suitable for emotion aware music recommendation.

By integrating the emotion recognition system to the music recommendation engine, a

user friendly user experience is achieved. The emotional state of the user is continuously
monitored and classified and as they engage with the system. This ongoing emotion
tracking is used to create dynamic suggestions of music that changes with the changing
emotional states of users as they naturally change as they listen.

Suppose that the system observes a change from sad to happy, it can automatically update
music playlist to have more upbeat tracks that help the user to change his mood. Such a
Chapter 4. Methodology 48

responsiveness both enhances user satisfaction and actuates a more emotional relationship
between the user and the music they are playing.

4.3.8 Future Directions

For looking further ahead, there are some ways to increase our facial emotion recognition
system. An exploration of another potential direction would be to incorporate multimodal
data such as audio and physiological signals as part of the emotion recognition process.
Once facial expressions are combined with other emotional data, a more thorough under-
standing of user sentiment is reached.

Also, we can explore applying techniques from transfer learning to make use of pre-trained
models on other datasets and hopefully get better performance as well as a reduction of
the training time. A major appeal of this approach is that it can likely benefit in rapid
adaptation to new domains or user populations with little added training.

Additionally, researching against explainable AI (XAI) could inform us on the decision

making processes of our emotion recognition system. Already understanding how the
model comes to classification gives us a second check for user trust and transparency as
that’s a crucial aspect of acceptance of AI technologies.

4.3.9 Conclusion

Finally, our facial emotion recognition model based on deep learning is a great achievement
in the field of emotion-aware applications. With a robust dataset, sophisticated prepro-
cessing methods, and correctly designed DCNN architecture, we created a system that
is able to provide natural emotions classification. Having this system integrated with a
music recommendation engine is then a great idea to improve user experiences by offering
dynamic and responsive music suggestions. We continue working on refining and expand-
ing our approach to tackle the problems of real world emotion recognition and exploring
new realms in this ever changing field.
Chapter 5

Implementation

Our emotion aware music recommendation system is a complex composition of several state
of the art technology and framework, architected carefully for a flawless and productive
user experience. Although this system recognizes the preferences of a user, it goes beyond
that to adapt to the emotional state of the user in real time, in an era where personalization
is paramount. In the context of music, this capability is extremely important as emotional
resonance enhances the listening experience so much.

The facial emotion recognition module, retrieval augmented generation with large language
model (RAG-LLM) based recommendation engine, and the user interface layer are the
three primary components of the system that plays a vital role in the application’s overall
functionality and subjective experience. All of these components have been created with
thought and care with the aim of providing top notch performance, reliability, and a great
users experience. The system does not only work, but it also is enjoyable to use, allowing
users to get closer to their music.

5.1 Facial Emotion Recognition Module

Our system is founded on facial emotion recognition module which realises real time de-
tection and classification of user emotions. To this end, MediaPipe Face Mesh is a
great framework for facial landmark detection. Realtime face calculation supported with
the 468 unique facial marks of a user’s face is also offered by MediaPipe. This significant
landmark detection is necessary to properly understand nuances of facial expressions that
are the thumb’s impetus for emotional states.

49
Chapter 5. Implementation 50

5.1.1 Real-Time Landmark Detection

The optimized pipeline on which MediaPipe is implemented uses machine learning models
to ensure that it is able to reliably detect even in varied lighting conditions and changing
head movements. As it’s a Python API, our implementation benefits from using Medi-
aPipe’s Python API, which helps us easily intergrate our Custom Deep Convolutional
Neural Network (Custom DCNN) model. The backend is optimized for MediaPipe and
this integration is meant to stay efficient on CPU usage so that the system can remain in
real time with minimal lag.

These facial landmarks are processed in real time by the emotion recognition pipeline, and
this facial landmark data is fed to the recommendation system in an ongoing manner to
signal the user’s emotional states. This continuous monitoring is crucial to ensuring that
the system can provide timely, and contextually relevant music recommendations since it
allows the system to adapt to the user’s emotional fluctuations at the time of the user
listening.

5.2 Deep Convolutional Neural Network (DCNN) for Emo-

tion Classification

TensorFlow and Keras are among the preeminent deep learning frameworks used for
such development and we utilize these to develop our DCNN for emotion classification.
As for the DCNN architecture, normalized facial landmarks are among the input of it
and these normalised facial landmark is fed through a series of convolutional blocks that
become progressively deeper with higher complexity.

5.2.1 Architecture Design

1. Low Level Features: The initial layers use 64 filters at 5x5 kernels that do a
good job of capturing low level features like edges and textures. By means of laying
a strong foundation for the subsequent feature extraction, this layer is deemed as
critical.

2. Filter Size: In the second block, the number of filters is increased to 128 during
the progress of data through the network using 3x3 kernels. Finally this design lets
the model capture more intricacy in the way the facial features and patterns behave
within them, to pick up on the more subtle emotional cues.
Chapter 5. Implementation 51

3. Higher Levels: The last block employs 256 consecutive filers to infer high order
emotional features. This is a determinant for the model to be able to distinguish
between complex emotional states and changes of emotional state.

To overcome the dying neuron issue we commonly see with traditional ReLU activations,
all across the network, we use Exponential Linear Unit (ELU) activation. This choice
of activation function ensures there is a smooth gradient flow during training which is
important to make the learning perform properly.

5.2.2 Regularization and Training

Batch normalization layers are strategically put after the convolutional layers to stabilize
training and speed up convergence. Also, the dropout layers with rates ranging from 0.4
to 0.5 are used which gives strong regularization to prevent over fitting. It is trained using
the model on the OAHEGA dataset using TensorFlow’s data pipeline optimizations for
efficient batch processing and augmentation during training. As a result this process of
training this thorough helps the model that it can generalize properly for unknown data
which is very crucial for real world applications.

5.3 RAG-LLM Based Recommendation Engine

Our recommendation engine is crucial in our system, which is an emotional state context
based music recommendations engine. Based on Llama 3.2, this engine is a state of the
art large language model that generates text that is consistently relevant to the context it
is given.

5.3.1 Model Inference and Prompt Engineering

For model inference, we use the Llama API and in particular, custom prompt engineering
for music recommendation tasks. LangChain is used to construct the RAG component
as it adds document retrieval and context management. The implementations has adapted
tailored chunking strategies for music metadata so that the system can process and retrieve
information that is relevant to the user in terms of their preference and emotional context.
Chapter 5. Implementation 52

5.3.2 Vector Storage and Similarity Search

Regarding vector storage and similarity search, we use Qdrant, a high performance vector
database which allows for efficient nearest neighbor search by high dimensional music
embeddings. Hence, this capability is essential because by rapidly identifying music tracks
that are relevant to the user’s current emotional state and preference, you are allowing the
user to make music appear in support of it in an effortless way.

5.4 Development and Testing Interface

Gradio was used to create a testing interface that enables us to develop and test the RAG
system. It paves the way for quick iteration and validation of many potential prompt
strategies and recommendation methods. It is crucial for quickly fine tuning the RAG
parameters and prompt engineering to have instant feedback on the quality of the recom-
mendations provided by the Gradio interface.

Figure 5.1: API Testing Interface using Gradio

5.4.1 User Interaction Features

As Gradio’s interface, this has components for manual emotion input, preference selection,
and real time preview of recommendations. The ability to tweak inputs and see the results
allows developers to test the true potential of the recommendation engine because of this
Chapter 5. Implementation 53

interactive setup. By simplifying the process of making quick adjustments and gaining
immediate feedback, Gradio greatly accelerates the development cycle, which ensures that
the recommendation engine is optimized against the best possible user experience.

5.5 User Interface Layer

Streamlit is a powerful framework used for developing responsive, interactive web ap-
plications with the least amount of boilerplate code. The application is well suited for
streamlit since it’s so easy to use and makes creating dynamic visualizations easy. This
implementation contains several necessary components that help to improve the user in-
teraction and engagement.

1. The Real Time Video Capture Module: This allows users to interact with this
system through your webcam, in real time receiving emotion detection. Since the
system has to adapt to user’s emotional state as it changes, it requires that the video
feed can be processed in real time.

2. Dynamic Emotion Visualization: The interface contains a dynamic visualization

that illustrates the current sensed emotional states the system is interpreting. In fact,
this feature not only gives the user info in terms of how they are emotionally, but
also helps to make the overall interactivity of the app better.

3. Integrated Spotify music player: Users can play recommended tracks through an
integrated Spotify music player with playback controls. This integration facilitates
the ease of use by people as they can provide music without leaving the app they
are currently on.

4. Music Preference Management Interface: Users are able to manage their mu-
sic preference through a simple interface that lets them rate the songs and giving
feedback on their recommendation. Consequently, it is crucial for the feedback loop
to fine tune the recommendation engine to the preferences of individual users and
offer suggestions while keeping them updated.

5. Real Time Recommendation Display: The system also provides real time dis-
play of music recommendations and explanation features about why such recommen-
dations are suggested given the emotional state of the user. Transparency creates
Chapter 5. Implementation 54

Figure 5.2: User Interface using Streamlit

confidence, ensures users “stick around” by participating more deeply with the sys-
tem.

6. Emotional and Music History Tracking: The system further provides for his-
torical tracking of emotions as well as historical tracking of music preferences, which
gives the users a chance to go back in time and reflect over their emotional journey
and how it corresponds to their music preferences.

5.6 Database Implementation

Since we want to manage our data efficiently, we chose to use Qdrant — a high perfor-
mance vector database that stores vectors and perform similarity searches on them. Our
application requires handling high dimensional data, therefore we have chosen Qdrant be-
cause it efficiently handles high dimensional data, something that is of value to our music
recommendation system that generates complex embeddings. This means that we can pick
Qdrant because of its advanced features to further optimize the overall performance and
responsiveness of our system.
Chapter 5. Implementation 55

5.6.1 Custom Indexing Strategies

We have implemented custom indexing strategies for optimal retrieval process in favour of
music recommendations. They are strategies that will guarantee best results for efficient
storage, both of the embedding vectors, which are the low dimensional space representation
of the music tracks and corresponding metadata, like artist names, genres, user ratings
etc. The organization of this data helps us to have fast access to the right information,
which is very necessary to facilitate users with rapid access to rapidly relevant music
recommendations in a timely manner.

5.6.2 Caching Mechanisms

On top of that, we setup caching mechanisms which minimize latency for frequently queried
problems. Caching is a crucial part of our architecture, as it enables us to keep stored
data in memory and therefore prevents an unnecessary database query to get the same
data over and over again. Not only does this increase the speed at which a user request
would be processed but also it reduces the strain on the database, which is able to continue
with a large number of simultaneous requests without any degradation on the database’s
performance.

5.6.3 Optimizing Index Structures

Our effort has been not only on caching but optimizing index structure used in Qdrant
for fast queries. Through the help of more advanced indexing mechanisms like ANN,
we achieve the power of finding music in a time-saving manner. Here the similar tracks
are determined mainly based on user preferences and their condition at the moment.
The emotions expressed by users and the sentiments are the driving factors of this type
of operation. Besides, the presented models are less computationally demanding and
they substantially speed up the process. While speed and flexibility are there, more
sophisticated tasks such as sorting input data according to relevance become more difficult
and so time-consuming. Really, it has never become a matter of how to do something if
technologies are efficient for the task but whether the problem is worth solving.
Chapter 5. Implementation 56

5.6.4 Scalability and Performance

The way we have constructed the system is quite in line with the idea of being scalable,
thus giving us the opportunity to serve more and more users as well as cover wider musical
ground without a single decline in quality. We can use our storage, network, and computer
capacities efficiently and keep multiple copies of data consistently so that we can still
provide instant recommendations and even have better data accuracy. Our system relies
on the TBDMS that employs data structure that we have designed and developed and can
thus remain a sensitive and powerful instrument in that we follow the same composition
of hardware uniquely tailored attention during this period. In addition, a huge storage
depends on big data processing technology so a considerable amount of work can be done
under the condition of searching for a specific track, and this is exactly when performance is
more of a determining factor than the minimum query response time. Sufficient efficiency,
fast response to calls, and distributed freedoms are all major facets of scalability.

5.7 Management of User Data

We use MongoDB as our main database solution in order to efficiently manage user data
and guarantee a flawless experience within our emotion-aware music recommendation sys-
tem. Our needs are especially well-suited to MongoDB, a NoSQL database that provides
a scalable and adaptable approach to data management. We can build optimized schemas
especially made to store a range of user-related data, such as user profiles, emotional
histories, and interaction logs, by using MongoDB collections.

5.7.1 Optimized Schemas for User Data

Our developed schemas are specifically designed to capture the distinctive features of user
interactions with the system. To help us customize the music recommendations, each
user profile contains vital information like user IDs, preferences, and demographics. In
order to help the system identify trends and modify recommendations based on past data,
we also keep emotional histories that monitor users’ emotional states over time. Under-
standing how emotional states affect musical preferences and making more sophisticated
recommendations are made possible by this longitudinal view of user emotions.

Another essential element of our data management plan is interaction logs. User activities
within the system, including song plays, skips, ratings, and comments on suggestions,
Chapter 5. Implementation 57

are recorded in these logs. We can learn more about user behavior and preferences by
examining this data, which helps the recommendation engine function better. We will
have a rich dataset to work with thanks to this thorough approach to data collection,
which will allow us to improve our algorithms and the user experience as a whole.

5.7.2 Flexible Querying Capabilities

The flexible querying is one of the stand out features of MongoDB. But the nature of the
document is such that it lets us easily query it and our way of retrieving user info from it
is no more than just retrieving whatever we want quickly. For example, we can efficiently
query user profiles to identify users that have the same emotional histories or preferences
that will serve us in generating music recommendations for a person. This flexibility is
vital in an environment where user preferences and emotional state rapidly change.

Furthermore, MongoDB also supports indexing which makes the data retrieval task easier.
Frequently queried fields can be indexed and this reduces the time to load user data to
a point such that the recommendation engine is able to operate in real time. This is
important as users are willing and expect immediate response as reactions to their current
emotional context.

5.7.3 Scalability and Performance

Mongodb’s scalability is a large strength as it grows our user base. The large volume of
data in the database is taken into consideration and it can therefore run on large data
volumes and scale horizontally as more machines are added to deal with increased demand.
This scalability guarantees that our system will work well and keeps growing even when
we are adding more and more users and adding more and more music to our library. To
keep the effectiveness of our recommendation engine, the scalability to our datasets has
to be maintained and the ability to manage large datasets while their performance is not
compromised.

5.7.4 Data Security and Privacy

Along with perfect performance and scalability, data security and the privacy of the user
are equally important to us during MongoDB implementation. As for sensitive user in-
formation, we take robust security measures like encryption and controls over access. We
Chapter 5. Implementation 58

build trust with our users by making sure that the data stored about them is secure and
is accessed only by independent components of the system, in order to comply with the
relevant data protection regulations.

5.8 System Integration

Direct API calls between components are used to system integrate along well defined inter-
face, making each module communicate with each other. Due to this modular architecture,
it is easy to update and maintain, as the functions of the system are not susceptible to
changes any component.

5.8.1 Error Handling and Logging

To allow the system to gracefully degrade in the case of component failures, we have put
in place robust error handling mechanisms throughout the system. It is crucial for to keep
the trust of the users and the customer satisfaction. Also, the implementation provides
comprehensive logging functionality to monitor system performance and user interactions.
For troubleshooting and also for the continuous improvement of the recommendation al-
gorithm, the logging of this has its place.

Figure 5.3: API Documentation

Chapter 5. Implementation 59

5.9 Future Enhancements

Looking ahead, we intend to add several more improvements to our emotion graspable
music recommendation system to make it better. In one potential direction the emotion
recognition process can be enriched in incorporating multimodal data, either audio
features and physiological signals. The combination of facial expressions with other sorts
of emotional data helps us better grasp in which way the user is feeling.

We are also studying the application of transfer learning techniques to take advantage
of pre trained models on large datasets, which might improve performance and decrease
training time.

5.9.1 Development Environment

To be sure of having a consistent dependencies and a consistent package version during

development and testing, I use python virtual environment. Using this we can make sure
that all of us can work as one as it will have no compatibility problem while working
with our project. Specifically defined are some of the hardware requirements for optimal
performance of the system, as well as for the component that is responsible for the real
time emotion detection, GPU acceleration offers benefits.

5.9.2 Regular Testing and Validation

The whole system is being tested regularly to understand its reliability, each part along
with that. Thus, this testing confirms the accuracy of the emotion detection as well as the
recommendations. The functionality and perform of system is validated using unit tests,
integration tests and also user acceptance testing. These are the testing phase where
receiving the feedback is crucial to understand what are the weak sides and ensure the
final product meets the user’s expectations.

Additionally, there are ongoing research works on explainable AI (XAI) to understand

the decision making process of our emotion recognition system. Knowing how the model
got to the classifications it made helps not only to improve user trust and transparency,
both key factors in the usage of such technologies, but also to increase their confidence
in the results. This understanding could also lead to improved user engagement, as users
may feel more connected to a system that explains its reasoning.
Chapter 5. Implementation 60

5.10 Conclusion

In summary, our emotion-aware music recommendation system has significant advantages

to deep learning integration with user experience design. We have built a system that
does more than that of reacting to user emotions in real time, but adds to the world
of music listening experience by blending a powerful facial emotion recognition module
with an advanced recommendation engine and an interactive user interface. With the
technologies, frameworks, and methodologies carefully selected, the system is efficient,
effective and capable of adapting itself to many different users’ needs.

While we are continuously working to adapt and grow, and to ultimately solve the chal-
lenges of real world emotion recognition as well as to open new frontiers in this very active
and popular field. Future enhancement is thus possible, for example, with multimodal
data integration and explainable AI widening the horizons of our system to an exciting
area of research and development. Through continuous improvement and innovation, we
seek to build a truly dynamic and responsive music recommendation experience that is
deeply emotional to the users.

Figure 5.4: User Interface using Kivy

Chapter 6

Testing and Results

6.1 Introduction

The technology has definitely played a role in the evolution of music consumption, as one
example having to do with recommendation systems. With increasingly people looking for
personalized experiences, emotion aware music recommendation systems have started to
gain attention as one solution. Instead, these systems curate music suggestions leveraging
real time emotional data to suggest music according to current emotional state of the
users. This report proposes a total analysis of our emotion aware music recommendation
system, which combines a facial emotion recognition component with a Retrieval Aug-
mented Generation (RAG) based recommendation engine. We further developed a hybrid
recommendation system so as to conduct comparative analysis for exploring the strengths
and weakness of the two approaches.

6.2 Overview of the Emotion-Aware Music Recommenda-

tion System

We built our emotion aware music recommendation system that provide real time sugges-
tion of music depending on the user’s emotional states. The system consists of two main
components.

1. Deep Convolutional Neural Network (DCNN): DCNN is used to investigate

facial emotion recognition system, and is trained on the OAHEGA dataset for the
61
Chapter 6. Testing and Results 62

classification of emotional expressions. The ability of the model to ’feel happy or

sad’ before the recommendation is made is crucial, for on this relies the very basis
on which the users’ preferences have to be understood.

2. Recommendation Engine by Based on RAG-LLM: A recommendation engine

based on RAG-LLM that makes use of contextual understanding and semantic pro-
cessing to provide personalized music recommendations. Consequently, this engine
is created in a way to adapt to user preferences and emotional states to furnish a
music discovery experience that is more compelling and connected.

6.2.1 Hybrid Recommendation System

Our RAG-LLM based recommendation system obtained additional effectiveness evalua-

tion through the creation of a hybrid recommendation system. The hybrid framework of
recommendation technology merges machine learning techniques with conventional recom-
mendation approaches for assessing their performance metrics comparisons. The hybrid
system employs a wider set of evaluation attributes which include popularity together with
duration alongside danceability and energy and key and loudness and mode and speechi-
ness and acousticness in addition to instrumentalness and liveness and valence and tempo
and time signature. The RAG dataset only contains three attributes comprising serial
number, track ID and emotion information.

Figure 6.1: Comparison of performance metrics between RAG-LLM and hybrid recom-
mendation systems

6.3 Evaluation Metrics

Several key performance metrics were evaluated on both the RAG-LLM based recommen-
dation system as well as the hybrid system.

1. Accuracy and Coverage: These metrics measure how precise and recall the rec-
ommendations are made by each system. Precision is the ratio of relevant recom-
mendation made to all the recommendation made, recall is the ratio of relevant
recommendation retrieved from total relevant games available.

2. Music Recommendation Diversity: The amount of suggestion variety that is

offered is important to maintain user engagement so this metric can be used to
Chapter 6. Testing and Results 63

evaluate this fact. Prevention from recommendation fatigue and satisfying the users
are some of the recommendations on a diverse set.

3. Recommendation Evaluation: Surveys and interviews were conducted with the

users to obtain overall satisfaction with the recommendations. This qualifies data
provides insight into the user preferences and experiences.

4. Ranking Quality: The Normalized Discounted Cumulative Gain (NDCG) scores

were used to assess the ranking quality of recommendation list. The higher the
NDCG score, the better the performance of most relevant items in a list of recom-
mendations.

5. Cold Start Performance: Evaluates how much the systems can deliver given the
amount of data about users. Recommendation systems are an area of recommen-
dation systems that suffer from cold start scenarios: generating relevant suggestions
without prior user history.

6. Query Complexity Performance: The systems are evaluated on how they will
respond when user queries become more and more complex. With growing ability of
users to express more nuanced preferences, adapting to provide relevant recommen-
dations is critical for the recommendation system.

6.4 Findings from the Evaluation

6.4.1 Facial Emotion Recognition System

Our system showed strong performance metrics during testing on the facial emotion aspect
of our system. Training accuracy of 88.63% was achieved by the DCNN model, trained on
OAHEGA dataset. The high accuracy of the model is due to the model’s ability to detect
emotional expressive on its own illustration under controlled conditions. The validation
accuracy of 76.65% does show some decrease, which is expected since emotion recognition
in real world scenario is a difficult task. Lighting conditions, head poses, and personal
differences in expression make a big difference in performance.

Music recommendation engine relies on the model to accurately predict emotions as the
basis for tailoring the web service to recommend music. This part’s integration into the
system enables the system to react rather dynamically to the users’ emotional states
resulting in a better experiences.
Chapter 6. Testing and Results 64

Figure 6.2: DCNN model accuracy and Loss

6.4.2 RAG-LLM Based Recommendation System

Our RAG-LLM based recommendation system was found to be superior compared to the
hybrid approach over all the key metrics especially in the case of complex queries as well
as when there is a cold start. The key performance findings summed up in the following
insights.

Figure 6.3: Comparison of Recall between RAG-LLM and hybrid recommendation sys-
tems

1. Superior Precision and Recall: Hybrid approach had Precision (0.51–0.68) and
Recall (0.49–0.63) while RAG showed better Precision (0.72–0.81) and Recall (0.59–
0.75). It indicates that RAG successfully generated more accurate and detailed
recommendation to capturing user preference.

2. Better Diversity: RAG worked out with diversity ranging from 0.74 to 0.89 higher
than the hybrid system (0.63–0.82). It is for this diversity that this keeps users
Chapter 6. Testing and Results 65

Figure 6.4: Comparison of Precision between RAG-LLM and hybrid recommendation

systems

engaged and relieves them of recommendation fatigue as they can look at more
music within their range of emotions.

3. Cold Start Handling: On cold start scenarios, with low number of user interaction
data, RAG yielded more relevant recommendations (0.58) than the hybrid system
(0.45). This is important, because it gives new users, who do not yet have preferences,
a taste of how meaningful their suggestions will be from the get go.

Figure 6.5: Comparison of Cold Start Performance between RAG-LLM and hybrid
recommendation systems

4. Advantage of complexity: for any particular query complexity, RAG’s perfor-

mance remained more robust (0.62–0.74) than the hybrid system (0.51–0.65). It is
the adaptability of such system, essential for the variety of musical taste of users
with nuanced tastes, who are permitted to cater the system through emotional and
contextual factors.

5. Higher User Satisfaction: 45 of the 55 participants were ’Very Satisfied’ with

RAG recommendations vs 35 for the hybrid system. Furthermore, users reported
less dissatisfaction (6 vs. 12 for hybrid) as an indicator that the RAG system satisfies
users’ expectations.
Chapter 6. Testing and Results 66

Figure 6.6: Comparison of User Satisfaction rate between RAG-LLM and hybrid rec-
ommendation systems

6. Better Ranking Quality: The NDCG scores of our RAG system were (0.69-0.82)
higher than (0.58-0.72) for the hybrid approach, which means RAG provides better
recommended item ranking. One of the key pieces to consider is that, when there are
higher ranked items available to the users, they are more likely to pick from them.

Figure 6.7: Comparison of NDCG between RAG-LLM and hybrid recommendation

systems

6.4.3 Key Performance Insights

Our tests results clearly show that hybrid approach is beaten by the RAG recommendation
system in all the most important metrics. This is especially apparent in complex queries as
well as cold start scenarios when the RAG system’s contextual understanding and semantic
processing are valuable to the query.

6.4.3.1 Why RAG Outperforms

1. Deeper Contextual Understanding: Moreover, as the RAG system has a deeper

contextual understanding of user queries and content, it is able to more accurately
Chapter 6. Testing and Results 67

Figure 6.8: Relevance vs Diversity Tradeoff between RAG-LLM and hybrid recommen-
dation systems

approach the problem of generating recommendations that better match the emo-
tional states of users. Since the system is able to analyze the context in which music
is consumed, this enables the system to provide more personalized suggestions.

2. Semantic Processing: RAG does this by performing semantic processing that aids
in interpreting nuanced emotional and thematic connections between RAG items,
which results in better RAG recommendations that are more personalized. It al-
lows the system to process semantic information and to recognize the emotional
undertones of songs and to correlate this with current feelings of users.

3. Adaptability: The RAG system also enables better adaptation to new items and
time changing user preferences to keep relevance in the recommendation recommen-
dations. This is crucial given the very quick change in a music landscape where new
tracks and genres keep popping up.

4. Balanced Exploration: RAG explores new content balance more better than other
methods, and it is necessary to maintain the user engagement and satisfaction. The
system promotes a more rich listening experience by encouraging users to listen to
new music that is conducive to the state they are in.

6.4.4 Limitations of the Hybrid System

In general, however, the hybrid recommendation system is superior to the RAG system in
terms of execution time but less consistently so in terms of attributes offered and perfor-
mance. Like the RAG, the hybrid system is quite reliant on traditional recommendation
Chapter 6. Testing and Results 68

techniques, therefore lacks the contextual understanding and adaptability like RAG. Fur-
thermore, it suffers in cold start scenario and complex query, where RAG system performs
well.

Figure 6.9: Comparison of Execution Time between RAG-LLM and hybrid recommen-
dation systems

The dataset of the hybrid system includes many attributes, such as popularity, duration,
danceability, energy, key, loudness, mode, speechiness, acousticness, instrumentalness, live-
ness, valence, tempo and time signature. Although these attributes offer a round view of
the music, they fail to convey such emotional nuances as RAG does. However, the RAG
dataset with only three attributes (serial number, track ID and emotion) deals with the
emotional part only which is of utmost importance in delivering personal recommendation.

Figure 6.10: Comparison of Query Complexity between RAG-LLM and hybrid recom-
mendation systems

6.5 Future Directions

Based on the findings of this evaluation, several future research and development oppor-
tunities are suggested.
Chapter 6. Testing and Results 69

1. Future work: Future work can expand the user study with a bigger sample size
for a greater statistical significance of our findings and for a deeper understanding
of user preference and satisfaction. To assist in identifying trends to preferences by
demographic, a more diverse participant pool could also be present.

2. Long Term Engagement Analysis: Long term engagement analysis would be

conducted to study the effect of the system on user satisfaction and music discov-
ery, and with that data further refining would be possible. A potential running of
this analysis could entitle the system to observe users’ interaction and preference
over, let’s say, time periods in order to determine this system’s effect on the music
consumption patterns.

3. Improving the Emotion Recognition Model: By refining the emotion recog-

nition model further the gap between training and validation accuracy would be
closed and performance would be stronger in real world scenarios. It could entail
loading more training data, considering alternate model architectures, or including
multi modal data sources like audio features.

4. Improving the Recommendations of the Hybrid System: Looking into ways

to improve the hybrid recommendation system, by adding to it some aspects of
machine learning and contextual understanding, may enhance it in its performance
and may make it more competitive with the RAG system. For instance, advanced
algorithms could be used which clearly can interpret user preferences and emotional
states much better to yield a higher quality recommendation.

5. Further Attributes: Additional attributes that could be explored in future itera-

tions of the hybrid system would include user generated content and social listening
data. The hybrid system, however, incorporated social dynamics and collaborative
filtering techniques, which could enable it to better account for collective preferences
of users to deliver more relevant suggestions.

6. Improved User Interface and Experience of Systems: Reducing the inter-

action with both systems could lead to better end user experience. We can make
the music discovery process intuitive and easier by designing the interfaces in such
a way that the users can easily interact with the recommendation systems. Other
possibilities include offering features like mood sliders or playlists based on mood,
which could give the users more control over their listening experience.
Chapter 6. Testing and Results 70

7. Real-Time Feedback Mechanisms: Users could be able to provide instant re-

sponses regarding the recommendations they receive if real time feedback mecha-
nisms were implemented. The feedback from this could be used to help the mech-
anisms that hand out these recommendations, to be fine tuned so that they react
quickly to changing psychological states and preferred user individual patterns.

8. Cross-Domain Recommendations: Cross domain recommendations might create

new doors to engage with users. For example, combining music recommendations
with other types of media including movies or podcasts based on users’ emotional
states provides a more comprehensive entertainment experience.

9. User Privacy and Ethical Considerations: As emotion aware systems evolve

we must grapple with user privacy and ethical considerations. In order to build trust
and earn up take from users, it will be essential to ensure that they are informed as
to how their emotional data is being used and that they have the option to control
some of their data.

6.6 Conclusion

So, we evaluated our emotion aware music recommendation system and got quite positive
results showing that integrating real time emotion recognition with RAG-LLM based rec-
ommendation indeed works well. Running the RAG system on a variety of music datasets,
key metrics are present to indicate potential use of RAG system in improving music dis-
covery process. Despite refining and broadening our systems, we’ve stayed dedicated to
making music recommending experiences better for people through novel ways of doing
so. Thus, this evaluation will provide valuable information for our future research and
development studies and will help to push the development of emotion-aware technologies
in music industry.
Chapter 6. Testing and Results 71

Figure 6.11: Comparison of overall performance metrics between RAG-LLM and hybrid
recommendation systems

In general, emotion recognition combined with advanced recommendation algorithms can

contribute to personalizing music experience. User emotions can help us create systems
that do not only recommend the music but also help create those emotional connections,
by developing a deeper relationship between listener and the music she is listening to. Un-
derstanding and responding to the emotional landscapes of users will be key to the future
of music recommendation, and our results open up further exploration and innovation in
this exciting field.

Table 6.1: Summary of performance metrics for RAG-LLM vs. Hybrid recommendation
systems

Metric RAG-LLM System Hybrid System

Precision 0.72–0.81 0.51–0.68
Recall 0.59–0.75 0.49–0.63
Diversity 0.74–0.89 0.63–0.82
Cold Start Performance 0.58 0.45
Query Complexity Performance 0.62–0.74 0.51–0.65
NDCG Scores 0.69–0.82 0.58–0.72
User Satisfaction (out of 55) 45 35
Chapter 7

Conclusion & Future Scope

7.1 Conclusion

Consequently, this research proposes an innovative approach for music recommendation

that uses real time facial emotion recognition along with a recommendation system based
on RAG-LLM. Using advanced deep learning techniques and sophisticated recommenda-
tion algorithms we have demonstrated the utility of emotion aware music recommenda-
tions. To address the main limitations of existing recommendation systems, which fre-
quently exclude emotions of users and their affect on music preferences, the system works.
We strengthen our application of DCNN with our emotion recognition component that
reached 88.63% in training and 76.65% in validation. User studies result revealed that no
less than 80% of people considered our system to be more beneficial than conventional
recommenders, taking into account the emotional sense of music.

However, the combination of our custom DCNN architecture with MediaPipe Face Mesh
for facial landmark detection has been very successful at reconstructing real time emotion
classification. The system can generate contextual and personal music suggestion that
respond to users emotional state in meaningful ways by using Llama 3.2 LLM within the
RAG framework. By implementing a combination of Streamlit for Core Interface and
Gradio for RAG Testing, we have managed to design an accessible and easy to use system
which is elegant, yet sophisticated enough to be used in real life.

Successful implementation and good user feedback also point to a lot of potential appli-
cations for emotion aware artificial intelligence in improving personalized content delivery
systems. This represents a real step in leveraging emotional states to more intuitive and

72
Chapter 7. Conclusion & Future Scope 73

personally related music discovery experiences. Real time emotion recognition along with
advanced language models for the generation of recommendations based upon them en-
ables us to find new ways of understanding and responding to users emotional needs in
music.

7.1.1 Key Performance Insights

Utilizing our tests, we show that the RAG (Retrieval Augmented Generation) recommen-
dation system consistently outperforms hybrid over all important metrics. Such relevance
and suitability is particularly worth noting in cases where the queries are complex and
there is cold start, where RAG system is able to harness the contextual understanding to
sustainably offer more relevant and diversified suggestions. It is more surprising that the
performance gap is largest for recommendation diversity and user satisfaction, that with
RAG users are more likely to find what they love.

7.1.1.1 Key Findings

1. The RAG system achieves superior precision and recall results than hybrid ap-
proaches in all query types especially when it comes to user-based recommenda-
tions. RAG produces precision scores between 0.72 to 0.81 yet the hybrid system
generates scores between 0.51 to 0.68. The RAG system achieves better recall rates
which span from 0.59 to 0.75 and the hybrid system reports recall rates between
0.49 to 0.63. RAG delivers precise recommendations that cover increased numbers
of relevant items when compared with other recommendation methods.

2. RAG achieves better diversity in its recommended products while maintaining strong
relevance to user needs. The diversity scores obtained by RAG span between 0.74
and 0.89 which exceeds the range of 0.63 to 0.82 obtained by the hybrid system. The
ability to offer diverse mood-based recommendations requires a system that provides
numerous music choices which match users’ emotions.

3. RAG delivers better recommendations during cold start scenarios even when there is
little available user interaction data or experience. RAG delivers a 0.58 performance
score in cold-start situations whereas the hybrid system provides 0.45. RAG delivers
essential recommendations to novices without preference history since it understands
their importance to new users at first encounter.
Chapter 7. Conclusion & Future Scope 74

4. RAG delivers improved recommendations across various levels of data complexity

because its performance remains stronger than hybrid recommendation engines. The
performance scores of RAG moderately increase from 0.62 to 0.74 as complexity
grows while hybrid systems experience more severe degradation down to 0.51 to
0.65. Users who enjoy diverse music benefit from RAG’s ability to handle complex
patterns since the system adjusts its operations according to different emotional and
situational needs.

5. The results from user satisfaction ratings show that the RAG recommendation sys-
tem achieved higher marks with 45 Very Satisfied users than the hybrid system which
obtained 35 Very Satisfied users. The RAG system obtained a superior satisfaction
level from users with only 6 dissatisfied participants whereas the hybrid system had
12 dissatisfied users demonstrating its capacity to fulfill user needs effectively.

6. Standardized NDCG scores from 0.69 to 0.82 indicate that RAG delivers superior
ranking quality compared to the hybrid model which scored between 0.58 and 0.72.
The enhanced ranking quality provided by the RAG system displays the most ap-
propriate music choices at the forefront of user lists to deliver an improved user
experience.

7.1.1.2 Why RAG Outperforms

In the words of these factors, the RAG system showed the superior performance.

• Deeper Contextual Understanding: This contextual understanding of user queries,

content and emotions is used to interpret more nuances in user preferences and emo-
tions with higher context. With this understanding, the system is then able to offer
recommendations that are so not only relevant to the user but also so personally.

• Semantic Processing: The RAG system does a great job understanding fine emo-
tional and thematic connections among items. RAG can further improve the lis-
tening experience by analyzing the semantic relationships between different music
tracks and suggest songs that resemble listener’s emotional state.

• Adaptability: RAGs are more adaptable to the changes of new items and of chang-
ing user preferences. RAG allows for users to interact with the system, which leads
RAG to learn and evolve in order to receive fresh, and relevant recommendations
every time a user interacts with the system.
Chapter 7. Conclusion & Future Scope 75

• Balanced Exploration: With respect to a given content base, this improves on

the knowledge of preferences over exploitation and knowledge of preferences over
exploring. This balance is necessary for sustaining user engagement by bringing
users to find new music while offering some familiar staples.

7.1.2 Implications for Future Research

Our performance analysis of the RAG system is the source of some interesting insight that
may be used via some future research and development possibilities. Now that we are
starting to refine and improve the emotion aware music recommendation system, there
exists several directions that we can explore:

1. Extension to Other Modalities (Audio Analysis, Text Sentiment Analy-

sis): The system could be further extended to incorporate additional modalities,
such as audio analysis and text sentiment analysis, to enhance the emotion recogni-
tion capability. Analysing the audio features of music and the sentiment of the user
generated content allows the system to understand the emotional connections from
a more holistic perspective.

2. User Centric Design: User centric design principles will also be essential to ensure
that the system will cater to the needs of the multiple users. User will be studied
to get feedback on the design of user interface, presentation of recommendations,
and overall experience with the app and these would be a strong value for helping
develop future updates.

3. Long term Assessment: To decode if music in emotion aware recommendation has

long term impact of users’ well being and emotional regulation; we can go ahead and
conduct long term longitudinal studies. As music recommendations can elicit many
powerful emotions, developing more effective interventions hinges on understanding
how users’ emotional states respond over time.

4. Cross-Cultural Adaptation: If we are looking to have a recommendation system

that is globally relevant, we can’t miss investigating how different cultural contexts
would impact emotional responses to music. We can increase the system’s appli-
cability and effectiveness in mixed populations by tailoring the system to a diverse
population.