2022 14th International Conference on Quality of Multimedia Experience (QoMEX)

Quality of Experience in the Metaverse: An Initial

Analysis on Quality Dimensions and Assessment
Simone Porcu1,2 , Alessandro Floris1,2 , Luigi Atzori1,2
1
DIEE, University of Cagliari, 09123 Cagliari, Italy
2
CNIT, University of Cagliari, 09123 Cagliari, Italy
{simone.porcu, alessandro.floris84, l.atzori }@unica.it

Abstract—The Metaverse provides a novel experience to the the analysis: “the QoE is the degree of delight or annoyance
user, by opening the doors to social-based multiuser environments of the user of an application or service which involves an
merging physical reality with digital virtuality. In this paper, we immersive media experience” [3]. This White Paper also
present an initial analysis of the Quality of Experience (QoE)
in the Metaverse. We first consider traditional influence factors defines the human, system, and context influence factors (IFs)
(human, system, and context). Then, we introduce the social and as the main factors potentially impacting the QoE of the users
economic dimensions of the Metaverse as additional factors to of an application or service. Based on these White Paper
be considered for QoE assessment. Finally, we discuss what QoE definitions, in [4], an overview of different IFs and dimensions
assessment methods can be more suitable for Metaverse appli- that may affect the overall QoE of interactive VR applications
cations, with a particular focus on implicit assessment methods
(e.g., physiological, human cognitive, affective behaviour). is provided. The QUALINET group has also presented a White
Index Terms—Metaverse, Quality of Experience, Social dimen- Paper specifically focused on immersive media experience [5].
sion, Economic dimension, Implicit QoE assessment. Immersion and sense of presence are particularly relevant QoE
features contributing to the quality perceived by the user in
VR and AR applications [4]. These QoE features allow to
I. I NTRODUCTION evaluate, respectively, the degree to which the range of sensory
channel is engaged by the virtual simulation [6] and the sense
The Metaverse is defined as “the post-reality universe, of being in the virtual world [7]. Thus, QoE studies focused
a perpetual and persistent multiuser environment merging on the experience perceived by using immersive applications
physical reality with digital virtuality. It is based on the con- are of extreme importance as the basis to assess the QoE of
vergence of technologies that enable multisensory interactions Metaverse applications. However, not all Metaverse are based
with virtual environments, digital objects and people, such as on immersive technologies. Also, there are additional aspects
virtual reality (VR) and augmented reality (AR)” [1]. There that may impact on the user’s QoE, such as economic factors,
are different types of Metaverse, such as gaming-based (e.g., engagement, and social involvement, which are of particular
Roblox, Second Life, Minecraft, Fortnite, Pokemon Go) or relevance for Metaverse applications.
blockchain-based (e.g., The Sandbox or Decentraland). Some In this paper, we first consider the traditional QoE IFs.
Metaverse (e.g., Horizon of Meta) are totally based on VR Then, we introduce the social and economic dimensions of the
technologies and require a VR headset to immerse the user Metaverse as additional factors for QoE assessment. Next, we
in the virtual world. However, all Metaverse have in common discuss what QoE assessment methods can be more suitable
one element, i.e., providing a virtual world where people can for Metaverse applications and we provide final conclusions.
interact with the digital environment and other users in real-
time. Nonetheless, with respect to AR and VR, the Metaverse II. T RADITIONAL Q O E I NFLUENCE FACTORS
has a strong aspect as a service with more sustainable content We consider the human, system, and context IFs as the main
and social meaning [2]. Therefore, the social dimension has a factors potentially impacting the QoE of the users of Metaverse
crucial role for the Metaverse applications. applications [3].
Since it is an emerging application, there are not studies 1) Human influence factors: Regard both static (e.g., sex,
in the literature that investigated how to evaluate the Quality age) and dynamic (e.g., affective state) information concerning
of Experience (QoE) in the Metaverse. However, similarly the user. The affective state may be measured with the methods
to other multimedia-based applications, the QoE definition discussed in Section IV. When VR-based technologies are
provided by the QUALINET White Paper can be considered involved, discomfort symptoms, such as cybersickness, must
valid also for Metaverse applications as the starting point for also be considered to evaluate the overall QoE.
2) System influence factors: The system IFs concern the
This work has been partially funded by the Ministero dell’Istruzione,
dell’Universita e della Ricerca (MIUR) with the PON “Ricerca e Innovazione” network infrastructure, the hardware, and the multimedia con-
2014-2020 (PON RI) “Azione IV.4 Dottorati e contratti di ricerca su tematiche tent. A solid network infrastructure is required as the baseline
dell’innovazione”, assigned with D.M. 1062 on 10.08.2021. for supporting the Metaverse applications, in particular when
978-1-6654-8794-8/22/$31.00 ©2022 IEEE high-resolution multimedia contents are involved (e.g., 360
video) [8]. In [9], the network architecture of the Metaverse Therefore, the social dimension of the Metaverse is very
is discussed, where the edge network is identified as a great important, and it is a unique characteristic that is not provided
support for offloading computations (e.g., multimedia render- by other multimedia-based applications. Although the QUA-
ing) for Metaverse applications to improve performance and LINET White Paper considers the social factors within the set
enhance the end user’s QoE. With regard to the hardware, of the context IF, we think that the social dimension of the
the VR headset is an important element that influences the Metaverse is a quite different concept. Indeed, the social as-
perceived quality because of its weight and screen quality. pect considered for traditional multimedia applications mostly
Hopefully, the VR headset manufacturers will be continuously regards consuming the service alone or with other people.
improving the headset, making it wireless and lighter. The Current multi-user applications have a final goal and some
screen quality, in terms of image resolution and frame-per- subjects collaborate or fight to reach that goal (e.g., gaming).
second rate, is directly connected to the perceived QoE. The social dimension in the Metaverse is a different concept as
Authors in [10] found that a 2K resolution is an important it regards sharing a digital place where people can interact or
threshold for an enhanced VR experience without affecting stay closer like the real world. For instance, a subject could
performance and increasing sickness levels. Other hardware go to a Metaverse place (e.g., a city, a shop, a pub, which
devices, such as motion inputs devices, hand-based input can even be the digital version of a real place) where he can
devices, haptic devices can also affect the quality of the meet friends but also find unknown people he can interact
Metaverse experience. Finally, multimedia contents also have to and merry the partner. This opens to the consideration
a direct impact on the perceived QoE. Authors in [11] suggest of particular social aspects related to people behaviours in
that the users’ visual attention is highly restricted as the VR the virtual world: i) the users behaviour may be annoying or
sickness increases. Therefore, the visual attention loses the even aggressive toward other people. For instance, according
focus from the multimedia content, affecting at the same to Meta, a beta tester was virtually groped by a stranger on
time the users’ perceived quality. In [12], the impact of Horizon Worlds. Further sexual assaults happened in the same
audio quality on gaming experience under different bitrate and Metaverse are reported in the SumOfUs report [16]. Such a
packet loss conditions is investigated. Results highlight that: behaviour may influence negatively the perceived QoE of the
the packet loss affected both the audio quality and the gaming Metaverse; ii) users can have different attitude: they can decide
QoE; the sound genre can affect the gaming QoE. to climb the social ladder of the Metaverse and become leaders
3) Context influence factors: Regard all those factors that of other people or they can decide to undertake a solitary
describe the user’s environment in terms of physical, temporal, experience of the virtual world by ignoring social interactions.
social, economic, task, and technical characteristics [3]. For Social aspects also regard how the user interact with other
instance, the influence of the surrounding environment is users. One-to-one and one-to-many interaction may happen
investigated in [13], where the subjects were exposed to in different ways, such as chatting, talking, or even gesture
different light and noise stimuli while consuming multimedia of the avatars. Also, the user has the opportunity to interact
contents. The social aspect in gaming has been treated in both with friends (from the real world) and strangers randomly
[14], where the social context is represented by players’ group met in the Metaverse. Thus, there is the need to define novel
composition based on previous player’s gaming experience. methods to assess how the social dimension of the Metaverse
impacts the user’s perceived QoE. This is not straightforward
III. T HE SOCIAL AND ECONOMIC DIMENSIONS OF THE because the Metaverse is an uncontrolled environment utilized
M ETAVERSE by other people (besides test participants) and it would not
We identified two additional aspects that should be taken be easy to quantify the impact of specific aspects (e.g., social
into account to assess the QoE of Metaverse applications: the interactions) with controlled assessments. User profiling can be
social and economic dimensions of the Metaverse. useful to lead the users to places and events of the Metaverse
close to the user’s interests by fostering meeting with new
A. The social dimension of the Metaverse people with common interests so as to optimize the social
In [2], it is highlighted that while VR-related studies focus experience in the Metaverse.
on a physical world acquisition and digital rendering, the
Metaverse has a strong aspect as a service with more sus- B. The economic dimension of the Metaverse
tainable content and social meaning. Also, it is stated that the With regard to the economic dimension, again we think that
Metaverse has a scalable environment that can accommodate it is a different concept from the economic factors defined in
many people, which is essential to reinforce social meaning. the QUALINET White Paper within the set of the context
In [15], the MetaSocieties are defined and the Metaverse is de- IFs. Indeed, the economic aspect considered for traditional
scribed as a parallel world that runs in the exact moment as the multimedia applications mainly regards the price (tariff) paid
real world. The authors state that the Metaverse expands the to utilize the service. Instead, with the economic dimension of
living and the working space for humans, allowing the humans the Metaverse we identify the possibility to buy digital prop-
to interact with each other in the virtual/real environment. erties in the Metaverse, such as lands (virtual spaces), NFT
Even Facebook has changed name to Meta as it considers (non-fungible token), or virtual accessories for the avatar with
the Metaverse as the next evolution of social connection. specific digital wallets and crypto-currency. The economic
 TABLE I
A SPECTS OF THE SOCIAL AND ECONOMIC DIMENSIONS OF THE M ETAVERSE THAT MAY INFLUENCE THE USER ’ S Q O E.

Dimension Aspects
User behaviour: polite, rude, aggressive
Social User attitude: solitary experience, leader or follower, interested in climbing the virtual social ladder
User-user interaction: one-to-one, one-to-many, with friends, with strangers
Money: need of crypto-currency to buy virtual goods (e.g., virtual lands, virtual house, virtual clothes for the avatar)
Economic
Job: user works in the Metaverse to earn crypto-currency

dimension also includes the possibility for the user to find QoE. Physiological methods try to assess the perceived QoE
a job in the Metaverse and to earn money as well (crypto- implicitly by means of perceptual and cognitive processes
currency in this case). Thus, the financial means of the user (e.g., electroencephalography, heart rate) [18]. In [19], a VR
may strongly influence the perceived QoE of the Metaverse assistive technology system based on physiological metrics
since some desired goods or services may not be affordable. is realized. Several parameters (i.e., electrodermal activity,
Even some places may be not accessible. User profiling may heart rate, blood volume pressure, temperature, accelerometer,
be useful also for the economic aspects, such as to find an electroencephalogram) are used as indicators of the subjects’
employment that requires the user’s skills or to suggest events perceived QoE. Although physiological methods provide valu-
affordable for the user’s financial means. able insight into QoE of advanced media technologies, their
Table I summarizes the main aspects of the social and eco- utilization may currently be still invasive for the users and
nomic dimensions of the Metaverse discussed in this section. not suitable for real-time QoE evaluation. Nonetheless, there
is another research field that has gained attention in the last
IV. Q O E ASSESSMENT
years, i.e., the affective computing, which investigates the
There are still no defined methodologies to assess the QoE relationship between the human perceived QoE and affective
of Metaverse applications, in particular with regard to the behaviours driven by human emotions. The emotions can be
social and economic dimensions. We discuss advantages and automatically inferred from facial expressions, speech, and
drawbacks of subjective and implicit QoE assessment methods. body gestures [20]–[22]. In [21], [23], facial expressions
A. Subjective assessment patterns and gaze directions of the user while watching videos
The subjective QoE assessment is the major approach to were used to train a machine learning-based QoE estimator.
measure the user’s perceived quality, since it considers the The quality of remote working was estimated in [22] based
user’s subjective experience, feelings and expectation. The on analysis of workers’ speech features. Speech features were
most prominent method to assess the subjective QoE is to used to estimate the QoE also in [20]. Thus, these implicit
use single stimulus tests with ratings on a 5-point Absolute methods can be used to infer the emotional state of the users,
Category Rating scale (ACR). The average of the ACR scores by means of their affective behaviors, while using Metaverse
(for stimulus) provided by the subjects is known as the Mean applications. This would: i) enable continuous unobtrusive
Opinion Score (MOS). This type of tests are usually orga- monitoring of user’s QoE; ii) avoid asking the user to provide
nized in lab environments with standardized and controlled feedback after/during the perceived experience. While several
conditions. For instance, the QoE of 360 videos is assessed in studies observed a correlation between the affective state and
[8], where the subjects had to rate the perceived QoE as well the QoE, further research investigations are needed to define
as further aspects, such as sense of presence, acceptability, precise QoE models based on these measurements.
and cybersickness. Emotions also play an important role in
how people perceive and approach the virtual world around
V. C ONCLUSION
them and react to stimuli. In order to quantitatively measure
emotions, the Self-Assessment Manikin (SAM) questionnaire
[17] can be used, which considers three discrete, nine-class We presented an initial analysis of the QoE in the Meta-
graphical rating scales for emotional dimensions, namely, verse. It emerged that the social and economic dimensions
valence, arousal and dominance. Often, the valence is found of the Metaverse may have an important impact on the user’s
to be positively correlated with the perceived QoE [13]. QoE. Social aspects include user’s behaviours and interactions
Nonetheless, subjective assessments have major drawbacks: with other people as well as their social position in the
i) the need to ask explicit feedback to the user; ii) time and Metaverse. Economic aspects include the need of financial
money consuming; iii) the rating scale may not reflect well means to afford goods and the job position in the Metaverse.
the user’s internal perception of quality. We identified implicit QoE assessment methods as those more
suitable to assess the user’s QoE in the Metaverse continuously
B. Implicit assessment and unobtrusively. However, there is still much to research
As an alternative to subjective methods, implicit measure- concerning the QoE assessment in the Metaverse and with
ments (e.g., physiological, human cognitive, affective be- this proposed initial analysis we hope to inspire discussions
haviour) can be used to objectively measure the perceived and ideas for future studies.
 R EFERENCES [21] S. Porcu, A. Floris, J.-N. Voigt-Antons, L. Atzori, and S. Möller,
“Estimation of the Quality of Experience During Video Streaming From
[1] S. Mystakidis, “Metaverse,” Encyclopedia, vol. 2, no. 1, pp. 486–497, Facial Expression and Gaze Direction,” IEEE Trans. on Network and
2022. Service Management, vol. 17, no. 4, pp. 2702–2716, 2020.
[2] S.-M. Park and Y.-G. Kim, “A Metaverse: Taxonomy, Components, [22] S. Porcu, A. Floris, and L. Atzori, “Analysis of the quality of remote
Applications, and Open Challenges,” IEEE Access, vol. 10, pp. 4209– working experience: a speech-based approach,” Quality and User Expe-
4251, 2022. rience, vol. 7, 12 2022.
[3] P. Le Callet, S. Möller, and A. Perkis. (2012) Qualinet White Paper [23] L. Amour, M. I. Boulabiar, S. Souihi, and A. Mellouk, “An improved
on Definitions of Quality of Experience (2012). European Network QoE estimation method based on QoS and affective computing,” in 2018
on Quality of Experience in Multimedia Systems and Services (COST Int. Symposium on Programming and Systems (ISPS), 2018, pp. 1–6.
Action IC 1003), Lausanne, Switzerland, Version 1.2, March 2013.
[4] S. Vlahovic, M. Suznjevic, and L. Skorin-Kapov, “A survey of chal-
lenges and methods for Quality of Experience assessment of interactive
VR applications,” Journal on Multimodal User Interfaces, pp. 1–35, 04
2022.
[5] A. Perkis, C. Timmerer, and et al., “QUALINET White Paper on
Definitions of Immersive Media Experience (IMEx),” in European
Network on Quality of Experience in Multimedia Systems and Services,
14th QUALINET meeting (online), 2020.
[6] G. Kim and F. Biocca, “Immersion in Virtual Reality Can Increase
Exercise Motivation and Physical Performance,” in Virtual, Augmented
and Mixed Reality: Applications in Health, Cultural Heritage, and
Industry, J. Y. Chen and G. Fragomeni, Eds. Springer International
Publishing, 2018, pp. 94–102.
[7] M. I. Berkman and E. Akan, “Presence and Immersion in Virtual
Reality,” in Encyclopedia of Computer Graphics and Games, N. Lee,
Ed. Springer International Publishing, 2019, pp. 1–10.
[8] H. T. T. Tran, N. P. Ngoc, C. T. Pham, Y. J. Jung, and T. C. Thang, “A
subjective study on QoE of 360 video for VR communication,” in 2017
IEEE 19th Int. Workshop on Multimedia Signal Processing (MMSP),
2017, pp. 1–6.
[9] W. Y. B. Lim, Z. Xiong, D. Niyato, X. Cao, C. Miao, S. Sun, and
Q. Yang, “Realizing the Metaverse with Edge Intelligence: A Match
Made in Heaven,” IEEE Wireless Communications, pp. 1–9, 2022.
[10] J. Wang, R. Shi, Z. Xiao, X. Qin, and H.-N. Liang, “Effect of Render
Resolution on Gameplay Experience, Performance, and Simulator Sick-
ness in Virtual Reality Games,” Proc. ACM Comput. Graph. Interact.
Tech., vol. 5, no. 1, 2022.
[11] J. Lee, W. Kim, J. Kim, and S. Lee, “A Study on Virtual Reality Sickness
and Visual Attention,” in 2021 Asia-Pacific Signal and Information
Processing Association Annual Summit and Conference (APSIPA ASC),
2021, pp. 1465–1469.
[12] S. Schmidt, S. Zadtootaghaj, S. Wang, and S. Möller, “Towards the
Influence of Audio Quality on Gaming Quality of Experience,” in 2021
13th Int. Conf. on Quality of Multimedia Experience (QoMEX), 2021,
pp. 169–174.
[13] S. Porcu, A. Floris, and L. Atzori, “Towards the Evaluation of the Effects
of Ambient Illumination and Noise on Quality of Experience,” in 2019
Eleventh Int. Conf. on Quality of Multimedia Experience (QoMEX),
2019, pp. 1–6.
[14] I. Slivar, L. Skorin-Kapov, and M. Suznjevic, “Cloud Gaming QoE
Models for Deriving Video Encoding Adaptation Strategies,” in Proc.
of the 7th Int. Conf. on Multimedia Systems. ACM, 2016.
[15] F.-Y. Wang, R. Qin, X. Wang, and B. Hu, “MetaSocieties in Metaverse:
MetaEconomics and MetaManagement for MetaEnterprises and MetaC-
ities,” IEEE Trans. on Computational Social Systems, vol. 9, no. 1, pp.
2–7, 2022.
[16] SumOfUs. Metaverse: another cesspool of toxic content. [Online]. Avail-
able: https://www.sumofus.org/images/Metaverse report May 2022.pdf
[17] M. M. Bradley and P. J. Lang, “Measuring emotion: The self-assessment
manikin and the semantic differential,” Journal of Behavior Therapy and
Experimental Psychiatry, vol. 25, no. 1, pp. 49–59, 1994.
[18] U. Engelke, D. P. Darcy, G. H. Mulliken, S. Bosse, M. G. Martini,
S. Arndt, J. Antons, K. Y. Chan, N. Ramzan, and K. Brunnström,
“Psychophysiology-Based QoE Assessment: A Survey,” IEEE Journal
of Selected Topics in Signal Processing, vol. 11, no. 1, pp. 6–21, 2017.
[19] D. P. Salgado, F. R. Martins, T. B. Rodrigues, C. Keighrey, R. Flynn,
E. L. M. Naves, and N. Murray, “A QoE Assessment Method Based on
EDA, Heart Rate and EEG of a Virtual Reality Assistive Technology
System,” in Proc. of the 9th ACM Multimedia Systems Conference.
ACM, 2018, p. 517–520.
[20] A. Bhattacharya, W. Wu, and Z. Yang, “Quality of experience evaluation
of voice communication: an affect-based approach,” Human-centric
Computing and Information Sciences, vol. 2, no. 1, 2012.