2019 IEEE 9th International Conference on Consumer Electronics (ICCE-Berlin)

Augmented reality for cultural heritage education

Cristian Perra Elisavet Grigoriou Antonio Liotta Wei Song Claudio Usai Daniele Giusto
University of University of Edinburgh Napier Shanghai Ocean University of University of
Cagliari, Italy Cagliari, Italy University, UK University, China Cagliari, Italy Cagliari, Italy
[email protected] [email protected] [email protected] [email protected] [email protected] [email protected]

Abstract—In the past few years, it is noticed a fast adoption and when visiting cultural heritage sites [4]. However, it is not
increase of mobile devices that are able to retrieve information adequately provided with the existing of traditional media [1].
quickly, regardless of the user’s physical environment. Thus, AR introduces a set of opportunities that should not be
application developers have leveraged it by developing a variety
of applications using mobile Augmented Reality (AR) in domains passed over. The learning experience of cultural heritage can
such as cultural heritage education, shopping, gaming, and more. benefit from an improvement in the availability of information
AR technology overlays a virtual object into the real world regarding particular museums, art galleries, and more, through
that enables users to have virtual and real-world experience the use of AR [3].
at the same time. During a visit to a cultural heritage site The authors in [2] have designed and developed an AR-
such as a museum or an art gallery, the learning experience
and enjoyment is necessary for a visitor and AR introduces based application towards enjoyable information learning at
new opportunities to improve them. We propose an AR-based cultural heritage sites to help visitors to learn. A multi-
framework for cultural heritage education. We aim to promote user mobile AR education platform that supports an indoor
cultural heritage education by enriching user’s experience and and outdoor environment and tries to enhance the visitors
at the same time to offer the knowledge with the most enjoyable experience via an engaging and playful game is proposed
way. In order to evaluate our framework, we have developed a
server application and an AR-based smartphone application. in [5]. An AR authoring tool which is accessible to cultural
Index Terms—augmented reality, cultural heritage, education, heritage experts is introduced in [6]. The authors in [6] explore
android, ARCore the relationship between AR technology and storytelling and
the feasibility of using AR in literary museums having little
I. I NTRODUCTION experience in the AR technologies.
We use mobile devices in our daily lives, anytime and The personalization of the digital guide is an essential
anywhere, retrieving information about anything and anyone. characteristic [7]. The [7] project proposed a tour assistant
Due to the fast development of communication and sensor to cultural site visitors, which supports onsite help and AR
technology, mobile devices are aware of the information reconstructions of ancient ruins, based on user’s position
nearby, the device’s orientation and context-related charac- and orientation in the area, and real-time image rendering.
teristics. Moreover, the current technologies and innovative Moreover, digital storytelling was considered on Acropolis
applications allow researchers and visual artists to investigate Museum [8], and the AR experience starts by forming the
a variety of application possibilities using mobile Augmented profile of the visitor and it dynamically is adapted with the
Reality (AR) in domains not commonly associated with com- AR activities based on user’s behaviour.
puter technologies such as cultural heritage and performing The main aim most of the related works was to help visitors
arts, by minimizing the distance between cultural spaces (e.g. to discover paintings playfully and excitingly by installing AR-
museums and art galleries) and citizens. Furthermore, the based systems in Museum’s premises. However, this comes
usage of Information and Communication Technologies (ICTs) with the challenge to deal with the large crowds [9]. The
can enhance users enjoyment, and people can learn more AR experience by groups can be complex because the visitors
effectively about culture [1]. are not familiar with the AR technology. Another challenge
AR technology overlays the virtual object into the real world that the AR creators need to deal with is the engagement with
that enables users to have virtual and real-world experience the visitors. A variety of requirements need to be considered
at the same time [2]. Video, audio, 2D and 3D images, during the development phase to allow active learning to
web and text are just some examples of content types that happen. These requirements are (i) fun ( the visitor must
can be overlaid on our perception of the real world. AR enjoy using the application), (ii) challenge (the visitor must
is an efficient way of tying together our physical context have an objective to use the application) and (iii) curiosity
and the digital information which we can retrieve through a (the application must stimulate the visitor to explore the topic
mobile device. The use of such a technology act as a bridge further) [10].
between the digital and the physical world and constitutes an In our work, we propose an AR-based framework for
excellent resource for educational purposes. Cultural heritage cultural heritage education. The aim of our research is to
education is one of the fields that could benefit more from promote cultural heritage education by enriching users expe-
this technology [3]. The learning experience and enjoyment rience and in the same time to offer the knowledge with the
at a cultural heritage site are highly necessary for a visitor most enjoyable way using AR technology. More precisely, we

Authorized licensed use limited to: UNIVERSIDAD VERACRUZANA. Downloaded on June 26,2024 at 02:32:12 UTC from IEEE Xplore. Restrictions apply.
333 978-1-7281-2745-3/19/$31.00 ©2019 IEEE
used Google ARCore [11] which powers the Android app to • AR Books. The books are augmented with the help of
enable the AR features and database of images made with the technological devices such as special glasses.
arcoreimg tool [12] in order to store, retrieve and recognize • Skills Training. The support of training individuals in
the objects. In order to evaluate our framework, we have devel- specific tasks. e.g. aeroplane maintenance.
oped an AR-based android application. The main contributions • AR games. AR technology enables the development of
of this work are: the proposal of an architecture that simplifies games which take place in the real world and are aug-
the management of learning units related to museums and art- mented with virtual information. They provide educators
gallery since all the content is managed through cloud services with highly interactive and visual forms of learning.
and applications; simplifies the user(learner) experience be- Some benefits of the AR usage in an education environment
cause the augmented reality services are provided by their own are the following [18] [19] [14] [17]:
smartphone devices; provides a more environmental-friendly • Using AR in the cultural heritage context will maximize
system because there is no need for printed tickets since virtual visitors satisfaction during the educational progress,
tickets are managed within the same smartphone application; because he/she has more fun running through a museum
the selection of the learning items is fully personalized because and solving tasks directed by an AR application than
each user can choose the information in any order during a reading labels next to the objects.
visit. • AR technology can enhance learning practices in terms
of the creation, distribution, and access to learning re-
II. T HE AR IN THE EDUCATION CONTEXT sources, time and location independence, etc.
• Mobile AR appealing because it can support the delivery
Educational researchers have increasingly recognized new
possibilities for teaching and learning provided by AR technol- of flexible, seamless and personalized learning activities
ogy. The coexistence of virtual objects and real environments to visitors. The emotional engagement of a museum
allows learners to visualize complex spatial relationships and visitor has excellent potential towards proposing a unique,
abstract concepts. Mobile AR provides designers with a new personalised experience that will fit not just to predefined
opportunity to start thinking more deeply about the mobile visitor profiles but to each visitor’s experience and the
learner’s context and its situation. AR technologies can take particularities of each museum visit.
• The users motivation is increased. The user is more
any situation, location, environment or experience to a whole
new level of meaning and understanding [13]. interested and engaged in dealing with new technol-
ogy and learning content compared to non-AR methods.
Thanks to AR applications, the vision can be enlarged and
Moreover, the users pay more attention to technology and
intensified utilizing dynamic modes, to experience and know
thus to the content.
the physical environment. The latest developments of AR
• The learning is user-based. A conventional visiting is
technology uncover unexpected scenarios related to learning
replaced by a new active and self-paced learning program.
contexts, in particular in cultural heritage sites. Moreover, AR
The visitors are more self-responsible for their progress
can be considered as a dynamic and interactive educational
in education.
tool which contributes to transform spaces, times and modes
• Users creativity is increased because AR supports creative
of learning. New educational strategies are explored, such
learning and increase the ability to explore new knowl-
as gamification, AR learning, mobile learning [14]. It’s
edge.
important to mention that the application of computer games
for education and training is defined as “Serious games” III. P ROPOSED FRAMEWORK
and has attracted extensive attention for the development of The AR-based android application was designed to be easy
AR applications for the cultural heritage sites [15]. Further- of use and educative. On the application, three characteris-
more, AR technology allows learners to enhance the exist- tics that will be available are (i) information about the Art
ing environment by augmented it with extensive knowledge. Gallery/Museum, (ii) camera access for AR experience and
Moreover, learning activities can follow different pedagogical (iii) AR Art Gallery/Museum map and navigation area. The
approaches. Apart from the subject content itself, educational first characteristic is already available, and the next two are
and psychological issues also need to be considered when in progress. Overall, there were “Points of Interest” to be
designing a higher education learning system [16]. discovered through AR, using different objects and QR code
AR technology can be implemented in education environ- recognition (in some cases).
ments by following one of these five ways [17]: The Figure 1 shows the high-level architecture with the
• Discovery-based Learning. A user is provided with in- basic features of the ARTeller project that are the AR-based
formation about a real-world place while simultaneously android application (right side) and the back-end (left side).
considering the object of interest. The, e.g. museum, • The AR-based application developed with Android Stu-
historical places, etc. dio and it uses ARCore version 1.9 and Sceneform to
• Objects Modeling. The learners receive immediate visual enable the AR features.
feedback on how a given item would look in a different • The back-end is composed of an authentication mid-
setting. e.g. architectural education. dleware, a database instance, a file repository and the

Authorized licensed use limited to: UNIVERSIDAD VERACRUZANA. Downloaded on June 26,2024 at 02:32:12 UTC from IEEE Xplore. Restrictions apply.
334
 Fig. 1. The ARTeller high level architecture.

arcoreimg tool. In our implementation we used a

PHP framework and a MySQL database. However, the
same result can be achieved with different languages.
The middleware ensures secure access to the REST API
by using HTTPS connection and client authentication by
utilizing JSON Web Token (JWTs) [20]. The arcoreimg
is a tool developed by the Google ARCore team that Fig. 2. ARTeller Android Application
creates a database from a set of images ready to be
used by any application that supports ARCore. Therefore
this database must be available to be downloaded on the
application to use the AR feature of the app. In order
to have the ability to update the Database (DB) from
time to time (in case a museum or art gallery changes its
collection), we chose not to embed the DB in the basic
version of the application in order to keep it lighter and
leave the user to decide whether or not to download the
AR package of the application. The file repository stores
files such as an explanatory video or audio files, about
each piece of the museum/art gallery collection, to expose
them when the application recognizes an object.
In Figure 4, we present the workflow of the ARTeller
application. It begins with the installation of the application
on the client’s device and with the registration or login of the
user to get an authenticated connection with the server. Then,
the user chooses to use the AR feature and downloads the
DB with the images that can be used by ARCore. Using the Fig. 3. Lab setup - Art Gallery using AR-based android application
device’s camera, ARCore scans the physical environment for
known (physical) objects, and if any object is recognized, it
proceeds with a request to the server. This request will include score, the highest possibility for the image to be recognized.
information related to the object (e.g. the painting) and then Each painting of the Collection is defined as “Point of Interest“
render the information on the screen through a visual effect. (POI) in the android application. So, when a user leaves from a
painting’s position, he will be able to see the same information,
when he returns to that painting.
A. Experimental setup In a laboratory environment, we set up a mini art-gallery using
As we mentioned before, the application was developed the paintings that we mentioned before. Then, we installed
using ARCore version 1.9 [11] software development kit in the the android application to the “Motorola G7” smartphone. We
Android platform, and a smartphone (Motorola G7) was used registered as a new user as shown in Figure 2 (left image)
for testing purposes. We used “arcoreimg” tool to identify the and through the Menu, we bought a new ticket as shown in
score of the images that we chose to use in our experiments. Figure 2 (right image) for an Art Gallery (e.g. our Laboratory
The three paintings that we decided to use were “Elasticity” Gallery). After that, we prompt the user to download the
by Boccioni, “Wanderer above the Sea of Fog” by Caspar “AR library” which extends the android application. The ”AR
and “Composition VII” by Kandinsky. We chose these three library” is the database with the images/paintings that will
painting because they contain many unique feature points and be recognized. The Museum/Art Gallery operator will be in
using ”arcoreimg” tool the score was 100. The highest the a position to update this database when it’s time to update

Authorized licensed use limited to: UNIVERSIDAD VERACRUZANA. Downloaded on June 26,2024 at 02:32:12 UTC from IEEE Xplore. Restrictions apply.
335
 ACKNOWLEDGMENT
This research activity has been partially funded within the
Cagliari2020 project (MIUR, PON04a2 00381), the ARTeller
project (POR FESR, 2014-2020), and the DigitArch Cluster
Fig. 4. The ARTeller workflow. Top-Down project (POR FESR, 2014-2020).
R EFERENCES
[1] F. Piccialli and A. Chianese, “Cultural heritage and new technologies:
Trends and challenges,” 2017.
[2] U. C. Pendit, S. B. Zaibon, and J. A. Bakar, “Mobile augmented reality
for enjoyable informal learning in cultural heritage site,” International
Journal of Computer Applications, vol. 92, no. 14, pp. 19–26, 2014.
[3] L. Villarejo, F. González-Reverté, O. Miralbell, and J. M. Gomis,
“Introducing augmented reality in cultural heritage studies,” eLearn
Center Research Paper Series, pp. 06–14, 2014.
[4] P. Zhao, S. Sintonen, and H. Kynäslahti, “The pedagogical functions of
arts and cultural-heritage education with icts in museums–a case study
of finna and google art project,” International Journal of Instructional
Technology and Distance Learning, vol. 12, no. 1, pp. 3–15, 2015.
[5] A. Angelopoulou, D. Economou, V. Bouki, A. Psarrou, L. Jin,
C. Pritchard, and F. Kolyda, “Mobile augmented reality for cultural
heritage,” in International Conference on Mobile Wireless Middleware,
Operating Systems, and Applications. Springer, 2011, pp. 15–22.
Fig. 5. The “Information Card” [6] C. Fenu and F. Pittarello, “Svevo tour: The design and the experimen-
tation of an augmented reality application for engaging visitors of a
literary museum,” International Journal of Human-Computer Studies,
its Collection, and the user will be informed when a new vol. 114, pp. 20–35, 2018.
[7] V. Vlahakis, M. Ioannidis, J. Karigiannis, M. Tsotros, M. Gounaris,
Collection is available through the application. Then, the user D. Stricker, T. Gleue, P. Daehne, and L. Almeida, “Archeoguide:
executes the AR-based application and scans the Paintings, as an augmented reality guide for archaeological sites,” IEEE Computer
shown in Figure 3 in the room. When the user is interested in a Graphics and Applications, vol. 22, no. 5, pp. 52–60, 2002.
[8] J. Keil, L. Pujol, M. Roussou, T. Engelke, M. Schmitt, U. Bockholt, and
Painting, he/she needs to press the “yellow” button as shown in S. Eleftheratou, “A digital look at physical museum exhibits: Designing
Figure 5 on the screen to see the information. This information personalized stories with handheld augmented reality in museums,” in
is presented in a “Information Card“. The “Information Card“ 2013 Digital Heritage International Congress (DigitalHeritage), vol. 2.
IEEE, 2013, pp. 685–688.
includes information related to the Artist, the Title of the [9] Y. Kolstee and W. van Eck, “The augmented van gogh’s: Augmented
Painting and the creation Year. In Figure 3, we show that the reality experiences for museum visitors,” in 2011 IEEE International
AR-based Android application can recognize the Painting and Symposium on Mixed and Augmented Reality-Arts, Media, and Human-
ities. IEEE, 2011, pp. 49–52.
it’s going to show the “Information Card” to the user as seen [10] H.-K. Wu, S. W.-Y. Lee, H.-Y. Chang, and J.-C. Liang, “Current
in Figure 5. status, opportunities and challenges of augmented reality in education,”
The proposed AR-based application is a Demo Version that Computers & education, vol. 62, pp. 41–49, 2013.
[11] P. Nowacki and M. Woda, “Capabilities of arcore and arkit platforms
was tested in a laboratory environment. Currently, there are for ar/vr applications,” in International Conference on Dependability
no results from its usage in a real environment regarding the and Complex Systems. Springer, 2019, pp. 358–370.
decrease of the operating costs or users’ opinions. Detailed [12] G. Developers. (2019) The arcoreimg tool. [On-
line]. Available: https://developers.google.com/ar/develop/c/augmented-
results will be described in our future work. images/arcoreimg
[13] R. K. Sungkur, A. Panchoo, and N. K. Bhoyroo, “Augmented reality,
IV. C ONCLUSION the future of contextual mobile learning,” Interactive Technology and
Mobile AR-based services can work anywhere, visualizing Smart Education, vol. 13, no. 2, pp. 123–146, 2016.
[14] C. Panciroli, A. Macauda, and V. Russo, “Educating about art by
the digital information related to the user’s real environment augmented reality: New didactic mediation perspectives at school and in
whenever desired. Hence, mobile AR can be extended to cover museums,” in Multidisciplinary Digital Publishing Institute Proceedings,
a variety of situations and scenarios in mobile computing. vol. 1, no. 9, 2018, p. 1107.
[15] L. Hoffmann, “Learning through games,” Commun. ACM,
In this paper, we introduce a novel framework for cultural vol. 52, no. 8, pp. 21–22, Aug. 2009. [Online]. Available:
heritage education using Augmented Reality technology. An http://doi.acm.org/10.1145/1536616.1536624
AR-based application was designed to help users to get [16] D. Joan, “Enhancing education through mobile augmented reality.”
Journal of Educational Technology, vol. 11, no. 4, pp. 8–14, 2015.
learning items at the cultural heritage site by enhancing their [17] P. Diegmann, M. Schmidt-Kraepelin, S. Eynden, and D. Basten, “Ben-
visiting experience. It assists users to understand the artwork efits of augmented reality in educational environments-a systematic
pieces of a museum or art grallery via engaging. We observe literature review,” Benefits, vol. 3, no. 6, pp. 1542–1556, 2015.
[18] A. Damala, T. Schuchert, I. Rodriguez, J. Moragues, K. Gilleade, and
that the mobile AR technology offers great opportunities to N. Stojanovic, “Exploring the affective museum visiting experience:
cultural organizations for provided added value for the learning adaptive augmented reality (a2r) and cultural heritage,” International
and lifelong learning areas. In our future work, we intend to Journal of Heritage in the Digital Era, vol. 2, no. 1, pp. 117–142, 2013.
[19] Z. Yovcheva, D. Buhalis, and C. Gatzidis, “Engineering augmented
extend the android application by supporting geolocation and tourism experiences,” in Information and communication technologies
user navigation to improve the user’s experience. in tourism 2013. Springer, 2013, pp. 24–35.
[20] Auth0. (2019) Json web token. [Online]. Available: https://jwt.io

Authorized licensed use limited to: UNIVERSIDAD VERACRUZANA. Downloaded on June 26,2024 at 02:32:12 UTC from IEEE Xplore. Restrictions apply.
336