GuideMe: A Mobile Augmented Reality System

to Display User Manuals for Home Appliances

Lars Müller1 , Ilhan Aslan2 , and Lucas Krüßen1

1
FZI Research Center for Information Technology, Karlsruhe, Germany
2
ICT&S Center, University of Salzburg, Salzburg, Austria

Abstract. In this paper we present GuideMe, a mobile augmented

reality application that provides assistance in using appliances. In or-
der to explore how users perceive GuideMe, as a design of an interac-
tive and digital manual, we conducted two user studies. We compared
GuideMe first with paper-based manuals and then with video-based
manuals. Our results indicate that the paper-based manuals were supe-
rior regarding typical usability measures (i.e. error rates and completion
times). However, participants reported a significantly higher perceived
task load when using paper-based manuals. Due to a better user expe-
rience, GuideMe was preferred by 9 of 10 participants over paper-based
manuals. We present our design in detail and discuss broader implica-
tions of designing digital manuals. Furthermore, we introduce a custom
format to define manual structures for mobile augmented reality enabled
manuals.

1 Introduction

Many of today’s challenges in designing interfaces relate to the spread of tech-

nology to our homes and everyday lives with a shift from a rather narrow task-
orientation to qualities of everyday experiences [3,2]. For example, ten years ago
a baking oven would be operated by choosing temperature and mode. Today,
an oven can be programmed to start at a predefined time with a predefined
temperature and several operation modes. Interfaces of everyday products in
our homes have gained in complexity. Getting familiar with all the features that
such a product provides is challenging. In order to assist users, to handle their
appliances and consumer electronics, manuals are created by technical writers
and illustrators, who carefully describe the operation of the product. However,
very few people enjoy interacting with a manual; i.e., going through a book con-
sisting of technical writings and illustrations to find out how to operate a system
or to solve an existing problem. Off the shelf mobile devices could be used as
digital manuals and thereby performance and the user experience of interacting
with manuals could be improved. However, designing a digital manual based on
inspirations taken from physical manuals can be a cumbersome task.
In this paper we present GuideMe, a design that aims at exploring alterna-
tives to traditional paper-based manuals. GuideMe is an interactive “digital”

D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 152–167, 2013.

c Springer International Publishing Switzerland 2013
 GuideMe: A Mobile Augmented Reality System 153

manual informed by properties of mobile devices and augmented reality technol-

ogy. Hereby, our intention was to make use of people’s familiarity with mobile
devices and create a design that would be timelier and exciting to use.
This paper is structured as follows. In the next section we provide the back-
ground on mobile augmented reality and current practices for user manuals.
Then we discuss in more general how GuideMe fits into current concepts and
notions in interaction design. Building on these fundamentals we present details
of the technical implementation of the prototype. Hereby, we introduce a custom
format to define manual structures, which we refer to as User Manual Markup
Language (UMML): an XML based format to define user manuals and especially
technical illustrations. The GuideMe prototype is evaluated using manuals for
two ovens and compared to a video tutorial and excerpts from the original printed
manual. The results are summarized and discussed.

2 Background

The development of GuideMe builds on knowledge from traditional user manual

design and the technical progress in the field of augmented reality (AR).

2.1 User Manuals

User manuals are a part of the technical documentation of a product. The Secure-
Doc guideline [18] states, “Products are not complete without documentation.”
The guideline interprets, an important standard in technical documentation, the
IEC 62079:2001 on “Preparation of instructions. Structuring, content and pre-
sentation” [10] and helps designing the technical documentation of a product. It
outlines the requirements that are induced by European law, such as to enable
customers to use all features of a product and to protect customers of potential
hazards. The guideline relates to factors that are often ignored when a product
is accompanied by a user manual of poor quality. For instance, “High quality
documentation helps reduce customer support costs,” because with the right in-
formation at hand it enables customers to solve many problems without further
assistance. Furthermore, “High quality documentation enhances customer sat-
isfaction,” because a poorly designed user manual can prevent customers from
exploring the full potential of a product. Another factor is that customers re-
late the quality of the product to the quality of its documentation. Therefore,
the need for high quality user manuals is given and the exploration of emerging
technologies, such as augmented reality, to improve their usability is worthwhile.
Although other formats are available, the standard format for user manuals is
still the printed handbook. Typically it offers an index that lists all functionalities
or use cases of a product. Each one of them is then described in a step-by-step
manner. Technical illustrations improve understandability by “expressive images
that effectively convey certain information via the visual channel to the human
observer” [20]. Mainly for cost reasons comprehensive user manuals are often
included as digital PDF documents on CD or for download. Although the content
154 L. Müller, I. Aslan, and L. Krüßen

is normally taken one-to-one from the printed version, it improves the aspects
of information retrieval concerning finding the user manual itself (if available for
download) and finding the relevant information in it by searching the document.

2.2 Augmented Reality

Initially research in AR-based user manuals has been conducted using head-
mounted displays (HMD) to provide information while having two free hands.
The KARMA project uses such an HMD to convey step-by-step instructions and
superimposed instructions to specialized mechanics [4]. Henderson et al. [9] built
an advanced HMD-based AR interface and showed that their system improves
the current documentation of mechanics. However, AR research currently ex-
plores new platforms. An increasing number of AR applications uses the mobile
phone to display virtual objects [7,5,13]. Furthermore, mobile projectors provide
new means to augmented objects with information by projecting the interface
directly on the surface of the object [11].
As early as 2001, the AR-PDA prototype [5] mentioned mobile AR manuals as
one possible application domain. Since this time, researchers have explored dif-
ferent technical implementations and hardware platforms. Hakkarainen et al. [7]
actually applied mobile AR to user manuals to showcase their developed system
that still relied on a server infrastructure to calculate the positioning of super-
imposed instructions. Liu et al. [13] evaluated a new mobile AR approach in
the domain of user manuals. They show that “real-time AR feedback on phys-
ical actions in the real world” is beneficial regarding usability and task load of
users. The test was based on an adapted MIDI station that could provide such
a feedback channel.
The technology to build AR-based manuals is developing rapidly. First ap-
plications are already available, e.g. an app to explain a new car model1 or a
prototype of Aurasma that uses augmented reality to explain how to connect
cables of a router2 . These approaches cannot be generalized to other home ap-
pliances. They are impressive technology demonstrations, and showcase state
of the art marker-less object recognition of 3D objects. However, they require
highly textured user interfaces and are tied to a specific car or appliance.
The research focus shifts from the technological feasibility to the design of
such applications. Several companies offer mobile AR SDKs that support the
development of AR applications for example Qualcomm3 and Metaio4 .

3 Designing Digital Manuals for and with Mobile Devices

From the beginning mobile devices have been recognized as personal digital
assistants, which help to manage personal information. Later mobile devices
1
Audi A1 user manual
http://itunes.apple.com/de/app/audi-a1-ekurzinfo/id436341817?mt=8
2
Aurasma Visual Browser http://www.aurasma.com/
3
Vuforia SDK http://www.qualcomm.com/solutions/augmented-reality
4
Metaio http://www.metaio.com/
 GuideMe: A Mobile Augmented Reality System 155

were also perceived as a tool that can manage contextual information. They have
pervaded our everyday life, and the number of mobile applications is growing.
As computational things become everyday things, what we design for can
not be restricted to how to enable people to become more productive.
Thus, there is a need for complementary design philosophies. [14]
Redström herby, refers to how HCI researchers used to focus on performance
and supporting people in accomplishing tasks. In the last decade many re-
searchers started to focus on creating new experiences enabled by mobile de-
vices. De Sa and Churchill [16] argue that the new affordances offered by mobile
augmented reality have potential to enhance users‘ experiences. More specifically
they discuss how this can be achieved through the provision of digital informa-
tion which is relevant for the user context. With mobile devices information that
is digitally stored can be processed anywhere anytime. Consequently, more and
more information is presented in digital form as and through mobile media.
One experiences the world through the technology, and the technology
inevitably becomes part of the way one relates to the world. While many
technologies appear in between the user and the world, not all are em-
bodied. For a technology to hold an embodiment relation it must be
technically transparent, it must allow its user to “see through” it. [3]
Based on philosopher Don Ihde‘s non-neutrality of technology-mediated ex-
perience, Fallman hereby points out that for technology to be embodied it needs
to recede into the background of experience. One could argue that this is true
for mobile devices and mobile augmented reality applications. Similar to how
one does not feel the weight of glasses after carrying them for a while, a mobile
device, although heavier, recedes into the background of experience during in-
teraction with the real world. However, this real world becomes augmented with
digital information.
Different lines of research within HCI have recognized the increasing blur be-
tween the digital and the physical in today’s designs. Efforts to improve the
understanding of those kinds of designs exist. For example, Vallgårda et al. dis-
cuss how knowledge of materials has been essential to design practice and how
computers share important characteristics with other materials that are used in,
for example, industrial design or architecture. They introduced computational
composites as a new type of composite material [19]. They argued that compu-
tational properties of a computer are difficult to exploit, but through combining
it with other material can come to use.
In mobile augmented reality applications, mobile devices are used to augment
real-world objects and materials. One could regard mobile augmented reality as
a computational property of a mobile device. Through augmenting a specific real
world object (e.g. an oven); i.e. combining a mobile device with the real world
object, augmented reality as a computational property comes to use.
While printed manuals are still the standard media for manuals, there is a
need for digital manuals and interaction that is more timely and uses features
of mobile devices that people have already become familiar with.
156 L. Müller, I. Aslan, and L. Krüßen

However the mobile device itself is a medium that supports through its prop-
erties (e.g. being mobile, lightweight, having a camera, having a screen etc.)
interaction that is fundamentally different from paper. Finding out how to best
interact with new media is in general a difficult task and requires exploration.
This is particularly true for digital counter parts of physical designs. For exam-
ple, Grasset et al. [6] try to answer the question if a mixed-reality book is still
a book and explore the design and development process of visually augmented
books. They argue that exploring design issues becomes more important as tech-
nology gets more mature; and that the development of prototypes requires time
and very specialized experts.
The intention of this section was to reflect on what it means to exploit mobile
devices for interaction in a broader sense and to remind the reader of current
design notions and problems. A mobile device is a new medium and mobile
augmented reality can provide rich sensory effects. The key challenge for AR-
based user manuals lies in building on the aforementioned technological advances
in AR and to connect the device and the required information in a manner that
is intuitive to the user and if possible even fun to use. It is unclear how much
of the knowledge on user manual design can be applied to mobile phones and
which new challenges arise when user manuals are adapted to the mobile device.
In order to provide some insights we now move to a concrete implementation
of a digital manual that is inspired by properties of a mobile device.

4 GuideMe Prototype

The GuideMe system identifies home appliances by using the camera and re-
trieves an interactive manual from a server. The interactive manual is specified
in the User Manual Markup Language (UMML). The following sections out-
line the method used to recognize the device, the UMML specification and the
resulting user interface.

4.1 Connect to the Physical User Interface

To use a mobile device to interact with an appliance a connection between those

has to be established. Both components should become a single user interface to
display the required information.
For this purpose, a marker-based object recognition similar to [15] was chosen
because the preferred marker-less recognition of an appliance requires textured
surfaces [17] and therefore cannot be applied properly to home appliances which
often have monochromatic or reflecting user interfaces. Marker-less approaches
struggle to distinguish two devices of similar color and form. If marker-less recog-
nition is applied the reflective user interfaces, the reflections will become part of
the calculated feature, thus making it difficult to recognize the same device with
different reflections. In a marker-based recognition the marker becomes part of
the user interface.
 GuideMe: A Mobile Augmented Reality System 157

4.2 Definition of the Digital User Manual

Appliances differ in size, form and functionality. In result each user manual has
to be customized. GuideMe aims at providing the best possible user manual
for a specific appliance and a specific mobile device. Therefore, we decided to
put the design of the user interface into the hands of the experts by providing
them a simple format to author user manuals. In consequence designers can
adapt existing manuals to a specific device and experiment with different layouts
and structures. We developed a new format to define manuals and refer to it
as User Manual Markup Language (UMML). UMML defines the layout of the
recognized user interface. In comparison to existing formats like APRIL [12],
UMML benefits from a clear focus on user manuals. APRIL aims at structuring
narrative content and defines cast, interactions and behavior. This flexibility
leads to complex definitions that require advanced knowledge.
The resulting schema builds on a smaller set of XML statements to define
the user manual. The user interface was split into the basic elements like text,
arrows or any other graphical element. Custom elements can be defined and used
as well. The grouping of these elements and the final structure of UMML was
inspired by the design of current printed user manuals.
An UMML file consists of two parts: the menu description and the list of func-
tionalities. Each functionality definition has a unique ID. The menu description
organizes the available functionalities into a hierarchical structure of menus that
can contain functionalities or submenus. Functionalities can be linked into a
menu several times at several levels, if desired. Functionalities contain a title tag
and consist of a number of steps. The steps are identified and linked by a unique
ID. Each step definition contains a list of virtual objects and their position in
relation to the frame marker, e.g. an arrow pointing to a button. Furthermore,
images, videos and audio files can be linked into the manual.

4.3 Magic Lens User Interface

The GuideMe application is an UMML interpreter that builds on the Vuforia

SDK to implement a so called “magic lens” interface [1]. The fixed components of
the user interface are minimal to leave the screen to the manual designer. Only a
small tool bar at the top supports the navigation and shows the current progress.
The remaining part of the screen can be used to display the augmented camera
image. This design aims at providing a maximum of possibilities to the manual
designer and restricts the design considerations to providing the appropriate
components to use the available space in an easy and intuitive manner.
Figure 1 shows how the camera image of the physical user interface (figure
a) is combined with instructions (figure b and c). Two hand icons point at the
available buttons. A frame is used to highlight the part of the display that will
change by pressing the buttons. The user sees a live image of the real oven.
Therefore, the user does not need to compare a depicted oven in a manual
with the real oven. Furthermore, if the participant moves the hand to press a
button, the camera will display this action on the screen as well. The virtual
158 L. Müller, I. Aslan, and L. Krüßen

(a) Oven with Vuforia (b) GuideMe highlights (c) GuideMe screenshot with
frame marker and enclosed the corresponding operat- highlighted operating elements
QR code ing elements

Fig. 1. GuideMe prototype in action

objects defined in the UMML file can be classified by their positioning on the
screen. The user has to hold the tablet in front of the user interface to recognize
the marker. GuideMe uses the size and position of the marker to adapt the
coordinate plane. Texture components, symbols, arrows, frames and user defined
graphics are placed relative to the recognized marker position. In general these
components are placed in the same plane as the marker but a 3-dimensional
placement is possible as well.
In some cases superimposed objects might not be sufficient to indicate the
correct action. For example, if one button has to be held while pressing other
buttons, an image of two arrows pointing to both buttons will not be enough.
The instructions that are printed in a manual have to be conveyed to the user as
well. Using UMML and GuideMe, they can be printed on the screen or played as
an audio or video clip. All three methods are possible with the existing UMML
elements. Videos, audio clips, Android webviews, buttons and option menus
are placed at fixed positions on the screen. Android webviews can load HTML
context into the Android application from a file or the Internet. Video and
audio clips start automatically when the corresponding step is rendered. Textual
descriptions are placed at the top of the screen. All components can be freely
combined, although a full screen video overlaps other components.
Navigation between user manual steps is possible by pressing buttons or by
selecting from an options menu. Options menus and buttons can be defined in
the UMML file. Users can always return to the last step using the Android back
button or return to the menu by using the navigation bar at the top.

5 Evaluation
We conducted two subsequent studies to obtain first insights on the user experi-
ence and task performance using GuideMe. The first study compared GuideMe
based manuals against excerpts from the original printed user manual and the
second against video tutorials.
We selected two state-of-the-art ovens, the Stoves SEB900MFSe and the
Bosch HBL78B7.1. Both provide sufficient complexity to require a manual for
 GuideMe: A Mobile Augmented Reality System 159

first usage of advanced functionality and provide several user interface elements,
such as knobs, buttons and displays that could be augmented. They are simi-
lar enough to allow a comparison but have different user interfaces to mitigate
learning effects during the study.

5.1 User Manual Design

One simple and one complex task were selected from the original printed user
manual. In a pre-test the two tasks were completed on both ovens with similar
effort. However, the complexity of the selected task was reduced by one step
after the pre-test. Participants needed more than 10 minutes to set the start of
the cooking period using the original manual. Hence, the complex task in the
study consisted of only four steps.
For the first study the relevant pages of the original manual were used as is.
All unrelated pages were removed to limit the scope of the evaluation to the
presentation of instruction. The original manual served as a basis for all other
manuals. For the second study, video tutorials were needed. The authoring of the
video tutorials was inspired by existing video tutorials. Each video tutorial was
implemented as one continuous video. The video contained the text as spoken
comment and as superimposed text on top of the screen. Furthermore, additional
annotations in the video highlighted important user interface elements of the
oven. Users can navigate the video by using the standard Android controls, e.g.
jumping to any point of the video sequence by selecting it from the progress bar.

Fig. 2. GuideMe example manual: explaining a microwave oven

The GuideMe manual was structured according to the numbered steps in the
manual. For each step, the corresponding explanation is shown at the top of
the screen and the step is visualized on the screen as shown in figure 2. Arrows
images and further markers are superimposed on the camera image to indicate
relevant buttons and displays. Navigation between steps was realized by buttons.

5.2 Study Design

The first conducted study compares (a) the original printed version and (b) a
manual based on GuideMe on a tablet. GuideMe ran on a Samsung Galaxy Tab
160 L. Müller, I. Aslan, and L. Krüßen

10.1 tablet with Android 3.1. We measured task completion times and used the
NASA TLX [8] to measure the subjective task load of participants. Participants
were asked to think aloud while completing the given tasks. Observers noted
down their statements, the perceived body language and the operation of the
appliance. A concluding questionnaire captured the subjective preferences of the
user. The subsequent second study replicates the design of the first study, but
compares (a) a video-based tutorial on a tablet and (b) GuideMe on a tablet.
The participants in the second study did not overlap with participants from the
first study.
During the user study, each participant performed overall 4 measured tasks,
two on each oven: (a) 1 simple and 1 complex task using GuideMe for one oven
and (b) 1 simple and 1 complex task using the printed original manual for the
other oven. Switching between devices and reversing the order of technologies
and devices aimed at mitigating learning effects of the oven’s user interface. Par-
ticipants could use as much time as necessary and should mention when they
deemed the task to be completed or if they want to give up on it. There was no
additional help offered except the manual to be tested. After each task partic-
ipants completed the NASA TLX questionnaire. The concluding questionnaire
collected overall feedback on both used technologies and the perceived complex-
ity of the used ovens.
This data was evaluated in a within subjects design to mitigate the individual
differences in performance and NASA TLX usage. The significance of quantita-
tive data (NASA TLX and task completion times) was analyzed using a paired
T-test and Cohen’s d.
2 groups of 10 young technology-savvy participants (22-31, 4 female, 16 male)
took part in the studies. All 20 participants used GuideMe and all of them
operated both ovens. The majority of the users already knew similar ovens before
the test (55 % Stoves, 70 % Bosch). 53 % agreed or strongly agreed when asked to
state their familiarity with Android device on a 5 point Likert scale. 74% agreed
or strongly agreed regarding their familiarity using tablet PCs. Only 10% had
any experience with augmented reality applications. Albeit this user group does
not represent the overall population, it represents the potential early adopters
of mobile media based user manuals.
Each of the 20 participants performed 4 tasks, resulting in a total number of
80 measurements. 5 of these measurements were classified as outliers, because
participants required exceptionally long to complete a given task. Participants
were stuck because they entered unforeseen menus and got lost in the mismatch
between instructions and available options. As we use a pairwise T-test the cor-
responding measurement using the second device was excluded from the analysis
as well.

6 Results

All participants in both studies reported that the two provided user manuals
were useful to complete the given tasks. Nevertheless, when asked if the oven
 GuideMe: A Mobile Augmented Reality System 161

was difficult to use the majority of users disagreed or strongly disagreed (65%
Stoves, 85% Bosch). According to this question, the Stoves oven was experienced
as more difficult.

6.1 GuideMe vs. Printed Manual

A first group of 10 participants completed a simple and a complex task using

GuideMe on one oven and a similar task using the original paper manual on
another oven. 90% of them stated in the concluding questionnaire that they
preferred GuideMe over printed manuals. As shown in figure 3(a), they needed
significantly more time (p < 0.0003, d = 1.46 ) to complete both tasks using
GuideMe than by using the printed manual. Although all 10 participants deemed
each task completed, the final result of the complex task included errors for the
majority of participants. One specific error stood out. Users of the Stoves oven
ignored the icon indicating the selected mode and ended up setting a timer
instead of setting the cooking time. In many of the observed cases this was due
to the timeout of the user interface after 5 seconds. After changing from main
menu to the timer menu, participants looked at the manual. They did not notice
when the oven display changed and subsequently showed the main menu again.
Pressing the next button, as advised in the manual, switched again from the
main menu to the time menu instead of from the timer menu to the cooking
time menu. The error was very common for GuideMe users (70%), while none
of the users of the paper manual made this mistake. A similar problem with a
timeout of menu occurred at the Bosch oven. However, the changes in the display
of the Bosch are more apparent so users recognized the problem and could avoid
a mistake.
In the original manual all steps of the desired functionality were printed on the
same page. Participants could scan all steps at once and quickly jump between
the different steps. Therefore, they could often perform several steps at once.
Afterwards they had to look at the device to synchronize their expectations
with the current behavior of the device in front of them. GuideMe users could
only see one step at a time. If users wanted to look at another step they had to

 #

 "


 





!




 

 


 

            
  

(a) Task comple- (b) NASA TLX score

tion time

Fig. 3. GuideMe (black, left columns) compared to the original printed manual (blue,
right columns)
162 L. Müller, I. Aslan, and L. Krüßen

press the forward/back button or return to the manual overview. It appeared to

be counterintuitive for participants to go one step back in the manual because
of the timeout in user interface.
Nevertheless, the NASA TLX scores depicted in figure 3(b), are significantly
lower (p < 0.046, d = 1.4) for GuideMe. The NASA TLX subscales show the
reasons. Participants felt that printed manuals induced an increased mental and
temporal demand as well as higher concerns about performance compared to
GuideMe. Participants stated in the concluding questionnaire that GuideMe
makes it easier to associate instruction symbols with user interface elements.
Using GuideMe, it was easier to follow the defined steps, e.g. “It shows an easier
way to act” or “You get only the relevant information”. The main drawback of
GuideMe was seen in the weight of the tablet (565 g) that had to be held with
the camera pointed at the marker. The detailed analysis of the NASA TLX di-
mensions in figure 3 shows how users of the printed user manual underestimated
their performance. Although, the observed results show a flawless and swift op-
eration of the device, they did not feel as confident in their performance using
the original manual. p No matter which technology was used, there was always
a disruption between gathering information from the manual and applying the
new knowledge by operating the oven. Participants using GuideMe did rarely
manipulate the oven controls while looking at the camera image on the tablet
screen. On the one hand, this behavior was induced by marker recognition prob-
lems, e.g. when the marker was no longer in the cameras field of view because of
the attempt to operate the oven. On the other hand, participants had problems
to operate a control in 3-dimensional space, e.g. pressing a button, while look-
ing at a 2-dimensional image of their hand movement. Furthermore, one user
said, “The tablet blocked operation of the oven because it had to be pointed at the
marker.”.

6.2 GuideMe vs. Mobile Video Tutorial

The remaining 10 participants used GuideMe in comparison to the video tu-
torial. There was not a clear preference of one technology as 6 out of 10 pre-
ferred the video tutorial over GuideMe. The task completion time depicted in
4 was significantly faster (p < 0.0056, d = 1.26) using the video than using
GuideMe. The video tutorial received a significantly lower NASA TLX score
(p < 0.0108, d = 0.9). Participants said again that the weight of the tablet and
the need to point the tablet at the user interface are the main obstacles when
using GuideMe “You don’t have to hold the tablet in front of the QR Code.” The
NASA TLX subscales in figure 4 show a difference in the physical and mental
demand, because users had to point the tablet at the marker. However, this
difference did not occur in the first study when GuideMe was compared to the
original printed manual.
In this study 50% of GuideMe users made again the same mistake while
operating the Stoves oven as in the first study. Moreover, 60% users of the video
tutorial made the same mistake. The mobile video tutorial does not provide a
visible separation into steps. Users had to use the standard video navigation
 GuideMe: A Mobile Augmented Reality System 163

 #

 "


 




!




 

 


 

            
  

(a) Time (b) NASA TLX

Fig. 4. GuideMe (black, left columns) compared to a video tutorial on a tablet (red,
right columns)

to go forward and backward. However, it was difficult for participants to scan

through the video, and go to one specific step. When GuideMe manuals were
read and applied in a step-by-step manner, the video tutorials were watched
as a whole, before operating the oven. Participants rarely stopped the video.
They rather watched the whole video again and aimed at memorizing the whole
procedure to perform it afterwards. The users that preferred video said “It’s
faster” or “Description is more detailed and easier to watch”. 4 user preferred
GuideMe because they felt that the video induced “time pressure” and they like
the “step-by-step” approach of GuideMe because the “steps are better separated”.
The body language in this study differed from the participants using the
paper manual in the first study. In the first study participants held the paper
manual only while reading and put it aside while operating the oven. Some
users put the manual on top of the oven while reading from it. Body language
suggested that participant were focused scanning the printed manual to extract
relevant information. In the second study, participants using the video tutorial
or GuideMe were looking expectantly at the tablet screen. They kept on holding
the tablet in their hands. For instance, even when operating the oven, one hand
was still holding the tablet. In the first study this behavior was attributed to the
need to point the camera at the oven, but the same behavior was now observed
for users of the mobile video tutorial.

7 Discussion

The two studies and the design of the GuideMe prototype provide insights into
the usage of different user manual formats, the developed prototype and the
design challenges when creating user manuals for and with mobile devices that
combine physical user interface elements and digital information.

7.1 User Manual Formats

Three different formats of user manuals were used in the two studies. The average
task completion time for the printed user manuals is the fastest and the number
164 L. Müller, I. Aslan, and L. Krüßen

of errors was the lowest among all technologies. Nevertheless, our technology-
savvy participants preferred the two technology based solutions. On the one
hand, this might be due to the nature of our user group. It appears that video
as well as GuideMe provide a special positive experience to this group. While
the paper was treated like a tool, the participants did not put the tablet away
during the study. The used platform, the interactive nature of the presentation
or the ongoing animation on the screen, contribute to this effect. On the other
hand, participants felt, although wrongly, more efficient when using video or
GuideMe as reflected in the performance scale of the NASA TLX. Video and
GuideMe allow users to simply imitate actions or follow detailed advice. The
paper manual had to be understood and mapped to the ovens’ user interface.
The participants have to decide more in the mapping process and each small
decision adds to their uncertainty.
Although GuideMe learned from the printed manual, some design aspects
became only clear by comparing both media in the study. For instance, the
step structure of GuideMe was copied from the original manual but not the
arrangement of steps on a single page. Each augmented image requires the whole
screen of the mobile device. Hence, content that is printed on one page in the
manual is split into a sequence of screens. This split is a critical point as users can
no longer scan through all steps on a page to recognize the current state of the
device. Future implementations should aim at supporting this type of scanning,
e.g. by employing similar methods as in mobile image browsing. Furthermore,
GuideMe could combine benefits of printed paper and digital augmented objects
by combining both media as explored by Grasset et al. [6].
A user manual does not only contain a series of steps, but aims at teaching
the user interface of the device. Hence, task descriptions contain additional in-
formation such as warnings on possible errors, the impact on other functionality
and a method to revert or abort the current task. Video tutorials often select
a simple subset of steps and neglect this additional information for the sake of
a shorter and clearer presentation. However, this background information is es-
sential to actually create an understanding of a new device. Albeit we included
all this information in the video, the video was praised for the clarity of the
explained steps. However, participants criticized the perceived “time pressure”.
Participants did not use the available controls to stop the video. Splitting the
instructions into a linked set of small videos may eliminate this behavior. Such
manuals can be realized using the current functionality of GuideMe.
Both studies were influenced by the weight and size of the used tablet. The
weight increased the physical demand. The size of the tablet was perceived as a
barrier between user and the oven. Further evaluations using smaller devices can
research the resulting trade-off between screen size and weight for this application
domain.

7.2 GuideMe Prototype

The marker-based object recognition did highlight the challenges induced by the
properties of the appliance and the employed tablet. The camera on the tablet
 GuideMe: A Mobile Augmented Reality System 165

is located on the top of the tablet thus creating a shift of the image that can be
confusing. The camera is expected to be in the middle of the tablet. Completely
different devices like wearable displays5 or personal projection devices [11] would
better align movements of the device and the created image.
The ovens’ user interface in our study was too wide for the used tablet camera.
Ovens were identified by a single marker in the middle of the user interface that
had to be within the cameras’ field of sight all the time. On the Stoves oven for
example, the knob to change the temperature is more than 20 cm away from the
marker. Participants often held the tablet too close to the user interface to see
marker and knob in the magic lens. Multiple markers or a new kind of distributed
markers could provide a solution. Another option would be to decorate the user
interface in a manner to enable marker-less object recognition. The participants
themselves came up with a third simple solution. They missed a button to freeze
the current state of the screen. This approach would combine augmented reality
and image based user manuals.
UMML enables GuideMe to be flexible and abstracts from the required pro-
gramming. Technical documentation experts can design augmented reality user
manuals by writing UMML. However, we did not evaluate the modeling capa-
bilities of UMML in this paper.

7.3 Digital Manuals for Mobile Devices

Future user manuals should emphasize the playful exploration of the function-
ality provided by mobile devices. The two conducted studies have provided first
evidence that video and AR-based manuals can support this change. Users felt
more secure using these manuals. Moreover, their body language indicated a
positive attitude towards this new medium. This preference was confirmed in
the concluding questionnaire.
However, the initial positive attitude may change, if the design of such user
manuals does not account for the specific properties of the mobile devices, e.g.
the position of the camera. Designers and technical documentation experts are
needed to facilitate this change. Until now technology has dominated the de-
velopment. UMML is a first approach to simplify development and include the
experts. The notion of computational composites as suggested by Vallgårda [19]
can help to bridge the gap between designers and technology driven developers.
In AR-based manuals the appliance becomes part of the user interface. For
instance, the color of superimposed graphics has to to blend into the camera
picture of the device. The design of the appliance is linked to the design of the
manual. Properties of both components influence each other and create the final
design. Therefore, GuideMe aims at a maximizing the flexibility of the design.
A further integration on the technical level was proposed by Liu et al. [13].
They compared AR manual implementations and concluded that “the conven-
tional AR approach performs almost as well as pictures” but feedback from the
device would “significantly improve task performance and user experience”. A
5
Project Glass http://plus.google.com/+projectglass/
166 L. Müller, I. Aslan, and L. Krüßen

direct feedback from the home appliance to GuideMe would prevent possible
discrepancies between the actual state of a device and the state that GuideMe
is assuming and thus significantly lowering error rates and frustration. Although
standardized methods for device feedback would be beneficial to applications
like GuideMe such a standardization of feedback mechanisms across appliances
and manufacturers is not foreseeable.

8 Conclusion
We described GuideMe a mobile design for AR-based user manuals, discussed
the underlying combination of mobile media and physical user interfaces and
presented the results of two small scale studies. Both studies compared GuideMe
manuals to current alternatives, the printed manual and mobile video tutorials,
with young users and real home appliances. The printed manual resulted in the
fastest task completion times and the lowest error rates. However, these classic
usability metrics were not decisive arguments for the participants; they preferred
video tutorials and GuideMe on the tablet. Both are interactive formats that
allow users to imitate actions. In result participants enjoyed using these manuals
on the tablet.
Augmented Reality and video provide starting points to new user manuals
that are fun to use. Both manual types can be designed, combined and tested
using UMML and GuideMe. A mobile application for consumers to adapt and
create simple UMML manuals by drag and drop is currently in development.
Applications like GuideMe and a simple editor would open the explored design
space for a broader audience. The current progress in mobile media has the
potential to transform the user manual but requires more research on the manual
design.

Acknowledgments. This work has been co-funded by the German Federal

Ministry of Education and Research in the CHICO project under grant number
16SV6186.

References
1. Bier, E., Stone, M., Pier, K., Buxton, W., DeRose, T.: Toolglass and magic lenses:
the see-through interface. In: SIGGRAPH 1993, pp. 73–80. ACM (1993)
2. Bødker, S.: When second wave hci meets third wave challenges. In: Proceedings of
the 4th Nordic Conference on Human-computer Interaction: Changing Roles, pp.
1–8. ACM (2006)
3. Fallman, D.: The new good: exploring the potential of philosophy of technology
to contribute to human-computer interaction. In: Proceedings of the 2011 Annual
Conference on Human Factors in Computing Systems, pp. 1051–1060. ACM (2011)
4. Feiner, S., Macintyre, B., Seligmann, D.: Knowledge-based augmented reality.
Communications of the ACM 36(7), 53–62 (1993)
5. Geiger, C., Kleinnjohann, B., Reimann, C., Stichling, D.: Mobile AR4ALL. In:
ISAR 2001, pp. 181–182. IEEE Comput. Soc. (2001)
 GuideMe: A Mobile Augmented Reality System 167

6. Grasset, R., Dunser, A., Billinghurst, M.: The design of a mixed-reality book: Is
it still a real book? In: 7th IEEE/ACM International Symposium on Mixed and
Augmented Reality, ISMAR 2008, pp. 99–102. IEEE (2008)
7. Hakkarainen, M., Woodward, C., Billinghurst, M.: Augmented assembly using a
mobile phone. In: Proceedings of the 7th IEEE/ACM International Symposium
on Mixed and Augmented Reality, ISMAR 2008, pp. 167–168. IEEE Computer
Society, Washington, DC (2008)
8. Hart, S., Staveland, L.: Development of nasa-tlx (task load index): Results of em-
pirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.) Human
Mental Workload, pp. 139–183. Elsevier, Amsterdam (1988)
9. Henderson, S., Feiner, S.: Exploring the benefits of augmented reality documen-
tation for maintenance and repair. IEEE Transactions on Visualization and Com-
puter Graphics 17(10), 1355–1368 (2011)
10. IEC International Electrotechnical Commission: IEC 62079:2001 - Preparation of
instructions. Structuring, content and presentation (July 2001)
11. Kawsar, F., Rukzio, E., Kortuem, G.: An explorative comparison of magic lens and
personal projection for interacting with smart objects. In: Mobile HCI 2010, pp.
157–160. ACM (2010)
12. Ledermann, F., Schmalstieg, D.: April: a high-level framework for creating aug-
mented reality presentations. In: Proceedings of the Virtual Reality, VR 2005, pp.
187–194. IEEE (March 2005)
13. Liu, C., Huot, S., Diehl, J., Mackay, W.E., Beaudouin-lafon, M.: Evaluating the
Benefits of Real-time Feedback in Mobile Augmented Reality with Hand-held De-
vices. In: CHI 2012, vol. 2012, pp. 2973–2976 (2012)
14. Redström, J.: Designing Everyday Computational Things. Ph.D. thesis, Göteborg
(2001)
15. Rohs, M.: Marker-based embodied interaction for handheld augmented reality
games. Journal of Virtual Reality and Broadcasting 4(5) (2007)
16. de Sa, M., Churchill, E., Isbister, K.: Mobile augmented reality: design issues and
opportunities. In: Proceedings of the 13th International Conference on Human
Computer Interaction with Mobile Devices and Services, pp. 749–752. ACM (2011)
17. Takacs, G., Chandrasekhar, V., Gelfand, N., Xiong, Y., Chen, W.C., Bismpigiannis,
T., Grzeszczuk, R., Pulli, K., Girod, B.: Outdoors augmented reality on mobile
phone using loxel-based visual feature organization. In: Proceedings of the 1st
ACM International Conference on Multimedia Information Retrieval, MIR 2008,
pp. 427–434. ACM, New York (2008)
18. TCeurope (Organisation) and European Commission: SecureDoc Guideline for Us-
able and Safe Operating Manuals for Consumer Goods: Fourth Brussels Colloquium
for User-friendly Product Information. TCeurope (March 29, 2004)
19. Vallgårda, A., Redström, J.: Computational composites. In: Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems, CHI 2007, pp.
513–522. ACM, New York (2007)
20. Viola, I., Meister, E.G.: Smart visibility in visualization. In: Computational Aes-
thetics in Graphics, Visualization, pp. 209–216 (2005)