Interaction, Usability and Aesthetics:

What Influences Users’ Preferences?

Antonella De Angeli, Alistair Sutcliffe & Jan Hartmann
Centre for HCI Design, School of Informatics, University of Manchester
P.O. Box 88, Manchester M60 1QD, UK
{antonella.de-angeli} {a.g.sutcliffe}@manchester.ac.uk; [email protected]

ABSTRACT success [2]. Not only is beauty an important quality of a

In this paper we describe an evaluation of two websites product but its effect seems to transcend the object and
with the same content but different interface styles influence other judgements, in what is known as the halo
(traditional menu-based and interactive metaphors). A effect. For example, not only do people associate positive
formative usability evaluation was carried out with heuristic personality traits with attractive individuals [4], but they
assessment of aesthetics, and questionnaire assessment of also tend to attribute more positive dimensions to
aesthetics, content, information quality, usability and post- individuals in the company of a beautiful friend [8].
test memory. The study revealed that perception of
information quality is affected by the interaction style Consistent with the halo effect, initial research suggested a
implemented in the interface, in a manner resembling the correlation between the aesthetic quality of an interface, its
halo effect in person perception. Implications for website perceived usability, and the overall user satisfaction with
design and evaluation are discussed. that system [7], [20]. More recently, however, these
findings were contradicted by experimental studies which
Author Keywords found no or only a weak correlation between aesthetic
Design Evaluation, Engagement, Interaction, Metaphors, quality of MP3-player skins and pragmatic attributes of the
Aesthetics product, thus suggesting that aesthetic appreciation may not
ACM Classification Keywords be strongly affected by experience [5]. This inconsistency
H5.2. Information interfaces and presentation (e.g., HCI): indicates the need for a better conceptualisation of what
User Interfaces. constitutes the “user experience” and in particular what is
beauty in interaction [13]. Lavie and Tracktinsky [6]
INTRODUCTION proposed a model of website aesthetics which differentiated
Since its first beginning HCI has been concerned with the between classical aesthetics, referring to traditional
evaluation of interactive systems, and considerations of aesthetic notions emphasising orderly and clear design, and
usability have had a great impact on the way interactive expressive aesthetics, which the authors associate with the
systems are designed and developed. With the recent shift design’s creativity and originality. However, with a few
from a functional vision (computers as tools for cognition) exceptions [18], few studies have been undertaken on how
towards an experiential vision (interactive systems as a different interaction styles and design features might
medium for emotions, sociability and pleasure) has come a influence aesthetics or users’ judgement of their interactive
realisation that usability may no longer be the only, or even experience.
the main, determinant of user satisfaction [3], [5], [20].
Increasing importance is now given to the interface “look Our initial research has suggested that the use of interactive
and feel”, its capability to engage the users in fulfilling metaphors contributes to users’ attitudes and rating of
interaction, and generating affective responses. website aesthetics, even when the usability of the system is
worse. This effect can be explained by affective
In his recent book, Norman [12] claims that aesthetic design interpretation of user judgement [11], in that an interactive
can be even more influential in affecting user preferences metaphor can induce curiosity and pleasure despite being
than traditional operational usability. This claim reflects more difficult to use. This paper builds upon our past
well-established knowledge in marketing, product design, research, attempting to define a model of user experience.
and even social psychology: namely, that beauty matters. We report an evaluation of two websites with identical
The aesthetic quality of a product influences consumers’ content (in terms of quality and quantity of information
attitudes, and is a major determinant of its marketplace provided) but very different interaction styles: one more
traditional and menu-based, the other more interactive,
Permission to make digital or hard copies of all or part of this work for exploiting metaphors and humour effects. In the study, we
personal or classroom use is granted without fee provided that copies investigated the relationships between content, presentation,
are not made or distributed for profit or commercial advantage and that usability and memory, their relative importance to the
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists, user’s preferences, and the effect of different interaction
requires prior specific permission and/or a fee. styles.
DIS 2006, June 26–28, 2006, University Park, Pennsylvania, USA.
Copyright 2006 ACM 1-59593-341-7/06/0006...$5.00.

271
METHOD reported a user-based evaluation run on small samples of
Two live websites (http://www.renaissanceconnection.org) pupils, concentrating on self-reports and observations to
were selected and adapted for the experiment. They capture qualitative data. In contrast, in our study we report
presented exactly the same information on Renaissance an expert-based comparative evaluation, which followed a
culture and history but with different user interface styles. much more structured approach, collecting and statistically
One was a traditional menu-based style; the other exploited comparing subjective and objective data.
animated metaphors and more aesthetic features (metaphor-
Websites architecture
based). The metaphor-based website adopted a playful and
The websites are structured in five sections, three of which
engaging interaction style, with animated picture characters
were the object of our evaluation. The Innovation 1400-
providing information by speech bubbles, and generating
2020 section was the most different as regards interaction
other pictures and information from inside their head. The
style (Figure 2) and was used to perform four of the eight
menu-based style adopted a more serious interaction style,
information retrieval tasks. In the metaphor-based style, it
displaying a static picture instead of the animated head, and
opened with a Flash-based introduction where a head
with no humorous effects. Examples are shown in Figure 1.
provided subject matter and functional help. The
introduction was controlled by the user and could be
skipped. The information was then presented following the
metaphor of a telescope: themes were selected by moving a
sequence of pictures at the bottom of the telescope (leading
to a change of the picture displayed in the lens) and the year
was selected by moving a slide on the lens. This last
selection led to an update of the text displayed in the video.
An example of the information presented on the Patrons
and Lifestyles theme in 2000 is shown in Figure 2 (lower).
In the menu-based style, the Innovation section begins with
a list of all the themes and after selection the user is taken
to a web page with all the information listed in
chronological order (Figure 2, upper).

Figure 1. Menu-based (upper), and metaphor-based (lower)

interfaces for the Renaissance Connection website.
The original websites were designed by Eduweb for the
Allentown Art Museum in Pennsylvania for use by middle
school students (10-14 years) and their teachers. These
websites were subjected to a comparative usability
evaluation, aimed at unveiling relative strengths and
weaknesses of Flash and HTML [15]. The study addressed
informal behavioural observations and the comments of
middle school and college students. The major difference
with the current study is that the HTML version (which we
call the menu-based style) was a static equivalent of the
Flash version (metaphor-based) and exploited the same
level of playfulness without any animation effect. In the
current study, we removed most of the fun features from the
menu-based interface, and replaced them with pictures of a Figure 2. The Innovations 1400-2020 section in the menu-
more serious content. Other important methodological based (upper) and metaphor-based (lower) interface
differences also need to be stressed. The original study

272
The Be a Patron of the Arts section proposed an interactive “educational”. The second measure builds on the Bernier
game requiring the user to commission a painting to Instructional Design Scale (BIDS), a psychometric
‘glorify God, your city and yourself, and your family’. To instrument developed to assess the quality of printed
play the game, the user had to make selections of education material. For our study we selected and adapted
alternatives in a form of interactive narratives. Different nine items, directly related to clarity of learning objectives,
choices led to different outcomes, with only one correct level of detail, quality of content, learning potential,
path (one leading to the achievement of the desired delivery of up-to-date information. Engagement was
painting). In the metaphor-based interface, the game measured by a short scale, previously used in [18].
consisted of vignettes, where comic-like characters talked
to the users with timed speech bubbles in a vignette. In the Usability • Performance analysis
menu-based interface, the same dialogue appeared as text • Self-report and severity rating of
associated to a character picture in a sequential fashion. usability problems (1=minor problem;
5=major problem)
One experimental task addressed this section.
• 5-item usability scale on a 7-point
The final section covered by the experiment was the The Likert scale [6]
artist’s life. It reported basic knowledge on famous Memorabilty • Free recall memory test and memory
Renaissance painters. In the metaphor-based condition, this rating (1=very negative, 5=very
information was structured on individual web pages positive)
through which the user navigated clicking on the icon of a Aesthetics • Heuristics for attractiveness [16], [17]
hand and/or selecting a menu at the top of the screen. In the • 10-item perceived website aesthetic
menu-based interface, the same information was displayed scale on a 7-point Likert scale [6]
on a web page through which the user had to scroll. Information quality • Scale A: six items on a 7-point Likert
scale adapted from [6] as service
In an informal assessment of the aesthetic/user engagement quality measure
differences between the two websites, we hypothesised that
• Scale B: nine items on a 7-point Likert
the variables that might influence users’ overall evaluation scale extracted and adapted from the
extended to the information quality assessment. Bernier Instructional Design Scale
(BIDS) [1]
Evaluation instruments
The websites were evaluated for usability, memorability of Engagement • 3 items on a 7-point Likert scale
(engaging, entertaining, pleasant)
content and interface features, aesthetics, information
quality, engagement and overall preferences. Several Overall preference • Dichotomous choice on the post-test
techniques and instruments were used to gather evidence on questionnaire, including different
scenarios
these dimensions (Table 1).
Usability was assessed by objective measures (performance Table 1. Summary of the instruments and techniques used
during the evaluation.
and self-report of usability problems) and subjective
measures (questionnaire). Aesthetics was assessed by two Overall preferences were assessed directly by a post-
distinct yet complementary approaches. A questionnaire experimental questionnaire, asking users to express their
collected holistic impressions on two apparently separate choice on a dichotomous question, for overall preference
dimensions, namely classical aesthetics and expressive and individual dimensions, in a number of different
aesthetics. The first dimension includes items such as scenarios. Indirect measures of preference were obtained by
pleasant, clear, clean, symmetrical and aesthetic design. questionnaire comparisons.
The dimension of expressive aesthetics is characterised by
Participants
qualities that capture the user’s perception of creativity and
Twenty-eight undergraduate and postgraduate students (23
originality of the site’s design. Relevant items in this
male and 5 female) from the School of Informatics,
dimension are “creative”, “fascinating”, “original”,
University of Manchester participated in the experiment.
“sophisticated design”, “use of special effects”. In addition,
They all had a basic knowledge of HCI, usability evaluation
we asked participants to evaluate the site applying the
techniques and the aesthetic heuristics used in the
heuristics for attractiveness, proposed by Sutcliffe [16].
experiment, from an advanced HCI course they had
These address the quality of individual design features
recently attended. All of the participants were expert web
linked to the perception of aesthetics (Table 3).
users but none of them had any prior knowledge of the two
Information quality addressed the educational impact of the websites.
website. Two measures were used. The first one builds Procedure
upon Lavie and Tractinksy’s [6] service quality construct Data were collected in a group setting, with each participant
(measured by the items “makes no mistakes”, “provides working individually for almost 3 hours. At the beginning
reliable information”, “reliable”), and localised to the of the experimental session, participants received written
evaluation of educational software by adding the following and verbal instructions followed by a brief pre-test
items: “provides enough details”, “informative”,

273
questionnaire recording personal data. Then, they were Several factors related to interaction design can be held
assigned to one of the two websites and had to perform responsible for the poorer performance of the metaphor-
eight information-retrieval tasks (e.g. finding events which based interface. In this condition, the information was
occurred in specific years, finding artists’ names and provided using an array of design elements, including
painting dates, engaging in on-line games, and reporting on speech bubbles and small pop-up windows, which could
specific picture details). Answers were written in an easily go unnoticed. Furthermore, information retrieval
experiment booklet. Optimal task performance required often required physical manipulation of interface objects.
visiting three sections of the websites in both conditions For instance, in the telescope metaphor, the year had to be
(Innovations, Artist’s Life, Be a patron of the arts). selected by moving a slider, and paintings needed to be
During task execution, participants recorded the usability clicked to retrieve details. These design solutions may have
errors they encountered and rated their severity. Once they disrupted participants’ performance by increasing the
had completed the tasks, they performed a free recall cognitive and physical load necessary for information
memory test, listing ten facts/ issues they could remember retrieval. On the contrary, the linear interaction style
about the website, and rating the quality of these memories supported by the menu-based interface was better suited to
on a five-point scale (as favourable, neutral or adverse). the experimental task.
Then, participants briefly revisited the site and completed Consistently, the usability evaluation clearly favoured the
the attractiveness heuristics and the questionnaires menu-based interface. On average, participants reported
addressing the remaining measures. After a short break, the significantly more problems when evaluating the metaphor-
same evaluation procedure was repeated with the other based website (mean= 3.68; standard error = .38) than when
website and a new set of tasks. Experimental tasks were evaluating the menu-based site (mean = 1.68; standard error
designed to be very similar in terms of cognitive workload = .37), t(27) = - 4.49 p < .001. The problems associated with
and navigation behaviour, but they addressed different the metaphor-based site were also scored as more severe
topics to minimise learning effects (e.g. they had to retrieve (mean/problem = 3.85 on a 5-point scale; standard error =
information related to different themes, or to commission a .12) than those associated with the menu-based site
painting for different targets). Finally, participants filled in (mean/problem = 2.95; standard error = .20). The difference
a post-test questionnaire, which captured their overall is significant (t(132) = -3.94, p < .001).
preference and the reasons behind it. They were also invited Usability problems were clustered into four categories
to select the ideal interaction style for different target according to their cause. Poor menu/navigation included
populations and environments (e.g. children aged 7-10 at critical incidents [10], i.e. usability problems which caused
home; children aged 7-10 at school). operational difficulties in finding the desired information.
Interaction style (2) was manipulated within-subjects, so Poor graphical design covered adverse comments on
that each participant evaluated both the menu-based and the aesthetic aspects of the site, including text layout and fonts,
metaphor-based interfaces. Evaluation order and tasks were as well as animations and pictures. Poor information
counter-balanced among experimental conditions. reflected negative comments on clarity and completeness of
the information architecture and content. The category
RESULTS
The results are summarised in six main sections according other included not understandable statements, in addition to
to the type of data analysed: the usability evaluation, comments reflecting unpredictable system behaviour and
aesthetic appreciation, information quality, engagement, the functionality failures. Table 2 reports frequencies and
memorability test, and finally the participants’ overall percentages of these categories in the two experimental
preference; a model to predict overall preference is conditions, along with their severity rating.
suggested. All scales showed high reliability (Cronbach Menu-based Metaphor-based
alpha > .78), therefore comparisons are based on the
Freq % Severity Freq % Severity
average of individual items.
Poor 21 45 2.95 47 47 3.87
Usability evaluation menu/navigation
In each condition, participants were given eight questions to
Poor graphical 13 27 2.93 38 38 3.97
answer, yielding a total of 224 tasks per condition. Overall, design
users were very accurate in information retrieval, with only
Poor information 7 15 2.50 9 9 3.67
3% of 448 tasks resulting in wrong information. Accuracy
was significantly affected by the interaction style of the Other 6 13 3.33 6 6 3.67
website. Most of the errors occurred with the metaphor- Total 47 100 2.94 100 100 3.87
based interface (N=10), whereas only 3 errors were
observed in the menu-based interface, a significant Table 2. Statistics of usability problems classified by cause in
difference according to a Wilcoxon Signed Rank test Z = - the two experimental conditions.
2.35 (N = 12) p < .05. Usability problems were homogeneously distributed in the
four categories independent of experimental conditions

274
(χ2(3) = 3.84, p = n.s.). The most common cause of usability Comparison of the usability index (average of individual
problems was poor menu navigation, followed by poor items) by a paired-samples t-test yielded significant results
graphical design. Fewer problems addressed the quality of (t(24) = 3.78 p < .01). Usability of the metaphor-based
the information. Despite this apparent similarity, the details interface was evaluated worse than the menu-based design.
of the usability problems were very different in the Consistently, 82% of the sample choosed the menu-based
experimental conditions. This difference was reflected in style as the most usable in the post-test questionnaire
the severity ratings, which were higher in the metaphor- (binomial test p < .001).
based interaction than in the menu-based one (F(5, 125) =
Aesthetic appreciation
15.11, p <.001). A questionnaire collected participants’ perception of
Menu/navigation issues in the metaphor-based conditions classical and expressive aesthetics. A heuristic evaluation
mainly occurred in the innovation section and the telescope reported a more detailed insight into the evaluation of
metaphor (N=20). Participants complained about the need specific design features of the websites.
for direct manipulation of interface objects, and for
scrolling down the information displayed in the small video Two indexes representing perception of classical and
(the backward arrow was always present, independent of expressive aesthetics were computed by averaging scores to
the text length). In contrast, the menu/navigation issues in relevant items. These indexes were entered as dependent
the menu-based interface were more general and variables in a repeated-measure ANOVA with aesthetic
differentiated, such as broken or missing links (N=8) and to dimension (2) and interaction style (2) as within-subjects
the need for scrolling (N=6). factors. Both the main effects and their interaction were
significant, namely aesthetic dimension F(1, 24) = 4.13, p
For poor information design, most participants (N=16) in =.05; interaction style F(1, 24) = 5.59, p < .05; 2-way
the metaphor-based condition complained about the interaction F(1, 24) = 50.34, p < .001. These results indicate
introductory flash sequence which was displayed every that the effect of the interaction style on aesthetics is
time they accessed specific sections allowing only limited modulated by the dimension considered Figure 4.
control. Another frequent complaint addressed text
readability (N=11) which, in the telescope section, was poor 7
classical
because of the need to fit the information into a small expressive
screen (Figure 2). In the menu-based condition, the most 6

specific complaint (N=3) was that the images were too big 5
and inconsistent. Other comments addressed general design
average score

(N=7), described as unattractive, and too simplistic. Typical 4

usability problems due to poor information concerned
information architecture in the metaphor-based interface 3

(N= 6) and issues related to misleading titles or lack of

2
captions (N=3) in the menu-based style.
The subjective evaluation of usability closely mirrored the 1

objective analysis of performance and usability problems. metaphor-based menu-based

website
Average scores and standard errors of the five usability
items in the Lavie and Tractinsky’s scale are illustrated in Figure 4: Average scores for the two aesthetic factors as a
Figure 3. function of experimental conditions
7 The metaphor-based interface was preferred on the
menu-based
metaphor-based expressive aesthetic continuum (simple effects, p < .001);
6
whereas the menu-based interface was slightly preferred on
5
the classical aesthetic dimension (simple effects, p = 06).
average score

4
A more detailed analysis addressed individual design
features associated with the website look and feel via a
3 heuristic evaluation. For each of the 16 heuristics, the user
had to quantify their relative importance in relation to the
2
overall quality of the website (1 = not important, 7 = very
important) and then evaluate the website based on that
1
clear easy to use easy to has easy convenient heuristic. Table 3 reports descriptive statistics for the 11
navigate orientation
heuristics which had an average importance value
significantly superior to 4 (mid-point of the scale) as
Figure 3: Usability ratings as a function of experimental assessed by a one-sample t-test. The importance index was
condition computed by averaging the scores given in the evaluation of
the menu- and metaphor-based websites. Discarded

275
heuristics addressed use of personality, logos, video, depth question, 75% of the sample indicated the menu-based style
of field, and sound, which were not considered important. as their favourite interface for content.
The remaining heuristics are listed in Table 3, in order of
Engagement
importance. Interaction style significantly affected the perceived level of
A series of repeated measures t-test were run to assess inter- engagement with the website (t(25) = -2.95 p < .01). The
sites differences. Significant results emerged only as metaphor-based site was perceived as more engaging
regards to the heuristics addressing layout structure, and (mean= 4.80; std. error = .27) than the menu-based interface
accessibility of information, which were considered the 2 (mean = 3.78; std. error = .24). Consistently, in the post-test
most important design elements. In both cases the menu- questionnaire 19 users picked the metaphor based interface
based interaction style was favoured. Note that these two as the most engaging (binomial test p = .09).
heuristics addressed design elements at the border between Memory
usability and classical aesthetic. Overall, participants reported 368 items in the free recall
memory of the two websites. No effect of interaction style
HEURISTICS Importance Menu-based Metaphor-based
emerged on the number of items retrieved, but the memory
Mean SE Mean SE N/A Mean SE N/A valency of the menu-based interaction style was in general
Structured layout* 6.17 .20 3.75 .32 0 5.18 .27 0 more positive (mean = 4.57; std. error = .15) than that of the
Info. accessibility* 6.12 .23 3.21 .31 0 4.75 .31 0 metaphor-based style (mean = 3.98; std. error = .14), t(360) =
Consistent layout 5.82 .22 4.89 .34 3 5.39 .30 0
2.93 p < .01.
Style 5.70 .22 4.29 .31 0 4.79 .25 0 Recalled items were categorised into reports relating to the
Images 5.60 .21 4.89 .29 0 4.93 .33 0
content of the website (including reference to the subject-
matter and information architecture), to the user-interface
Colour 5.42 .25 5.00 .27 0 5.00 .31 0
(sub-divided into link and navigation, multimedia,
Interactivity 5.36 .31 4.54 .35 0 4.29 .33 4 labelling/feedback, graphical design), and others. A cross-
Matching expectn 5.22 .22 4.08 .32 4 4.73 .26 2 tabulation analysis (χ2(3) = 33.20, p < .001) indicated that
Use of space 5.07 .23 4.32 .32 0 4.33 .30 1 the interaction style affected the content of these memories
(Table 5). In particular, the menu-based style triggered
Arousal 4.87 .24 3.93 .29 0 3.74 .34 1
memory related to the content of the website (40% within
Identity 4.58 .27 4.50 .22 2 4.73 .24 2 condition); whereas the metaphor-based style induced
memory related to multimedia elements (38% within
Table 3. The 11 heuristics considered of importance in website
evaluation. Significant results are marked with an asterisk; SE condition). No difference in valency for content and
= Standard Error, N/A = not applicable multimedia emerged within the two websites. In contrast,
the metaphor based interaction style elicited more negative
When asked to indicate the most attractive interface in the memories on link and navigation, labelling and feedback,
post-test questionnaire, 22 users selected the metaphor- and graphical design than the menu-based style.
based interface (binomial test p < .01).
Information quality
Menu-based Metaphor-based
Overall, participants evaluated the information provided by
the menu-based system as more educational than the Freq Valency SE Freq Valency SE
information provided by the metaphor-based interface. The Content Subject-matter 37 4.38 .35 17 4.59 .39
effect consistently emerged in both scales analysed (scales Info Architecture 35 4.77 .39 18 4.56 .39
A and B; see Table 1).
Link and Navign 32 4.62 .34 24 2.96 .43
Menu-based Metaphor-based t-test Multimedia 32 4.16 .39 72 4.31 .22
Mean Std. Error Mean Std. Error UI
Labelling/feedback 9 4.89 .75 8 3.87 .54
Scale_A- 5.03 .16 4.31 .18 t(24) = 2.99 p < .01 Graphical design 31 4.93 .30 38 3.59 .30
Qual
Other 4 3.75 1.37 11 3.63 .59
Scale_B 4.87 .21 4.35 .20 t(23) = 11 p < .05
Total 180 4.57 .15 188 3.98 .14
BIDS
Table 5. Cross-tabulation analysis of memory content and user
Table 4. Results of the paired-sample t-test and descriptive
interaction style.
statistics of overall indexes.
Overall preference
Results of the paired-sample t-test and descriptive statistics Overall preference was tied, with 14 subjects favouring
of overall indexes (average of individual items) are given in
each interface. When asked to justify their decision,
Table 4. Consistently, in the post-test forced choice
participants who expressed their preference for the menu-
based interface used almost twice as many words (N=298)

276
 as those who chose the metaphor-based interface (N=145). exactly half of the sample preferred the metaphor-based
This seems to suggest that people choosing the less version and the other half the menu-based one.
engaging website felt the need to better justify their
decision. In doing so, they often (N=9) explicitly referred to Menu-based Metaphor-based
negative features of the Flash website. In other words, their Usability + -
preference appeared to be the result of a general dislike for Classical aesthetic = =
the metaphor-based version. In contrast, only two people
Expressive aesthetic - +
explicitly referred to negative features of the menu-based
website when stating their preference for the metaphor- Information quality + -
based site. Engagement - +

The reasons driving participants’ preferences were very Memory valency + =

different according to the selected website. The most Overall preference = =
common reason for preferring the metaphor-based interface
made explicit reference to a more engaging (N=8) and more Table 7. Summary of the differences between the two
interactive styles.
interactive (N=7) style. Only two participants declared it
was easier to use. On the other hand, all but one of the The correlations between the main dimensions for the two
participants who preferred the menu-based style made interaction styles are reported in Table 8. Correlation values
explicit reference to usability issues. for the metaphor-based styles are reported in the upper part
of the matrix, and menu values are in the lower part. The
Participants’ preferences of the website changed drastically
overall correlation trends demonstrate the inter-
according to the target population and the scenario of use.
relationships between the different evaluation dimensions.
A clear majority agreed that the metaphor-based style was
The most striking difference between the two interaction
better for children interacting at home (leisure time),
styles is the correlation between usability and expressive
whereas less agreement was found when the system was
aesthetics, which is highly significant in the menu-based
meant to be used in a classroom environment (formal
interaction styles but not in the metaphor-based style.
education). Similarly, the metaphor-based style was deemed
Furthermore, memory valency tends to correlate with more
inappropriate for more mature and knowledgeable target
dimensions in the menu-based interface than in the
populations (see Table 6).
metaphor based one.
Scenario of use Menu-based Metaphor-based Binomial test
% % p Metaphor measure combination
1 2 3 4 5 6
Children (7-10), school 20 71 .05
Children (7-10), home 14 86 .001 1. Memory valency * ** *

Children (10-14), schl 32 68 n.s 2. Usability ** ** *

Menu
Children (10-14), home 21 79 .01 measure 3. Classical aesthetics ** * * * **
combination
Teenagers 43 57 n.s 4. Expressive aesthetics ** * ** ** **

University students 85 15 .001 5. Information quality ** ** *

Arts experts 82 18 .001 6. Engagement ** ** ** ** *

Elderly 89 11 .001 Table 8. Correlation matrix of the evaluation measures. * = p

< .05; ** = p < .001.
Table 6. Participants’ preferences for different target
populations. In order to understand what factors predict overall
preference, a binary logistic regression was conducted,
A model of participants’ preference
The experimental results discussed so far are summarised in applying the forward stepwise method based on likelihood
Table 7 (+ denotes better style, - worse style, = no ratio. The analysis is similar to a linear regression but it is
difference between the styles). It appears that both styles better suited to a model where the dependent variable is
had unique strengths and weaknesses. dichotomous. It predicts whether an event will or will not
occur and identifies the variables useful in making that
The menu-based interface received more positive prediction. The dependent variable in our model is the
evaluation for usability and information quality (even overall preference of the user, as expressed by their choice
though the two sites had exactly the same content). It also in the post-test questionnaire (metaphor-based vs. menu-
tended to elicit more positive memory. On the other hand, based). We selected three main covariates for the model,
the metaphor-based interface was perceived as more corresponding to the evaluation dimensions which strongly
engaging and better in the expressive aesthetic dimension. differentiated the two interaction styles (usability,
Despite these clear differences, no clear winner emerged as expressive aesthetic and information quality). Engagement
was discarded as it is highly correlated to expressive

277
aesthetics in both interaction styles (r = .89 and r = .71) and Figure 5 reports average values for content and usability as
our sample is not large enough to reliably accommodate a function of preferred interaction style. People who
four factors. The model predictors corresponded to the preferred the menu-based style were much more severe in
within-subjects differences on the evaluation of the evaluating usability and content of the metaphor-based style
websites (e.g. usability_predictor=usability_menu-usability and more positive in evaluating these dimensions of their
_metaphor). Usability was found to be an important favourite style.
predictor of final evaluation (Nagelkerke R Square = .50),
Similarly, people who preferred the metaphor-based style
and together with expressive aesthetic it explained 88% of
were much more negative in evaluating both expressive
the final preferences (total Nagelkerke R Square = .77). The
aesthetics and engagement of the menu-based style (Figure
model appears to fit well the data (χ2(3) = 22.97, p < .001),
6).
and it becomes less reliable if information quality is added.
From the analysis one could conclude that differences in CONCLUSION
evaluation of usability and expressive aesthetic were good The evaluation has revealed that the websites differed along
predictors of the overall preference, whereas information several dimensions, as summarised in Table 7. The menu-
quality is not. To further investigate these results we ran a based interface had better usability; it elicited more positive
series of mixed model Anovas with interaction style (2) as memories, and was perceived as providing better content,
the within-subjects factor and preference (2) as the although the content was exactly the same in the two
between-subjects one. The four dimensions differentiating interaction styles. The metaphor-based interface was
the two styles (usability and information quality, on the one preferred on the expressive aesthetic dimension and rated as
hand, and expressive aesthetics and engagement on the more engaging. Differences in evaluation of usability and
other) were entered as dependent variables. expressive aesthetics were good predictors of overall
preference, as participants appeared to discount negative
In all the analyses we found a significant preference for attributes in their favourite style. The overall preference
interaction style (p < .001) showing that participants tended was context dependent, and changed dramatically according
to discount negative attributes in the favourite style. to the target population and the scenario of use.
Examples of this effect are illustrated in Figure 5 and 6.
DISCUSSION
7
Our results show that the link between aesthetics and
preferred usability reported by Tractinsky’s studies [20], [6] is more
6
average scores

metaphor complex that than the strong claim that “what is usable is
5 preferred menu beautiful” [20]. In the websites we studied the metaphor-
4 based design was perceived as having better expressive
3 aesthetics using Lavie and Tractinsky’s scale, yet it had
2 worse objective and perceived usability. This is consistent
1 with our findings on other live websites [18], that
participants preferred an interface evaluated as more
menu

menu
metaphor

metaphor

attractive on the expressive aesthetics dimension (a concept

strongly related to engagement and fun) despite an
usability content acknowledged inferior usability.
Figure 5. Average values for content and usability as a Even though the metaphor design had superior expressive
function of preferred interaction style aesthetics there was evidence of poor graphical design in
the usability measures and no difference was found on the
7 interactivity heuristics, although the metaphor design
preferred
6 metaphor clearly incorporated more interactive effects. In contrast,
average score

5 preferred menu the metaphor design was rated better on engagement and
4 was preferred when the framing question pointed to
3 younger users and more playful applications.
2 Our explanation of these apparent contradictions in users’
1 judgement is that levels of attitudes are being formed from
menu

menu
metaphor

metaphor

the experience. The more general attitude level reflects the

overall design concept, in the metaphor case one of
playfulness and engaging interaction. More detailed
expressive aeshetic engagement attitudes are based more directly on experience and design
features, reflected in the adverse graphical design, usability
Figure 6. Average values for expressive aesthetics and and classical aesthetics ratings for the metaphor design.
engagement as a function of preferred interaction style More general attitudes seem to have a halo effect in
overriding the adverse influences of more detailed attitudes,

278
which supports Tractinsky’s findings [19, 20] on aesthetic features. In our future work we intend to extend and refine
preferences in mobile phone personalisation and that more the evaluation approach we have developed to further
aesthetic ATM designs are also perceived to be more understand the relative strengths of different influences on
usable. In Tractinsky’s experiments the usability effect was users’ overall quality judgement, and investigate the
small (mainly response times), whereas our findings interactions between them.
demonstrate the strength of the effect.
REFERENCES
The metaphor site experienced severe problems and much 1. Bernier, M. J. Establishing the psychometric properties
worse usability ratings. The preference analysis provides of a scale for evaluating quality in printed educational
more evidence for attitudes, since users suppressed adverse materials. Patient Education and Counselling 29,3
judgements for their preferred sites. Users’ memory also (1996), 283-299
supports this interpretation; the metaphor site was 2. Bloch, P. Seeking the ideal form: Product design and
remembered for its multimedia appeal and for its problems, consumer response. Journal of Marketing 59 (1995)
while information content was remembered for the more 16-29.
serious menu design. While we interpret users’ judgements
in terms of attitude (i.e. valenced memory), one could also 3. De Angeli, A., Lynch, P., and Johnson G.I. Pleasure
view the levels as emotional reactions at the visceral versus efficiency in user interfaces: Towards an
(general) and reflective (detailed) level [12], although we involvement framework. In W. S. Green, & P. W.
had little evidence of affective reaction in our qualitative Jordan (Eds.), Pleasure with products: Beyond
data. usability London: Taylor & Francis, (2002), 97-111.
We argue that judgements of aesthetics and engaging 4. Dion, K., Berscheid, E., and Walster, E. What is
designs are highly contextually dependent [5, 9, 18] beautiful is good. Journal of Personality and Social
showing framing effects of the target user audience and Psychology 24 (1972), 285-290.
application type. When the context is less serious, aesthetics
5. Hassenzahl, M. The thing and I: understanding the
can have a strong halo effect; however, this may not
relationship between user and product In M. Blythe, C.
generalise to other more serious user/application contexts.
Overbeeke, A. Monk, and P.C. Wright (Eds),
The menu design in our study had better usability, was
Funology: From usability to enjoyment, (2003), 31-42.
preferred for serious usage contexts, and had better rating
for information quality, even though the content of both 6. Lavie, T., and Tractinsky, N. Assessing dimensions of
sites was equivalent. In this case it appears that usability perceived visual aesthetics of web sites. International
can also have a halo effect on information content. Journal of Human-Computer Studies 60, 3 (2004) 269-
298.
In their classic study, Dion, Berscheid and Walster [4]
asked subjects to choose which personality traits applied to 7. Lindegaard, G., and Dudek, C. What is this evasive
pictures of people, varying on the attractiveness beast we call user satisfaction? Interacting with
dimensions. Results showed that more positive traits were Computers 15, 3 (2003), 429-452.
ascribed to attractive individuals, as compared to less
8. Meiners, M.L., and Sheposh, J.P. Beauty or brains:
attractive ones. This halo effect was obtained consistently
Which image for your mate? Personality and Social
over a wide range of personal qualities and was also
Psychology 3 (1977), 262-265.
demonstrated in users’ perceptions of human images
displayed on computers [14]. An interesting question for 9. Monk, A. The product as a fixed effect fallacy.
further research is the relative strength of halo effects from Human-Computer Interaction 19, 4 (2004), 371-375.
different variables. From the evidence in this study we
10. Monk, A., Wright, P., Haber, J., and Davenport, L.
speculate that the concept of interaction (via metaphors,
Improving Your Human-Computer Interface: A
animations, pop-ups etc.) makes the “feel-good factor” that
Practical Technique. Prentice Hall, London, 1993..
might outweigh the “look” of visually aesthetic design.
11. Norman, D.A. Beauty, goodness and usability:
However, interaction may only “feel good” when the
Introduction to the special section. Human-Computer
context is less serious. We have made a small advance in
Interaction 19, 4 (2004), 311-318.
measures of aesthetics by introducing related phenomena of
interaction and engagement with a rigorous evaluation 12. Norman, D.A. Emotional Design: Why We Love (or
methodology. This exposed the conflicting opinions held by Hate) Everyday Things. Basic Books, New York, 2004.
our users and indicates that expressive aesthetics [6] have to
be assessed in general attitude, which in the metaphor site 13. Overbeeke, K.C., Djajadiningrat, J.P., Hummels, C.C.
conflicts with opinion on more detailed aspects of M., and Wensveen, S.A.G.. Beauty in usability: forget
aesthetics. The attractiveness heuristics we used in previous about ease of use.In W. S. Green, & P. W. Jordan
studies [18], [17] attempt to link more general impressions (Eds.), Pleasure with products: Beyond usability.
to assessment of specific aesthetics and interactive design London: Taylor & Francis, (2002), 97-111

279
14. Reeves, B., and Nass, C. The Media Equation: How Conference on System Sciences (Hawaii). IEEE
People Treat Computers, Television and New Media Computer Society Press, Los Alamitos, CA, 2002,
Like Real People and Places. CLSI/Cambridge 1838-1847.
University Press, Stanford CA/Cambridge, 1996
18. Sutcliffe, A.G., and De Angeli, A. Assessing
15. Schaller, D.T., Allison-Bunnell, S., Chow, A., Marty, interaction styles in web user. In Proceedings of
P., and Heo, M.,. To FlashFlash or not to FlashFlash? Human Computer Interaction - Interact 2005 (Rome)
Usability and user engagement of HTML vs. M. F. Costabile & F. Paterno, Eds. Springer-Verlag,
FlashFlash. In Proceedings Museums and the Web Berlin, 405-417.
2004 International Conference. Available on-line at
19. Tractinsky, N., and Zmiri, D. Exploring attributes of
http://www.eduweb.com/ToFlashFlashornot.pdf.
skins as potential antecedents of emotion in HCI. In P.
16. Sutcliffe, A.G. Heuristic evaluation of website Fishwick (Ed.), Aesthetic computing. Cambridge MA:
attractiveness and usability. In GIST technical report MIT Press, in press
G2001/1: Proceedings 8th Workshop on Design,
20. Tractinsky, N., Shoval-Katz, A., and Ikar, D. What is
Specification and Verification of Interactive Systems
beautiful is usable. Interacting with Computers 13. 2,
(Glasgow) Dept of Computer Science, University of
127-145, 2000.
Glasgow, Glasgow, 2001, 188-199.
17. Sutcliffe, A. G. Assessing the reliability of heuristic
evaluation for website attractiveness and usability. In
Proceedings HICSS-35 Hawaii International

280