Evaluation
Heuristic
Evaluation
Acknowledgment :Mahmood Jasim
© Mahyar with acknowledgements to
Joanna McGrenere and Dongwook
Heuristic
evaluation
What for:
Identifying (listing & describing) problems with existing
prototypes (any kind of interface); for any kind of user, new or
proficient
Cost-benefit:
Usability engineering activities are often expensive / slow; but
some can be quick / cheap, and still produce useful results
Focused less on what is “correct” than on what can be done
within
development constraints
2
Ultimate trade-off may be between doing no usability
How to preform a heuristic
evaluation
Design team supplies scenarios, prototype, list of
heuristics Need 3-5 evaluators: train in method if non-
expert
Single evaluator catches ~35% of the usability
problems
Five evaluators catch ~75%
Each evaluator independently produces list of justified, rated
problems by stepping through interface and applying heuristics
at each point
use heuristics list & severity rating convention
Team meets and compiles report that organizes and
categorizes problems 3
Individuals vs.
teams
Nielsen recommends individual evaluators inspect the interface
alone. evaluation is not influenced by others
independent and unbiased
greater variability in the kinds of errors found
no overhead required to organize group meetings
4
Step 1: briefing
session
Get your experts together
Brief them on what to do, goals of system,
etc. Discuss heuristics to be applied
May also want to provide experts
with: Some examples of tasks
Descriptions of user personas
Simple instructions/guidance
Especially if NOT a fully
functioning system
5
Step 2: individual
evaluation
At least two passes for each evaluator
First to get feel for flow and scope of
system Second to focus on specific
elements
Each evaluator produces list of problems
Explain problem w/reference to heuristic or other
info Be specific and list each problem separately
Assign rating of severity to each violation
6
Evaluation
form
7
Severity
ratings
Each violation is assigned a severity rating
Many other methods of doing this
Usually some combination
of: Frequency
Impact
Persistence (one time or
repeating)
Used to:
Help prioritize problems
Allocate resources to fix problems
Estimate need for more usability
efforts
8
Can be done independently by all
Example severity & extent
scales
one severity scale:
1 - don’t agree that this is a usability
problem
2 - cosmetic problem
3 - minor usability problem
4 - major usability problem; important to fix
4 - usability catastrophe; imperative to fix
one extent scale:
1 = single case
2 = several
places 3 =
widespread 9
Step 3: aggregating results & making
recommendations
Evaluation team meets and compares results
Through discussion and consensus, each violation is documented
and categorized in terms of severity, extent
Violations are ordered in terms of severity
E.g., Use an excel spreadsheet (which can be
sorted) Combined report goes back to design team
10
Heuristic
evaluation
Advantages
Contributes valuable insights from objective observers
The “minimalist” approach
General guidelines can correct for majority of usability
problems Easily remembered, easily applied with modest
effort Systematic technique that is reproducible with care
Discount usability engineering
Cheap and fast way to inspect a system
Can be done by usability experts and rapidly-trained end
users
11
Heuristic
evaluation
Problems:
Principles must be applied intuitively and carefully
Can’t be treated as a simple checklist
Heuristics can narrow focus on some problems at cost of others
Can reinforce existing design (not for coming up with radical
ideas) Doesn’t necessarily predict users/customers’ overall
satisfaction May not have same “credibility” as user test data
12
Combining Heuristic Evaluation and Cognitive
Walkthrough
HCI practitioners often use a combination of both that might
vary
based on what they’re trying to learn
e.g., While doing a walkthrough for a task, apply the
heuristics at each step, in addition to the CW questions.
13
One popular list of heuristics (Nielson,
‘93)
H1: Visibility of system status
H2: Match between system & the real
world H3: User control & freedom
H4: Consistency and standards
H5: Recognition rather than
recall H6: Error prevention
H7: Flexibility and efficiency of use
H8: Aesthetic and minimalist
design
H9: Help users recognize,
diagnose & recover
H10:Help and documentation 14
15
H1: Visibility of system
status
The system should always keep users informed about what is
going on, through (appropriate feedback within reasonable
time)
Example: consider system response time (user must wait)
0.1 sec: no special indicators needed, why?
1.0 sec: user starts to lose track of data, objects,
etc. 10 sec: max duration if user to stay focused
on action for longer delays, use percent-done
progress bars
searching database for matches
16
H1: Visibility of system
status
Keep users informed about what is going
on Appropriate visible feedback
What mode
am I in
now?
What did I How is the
select? system
interpretin
g my
actions? 17
H2: Match between system & real
world
The system should speak the users' language, with words, phrases
and concepts familiar to the user, rather than system-oriented
terms.
Follow real-world conventions, making information appear in a
natural and logical order.
e.g. withdrawing money from a bank machine
18
H2: Match between system & real
world
19
H3: User control &
freedom
Users often choose system functions by mistake and will need a
clearly marked "emergency exit" to leave the unwanted state
without having to go through an extended dialogue. Support undo
and redo.
How
do I
get
out of
this?
20
H3: User control &
freedom
“Exits” for mistaken choices, undo,
redo Don’t force down fixed paths
Strategies:
Cancel button (for dialogs waiting for user
input) Universal undo (can get back to
previous state) Interrupt (especially for lengthy
operations)
Quit (for leaving the program at any
time) Defaults (for restoring a property
21
sheet)
H4: Consistency &
standards
Consistency of effects = predictability
Same words, commands, actions should always have the same
effect in equivalent situations
Consistency of language and graphics
Same info/controls in same location on all screens/dialog boxes:
Same visual appearance across the system (e.g., widgets)
e.g., not different scroll bars in a single window system
Consistency of input
require consistent syntax across complete system
22
H4: Consistency &
standards
Consistency of language and graphics
Same info/controls in same location on all screens/dialog
boxes
23
H5: Error
prevention
Try to make errors impossible
Even better than good error messages is a careful design which
prevents a problem from occurring in the first place. Either
eliminate error-prone conditions or check for them and present
users with a confirmation option before they commit to the
action.
Modern widgets: only “legal commands” selected, or “legal data”
entered
24
H5: Errors we
make
Mistakes
Arise from conscious deliberations that lead to an error
instead of the correct solution
Slips
Unconscious behavior that gets misdirected en route to
satisfying goal
e.g., Drive to store, end up in the
office Shows up frequently in skilled
behavior
Usually due to inattention
Often arises from similarities of
actions 25
H5: Types of slips
Capture error
Frequent response overrides [unusual] intended one
Occurs when both actions have the same initial
sequence
Confirm saving of a file when you don’t want to
delete old
version
I can’t believe
I pressed
Yes...
26
H5: Types of slips
Description error
Intended action has too much in common with others possible
E.G. When right and wrong objects physically near each other
Pour juice into bowl instead of glass
Go jogging, come home, throw sweaty shirt in toilet instead of
laundry
Move file to trash instead of to folder
Loss of activation
Forgetting the goal while carrying out the action sequence
e.g., Start going to a room and forget why by the time you get there
Navigating menus/dialogs, can’t remember what you are looking
for
But continue action to remember (or go back to beginning)!
Mode errors
People do actions in one mode thinking they are in another
Refer to file that’s in a different directory
Look for commands / menu options that are not relevant
27
H6: recognition rather than
recall
Computers are good at remembering things, people aren’t!
Minimize the user's memory load by making objects, actions, and
options visible. The user should not have to remember
information from one part of the dialogue to another. Instructions
for use of the system should be visible or easily retrievable
whenever appropriate.
28
H7: flexibility and efficiency of
use
Experienced users should be able to perform frequently
used operations quickly
Strategies:
Keyboard and mouse
accelerators Abbreviations
Command completion
Menu shortcuts & function keys
Double clicking vs. Menu
selection
Type-ahead (entering input before the
system is ready for it)
Navigation jumps
Go to desired location directly, avoiding intermediate nodes
History systems
WWW: ~60% of pages are revisits 29
H8: aesthetic and minimalist
design
Dialogues should not contain information which is irrelevant or rarely
needed. Every extra unit of information in a dialogue competes with
the relevant units of information and diminishes their relative
visibility.
30
H9: help users recognize, diagnose, and recover from
errors
Error messages should be expressed in plain language (no
codes), precisely
Indicate the problem, and constructively suggest a solution.
31
H10: help and
documentation
Help is not a replacement for bad design!
Simple systems: walk up and use; minimal
instructions Most other systems:
Feature-rich
Some users want to become “expert” rather
than “casual” users
Intermediate users need reminding, plus a
learning path
Many users do not read manuals
Usually used when users are panicked & need help
NOW Need online documentation, good
search/lookup tools Online help can be specific to
current context
Sometimes used for quick 32
reference Syntax of actions,
H10: types of
help
Tutorial and/or getting started manuals
Short guides that people usually read when first encounter
system Encourage exploration and getting to know the
system Communicate conceptual material and essential
syntax
On-line “tours”, exercises, and demos
Demonstrate very basic principles through working examples
Reference manuals
Used mostly for detailed lookup by experts
Rarely introduces concepts
Thematically arranged
On-line hypertext
Search / find
Table of
contents Index
Cross-index
33
H10: types of help
(cont’d)
Reminders
Short reference cards
Expert user who just wants to check facts
Novice who wants to get overview of system’s
capabilities
Keyboard templates
Shortcuts/syntactic meanings of keys; recognition vs.
Recall; capabilities
Tooltips
Text over graphical items
indicates their meaning or
purpose
34
H10: types of
help
Context-sensitive help
System provides help on the interface component the user is
currently working with
Macintosh “balloon help”
Microsoft “what’s this” help
Wizards
Walks user through typical
tasks Reduces user autonomy
35
In-class
activity
Tripadvisor website
Booking a hotel for a tour destination
Optional
Reading
Ten Usability Heuristics for User Interface Design https://
www.nngroup.com/articles/ten-usability-heuristics/
Heuristic Evaluation
https://drive.google.com/file/d/1LFOZhjoufAcAXkBhYF3t7gUlDv8y
ZniN/view?usp=sharing
40
