Scribd
Upload
86 views

Evaluation in HCI

The document discusses different methods for evaluating user interfaces during development, including formative evaluation, summative evaluation, expert evaluation techniques like heuristic evaluation and cognitive walkthroughs, and user testing. Formative evaluation involves identifying usability issues early in development by testing prototypes, while summative evaluation assesses the final product. Expert evaluation relies on usability experts to assess a design against established usability principles, whereas user testing directly observes intended users completing typical tasks.

Uploaded by

Minahel Noor Fatima
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
86 views

Evaluation in HCI

The document discusses different methods for evaluating user interfaces during development, including formative evaluation, summative evaluation, expert evaluation techniques like heuristic evaluation and cognitive walkthroughs, and user testing. Formative evaluation involves identifying usability issues early in development by testing prototypes, while summative evaluation assesses the final product. Expert evaluation relies on usability experts to assess a design against established usability principles, whereas user testing directly observes intended users completing typical tasks.

Uploaded by

Minahel Noor Fatima
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 41

Evaluation in HCI

Evaluation
 Nowadays users expect much more than just a
usable system; they also look for a pleasing
and engaging experience. This means it is even
more important to carry out an evaluation.
 Imagine you have designed an app for teenagers to

share music, gossip, and photos. You have


prototyped your first design and implemented the
core functionality. How would you find out
whether it would appeal to them and if they will
use it? You could need to evaluate it – but how?
Evaluation
 In our human-centred approach to design, we
evaluate designs right from the sketches and then
as the project progresses we keep improving our
prototypes and evaluate those.
Evaluation
Types of Evaluation
 Basically two types evaluation:
◦ Formative Evaluation
◦ Summative Evaluation
Formative Evaluation
 Formative evaluations involve evaluating a
product or service during development. The goal
is to make improvements in the design prior to
release. This means identifying or diagnosing the
problems, making and implementing
recommendations, and then evaluating again.
Formative usability is always done before the
design has been finalized. In fact, the earlier the
formative evaluation, the more impact the
usability evaluations will have on the design.
Formative Evaluation
 Here are a few key questions you will be able
answer with a formative approach:
• What are the most significant usability issues
preventing users from accomplishing their goals or
resulting in inefficiencies?
• What aspects of the product work well for the
users? What do users find frustrating?
• What are the most common errors or mistakes
users are making?
Formative Evaluation
 Here are a few key questions you will be able
answer with a formative approach:
• What are the most significant usability issues
preventing users from accomplishing their goals or
resulting in inefficiencies?
• What aspects of the product work well for the
users? What do users find frustrating?
• What are the most common errors or mistakes
users are making?
Formative Evaluation
• Are improvements being made from one design
iteration to the next?
• What usability issues can you expect to remain
after the product is launched?
Summative Evaluation
 Summative usability is about evaluating the dish
after it comes out of the oven. The usability
specialist running a summative test is like a food
critic who evaluates a few sample dishes at a
restaurant or perhaps compares the same meal in
multiple restaurants. The goal of summative
usability is to evaluate how well a product or piece
of functionality meets its objectives. Summative
testing can also be about comparing several
products to each other.
Summative Evaluation
 Summative usability evaluations answer these
questions:
• What is the overall usability of our product?
• How does our product compare against the
competition?
• Have we made improvements from one product
release to the next?
. Summative testing can also be about comparing
several products to each other.
Usability Evaluation Methods
 Two main types
◦ Expert Evaluation
 Heuristic Evaluation
 Cognitive Walkthroughs
◦ User Testing
Expert Evaluation

 A simple, relatively quick and effective method of


evaluation is to get an interaction design, or
usability, expert to look at the system and try
using it
 this is no substitute for getting real people to

use your design


 expert evaluation is effective, particularly early in

the design process.


Heuristic Evaluation
Jakob Nielsen

Most usability engineering methods will


contribute substantially to the usability of an
interface …
…if they are actually used.
Heuristic Evaluation

 What is it?
Heuristic Evaluation

 What is it?
A discount usability engineering method
- Easy (can be taught in ½ day seminar)
- Fast (about a day for most evaluations)
- Cheap
Heuristic Evaluation

 How does it work?


Heuristic Evaluation

 How does it work?


◦ Evaluators use a checklist of basic usability
heuristics
◦ Evaluators go through an interface twice
 1st pass get a feel for the flow and general scope
 2nd pass refer to checklist of usability heuristics
and focus on individual elements
◦ The findings of evaluators are combined and
assessed
Nielsen: 10 Usability Heuristics
(based on extensive empirical testing)

• Visibility of system status • Recognition not recall


• Match between system • Flexibility and efficiency
and the real world • Aesthetic and minimalist
• User control and freedom design
Consistency • Help users diagnose and
• Error prevention recover from errors
• Help and documentation
Heuristic Evaluation

 One expert won’t do


 Need 3 - 5 evaluators
 Exact number

needed depends on
cost-benefit analysis
Heuristic Evaluation

 Debriefing session
◦ Evaluators rate the severity of all problems
identified
◦ Use a 0 – 4, absolute scale
 0 I don’t agree that this is a prob at all
 1 Cosmetic prob only
 2 Minor prob – low priority
 3 Major prob – high priority
 4 Usability catastrophe – imperative to fix
Heuristic Evaluation

 How does H.E. differ from User Testing?


Heuristic Evaluation

 How does H.E. differ from User Testing?


◦ Evaluators have checklists
◦ Evaluators are not the target users
◦ Evaluators decide on their own how they want to
proceed
◦ Observer can answer evaluators’ questions about
the domain or give hints for using the interface
◦ Evaluators say what they didn’t like and why;
observer doesn’t interpret evaluators’ actions
Heuristic Evaluation

 What are the shortcomings of H.E.?


Heuristic Evaluation

 What are the shortcomings of H.E.?


◦ Identifies usability problems without indicating
how they are to be fixed.
 “Ideas for appropriate redesigns have to appear
magically in the heads of designers on the
basis of their sheer creative powers.”
Example
A few problems with the Interface
 Red is used both for help messages and for error
messages (consistency, match real world)
 “There is a problem with your order”, but no

explanation or suggestions for resolution (error


reporting)
Continued….
 No “Continue shopping" button (user control &
freedom)
 Recalculate is very close to Clear Cart (error

prevention)
 “Check Out” button doesn’t look like other

buttons (consistency, both internal & external)


 Must recall and type in cart title to load

(recognition not recall, error prevention,


efficiency)
Evaluation
 Expert Evaluation
◦ Heuristic Evaluation
◦ Cognitive Walkthroughs
 User Testing
Cognitive Walkthrough
Cognitive Walkthrough
 The cognitive walkthrough was originally
designed as a tool to evaluate walk-up-and-use
systems like postal kiosks, automated teller
machines (ATMs), and interactive exhibits in
museums where users would have little or no
training. However, the cognitive walkthrough has
been employed successfully with more complex
systems like CAD software and software
development tools to understand the first
experience of new users.
Cognitive Walkthrough
 The cognitive walkthrough is a usability
evaluation method in which one or more
evaluators work through a series of tasks and ask a
set of questions from the perspective of the user.
 What is the user goal and why?
 Is the action obviously available?
 Does the action or label match the goal?
 Is there good feedback?
Cognitive Walkthrough
User Evaluations
User testing
 Expert evaluation is useful but there is no
comparison to using the actual users for
evaluation.
 There are many different types of user evaluations

for example
◦ Formative Evaluation
◦ Field Study
◦ Controlled Experiments
Usability Testing
 Usability testing involves collecting of data using a
combination of methods in a controlled setting,
for example, experiments that follow basic
experimental design, observation, interviews, and
questionnaires. Often, usability testing is
conducted in labs, although increasingly
interviews and other forms of data collection are
being done remotely via phone and digital
communication (for instance, through Skype or
Zoom) or in natural settings.
Usability Testing
 The primary goal of usability testing is to
determine whether an interface is usable by the
intended user population to carry out the tasks for
which it was designed. This involves investigating
how typical users perform on typical tasks. By
typical, we mean the users for whom the system
is designed (for example, teenagers, adults, and so
on) and the activities that it is designed for
them to be able to do (such as, purchasing the
latest fashions).
Usability Testing
 As users perform the tasks, they may be recorded
on video. Their interactions with the software may
also be recorded, usually by logging software. User
satisfaction questionnaires and interviews can
also be used to elicit users’ opinions about how
they liked the experience of using the system.
Usability Testing
Usability Testing

You might also like