Lecture Set 08

UID Evaluation and testing

2
3
4
5
6
7
 Why, what, where, and when
to evaluate
Iterative design and evaluation is a continuous process
that examines:
Why: To check users’ requirements and confirm that
users can utilize the product and that they like it
What: A conceptual model, early and subsequent
prototypes of a new system, more complete prototypes,
and a prototype to compare with competitors’ products
Where: In natural, in-the-wild, and laboratory settings
When: Throughout design; finished products can be
evaluated to collect information to inform new products
www.id-book.com 8
 Types of evaluation
• Controlled settings Controlled settings involving users
defines evaluation schemes where users activities are
controlled in some way with the end goal of testing a
hypothesis and gathering data regarding certain behaviors
• Natural settings involving users (for instance, online
communities and products that are used in public places)
▪ Often there is little or no control over what users do, especially
in in-the-wild settings

• Any setting that doesn’t directly involve users (for

example, consultants and researchers critique the
prototypes, and may predict and model how successful they
will be when used by users)

www.id-book.com 9
 Participants’ rights and
getting their consent
• Participants need to be told why the evaluation is
being done, what they will be asked to do and
informed about their rights
• Informed consent forms provide this information
and act as a contract between participants and
researchers
• The design of the informed consent form, the
evaluation process, data analysis, and data
storage methods are typically approved by a high
authority, such as the Institutional Review Board

www.id-book.com 10
 Things to consider when
interpreting data
Reliability: Does the method produce the
same results on separate occasions?
Validity: Does the method measure what it is
intended to measure?
Ecological validity: Does the environment of
the evaluation distort the results?
Biases: Are there biases that distort the
results?
Scope: How generalizable are the results?
www.id-book.com 11
 What is Usability?

• Usability Is a measure of how easy it is to

use something:

– How easy will the use of the software be for

a typical user to understand, learn, and
operate
– e.g., “user-friendliness”
 ISO Definition (9241-11) for Usability:

• “...the extent to which a product can be

used by specified users to achieve
specified goals with effectiveness,
efficiency and satisfaction in a specified
context of use.”
 What is Usability Testing?
• Usability testing is an effort to ascertain
the degree to which software has met the
usability needs of its intended user base
• Usability is difficult to evaluate and
measure
 What is Usability Testing?
• Usability Testing is an attempt to quantify
software user-friendliness according to:
1. Skill needed to learn the software
2. Time required to become efficient in using the
software
3. The measured increase in user productivity
4. A subjective assessment of a user’s attitude
toward using the software
 What is Usability Testing?
• The idea is to place users in front of some
version of the software under test and watch
how these users try to use it
• Can be expensive depending on what tasks you
have users try and on what you are watching for
• Not cost-effective if done too late in dev cycle
• Can uncover usability problems that design
guidelines and inspections may have missed
 Formal vs. Informal Testing
• Formal testing might entail building a
usability testing lab, equipping it with an
array of computers, audio-video
equipment, then staffing it with
psychologists, technicians, and human-
computer interaction specialists
 Formal vs. Informal Testing
• Informal approach: No fancy lab or expensive
equipment
• A simple test plan and task list are prepared,
notepad and pencil
• Participants are observed by an impartial
moderator
• The advantage is that informal testing looks at
what people actually do when they are doing
real work in an ordinary setting
Testing Basics: 5-Step Process
Step 1:
PLAN & PREP

Step 2:
SELECT PARTICIPANTS

Step 3:
CONDUCT TESTS

Step 4:
ANALYZE RESULTS

Step 5:
DEVELOP
RECOMENDATIONS
 Step 1: Plan & Prepare
Develop a test plan:
– For simple testing, prepare a list of questions
– For more detailed testing, have a script prepared

• Test Plan is important because you can

create a framework for your testing
process

• It allows you to communicate your goals

with the client & align expectations
 Step 1: Plan & Prepare
Create a Task List:
• Create lists of tasks or questions that a
typical user should be able to complete in
an hour
• Tasks should not be too simple nor too
difficult to accomplish
 Test-Plan
Quantitative performance measures
• Number of users successfully completing the task
• Time to complete task
• Time to complete task after time away from task
• Number and type of errors per task
• Number of errors per unit of time
• Number of navigations to online help or manuals
• Number of users making a particular type of error
Source: Wixon and Wilson, 1997

www.id-book.com 22
 Step 1: Plan & Prepare
• Informal usability tests only require a
pencil, paper, computer and browser
• Sometimes might use a video camera and
record each session
• Sometimes watched by development team
• Often usability tests can be conducted
within the user’s own environment
• Keep a printed version of the site for note
taking, and then watch and learn . . .
 Step 2: Find Participants
• A challenging aspect in usability testing is
finding suitable participants
• Important to gather on ongoing user base
• Test outside the team—testing with people
who are not associated with your company
or your Web site
 Step 2: Find Participants
Prior to conducting sessions with
participants:
• Test out your test plan beforehand with co-
workers or friends that have an acceptable
degree of Web user experience
• The first usability test should be fun,
informative, and low-stress
 Step 3: Conduct the Session
• Introduce yourself, explain the process to
the user
• User will be asked to perform a set of pre-
defined tasks (but do not tell them how
many or how long each will take)
• Make the user feel comfortable
• Speak only to give a new task and take
notes during the process
 Step 3: Conduct the Session
• Once the usability test session is over,
prepare a short summary of the session
and the results
• Outline specific problem areas and any
unexpected results
• Include any personal observations
 Step 3: Conduct the Session
• Collect basic data:
– Could the user complete the task?
– Did they need help?
– Track how much time it took them
– Note any stumbling blocks
(problems/obstacles)
– Overall observations, commentary
– Debrief the user, allow user to speak their
mind
– Prepare a post-test survey
 Step 3: Conduct the Session
Post-Test Survey:
• Prepare a survey online or in paper form for the
user to fill out after they have completed the
testing process
• Questions should include what the user thought
the Web site was like: graphics, logic, content,
navigation, and their overall satisfaction
• Gather data about overall effectiveness of the
site in relation to the goals of each task
 Step 4: Analyze Results
• Compile and summarize data
• Transfer handwritten notes to computer
• Write your reports while they are fresh in
your mind,
• Create a summary after testing is
complete, into a table that shows the
results of each test, include problem
areas, comments and user feedback from
the survey
 Step 4: Analyze Results
• Identify difficulties and problem areas
• Identify why there was difficulty or the
source of any problems (specific factors
such as navigation, text, graphics, etc.)
• Identify any specific task-oriented issues
 Step 5: Make Recommendations
• Compile and recommend
– Gather all your compiled information and
translate into recommendations
– Concentrate on high-level functionality first
– Then focus on recommendations for improved
user experience (what works and what does
not work well for users!)
– Determine the implementation plan
• Write up a formal report
 Usability testing conditions
• Usability lab or other controlled space
• Emphasis on:
▪ Selecting representative users
▪ Developing representative tasks
• 5-10 users typically selected
• Tasks usually around 30 minutes
• Test conditions are the same for every
participant
• Informed consent form explains procedures and
deals with ethical issues
www.id-book.com 33
 How many participants is
enough for user testing?
• The number is a practical issue
• Depends on:
▪ Schedule for testing
▪ Availability of participants
▪ Cost of running tests
• Typically 5-10 participants
• Some experts argue that testing should
continue until no new insights are gained
www.id-book.com 34
 1. Inspections: Heuristic
Evaluation and Walkthroughs
• Several kinds.

• Experts use their knowledge of users & technology to

review software usability.

• Expert critiques can be formal or informal.

a. Heuristic evaluation is a review guided by a set of
heuristics.

b. Walkthroughs involve stepping through a pre-planned

scenario noting potential problems.

www.id-book.com 35
 1.a. Heuristic Evaluation
• Developed by Jacob Nielsen in the early 1990s.
• Based on heuristics distilled from an empirical
analysis of 249 usability problems.
• These heuristics have been revised for current
technology by Nielsen and others for:
– mobile devices,
– wearables,
– virtual worlds, etc.
• Design guidelines form a basis for developing
heuristics.

www.id-book.com 36
Revised version (2014) of
Nielsen’s original heuristics

www.id-book.com 37
 Number of Evaluators and
Problems

Curve showing the proportion of usability problems in an interface

found by heuristic evaluation using different numbers of evaluators
Source: Nielsen and Mack, 1994. Courtesy of Wiley.
www.id-book.com 38
 Number of Evaluators
• Nielsen suggests that on average 5
evaluators identify 75-80% of usability
problems.

• Cockton and Woolrych (2001) point out that

the number of users needed to find 75-80%
of usability problems depends on the context
and nature of the problems.

www.id-book.com 39
Heuristics For Websites Focus On
Key Criteria (Budd, 2007)
• Clarity

• Minimize unnecessary complexity &

cognitive load
• Provide users with context
• Promote positive & pleasurable user
experience

www.id-book.com 40
 3 Stages For Doing Heuristic
Evaluation
• Briefing session to tell experts what to do.

• Evaluation period of 1-2 hours in which:

– Each expert works separately;
– Take one pass to get a feel for the product;
– Take a second pass to focus on specific features.

• Debriefing session in which experts work

together to prioritize problems.

www.id-book.com 41
 Advantages and Problems
• Few ethical & practical issues to consider
because users not involved.
• Can be difficult & expensive to find experts.
• Best experts have knowledge of application
domain & users.
• Biggest problems:
– Important problems may get missed;
– Many trivial problems are often identified;
– Experts have biases.

www.id-book.com 42
 1.b. Walkthroughs
• Focus on ease of learning.

• Designer presents an aspect of the design & usage

scenarios.

• Expert is told the assumptions about user population,

context of use, task details.

• One or more experts walk through the design

prototype with the scenario.

• Experts are guided by 3 questions.

www.id-book.com 43
 The 3 Questions
• Will the correct action be sufficiently evident to the user?
(Will the user know what to do to achieve the task?)
• Will the user notice that the correct action is available?
(Will users see how to do it?)
• Will the user associate and interpret the response from
the action correctly? (Will users know from the feedback
that they have made a correct or incorrect choice of
action?)
As the experts work through the scenario they note
problems.

www.id-book.com 44
 2. Web Analytics
• Designers use the analysis to improve their designs

• When designs don’t meet users’ needs, they will not return
to the site. They become one-time users
• Web analytics enable designers to track the activities of
users on their site.
• They can see how many people come to the site, how long
they stay, and where they go
• Web analytics offer designers the “big picture” about how
their site performs based on user activity
• One of the most well-known analytics

www.id-book.com 45
Segment of Google Analytics for Interaction
Design 5e website, December 2018

www.id-book.com 46
 Segment of Google Analytics for Interaction
Design 5e website, December 2018 (continued)

www.id-book.com 47
 Key points
• Inspections can be used to evaluate requirements, mockups,
functional prototypes, or systems.

• User testing & heuristic evaluation may reveal different usability

problems.

• Design guidelines can be used to develop heuristics.

• Walkthroughs are focused so are suitable for evaluating small

parts of a product.

• Analytics involves collecting data about users activity on a

website or product.

www.id-book.com 48