MEASURING UX

Nie Zhenzhi
Lead Instructor, User Experience Design
AGENDA 2

‣ What is UX metrics and Why

‣ How to measure UX
‣ Descriptive Statistics
‣ Calculating ROI
‣ A few tips
MEASURING UX 3

WHAT IS UX METRICS
AND WHY
MEASURING UX 4

DISCUSS
‣ Singapore Airlines approached you to revamp their website
‣ Question:
‣ How will you know that your design is in fact better than
the current one?
MEASURING UX 5

1 Measure the
initial site 2 Redesign

4 3
Compare initial Measure the
with redesign redesign
WHAT & WHY 6

A metric is a way of measuring

or evaluating a particular
phenomenon or thing.

Measuring the User Experience

by Tom Tullis and Bill Albert
WHAT & WHY 7

WE ARE FAMILIAR WITH MANY METRICS

‣ Time
‣ Distance
‣ Speed
‣ Weight
‣ Height
‣ Temperature
‣ Volume
WHAT & WHY 8

WHAT ARE UX METRICS

‣ Must reveal something about the user experience
‣ Must be observable
‣ Must be quantifiable
‣ Measure the effectiveness, efficiency, or satisfaction

‣ E.g. task success, user satisfaction, errors, etc.

WHAT & WHY 9

CHALLENGES
‣ UX metrics are about people and their behaviour or attitude
‣ which can be difficult to measure

‣ Apply Confidence Intervals to reflect the variability of the data

WHAT & WHY 10

WHY WE MEASURE
‣ Add structure to the design and evaluation process
‣ More effectively communicate the research findings
‣ Provide information to decision makers
‣ Remove “gut feeling” or hunches
WHAT & WHY 11

UX METRICS HELP YOU ANSWER:

‣ Will the users recommend the product?
‣ Is this new product more efficient to use than the current product?
‣ How does the user experience of this product compare to the
competition?
‣ Do the users feel good about the product or themselves after using it?
‣ What are the most significant usability problems with this product?
‣ Are improvements being made from one design iteration to the next?
WHAT & WHY 12

SUMMATIVE VS FORMATIVE
‣ Summative:
‣ Food critic evaluates sample dishes and
compare them with other restaurants to
determine how well the food is.

‣ Formative:
‣ Chef checks the dish periodically while it’s
being prepared and make adjustments to
impact the end result positively
WHAT & WHY 13

SUMMATIVE VS FORMATIVE
Summative Formative

What Evaluation and Comparison Make iterative changes

When Final design Early in the design process

Funding for a major revamp or

Outcome Improve the design
launch of a new project
WHAT & WHY 14

Summative Formative

1 Measure the
initial site 2 Redesign

4 3
Compare initial Measure the
with redesign redesign

Summative
MEASURING UX 15

HOW TO MEASURE UX
HOW TO MEASURE 16

Users perform Tasks

HOW TO MEASURE 17

Users perform Tasks

Metrics
HOW TO MEASURE 18

Users perform Tasks Measurements

Metrics
HOW TO MEASURE 19

DISCUSS
What metrics would
you choose to
measure the usability
of Singapore Airlines
website?
HOW TO MEASURE 20

UX METRICS
‣ Time on task
‣ Task success
‣ Efficiency
‣ Self-reported ratings
HOW TO MEASURE 21

TIME ON TASK
‣ Time on Task = End Time − Start Time
‣ A good way to measure the efficiency of a product
‣ Also a good way to calculate cost savings and ROI
‣ In most situations, shorter time on task = better experience
‣ Exceptions: games, e-learning, etc.
HOW TO MEASURE 22

HOW TO MEASURE TIME ON TASK

‣ Measure using a stopwatch
‣ Measure by timestamp in recorded video
‣ Automated tools:
‣ Usability Activity Log ‣ Morae
‣ Observer XT ‣ Usability Testing Envrionment
‣ Ovo Logger ‣ Usability Test Data Logger
HOW TO MEASURE 23

TASK SUCCESS (BINARY)

‣ Binary data (0 for fail, and 1 for success)
‣ The most common usability metrics
‣ The task must have a clear end state or goal
‣ Usually analysed by task, shown as bar graphs
HOW TO MEASURE 24

TASK SUCCESS (BINARY) - EXAMPLE

HOW TO MEASURE 25

TASK SUCCESS (LEVELS)

‣ Suitable when there are reasonable shades of grey associated with
task success
‣ Need to clearly define the levels
‣ May assign numbers to each level for quantitative analysis
HOW TO MEASURE 26

TASK SUCCESS (LEVELS) - EXAMPLE

‣ 1 = No problem
‣ 2 = Minor problem
‣ 3 = Major problem
‣ 4 = Failure / give up
HOW TO MEASURE 27

EFFICIENCY (AS TASK SUCCESS / TIME)

‣ It expresses task success per unit time
‣ The higher the more efficient
HOW TO MEASURE 28

EFFICIENCY (AS LOSTNESS)

‣ N: The number of different web pages visited while performing a task
‣ S: The total number of pages visited while performing the task,
counting revisits to the same page
‣ R: The minimum (optimum) number of pages that must be visited to
accomplish the task

‣ Lostness L = sqrt[(N/S-1)2 + (R/N-1)2]

HOW TO MEASURE 29

EFFICIENCY (AS LOSTNESS) - EXAMPLE

2
1 Membership Sign up
Home
Programme A Programme A

R=3
HOW TO MEASURE 30

EFFICIENCY (AS LOSTNESS) - EXAMPLE

2
1 Membership Sign up
Home
3 Programme A 4 Programme A
1

2 Membership
Programme B

R=3 N=4 S=5 L = sqrt[(N/S-1)2 + (R/N-1)2] = 1.414

HOW TO MEASURE 31

EFFICIENCY (AS LOSTNESS)

‣ What does it mean?
‣ Ideally, L = 0
‣ L<0.4, no observation of being lost
‣ L>0.5, definitely appear to be lost
HOW TO MEASURE 32

SELF-REPORTED RATINGS
‣ Self-reported data gives users’ perception of the product
‣ It tells you how users feel about the product
‣ You can collected these data when:
‣ Post-task
‣ Post-session

‣ Be careful about the bias - Social Desirability Bias

HOW TO MEASURE 33

POST-TASK RATINGS
‣ Ease of use for the task
‣ Single Ease Question (SEQ)

http://www.measuringu.com/blog/single-question.php
HOW TO MEASURE 34

POST-SESSION RATINGS
‣ System Usability Scale (SUS)
‣ Net Promoter Score (NPS)
HOW TO MEASURE 35

SYSTEM USABILITY SCALE (SUS)

http://www.measuringu.com/sus.php
HOW TO MEASURE 36

CALCULATING SUS
‣ For odd-numbered items: subtract 1 from the user response.
‣ For even-numbered items: subtract the user responses from 5
‣ Add up the converted responses, and multiply that total by 2.5

‣ < 50: Not acceptable

‣ 50 - 70: Marginal
‣ >70: Acceptable
HOW TO MEASURE 37

NET PROMOTER SCORE (NPS)

How likely is it that you'll recommend this product to a friend or
colleague?

The response options range from 0 (Not at all likely) to 10 (Extremely

likely). Responses are then bucketed into the following segments.

Promoters: Responses from 9-10

Passives: Responses from 7-8
Detractors: Responses from 0 to 6 http://www.measuringu.com/blog/nps-ux.php
HOW TO MEASURE 38

NET PROMOTER SCORE (NPS)

‣ Subtracting the proportion of detractors from the proportion of
promoters and converting it to a percent gets you the Net Promoter
Score.
‣ For example, 100 promoters and 30 passive and 80 detractors gets you
a Net Promoter Score (NPS) of 9.5% (20 divided by 210). This means
there are 9.5% more promoters than detractors. A NPS of -10% means
you have 10% more detractors than promoters.
MEASURING UX 39

DESCRIPTIVE STATISTICS
DESCRIPTIVE STATISTICS 40

TRUE-SCORE THEORY

Observed Score = True Score + Measurement Error

Based on experiment Based on all users (Calculated)

(observed) (unknown)
DESCRIPTIVE STATISTICS 41

MEASURES OF CENTRAL TENDENCY

‣ Mode = the value that appears most often in the distribution
‣ Excel: MODE(array)
‣ Median = center of distribution when values are ranked in the order
of magnitude
‣ Excel: MEDIAN(array)
‣ Mean = average score of the distribution
‣ Excel: AVERAGE(array)
DESCRIPTIVE STATISTICS 42

MEASURES OF VARIABILITY OR SPREAD

‣ How far away from the mean are your data points?
‣ Standard Deviation
‣ Excel: STDEV.S (array)
DESCRIPTIVE STATISTICS 43

CONFIDENCE INTERVALS
‣ An estimate of a range of values that includes the true score
‣ Excel: CONFIDENCE (alpha, standard deviation, sample size)

‣ Alpha = 1 - Confidence Level

‣ Confidence Level: 99%, 95%, 90%, etc.
DESCRIPTIVE STATISTICS 44

CONFIDENCE INTERVALS
‣ Look at example in Excel “Single” tab
 DESCRIPTIVE STATISTICS 45

CONFIDENCE INTERVALS
Observed score +
56.2 Margin of error

Margin of 53.9
2.3 error

95%
Confidence Observed
Interval = 53.9 score
4.6

Observed score -
51.6 Margin of error
DESCRIPTIVE STATISTICS 46

CONFIDENCE INTERVALS
56.2
55.8
55.4

53.9 53.9 53.9

52.4
52.0
51.6
95% confidence 90% confidence 80% confidence
DESCRIPTIVE STATISTICS 47

CONFIDENCE INTERVALS
‣ The size of Confidence Interval depends on:
‣ Confidence level
‣ Sample size
‣ Variability (Standard Deviation)

Higher confidence level = Bigger confidence intervals

Bigger sample size = Smaller confidence intervals
Bigger SD = Bigger confidence intervals
DESCRIPTIVE STATISTICS 48

COMPARING MEANS: SIGNIFICANT DIFFERENCE

‣ If the confidence intervals don’t overlap, the two means are
significantly different from each other (at the confident level chosen).
‣ If the confidence intervals overlap slightly, run a t test to determine
‣ If the confidence intervals overlap widely, the two means are not
significantly different
DESCRIPTIVE STATISTICS 49

COMPARING MEANS: T TEST

‣ Independent samples: different set of users
‣ Paired samples: the same set of users

‣ Excel: T.TEST (array 1, array 2, tails, type)

‣ Tails = 2 almost all times
‣ Type: 1 for paired, 2 for independent
DESCRIPTIVE STATISTICS 50

COMPARING MEANS: T TEST

‣ What does it mean?
‣ If t test returns 0.03, it means there’s a 3% chance that this
difference is not significant.
‣ Compare this to alpha level. E.g. if your confidence level is 95%, then
alpha = 5%. In this case, 3% < 5%, which means the difference is
significant at 95% confidence level.
MEASURING UX 51

CALCULATING ROI
CALCULATING ROI 52

INTERNAL ROI
‣ Increased user productivity
‣ Decreased user errors
‣ Decreased training cost
‣ Savings gained from making changes earlier in the design
‣ Decreased user support
CALCULATING ROI 53

EXTERNAL ROI
‣ Increased sales
‣ Decreased customer support cost
‣ Savings gained from making changes earlier in the design
‣ Reduced cost of providing training
CALCULATING ROI 54

CALCULATE THE ROI

‣ Task failure rate drops from 28% to 5%
‣ $2 transaction fee per transaction

‣ What’s the yearly revenue increase?

CALCULATING ROI 55

CALCULATE THE ROI

‣ Task failure rate drops from 28% to 5%
‣ $2 transaction fee per transaction
‣ Assumption: 100,000 transactions per month
‣ What’s the yearly revenue increase?
CALCULATING ROI 56

CALCULATE THE ROI

‣ Task failure rate drops from 28% to 5%
‣ $2 transaction fee per transaction
‣ Assumption: 100,000 transactions per month
‣ What’s the yearly revenue increase?
‣ $552,000
CALCULATING ROI 57

CALCULATE THE ROI

‣ Average time savings: 79 seconds per user
‣ Average annual salary per user: $14,700
‣ Average working hours per week: 40

‣ What’s the yearly benefit?

CALCULATING ROI 58

CALCULATE THE ROI

‣ Average time savings: 79 seconds per user
‣ Average annual salary per user: $14,700
‣ Average working hours per week: 40
‣ Assumption: a quarter of 2.7 million users will use it once a month
‣ What’s the yearly benefit?
CALCULATING ROI 59

CALCULATE THE ROI

‣ Average time savings: 79 seconds per user
‣ Average annual salary per user: $14,700
‣ Average working hours per week: 40
‣ Assumption: a quarter of 2.7 million users will use it once a month
‣ What’s the yearly benefit?
‣ $1.256 million
MEASURING UX 60

A FEW TIPS
A FEW TIPS 61

A FEW TIPS
‣ Utilise your data
‣ Don’t wait to be asked to measure
‣ Measurement is not as expensive as you thought
‣ Plan early
‣ Speak the language of business
‣ Simplify your presentation
MEASURING UX 62

Q&A
FURTHER READING 63