Usability

Usability derived from the word usable which means can be used very well.
According to ISO 9241-11, usability is defined as how far can the product be used by certain
users based on effectivity, efficiency, and satisfaction in a usability context in order to reach
certain outcomes (Sabandar & Santoso, 2018). Users’ wants to achieve their goal with
accuracy and speed. A product needs to support users’ goals and needs. Hence, it is why
effectiveness and efficiency are used to measure the usability of a product. Another critical
criterion is satisfaction. Users’ perception of satisfaction determines how users feel about a
certain product and it measures the satisfaction level of a product. When users are satisfied
with a product, it means that their expectation is fulfilled. Therefore, those criteria become
the critical measures of usability (Barnum, 2020).

Users’ perception is measured by using System Usability Scale (SUS). System

Usability Scale is a subjective usability that is measured after users have worked through
evaluation tasks. Users are asked to rate their agreement level with statements that cover
usability characteristics, such as complexity and effectivity. Usability is measured by
distributing a 10-item post-test questionnaire to the users that can quickly assess product
usability without the need to conduct further analysis (Drew, et al., 2018). The System
Usability Scale (SUS) questionnaire can be seen in Table 2.2 below (Sharfina & Santoso,
2016). The SUS score can be seen in Figure 2.8 below (Aulia, et al., 2021).

Table 2. 2 SUS Questionnaire

No Question

1. I think that I would like to use this system.

2. I found the system unnecessarily complex.

3. I thought the system was easy to use.

4. I think that I would need the support of a technical person to be able to use
this system.

5. I found the various functions in the system were well integrated.

6. I thought there was too much inconsistency in this system.

7. I would imagine that most people would learn to use this system very
quickly.

8. I found the system very cumbersome to use.

9. I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

Users are asked to fill in the questionnaire with five choices available starting from
strongly disagree to strongly agree. The score ranged from 0 to 100. The calculation of SUS
score is based on the following provisions (Kardbri, et al., 2021).
1. For every statement in odd numbers, the scale is subtracted by 1 (x-1).
2. For every statement in even numbers, the scale is subtracted from 5 (5-x).
3. The scales values from range 0 to 4, with 4 being the most positive responses.
4. The converted responses are added up and multiplied by 2.5.
5. Define the average score of all respondents.

Figure 2. 8 SUS Score

Figure 2.8 shows how the SUS score is measured. There are three points of view;
acceptability ranges, grade scale, and adjective ratings. Aside from that, there is another way
to measure SUS score by using the percentile rank. Table 2.3 shows the percentile rank of
SUS score (Kardbri, et al., 2021).

Table 2. 3 SUS Percentile Rank

Percentile Rank Average SUS Score

Grade A More than or equal to 80.3

Grade B Less than 80.3 and more than 74

Grade C Less than 74 and more than 68

Grade D Less than 68 and more than 51

Grade F Less than 51

2.6 Usability Testing

Usability testing is a method in which representative tasks are being attempted to be

completed by representative users in representative environments, such as early prototypes.
The goal of usability testing is to improve the interface quality based on the findings such as
flaws-areas of the interface that needs improvement. Usability testing is expected to be
practical and have major influence (Lazar, et al., 2017).

2.6.1 Types of Usability Testing

There are many types of usability testing. Usability testing is classified into three
different categories: expert-based testing, automated usability testing, and user-based testing
(Lazar, et al., 2017).

2.6.1.1 Expert-Based Testing

An expert-based testing is done by interface experts. It uses several structured

methods in order to find the flaws of the interface. The user interface designer should not be
involved in the testing as it may result in a biased result (Lazar, et al., 2017).

One of the expert-based testing methods is by using heuristic review.

Shneiderman’s 8 Golden Rules of Interface Design is one of the best-known sets of heuristic
review (Lazar, et al., 2017). The eight golden rules are as follows (Shneiderman, et al., 2018).
1. Strive for consistency
2. Seek universal usability
3. Offer informative feedback
4. Design dialogs to yield closure
5. Prevent errors
 6. Permit easy reversal of actions
7. Keep users in control
8. Reduce short-term memory load

2.6.1.2 Automated Usability Testing

An automated usability testing is done by a software. The software is developed by an

expert and applies a sequence of guiding principles to an interface.
The software determines the flaws in which the interface doesn’t meet the guiding principle.
The automated usability testing excels in reading through code rapidly.

The software looks for usability problems that can be gathered. However, there are many
aspects of usability that cannot be discovered automatically, such as layout, labels, and
wording (Lazar, et al., 2017).

2.6.1.3 User-Based Testing

A user-based testing is done to the representative users. Representative tasks are

prepared and tested to the representative users (Lazar, et al., 2017). The number of the
representative users based on various rules of thumb is three to five users (Hartson & Pyla,
2019). A study by Nielsen and Landauer proved that five users are enough to do usability
testing. According to Nielsen, 85% of the findings will be uncovered after testing to five
users and the same findings will be repeated. Hence, the testing should stop after the fifth
user. A graph regarding the study can be seen in Figure 2.9 (Barnum, 2020).

Figure 2. 9 Relation Between Number of Test Users and Usability Problems Found (Barnum,
2020)
 The curve shows that there are no problems found at the beginning of the test with no
user. However, after a single user test, almost a third of the problems can be discovered. This
shows that there is a significant difference between no test user and one test user. Therefore,
testing only five users is effective enough to discover usability problems (Barnum, 2020).

2.6.2 Stages of Usability Testing

Usability testing requires planning beforehand because it’s not something that just
happened (Lazar, et al., 2017). Rubin (2008) stated the stages of usability testing consist of
eight stages. The stages are as follows (Rubin & Chisnell, 2008).
1. Develop the test plan
2. Setup the test environment
3. Find and select participants
4. Prepare test materials
5. Conduct the test sessions
6. Debrief the participants
7. Analyse data and observations
8. Report findings and recommendations
Lazar also described different steps of usability testing. There are eight steps of
usability testing. The steps are as follows (Lazar, et al., 2017).
1. Select representative users
2. Select the setting
3. Decide what tasks users should perform
4. Decide what type of data to collect
5. Before the test session (informed consent, etc.)
6. During the test session
7. Debriefing after the session
8. Summarize results and suggest improvements

2.6.3 Maze

Maze is an online software that is used to conduct usability testing. The usability
score is calculated by using several formulas. The first formula used is Screen Usability Score
(SCUS) is a score that calculates the usability of a product based on the activities that are
done on the product’s screen. The formula is as follows (Kinney, 2020).

SCUS = MAX(0,100-(DOR x dW)-(MCR x mW)- (MIN (10,MAX (0, AVGD -5

)))) …(1)
2

where,

SCUS = Screen Usability Score DOR = Drop-Off

and Bounce Rate (%)
dW = DOR weight; The dW equals 1 point for every drop-off/bounce MCR = Misclick
Rate (%)
mW = MCR weight; The mW equals 0.5 points for every misclick
AVGD = Average Duration (second)

The second formula used is Mission Usability Score (MIUS). It is a score that reveals
how easy it is for the users to finish a mission. The principles used in calculating the MIUS
score are that the direct success must have strong relation to the MIUS, the indirect success
must not be considered as a failure, and the average usability metrics are not supposed to
influence the mission score. The formula is as follows (Kinney, 2020).

MIUS = DSR + (IDSR/2) - avg(MC_P) - avg(DU_P) …(2)

where,

MIUS = Mission Usability Score

DSR = Direct Success Rate (%)
IDSR = Indirect Success Rate (%)
Avg = Average
MC_P = Misclick Penalty = MCR * 0.5
DU_P = Duration Penalty = (MIN(10, MAX(0,(AVGD-5)/2)))
 The third formula used is Maze Usability Score (MAUS). The score doesn’t affect the
other mission scores. It calculates the average of the Mission Usability Score. The formula is
as follows (Kinney, 2020).

MAUS = avg(MIUS) …(3)

where,

MAUS = Maze Usability Score avg =

Average
MIUS = Mission Usability Score