Standardised Questionnaires in Usability Evaluation.

Applying Standardised Usability Questionnaires in digital

products evaluation

Oana Alexandra Rotaru 1[0000-0003-0550-6871], Silviu Vert 1[0000-0003-2394-4859],

Radu Vasiu 1[0000-0003-1185-1997], Diana Andone 1[0000-0001-6600-7860]
1
Politehnica University of Timișoara, Timișoara 300006, Romania
[email protected],[email protected],
[email protected], [email protected]

Abstract. The usability evaluation plays a crucial role in the human-computer

interaction. It is one of the basic elements used to verify the user interface
quality and also, the quality of the system, as a whole. The goals of usability
testing can vary by study, but usually they include: identifying problems in the
design of a product or service, uncovering opportunities to improve, learning
about the target user’s behaviour and preferences. In this paper, we will present
an analysis over the most commonly used standardised questionnaires. Also,
based on a comparison between them, we will present the results of an analysis
done by a group of students, who were asked to compare and decide which
standardised questionnaire would be appropriate for their usability evaluation
over a certain project. The students were split in teams, each team having a
different project to analyse.Their activity is part of the “Interactivity and
Usability” subject of Multimedia Technologies Master Degree Program of
Politehnica University of Timișoara.We will also present a score of choosing
some of the surveys and emphasise the pros and cons of the preferred
questionnaires.
Keywords: Usability, Usability Evaluation, Questionnaire, Usability Methods,
SUS, PSSUQ, SUMI, QUIS.

1 Introduction
According to [1],usability is a web design (or an application design in general)
approach, by which we decide how difficult for a user is learning and accessing an
application.Developers should take into consideration that users are frequently
familiar with some user interface patterns.Keeping this in mind,complicated designs
and strange functionalities can be very confusing to the potential customers of any
digital product.[1]
Usability is defined by International Organization for Standardization (ISO) as “The
extent to which a product can be used by specified users to achieve specific goals with
effectiveness, efficiency, and satisfaction in a specified context of use”. [1]
There are five main components which define usability, according to [1]:
1. Learnability – How easy is to perform basic tasks for new users?
2. Efficiency – What time does it take for users to find what they came for?
3. Memorability – How hard is it for users to repeatedly perform their tasks?
2

4. Error rate – Errors made by users.

5. Satisfaction – The comfort users feel when using the design.
In this paper, we will present an analysis of the most commonly used standardised
questionnaires for usability evaluation. Also, based on a comparison between them,
we will statistically present the results of an analysis done by a group of students, who
were asked to compare and decide which standardised questionnaire would be
appropriate for their usability evaluation over a certain project. The students were split
in teams, each team having a different project to analyse. Their activity is part of the
“Interactivity and Usability” subject of Multimedia Technologies Master Degree
Program of Politehnica University of Timișoara.

2 Standard Questionnaires used in Usability Evaluation

The usability testing has a key importance in the human-computer interaction. It is
one of the basic elements used to verify the user interface quality [2]. There are many
definitions of usability. The actual definition of usability is based on the assumption
that users are rational agents,who can interact with a system,being capable to use their
knowledge and receiving information from the system’s reactions to achieve their
specific goals [3].
One of the popular applied methods in usability testing is the user observation, where
participants are handed tasks on an product and their behaviour is observed and
analysed by the people organising the usability testing.
Also, the questionnaire method is another good option when it comes to usability
testing. It can briefly sum up the feedback of a possible user after navigating through
an app, either mobile or web app.
As being mentioned in [4],people often think that a high level of usability can be
accomplished just by adding a simple and clear interface. However, usability is much
deeper than the superficial features of the user interface.The user interface features
have an important impact on the usability of every product,yet an important role is
played by the clear information which is presented to the user.So,a simple graphic
user interface,with few and very easy-to-use elements,may not satisfy completely the
user needs sometimes.An acceptable level of usability is reached when the design
meets the user needs on a higher level. [4]
However, in order to improve and also, somehow standardize the entire usability
testing process, there were developed targeted questionnaires, based on the essential
components of the usability itself : learnability, efficiency, memorability, error rate
and user satisfaction.
Some well known surveys for post-study are System Usability Scale (SUS), Post
Study System Usability Questionnaire (PSSUQ), Questionnaire for User Interaction
Satisfaction (QUIS), and Software Usability Measurement Inventory (SUMI).

2.1 SUMI
According to [5],SUMI is a solution to the measurements of the user perception of the
usability of a digital product.It offers the possibility to compare competing products
and similar products,so it can give useful information for future updated on the
software.It consists of a 50-item survey and it’s intended to be managed by people
 3

with some previous experience using similar products.”The concept of usability as

assessed by SUMI draws on the definition in ISO 9241 and relates to the European
Directive on Health and Safety Standards for Workers with VDU Equipment” [5]
One important aspect of SUMI has been the development of a standardisation
database, which consists of usability profiles of over 200 different kinds of
applications such as: word processors,CAD and graphics packages, travel reservation
systems,on-screen help systems.
According to [6],SUMI is especially used for evaluating product-against-product
comparisons or for product against the standardisation database comparisons,in order
to see how the product which is being rated compares against the average state-of-the-
art market profile.
SUMI has also strengths and weaknesses. Some of its strengths are the following :
it’s a validated instrument, it is complex and it consists of a database with results
available for comparison of own test results. The weaknesses can be : same drawbacks
as with all subjective scales; focus mainly on software; scale mostly addresses
classical usability issues, smaller part is about affect; the results are not highly
informative for the future possibility of redesigning products.Also, SUMI is not free.
[7]

2.2 SUS
Another type of standardized questionnaire we will present is SUS (System Usability
Scale), which is the most frequently used questionnaire to measure usability. It was
created by John Brooke in 1986.[8] In UX, SUS is always used in an online survey or
after each usability testing session for users to fill in.
According to [8],SUS was not initially created for the websites usability testing
measurements,it was developed at Digital Equipment Corporation in the UK in 1986
as a tool dedicated for electronic office systems usability engineering.While SUS is
frequently used today to measure the usability of websites, its usage is not limited to
websites. It can be used to measure any systems and applications,from digital products
such as mobile apps, digital kiosks, laptops to machinery.
SUS is known as a simple, frequently used 10-statement questionnaire..The tool asks
users to rate their perception of agreement or disagreement to the 10 statements—half
worded positively, half negatively—about the software under review. [8]
SUS has been proven as an efficient questionnaire and it is also a free tool.

2.3 PSSUQ
The PSSUQ (Post-Study System Usability Questionnaire) is a 16-item standardized
questionnaire.It is frequently used to measure users perceived satisfaction of a
website, software, system or product at the end of a study.
The items are seven-point graphic scales, anchored at the end points with the terms
“Strongly agree” for 1 and “Strongly disagree” for 7 and a “Not applicable” point
outside the scale.A group of usability evaluators selected the items on the basis of
their comprehensive content regarding hypothesized constituents of usability. For
example, the items assess such system characteristics as ease of use, ease of
learning,simplicity, effectiveness, information and the user interface. [9]
4

One of the advantages of using standardized usability questionnaires such as PSSUQ

is its replicability. It can be easily replicated PSSUQ across the studies. Also, PSSUQ
is free. [10]

2.4 QUIS
According to [11],the Questionnaire for User Interaction Satisfaction(QUIS) is a
usability testing tool created to measure the users subjective satisfaction with any
computer interface.It contains a demographic questionnaire, a measure of general
satisfaction over the entire product and also,it measures the users satisfaction on 4
certain threads, such as the screen factors,terminology and system
feedback,learnability and system capabilities.
Also,the QUIS is commonly used in areas as commerce/industry,international
education and research,domestic education and research users.Most of them use the
QUIS in conjunction with a usability testing laboratory.[11]
In the next chapter, we will present the case study applied on a group of students who
had the task to choose between the presented surveys in order to customise and use
them for their assigned projects usability evaluations.

3 Applied case study

A group of Multimedia Technologies master degree students were asked to compare
all of these four types of questionnaire and they had to analyse which one is the most
appropriate to use for their project. The students were split in 5 teams, each of the
teams having assigned a certain mobile application or web platform.

3.1 Spotlight Heritage Timisoara project

Team 1 had to realise a study on the Spotlight Heritage Timisoara mobile application
and decide which standardised questionnaire would be the most appropriate option in
order to use it for the questionnaire method applied in this case.
Spotlight Heritage Timișoara is a cultural project developed by the Politehnica
University of Timișoara and the National Museum of Banat in collaboration with
Timișoara 2021 European Capital of Culture Association.
The mobile app contains several stories about the city that highlights neighbourhoods
and communities of the city. The first story is already implemented and it’s called
“Iosefin”. In Figure 1, there is a capture of a section from the mobile app.
The students decided to use the SUS questionnaire because it’s fast and easy to use by
the participants of the usability evaluation and also, for its rationale of the score
calculation.
 5

Figure 1. Spotlight Heritage Figure 2. OpenVMLH

Timisoara mobile application1 mobile application2

The customized questions/statements of the SUS questionnaire applicable to the

Spotlight Heritage Timișoara mobile app usability evaluation is presented as it
follows:
1. I think I would frequently use this mobile app in Timișoara.
2. I think the app is easy to use.
3. The information in this app is useful.
4. The information regarding the Iosefin neighbourhood is really interesting.
5. There is a lot of useless information on this app.
6. I think the information on the app is quite incomplete.
7. This app won’t help me in the future.
8. I can easily find the landmarks once selected as favourites.
9. This app is fast.
10. I’m comfortable with the design of this app.

3.2 Open Virtual Mobility Learning Hub project

Team 2 was responsible for finding the appropriate type of survey for the Open
Virtual Mobility Learning Hub mobile application2.
According to [12], The OpenVMLH (Virtual Mobility Learning Hub) is an innovative
multilingual environment which was created as part of the Open Virtual Mobility, a
European-funded project, with the purpose to promote collaborative learning, social
connectivism and networking as an instructional method, OERs as the main content

1 https://play.google.com/store/apps/details?id=eu.spotlighttimisoara&hl=en_US
2 https://play.google.com/store/apps/details?id=eu.openvirtualmobility.hub&hl=en_US
6

and open digital credentials. In Figure 2 presented above, there is a capture from the
mobile application first page.
For this mobile app, the team decided to use the PSSUQ - Post-Study System
Usability Questionnaire. The reasons for their choice are the following ones : its
replicability, the PSSUQ is more complex than SUS and it would be helpful for
finding precise results regarding the usability problems of the app and it’s free.
As it follows,the students customised the questionnaire and the structure is presented
below.
1. The login section was very easy and intuitive.
2. I find the app easy to use.
3. I didn’t have many problems using the app.
4. I felt comfortable with the design of this app.
5. The menu is too complex.
6. The information on OpenVMLH app is useful.
7. I find the graphic interface easy to use.
8. The video tutorials on the app are useful and precise.
9. I enjoyed the entire design of the app.
10. The information on the app is well structured.
11. I didn’t find on the app everything I searched for.
12. Communicating through this app is fast.
13. The OpenVMLH app sends suggestive error messages so I can easily
understand what I should do.
14. I think this app helps you being productive.
15. The calendar sections on this app is easy to use.
16. I think this app will help me in the future.

3.3 E-learning Center website project

Team 3 had to study the standardised questionnaires in order to decide which one
would suit their own project usability evaluation regarding the E-learning Center
website 3. Also, in Figure 3, there is a capture of the homepage of this website.

Figure 3. E-learning Center website3 Figure 4 DigiCulture courses on

UniCampus platform4

The students decided that SUS would be appropriate for their usability study over this
website because the standard questions of the questionnaire are suitable for websites
usability testing,so the major changes of the questions would be useless as long as it
has been proven as a successful questionnaire in the past.

3 https://elearning.upt.ro/ro/
4 https://unicampus.ro/
 7

Also,the students customised the survey in such a way so it could be more specific on
the E-learning Center website.
1. I think I could frequently use the E-learning Center website.
2. I think that this website is way too complex.
3. This website is easy to use.
4. I would need technical help in order to use this website.
5. The information on this website is well structured.
6. I think the menu of the website is too complicated.
7. The website is easy to learn to use.
8. I think that this website is way too difficult to use.
9. I felt comfortable using this website.
10. I think I have to improve my skills before using the E-learning Center
website.

3.4 UniCampus website project

Team 4 had to realise a study on the UniCampus platform 4 in order to create their
own questionnaire.
UniCampus is an initiative of the eLearning Center (CeL) of the Politehnica
University of Timișoara, which aims to strengthen the recognition of the Romanian
universities, the power of support and penetration in the social and educational life in
Romania of quality, academic education, by promoting access to knowledge.
Its purpose is to create and develop the first MOOC (Massive Open Online Course) in
Romania, as an online virtual platform for free, free courses, open to everyone. [13]
In Figure.4 presented above, there is a capture of the existing courses on UniCampus
website.
The students concluded that using their customised version of the SUS questionnaire
would be the best option. The reasons for their choice consists of the fact that it’s a
short questionnaire and it’s easier to be filled by the participants, rather than a longer
questionnaire which can be overwhelming for them. Also, they chose this standardised
questionnaire for its balance between the positive and negative questions, which
brings more objectiveness.
As it follows,this is the customised version of the SUS survey applicable for their
project.
1. I would frequently use this website to improve my digital skills.
2. I think that the UniCampus website is too complex.
3. This website is easy to use.
4. I think that the video tutorials are really useful.
5. I think that the website is inconsistent.
6. I need technical support to use this website.
7. I think I learned this website very quickly.
8. I need more time to get used to this website.
9. I felt confident using the UniCampus website.
10. I need more technical background to use this website.

3.5 Virtual Museum of Politehnica University of Timisoara project

The project assigned to Team 5 was a VR application developed for the Politehnica
University of Timisoara Technical Museum.The application consists of a journey
8

through the history of the university such as a virtual museum.Also, the application is
not a final product yet. In Figure 5, there is a capture from the application.

Figure 5. Virtual museum application

The students from Team 5 concluded that the appropriate questionnaire for this
application is the PSSUQ.
Through the reasons for their choice we noticed the number of questions which can
cover the main possible usability problems of the application and also, choosing this
type of survey rather than the SUS brings more specificity to the usability problems of
the application.
The PSSUQ questionnaire proposal from Team 5 is presented below.
1. The application is complex and creative..
2. I didn’t have problems at the selection of rooms in the application.
3. Navigating through this app is easy.
4. The historical content presented in the app was intelligible.
5. The objects descriptions from the app was easy to understand.
6. The sound implementation would be a real improvement.
7. The number of graphical elements was good enough.
8. All over, the objects recognition was easy without description.
9. The objects sorting by categories is well implemented.
10. The graphical elements were clear.
11. The information was clear and readable.
12. I didn’t have problems with the interactions speed.
13. The level of interactions with the elements was enough.
14. The interaction with some elements was intuitive.
15. The entire graphical interface is friendly.
16. The text information is useful.

3.6 Discussion
As presented above, we can observe that 3 out of 5 teams tend to choose the SUS
questionnaire for their usability evaluation of the projects so we can conclude that in
our case, there is 60% predilection of using the SUS questionnaire and 40%
predilection of using the PSSUQ.
Also,we could be able to previously predict the fact that after analysing all these four
standardised questionnaires presented above, the students will tend to use these two
types because both of them are free.
 9

We can also observe that both of the teams which chose the PSSUQ have to test the
VR application and the OpenVMLH mobile application, these products requiring
more specific research for the possible usability problems.
Through the reasons mentioned by the students decided to use the SUS,we can notice
that this standardised questionnaire offers fewer questions,it’s an easy and objective
type of questionnaire and also,being short encourages the participants to be honest
and fast. Another reason for choosing the SUS is its replicability and the fact that it
can be customised or not.An advantage of using the SUS is also the ease of the score
calculation,demonstrating a logic rationale,with an equal number of positive and
negative statements,with answers in a range between “Strongly Disagree” to
“Strongly Agree”.The interpretation of the obtained score can be found in [14].
Likewise,the students embracing the use of PSSUQ in their usability evaluations
brought significant reasons for their analysis, such as the ease of use, the proper
number of questions which is also an important aspect in the questionnaire method.
The PSSUQ offers 16 questions/statements that can be properly customisable so the
survey is able to cover important aspects of the tested product.Contrarily,a longer
questionnaire can be overwhelming for the participants,inducing thoughtless answers
and subjectiveness.
Comparing the options of the teams participating in this case study,we can conclude
that choosing one of the standardised questionnaires in usability evaluations depends
on the complexity of any product,the stage of development.An example supporting
these statements can be the decision to use the PSSUQ for the Virtual Museum VR
application,which is not a final product and it’s also a complex application.
Furthermore,choosing one of the questionnaires presented in this paper depends on
the budget of the team organising the usability evaluation.As being said above,the
students picked the free questionnaires-SUS,PSSUQ, instead of the paid ones - SUMI
and QUIS.
An important point when it comes when picking the appropriate questionnaire is its
complexity.As mentioned above,compared with a short questionnaire,a long survey
can overwhelm the participants and induce thoughtless answers.

4 Conclusion and Future work

As specified in the beginning of the paper, the goals of usability testing can vary by
study, but usually they include: identifying problems in the design of a product or
service, uncovering opportunities to improve, learning about the target user’s
behaviour and preferences.
Also,in this paper we researched and presented various types of commonly used
standard questionnaires used in the user evaluation phase : SUS, PSSUQ, QUIS and
SUMI.
Likewise,we presented a case study where five teams of master degree students had to
choose between those four surveys in order to accomplish their usability evaluations
work by the end of semester.
10

We obtained a score of 60% for using the SUS questionnaire and 40% for using the
PSSUQ.We also emphasised above the pros and cons of using any of these two types
of usability evaluation surveys.
As future work,the students will have to apply the questionnaires within the
Interactivity and Usability subject at the Politehnica University of Timișoara and
afterwards,we will analyse the impact of the chosen questionnaires over the assigned
projects usability evaluations.

References

1. Ondrej Krejcar,University of Hradec Kralove:Principles of Usability in Human-Computer

Interaction. Journal 2(5), 99–110 (2016).
2. Nielsen, Jakob. “Usability engineering.” The Computer Science and Engineering
Handbook (1993).
3. Law, E., Jerman-Blažič, B., Assessment of user rationality and adaptivity: a case study. V:
KINSHUK (Ed.), SAMPSON, Demetrios G. (Ed.), ISAÍAS, Pedro (Ed.). Proceedings of
the IADIS International Conference on Cognition and Exploratory Learning in Digital Age
(CELDA 2004), Lisbon, Portugal, December 15-17, 2004. IADIS, 2004.
4. MacLeod, Miles. “Usability: Practical Methods for Testing and Improvement.” (1994).
5. Kirakowski, Jurek and Mary Corbett. “SUMI: the Software Usability Measurement
Inventory.” BJET 24 (1993): 210-212.
6. Kulkarni, Rajesh, P. Padmanabham, Varsha Sagare and Vipul Maheshwari. “Usability
evaluation of PS using SUMI (Software Usability Measurement Inventory).” 2013
International Conference on Advances in Computing, Communications and Informatics
(ICACCI) (2013): 1270-1273.
7. SUMI Homepage, http://www.allaboutux.org/sumi, last accessed 2020/28/03.
8. McLellan, Sam, Andrew Muddimer and S. Camille Peres. “The effect of experience on
system usability scale ratings.” (2012).
9. Lewis, James R.. “Psychometric Evaluation of the Post-Study System Usability
Questionnaire: The PSSUQ.” (1992).
10. PSSUQ Homepage, https://uiuxtrend.com/pssuq-post-study-system-usability-
questionnaire/, last accessed 2020/28/03.
11. Harper,Peter D. and Kent L. Norman. “Improving user satisfaction: the questionnaire for
user interaction satisfaction version 5.” (1993).
12. Diana Andone,Silviu Vert,Vlad Mihaescu,Daniela Stoica,Andrei Ternauciuc,
Politehnica University of Timișoara,“Evaluation of the Virtual Mobility Learning
Hub”.
13. UniCampus Homepage, https://unicampus.ro/, last accessed 2020/28/03.
14. SUS Homepage, https://uiuxtrend.com/measuring-system-usability-scale-sus/, last
accessed 2020/28/03