Original Article

Healthc Inform Res. 2024 October;30(4):324-332.

https://doi.org/10.4258/hir.2024.30.4.324
pISSN 2093-3681 • eISSN 2093-369X

Evaluation of the Usability and User Experience

of the Jaminan Kesehatan Nasional Mobile
Application in Indonesia
Savira Putri Handasari, Respati Wulandari, Haikal
Department of Public Health, Faculty of Health Science, Universitas Dian Nuswantoro, Semarang, Indonesia

Objectives: The National Health Insurance (Jaminan Kesehatan Nasional or JKN) mobile application represents an adminis-
trative transformation of health social insurance in Indonesia. The application’s user base has quadrupled in the past 2 years.
The study aimed to evaluate the usability and user experience of the JKN mobile application in Indonesia. Methods: The
study administered an online survey featuring the System Usability Scale (SUS) questionnaire alongside a user experience
assessment. The survey was disseminated via social media and garnered responses from 370 participants. The variables ex-
amined included usability and user experience, specifically focusing on attractiveness, perspicuity, efficiency, dependability,
stimulation, and novelty. Data analysis was performed descriptively, and the results were benchmarked. Results: The JKN
mobile application had an average SUS score of 60.9 ± 13.47 (adjective rating “good,” Grade “F,” acceptability rating of “low
marginal,” and a percentile range of 34). User experience testing yielded the following results: attractiveness (1.09 ± 1.09), per-
spicuity (1.05 ± 1.17), efficiency (1.07 ± 1.20), dependability (1.03 ± 1.12), stimulation (0.88 ± 1.20), and novel-ty (0.58 ± 1.09).
The benchmark for the JKN mobile application placed the user experience in the “below average” category. Conclusions: The
evaluation of the usability and user experience of the JKN mobile application was generally positive. However, there is a need
to improve the usability and all parameters of the user experience. Broader-scale research is needed to explore the factors that
influence usage and user experience to promote the adoption of the JKN application and increase user satisfaction.

Keywords: Mobile Applications, National Health Programs, Benchmarking, Indonesia, Surveys and Questionnaires

Submitted: March 22, 2024

I. Introduction
Revised: July 28, 2024
Accepted: September 18, 2024 The National Social Security System in Indonesia is a gov-
ernment initiative designed to ensure protection and social
Corresponding Author welfare for all Indonesian citizens. This system is managed
Respati Wulandari
Department of Public Health, Faculty of Health Science, Universitas by the Social Security Organizing Agency (SSOA). One of its
Dian Nuswantoro, Jalan Pamularsih Buntu no 18, Semarang 50141, programs, the Jaminan Kesehatan Nasional (JKN), operates
Indonesia. Tel: +628179503121, E-mail: respati.wulandari@dsn. on the principles of social insurance and equity. The JKN
dinus.ac.id (https://orcid.org/0000-0002-5098-3792)
offers a range of health services to its participants, including
This is an Open Access article distributed under the terms of the Creative Com- preventive, promotive, curative, and rehabilitative care, as
mons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-
nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduc-
well as access to medications and disposable medical sup-
tion in any medium, provided the original work is properly cited. plies. The program has seen a substantial increase in partici-
ⓒ 2024 The Korean Society of Medical Informatics pation, with the total number of JKN participants reaching
257,375,800 as of June 30, 2023 [1].
 Evaluation of JKN Mobile App

The JKN mobile application was launched in November friendly an application is. It encompasses several compo-
2017 as part of SSOA Health’s digital transformation of ad- nents: learnability, efficiency, memorability, errors, and sat-
ministrative activities. It allows users to access its services isfaction [4]. Evaluating an application’s usability is deemed
anytime and anywhere. The application is available on both essential for the successful development of the application
the Google Play Store and the Apple Store. It is recom- [5]. A variety of tools are available for measuring the usabil-
mended that users access the application on Android phones ity of mobile health applications, including usability evalu-
with version 4.0 or higher, as well as on iOS 10. The JKN ation scales, application usability measurement scales, and
mobile application features several menus: (1) Participant application usability ease levels, which are among the most
menu, which includes participant features, participant card rigorously validated tools [6]. The review results show that
features, participant data modification feature, and partici- 40% of studies assessing mobile health applications rely on
pant registration feature; (2) Billing menu, which comprises questionnaires to evaluate usability [7]. The effectiveness of
a premium feature, payment feature, and virtual account a mobile application largely depends on its usability. This
check feature; (3) Service menu, which offers a service his- implies that the application should be easy to use, efficient,
tory feature, service registration feature, and screening fea- and satisfying [8]. Evidence from a review study on mobile
ture; (4) General menu, which provides a JKN information health applications highlights that the most commonly as-
feature, location feature, complaint submission feature, and sessed usability aspects are efficiency (70%), satisfaction
settings feature. The user interface of the JKN application is (66%), and effectiveness (58%) [9].
illustrated in Figure 1. In early 2022, the application was en- User experience evaluation is recognized as a standard
hanced with a new feature that supports online queues. This because it extends usability evaluation [10]. According to
includes the capability for sending new patient identities and the ISO 9241-210 standard, user experience is defined as
facilitating check-ins when patients arrive at healthcare facil- an individual’s perceptions and responses to the use of a
ities. The feature supports a continuum of care from primary product or service system. It reflects the quality of a product
healthcare facilities to advanced referral healthcare facilities, based on subjective perceptions. Additionally, the concept of
including internal referrals and follow-up appointments [2]. user experience includes considerations of the time before
Over the past 2 years, the number of JKN mobile application the product is actually used [11]. User experience relates to
users in Indonesia has seen a substantial increase, growing two main aspects: (1) the quality of user interaction with the
from 432,000 users in 2020 to 16,346,826 users in 2022 [3]. product in terms of achieving goals, which includes efficien-
This surge in users raises an important question: Does the cy, learnability, dependency, and adaptability; and (2) the
application meet the acceptance criteria of its users? overall subjective impression of the product, encompassing
Usability is a quality metric used to evaluate how user- stimulation, attractiveness, and novelty [12]. The findings
from usability and/or user experience evaluations can pro-
vide valuable insights for the development of mobile health
applications [10,13-15].

II. Methods
1. Study Design and Sampling Techniques
This cross-sectional study was conducted from February to
July 2023 using the practice sampling method. The sampling
technique involved distributing an online survey through
social media platforms (WhatsApp, Telegram, and Insta-
gram). The survey, hosted on Google Forms, initially asked
respondents if they had ever used the JKN mobile applica-
tion. Those who answered “yes” were directed to answer
evaluation questions about their experience with the JKN
mobile application. Conversely, respondents who answered
Figure 1. ‌User interface display of the National Health Insurance “no” received a link to download the application, along with
(Jaminan Kesehatan Nasional or JKN) mobile application. instructions for its download and use, available directly on

Vol. 30 • No. 4 • October 2024 www.e-hir.org 325

Savira Putri Handasari et al

the Google Form. This approach helped validate the re- The instrument used to measure user experience is the
cruitment of respondents and aims to minimize potential User Experience Questionnaire (UEQ), which has been
response bias. The inclusion criteria for participation in the translated into Indonesian following Santoso et al.’s work [17].
study required respondents to have used the JKN application This questionnaire comprises 26 statements, organized into
at least once and to be at least 18 years old. six parameters as shown in Table 2. The dependent variables
in this study are these six user experience parameters: at-
2. Population and Sample tractiveness, perspicuity, efficiency, dependability, stimula-
The population in this study comprised all Indonesian citi- tion, and novelty [18]. Each item in the UEQ is represented
zens who use the JKN mobile application. The technique by a pair of opposing words, and participants rate them on a
for calculating the sample size was based on the Isaac and 7-point scale. For instance, the efficiency of the application
Michael sample size determination table. Assuming there is measured on a scale from “fast” to “slow.” The instrument
are 16,346,260 JKN mobile application users in 2022 [2], and used to assess respondent characteristics is a closed-ended
allowing a margin of error of 5%, the minimum sample size questionnaire. These characteristics include sex, age, educa-
required was 349. For this study, a sample size of 370 was tion, marital status, occupation, JKN membership, domicile,
deemed sufficient to represent the population. and income.

3. Instrument and Variables

This study used the System Usability Scale (SUS) to measure Table 2. Grouping of parameters from the user experience variable
usability. This instrument is structured as a questionnaire,
Parameter Item
adapted from Bangor et al. [16]. The SUS questionnaire in-
1 Attractiveness annoying/enjoyable
cludes 10 statements designed to assess the usability of the
JKN mobile application (Table 1). Each statement on the good/bad
SUS features a 5-point scale ranging from “strongly disagree” unlikable/pleasing
to “strongly agree.” The statements in the SUS are composed unpleasant/pleasant
of five negative and five positive statements, which are pre- attractive/unattractive
sented in an alternating sequence. friendly/unfriendly
2 Efficiency fast/slow
Table 1. System Usability Scale statement items [19] inefficient/efficient
impractical/practical
Statement items
organized/cluttered
1 I think that I would like to use the JKN mobile
3 Perspicuity not understandable/understandable
application more often.
easy to learn/difficult to learn
2 I found the JKN mobile application to be complex to use.
complicated/easy
3 I found that the JKN mobile application is easy to use.
clear/confusing.
4 I need assistance from others or technicians to use the
4 Dependability unpredictable/predictable
JKN mobile application.
obstructive/supportive
5 I found various functions in the JKN mobile applica-
secure/not secure
tion to be well-integrated.
meets expectations/does not meet
6 I encountered a lot of inconsistencies in the JKN mobile
expectations
application.
5 Stimulation valuable/inferior
7 I imagine people would learn to use the JKN mobile
application easily. boring/exciting

8 I found the JKN mobile application to be very difficult not interesting/interesting

to use. motivating/demotivating
9 I feel very confident using the JKN mobile application. 6 Novelty creative/dull
10 I need to learn a lot of things before I can start using inventive/conventional,
the JKN mobile application. usual/leading edge
JKN: Jaminan Kesehatan Nasional (National Health Insurance). conservative/innovative

326 www.e-hir.org https://doi.org/10.4258/hir.2024.30.4.324

 Evaluation of JKN Mobile App

4. Data Analysis 5. Ethics

All respondents’ answers were individually reviewed. This This study was approved by the Health Research Ethics
process included cleaning, which involves identifying outlier Committee Faculty of Health Dian Nuswantoro University
data, removing data that do not meet the inclusion criteria, (No. 349/EA/KEPK-Fkes-UDINUS/I/2023). The Health Eth-
checking for possible data duplication, and examining other ics Committee declares that the proposed research protocol
data errors. The analysis of the SUS questionnaire yields an is ethically feasible, in accordance with the seven standards
average usability score for the JKN mobile application. The of the World Health Organization in 2011.
SUS score is calculated as follows [8]:
• Subtract 1 from the respondent’s answer score for positive III. Results
statement items (odd-numbered statements). For example,
if a respondent scores a 4 on statement number 3, the ad- 1. Respondent Characteristics
justed score is 4 – 1 = 3. A total of 370 individuals completed the questionnaire. Table
• For negative statement items (even-numbered state- 4 summarizes the frequency distribution of respondent
ments), subtract 1 from the respondent’s answer score, characteristics. The majority of respondents were women
denoted as X. For instance, if a respondent scores a 2 on (75.9%), under 25 years old (88.1%), and possessed a low to
statement number 4, the adjusted score is 2 – 1 = 1. medium level of education. Most were unmarried or single
• Sum these adjusted scores and multiply the total by 2.5. (91.9%), unemployed (79.5%), and had no income (64.3%).
• The final result, which ranges from 0 to 100, represents Additionally, the majority of respondents were JKN partici-
the average usability score of the JKN mobile application. pants (85.4%) and lived on the island of Java (83.5%).
The average usability score was interpreted using the SUS
score, which ranges from 0 to 100, to determine its classifi- 2. Usability
cation in terms of adjective ratings, grade scale, and accept- The SUS scores collected from 370 respondents totaled 2,254,
ability ranges (Figure 2). A SUS score of 50 is considered with an average of 60.9 and a standard deviation of 13.47.
poor, a score above 70 is considered good, and a score of 85 Table 5 presents the frequency distribution of SUS evalua-
or higher is considered excellent [8,16]. According to Table
3, the average usability score is interpreted within specific
Table 3. Curved grading scale for the SUS [21]
percentile ranges [19].
The UEQ analysis, available on the official website (https:// Grade SUS Percentile range
www.ueq-online.org/), was utilized to assess the user experi- A +
84.1–100 96–100
ence of the JKN mobile application. This analysis produces A 80.8–84.0 90–95
average scores for each user experience parameter. These A –
78.9–80.7 85–89
scores are interpreted as follows: an average score of -0.8 B+ 77.2–78.8 80–84
signifies a negative evaluation, a score ranging from 0 to +8
B 74.1–77.1 70–79
denotes a neutral evaluation, and any score above +8 reflects –
B 72.6–74.0 65–69
a positive evaluation. Scores between 1 and 2 are indicative +
C 71.1–72.5 60–64
of a very good evaluation. Additionally, the UEQ analysis
generates benchmark graphics that compare the JKN mobile C 65.0–71.0 41–59
application with other applications. C– 62.7–64.9 35–40
D 51.7–62.6 15–34
F 0–51.6 0–14
SUS: System Usability Scale.

Not acceptable Marginal Acceptable

Acceptability High
Low
ranges
Grade scale F D C B A

Adjective Worst Best

imaginable Poor Ok Good Excellent imaginable
ratings
Figure 2. ‌Users’ System Usability Scale
0 10 20 30 40 50 60 70 80 90 100 score [18].

Vol. 30 • No. 4 • October 2024 www.e-hir.org 327

Savira Putri Handasari et al

tion results, categorized by acceptability rating, grade, and The interpretation of the SUS score, according to Bangor
adjective rating. The analysis of the JKN mobile application et al., indicated that the JKN mobile application was rated as
revealed that the responses were predominantly categorized “good” in the adjective rating. However, it received a grade
as marginal in the acceptability rating (52.4%), received a of “F” and an acceptability rating of “marginal.” Based on the
grade of “F” (43%), and were described by the adjective rat- Lewis scale, this average score fell within the 34th percentile,
ing as “good” (52%). suggesting substantial room for improvement in the applica-
tion’s usability.
Table 4. Respondent characteristics (n = 370)
3. User Experience
Variable Frequency (%) Table 6 presents the results of the JKN application’s user
Sex experience evaluation. It illustrates the user experience in
Female 281 (75.9) terms of attractiveness, perspicuity, efficiency, dependabil-
Male 89 (24.1) ity, stimulation, and novelty. Overall, the user experience
Age (yr) parameters indicated positive values, with an average score
≤24 326 (88.1)
>25 44 (11.9)
Table 5. Usability score of the JKN mobile application (n = 370)
Education
Low 5 (1.4) Components n (%)
Medium 225 (60.8) Acceptability rating
High 140 (37.8) Not acceptable 101 (27.3)
Status Marginal 194 (52.4)
Unmarried 340 (91.9) Acceptable 75 (20.3)
Married 30 (8.1) Grade
Occupation F 159 (43.0)
Unemployed 294 (79.5) D 105 (28.0)
Employed 76 (20.5) C 77 (21.0)
Participation in JKN B 17 (5.0)
Not a participant 54 (14.6) A 12 (3.0)
Participant 316 (85.4) Adjective rating
Domicile Worst imaginable 3 (1.0)
Outside of Java Island 61 (16.5) Poor 14 (4.0)
Java Island 309 (83.5) OK 106 (29.0)
Income Good 194 (52.0)
No income 238 (64.3) Excellent 39 (11.0)
Having income 132 (35.7) Best imaginable 14 (4.0)

Table 6. Evaluation of the user experience of the JKN mobile application and benchmark categories (n = 370)

95% CI Benchmark category

Parameter Mean SD Description
Lower Upper 25% 50% 75% 90%
Attractiveness 1.09 1.08 0.98 1.20 0.69 1.18 1.58 1.84 Below average
Perspicuity 1.05 1.17 0.93 1.17 0.72 1.20 1.73 2.00 Below average
Efficiency 1.07 1.20 0.94 1.19 0.60 1.05 1.50 1.88 Above average
Dependability 1.03 1.12 0.91 1.14 0.78 1.14 1.48 1.70 Below average
Stimulation 0.88 1.20 0.76 1.01 0.50 1.00 1.35 1.70 Below average
Novelty 0.58 1.08 0.47 0.69 0.16 0.70 1.12 1.60 Below average

328 www.e-hir.org https://doi.org/10.4258/hir.2024.30.4.324

 Evaluation of JKN Mobile App

2.5 Excellent among a sample of 370 respondents from across Indonesia.

2.0
Good The evaluation, conducted using the SUS, provides insights
Above average
1.5 Below average into user perceptions regarding satisfaction, efficiency, learn-
Bad
Mean
ability, errors, and memorability [6]. The average SUS score
1.0
obtained was 60.9, with a standard deviation of 13.47. Accord-
0.5
ing to Bangor’s interpretation [16], this average score would
0
be categorized as “good” (194 individuals; 52%). However,
0.5 the acceptance of the JKN mobile application was catego-
1.0 rized as “low marginal.” This assessment is based on the
acceptability rating, where the majority of responses (194
n
s

ty

en cy

im ty

ty
es

io
ui

li

el
n

at
bi
en

ic

ie

ov
da

ul

individuals; 52%) fell into the “low marginal” category. “Low

sp

fic
iv

N
ct

Ef
r
Pe

St
tra

ep

marginal” is defined as a level between “not acceptable”

At

Figure 3. ‌National Health Insurance (Jaminan Kesehatan Nasi- and “high marginal,” indicating a status that is considerably
onal or JKN) mobile application benchmark. below “acceptable.” This aligns with the SUS score being in
the lowest grade, which is grade F, with most respondents’
greater than 0.8. However, novelty received the lowest aver- assessments (159 individuals; 43%) falling into this category.
age score, at 0.58. These findings suggest that while the JKN mobile applica-
The evaluation results for the user experience of the m- tion was considered “good” by a majority of users, there is a
JKN application indicate that, in terms of attractiveness, substantial need for improvements in its overall usability.
its average score was 1.09, placing it in the “below average” Several previous studies on the evaluation of the JKN mo-
benchmark category (25%–50% scale). The mean score of bile application have reported findings similar to those of
1.09, with a standard deviation of 1.09 at a 95% confidence this research. A study on the JKN mobile application [20] re-
level (Z-score of 1.96) and a sample size of 370, resulted in ported an initial average SUS score of 59 and recommended
a confidence interval (CI) of 0.111, ranging from 0.976 to further development of the application. Likewise, Lubis et
1.197. This CI range remained within the “below average” al. [21] highlighted both strengths and weaknesses of the
category, with the upper limit slightly exceeding the catego- JKN mobile application, recommending enhancements to
ry’s upper boundary. Detailed calculations can be found in improve the system. A study involving local respondents in
Table 6. The standard deviation is linked to the CI value and Malang, Indonesia, found low scores in control, learnability,
the sample size. Specifically, a larger sample size results in a and efficiency aspects of the JKN mobile application [22].
narrower CI, which increases estimation accuracy and yields When compared to other mobile health applications, the
more reliable results. The standard deviation values for all JKN mobile application has a lower average SUS score. In
parameters in the user experience evaluation were reason- contrast, other mobile health applications such as Oncodoc
ably small and within acceptable limits, as the CI values fell [23], JAGA SEHAT app [24], “Pedulilindungi” app [25],
within the benchmark category range. Therefore, it can be and TOGETHERCare app [26] all boast higher average SUS
concluded that the user experience of the JKN mobile appli- scores.
cation, across parameters such as attractiveness, perspicuity, User feedback on the JKN mobile application has been
dependability, stimulation, and novelty, was categorized as positive across nearly all dimensions—including attractive-
“below average” (25%–50% scale). However, the efficiency ness, perspicuity, efficiency, dependability, stimulation, and
parameter ranked in the “above average” benchmark cat- novelty—with mean scores above 0.8. User satisfaction with
egory (50%–75% scale), suggesting that the application an application hinges on its attractiveness, pragmatic quali-
performs efficiently (Figure 3). The benchmark used in the ties (such as perspicuity, efficiency, and dependability), and
UEQ compared the JKN application with other applications hedonic qualities (including stimulation and originality) [27].
assessed using the UEQ, including business software, web Hedonic qualities, which include features of the applica-
pages, web shops, and social networks. tion, visual resources, and the ease of monitoring treatment,
positively influence the user experience of mobile health
IV. Discussion applications [28]. Conversely, factors such as comfort, re-
sponsiveness, and availability are positive predictors of user
This study evaluates the use of the JKN mobile application experience [29]. However, perceptions of user experience

Vol. 30 • No. 4 • October 2024 www.e-hir.org 329

Savira Putri Handasari et al

regarding mobile applications encompass all aspects of in- age” range.

teraction with the product, including feelings, thoughts, and Based on the evaluation results for usability and user expe-
perceptions related to the use of the product [30]. rience, it can be concluded that the JKN mobile application
The pragmatic quality of this study was positively assessed exhibits a marginally low level of usability. While the user
by the respondents. The majority of users of the JKN mobile experience, in terms of attractiveness, perspicuity, efficiency,
application, approximately 85.4%, are members of the SSOA dependability, stimulation, and novelty, receives positive
health insurance program. The application’s efficiency, de- ratings, the overall user experience evaluation of the JKN
pendability, and stimulation were also favorably evaluated by mobile application remains in the “below-average” category.
the respondents, indicating that the application is practical, This indicates a need for improvements in both usability and
fast, secure, and beneficial. Participants from the SSOA re- all user experience parameters of the JKN mobile applica-
ported significant benefits from using the JKN mobile appli- tion.
cation. Before the application was available, participants had The limitations of this study include the use of an online
to physically visit local SSOA offices to handle tasks such as survey method, which may introduce selection bias in sam-
registration, adding new members, and filing service-related ple selection. Despite validation efforts to ensure that only
complaints. With the introduction of new features, SSOA users of the JKN mobile application were eligible to complete
participants now enjoy greater convenience in managing the questionnaire, it was not possible to completely eliminate
healthcare services, including registration (queueing), ac- respondent answer bias. Conducting further research on a
cessing facility location information, and consulting with larger scale would enhance the reliability and generalizability
doctors. This has resulted in time and cost savings for the of the findings and positively influence improvements in the
participants, which is consistent with previous research adoption and user satisfaction of the JKN mobile applica-
findings that highlight the significant influence of time and tion.
money on user experience [28].
The perspicuity of the JKN mobile application was rated Conflict of Interest
highly by respondents, who found it simple, straightforward,
and user-friendly. This can be attributed to the demographic No potential conflict of interest relevant to this article was
characteristics of the respondents: the majority (60.8%) had reported.
completed education up to senior high school, and a signifi-
cant portion (88.1%) were aged 24 or younger, suggesting Acknowledgments
that they were predominantly students or college students.
These respondents are typically part of Generation Z, a All researchers would like to thank Savira Putri Handasari
group known for its openness to new developments, includ- who has contributed to her thesis at the Department of Pub-
ing advancements in information technology. They are gen- lic Health, Faculty of Health Science, Dian Nuswantoro Uni-
erally well-versed in computer use, adept at navigating social versity, Semarang, Indonesia
media, and consequently, comfortable with mobile applica-
tions. This familiarity likely influenced their positive assess- ORCID
ment of the application, a finding that aligns with previous
research conducted on the Oncodoc application [23]. Savira Putri Handasari (http://orcid.org/0009-0006-5963-9345)
The hedonic quality, particularly the novelty aspect, re- Respati Wulandari (http://orcid.org/0000-0002-5098-3792)
ceived the lowest average score (0.580 ± 1.09) from respon- Haikal (http://orcid.org/0000-0002-7550-7995)
dents when compared to attractiveness, efficiency, perspicu-
ity, dependability, and stimulation. This lower score may be References
due to Generation Z’s reduced perception of novelty, as their
frequent use of social media and applications could lead 1. The Social Health Insurance Organizer. Participants of
them to perceive the experience of using a new application the National Health Insurance Program (Program JKN)
as commonplace. Consequently, they may hold higher stan- [Internet]. Jakarta, Indonesia: The Social Health Insur-
dards for what constitutes a truly novel and distinct applica- ance Organizer; 2023 [cited at 2023 Jul 7]. Available
tion. Despite this, the benchmark for the novelty aspect of from: https://www.bpjs-kesehatan.go.id/#/.
the JKN mobile application still fell within the “below aver- 2. Ministry of Communication and Digital Republic of

330 www.e-hir.org https://doi.org/10.4258/hir.2024.30.4.324

 Evaluation of JKN Mobile App

Indonesia. Access services at your fingertips via JKN 13. Wulandari R, Suwandono A, Kartasurya MI, Nugraheni
Mobil [Internet]. Jakarta, Indonesia: Ministry of Com- SA. Postpartum care behavior improvement during
munication and Digital Republic of Indonesia; 2020 COVID-19 pandemic in indonesia using mobile-health
[cited 2024 Oct 28]. Available from: https://www. interactive message. Ethiop J Health Sci 2022;32(2):243-
kominfo.go.id/berita/pengumuman/detail/mengakses- 54. https://doi.org/10.4314/ejhs.v32i2.4
pelayanan-dalam-genggaman-melalui-jkn-mobile/. 14. Kushendriawan MA, Santoso HB, Putra PO, Putra PO,
3. Bahri S, Amri A, Siregar AA. [Analysis of service qual- Schrepp M. Evaluating user experience of a mobile
ity of the mobile JKN application by the social health health application halodoc using user experience ques-
insurance organizer using the service quality (Servqual) tionnaire and usability testing. J Sist Inf 2021;17(1):58-
method]. Ind Eng J 2022;11(2):1-7. 71. https://doi.org/10.21609/jsi.v17i1.1063
4. Nielsen J. Usability 101: introduction to usability [Inter- 15. Islam MN, Karim MM, Inan TT, Islam AK. Investigating
net]. Dover (DE): Nielsen Norman Group; 2012 [cited usability of mobile health applications in Bangladesh.
at 2023 Jul 19]. Available from: https://www.nngroup. BMC Med Inform Decis Mak 2020;20(1):19. https://doi.
com/articles/usability-101-introduction-to-usability/. org/10.1186/s12911-020-1033-3
5. Lewis JR. The system usability scale: past, present, and 16. Bangor A, Kortum P, Miller J. Determining what in-
future. Int J Hum Comput Interact 2018;34(7):577-90. dividual SUS scores mean: adding an adjective rating
https://doi.org/10.1080/10447318.2018.1455307 scale. J Usabil Stud 2009;4(3):114-23.
6. Muro-Culebras A, Escriche-Escuder A, Martin-Martin 17. Santoso HB, Schrepp M, Isal RY, Utomo AY, Priyogi B.
J, Roldan-Jimenez C, De-Torres I, Ruiz-Munoz M, et al. Measuring user experience of the student-centered e-
Tools for evaluating the content, efficacy, and usability learning environment. J Educ Online 2016;13(1):58-79.
of mobile health apps according to the consensus-based 18. Laugwitz B, Held T, Schrepp M. Construction and eval-
standards for the selection of health measurement in- uation of a user experience questionnaire. In: Holzinger
struments: systematic review. JMIR Mhealth Uhealth A, editor. HCI and usability for education and work.
2021;9(12):e15433. https://doi.org/10.2196/15433 Heidelberg, Germany: Springer, 2008. p. 63-7. https://
7. Hajesmaeel-Gohari S, Khordastan F, Fatehi F, Samzadeh doi.org/10.1007/978-3-540-89350-9_6
H, Bahaadinbeigy K. The most used questionnaires for 19. Lewis JR, Sauro J. Item benchmarks for the system us-
evaluating satisfaction, usability, acceptance, and qual- ability scale. J Usabil Stud 2018;13(3):158-67.
ity outcomes of mobile health. BMC Med Inform Decis 20. Parera G, Darwis M. Usability of JKN mobile applica-
Mak 2022;22(1):22. https://doi.org/10.1186/s12911-022- tion using system usability scale (SUS) method. J Bit-
01764-2 Tech 2023;5(3):138–45. https://doi.org/10.32877/bt.v5i3.
8. Brooke J. SUS: a retrospective. J Usabil Stud 2013;8(2): 691
29-40. 21. Lubs BO, Salim A, Jefi. [Evaluation of usability of the
9. Weichbroth P. Usability of mobile applications: a sys- mobile JKN application system using the use question-
tematic literature study. IEEE Access 2020;8:55563-77. naire]. J Saintekom 2020;10(1):65-76.
https://doi.org/10.1109/ACCESS.2020.2981892 22. Annisa N, Pradana DS, Suharso W. [Evaluation of the
10. Lobo EH, Abdelrazek M, Frolich A, Rasmussen LJ, Liv- National Health Insurance (JKN) mobile application in
ingston PM, Islam SM, et al. Detecting user experience Malang City from the usability aspect]. J Repos 2020;
issues from mHealth apps that support stroke caregiver 2(12):1689-99. https://doi.org/10.22219/repositor.v2i12.
needs: an analysis of user reviews. Front Public Health 30999
2023;11:1027667. https://doi.org/10.3389/fpubh.2023. 23. Wulandari R, Pangarsa EA, Andono PN, Rachmani E,
1027667 Sidiq M, Setyowati M, et al. Evaluation of usability and
11. Hornbæk K, Hertzum M. Technology acceptance and user experience of Oncodoc’s M-health application for
user experience: a review of the experiential component early detection of cancer. Asian Pac J Cancer Prev 2022;
in HCI. ACM Transact Comput Hum Interact 2017; 23(12):4169-4176. https://doi.org/10.31557/APJCP.
24(5):1-30. https://doi.org/10.1145/3127358 2022.23.12.4169
12. Sharp H, Rogers Y, Preece J. Interaction design: beyond 24. Iskandarsyah A, Shabrina A, Siswadi AG. Usability and
human-computer interaction. 2nd ed. Chichester, UK: acceptability of JAGA SEHAT: mobile application to
John Wiley & Sons; 2007. improve knowledge about healthy lifestyle. J Multidiscip

Vol. 30 • No. 4 • October 2024 www.e-hir.org 331

Savira Putri Handasari et al

Healthc 2022;15:115-24. https://doi.org/10.2147/JMDH. Marchi AC. Assessing long-term user experience on a

S342913 mobile health application through an in-app embedded
25. Dyayu AL, Beny YH, Yani H. [Evaluation of the us- conversation-based questionnaire. Comput Hum Behav
ability of the PeduliLindungi application using usability 2020;104:106169. https://doi.org/10.1016/j.chb.2019.
testing and the system usability scale (SUS) method]. J 106169
Manaj Teknol Dan Sist Inf 2023;3(1):395-404. https:// 29. Pal S, Biswas B, Gupta R, Kumar A, Gupta S. Exploring
doi.org/10.33998/jms.2023.3.1.720 the factors that affect user experience in mobile-health
26. Oakley-Girvan I, Yunis R, Fonda SJ, Neeman E, Liu R, applications: a text-mining and machine-learning ap-
Aghaee S, et al. A novel smartphone application for the proach. J Bus Res 2023;156:113484. https://doi.org/10.
informal caregivers of cancer patients: usability study. 1016/j.jbusres.2022.113484
PLOS Digit Health 2023;2(3):e0000173. https://doi.org/ 30. Ramachandran A, Snehalatha C, Ram J, Selvam S, Si-
10.1371/journal.pdig.0000173 mon M, Nanditha A, et al. Effectiveness of mobile phone
27. Hassenzahl M. The effect of perceived hedonic quality messaging in prevention of type 2 diabetes by lifestyle
on product appealingness. Int J Hum Comput Interact modification in men in India: a prospective, parallel-
2001;13(4):481-99. https://doi.org/10.1207/S15327590I- group, randomised controlled trial. Lancet Diabetes
JHC1304_07 Endocrinol 2013;1(3):191-8. https://doi.org/10.1016/
28. Biduski D, Bellei EA, Rodriguez JP, Zaina LA, De S2213-8587(13)70067-6

332 www.e-hir.org https://doi.org/10.4258/hir.2024.30.4.324