Analyzing Gamification of “Duolingo”

with Focus on Its Course Structure

Duy Huynh, Long Zuo, and Hiroyuki Iida(B)

School of Information Science, Japan Advanced Institute of Science and Technology,

1-1 Asahidai, Nomi, Ishikawa 923-1211, Japan
{duyhuynh,zuolong,iida}@jaist.ac.jp

Abstract. Gamification is the application of game-based elements and

game design techniques in non-game contexts. Many learning plat-
forms have applied gamification to increase motivation and engagement.
Duolingo is a popular language learning platform which has applied
gamification. In this paper, game refinement measure is employed to
evaluate the gamification of Duolingo. The results show that the range
of game refinement value of Duolingo is reasonable in such as serious
environment. By assuming a milestone in a language course as a sub-
game, it is supposed that the challenges in each milestone could adapt
the advancement of learners’ skill.

Keywords: Duolingo · Gamification · Language learning platform ·

Game refinement theory · Attractiveness

1 Introduction
Gamification is a term that refers to the use of game-based elements such as
mechanics, aesthetics, and game thinking in non-game contexts aimed at engag-
ing people, motivating action, enhancing learning and solving problem [4,11].
The benefit of game and game-based approaches in education has been investi-
gated since 1980s [7,12,16]. In recent years, there is a growing interest in gamifi-
cation as well as its applications and implications in the field of education since
it provides an alternative to engage and motivate student during the process of
learning. While gamification is gaining ground in some fields such as business,
marketing, management and wellness initiatives, its application in education is
still an emerging trend. In this study, we aim to investigate the attractiveness
of Duolingo, which is one of the most popular language learning platforms by
applying game refinement theory. Due to the lack of research on the gamification
in the education domain, numerous questions arise as to clarify how gamification
can be used and how it benefits us the most. Therefore, the research question
Duolingo R
and all other trademarks, service marks, graphics and logos used in
connection with the Service are trademarks or service marks of Duolingo or their
respective owners, and certain of them are registered with the United States Patent
and Trademark Office.

c Springer International Publishing AG 2016
R. Bottino et al. (Eds.): GALA 2016, LNCS 10056, pp. 268–277, 2016.
DOI: 10.1007/978-3-319-50182-6 24
 Analyzing Gamification of “Duolingo” with Focus on Its Course Structure 269

for this study is “How do game elements make an effect when applying it into an
education situation?”. To answer this question, we try to quantify the attractive-
ness of the language course based on the game refinement theory. Specifically,
we analyze the game refinement value zone of Duolingo.
Game refinement theory has been proposed earlier by Iida et al.. [10] to
measure the attractiveness and sophistication of games under consideration.
A game refinement measure is derived from a game information progress model
and has been applied in various games. Classical game theory concerns the opti-
mal strategy from the viewpoint of players, whereas game refinement theory
concerns the optimization from the game developer’s point of view. The game
refinement theory was successfully applied in the domain of board games [10],
for Mah Jong [9], and sports games [17]. In fact, there are many challenging
questions, especially applying this theory to gamification of serious games.
This paper is organized as follows. Section 2 presents the previous studies in
gamification in education and a short sketch of free language-learning platform-
Duolingo which is our case study in this paper. In Sect. 3, we present the basic
idea of game refinement theory and show our analysis using this way. In Sect. 4,
we discuss about our experiment and a structure of Duolingo language course.
Finally, Sect. 5 gives concluding remarks including some future works.

2 Gamification and Education

In this section, we first show the previous study of gamification in education.

Then, we give a short sketch of the gamified free language-learning platform
Duolingo.

2.1 Related Works

We review literatures related to the use of gamification in education. Although

some researchers are working on it, there currently is still few works on this
subject. Gamification, as a term, was mentioned for the first time in 2008. There
are works which have been done, and many papers are written on this topic [1].
Unlike game-based learning which is clearly a game, the essence of gamification
is that it occurs in a non-game context; therefore, it would be applied in such a
way that would not change the existing practice of learning, and instead focus on
making it more engaging and challenging for student [15]. There are increasing
number of case studies and researches dealing with gamification in general [8]
and in education contexts [2].
Besides, some successful gamification applications, many studies presented
a gamification mechanic and its effect when applied into the system. Muntean
[13] has shown a theoretical analysis of gamification as a tool to engage users in
e-learning platforms. In Fogg’s Behaviour Model, gamification mechanics is used
to motivate and trigger desired behaviours on learners. He has provided a list
of gamification elements and explained how they could apply into an e-learning
course. Recently, Scepanovic et al. [15] also give a discussion and evaluation of
270 D. Huynh et al.

gamification mechanics based on literature review and analysis of implementing

gamification concepts in Higher Education learning.
While majority of studies report overall positive results of applying game
elements and mechanics into the system, and their effects in motivating and
engaging learners. Our study aims at analyzing and measuring the attractive-
ness of game elements and its effect when combined with a language course’s
structure. By applying game refinement theory as an assessment method, we
could figure out specific factors, which directly make an effect on a platform,
and a degree of their effectiveness. By those factors mentioned above, we could
adjust them to increase their attractiveness and their making motivation in a
platform. Moreover, we could compare the impact between the elements based
on a degree of their attractiveness to list elements, which make the most effec-
tiveness in a platform, and enrich them. Those are different points of this study
compared to previous works. This contribution will lead to a better understand-
ing of the effects of gamification in learning domain.

2.2 Duolingo

Duolingo is a gamified free language-learning platform created by professor

Luis Von Ahn and his graduate student Severin Hacker. The system is designed
so that users could learn languages, while helping Duolingo to translate doc-
uments. Duolingo became publicly available in 2012 with more than 300000
users. As of April 2016, it offers 59 different language courses across 23 languages
[5]. The authors of Duolingo skillfully apply gamification into their system in
order to engage and motivate their users. They have used some game-elements
in their system. They are shown as follows [6]:

– Reward: In Duolingo, lingot is a reward. Users can easily collect dozen of

lingots by completing their lesson or skill.
– Leader-board: which shows users how they are doing compared to their friends,
these are some great motivators to help the users coming back and learning.
– Level-system: providing a way to track users daily activity and compete
with their friends, XP (experience point) also determines users language
“level”, which is displayed on their profile page and above their comments in
discussions.
– Badges: Duolingo has some achievement tokens which are the skills a user
has gained. They are displayed on user’s profile. Badges is an excellent tool
for making people feel invested in their study.

Although Duolingo has used many game elements in their platform, we specifi-
cally analyze only “badges” in this study because the important part of successful
gamification is content and learning material [15]. Moreover, a main purpose of
learners in Duolingo is learning a second language, while badges is used to lift
up the motivation of learners when they study. Hence, “badges” is an important
element in the platform. The game element “badges” is combined in harmony
with a learning content to construct the main structure of a language course.
 Analyzing Gamification of “Duolingo” with Focus on Its Course Structure 271

The structure of a language course includes some elements as follows. The core
element in a course is its lessons. The lesson is well-designed, drilling skills of
user with several different kinds of challenges. They do not get too repetitive,
and they are just easy and fun to do [6]. They are categorized into small sets
which are called skills by part of speech or lesson vocabulary meaning such as:
verb, adjective, sport, food, etc. Each skill has a strength bar, which will be full
only when users have passed all lessons in the skill. However, the strength will be
decreased overtime to represent “words fading from memory”. At the beginning,
only basic skill is available, other skills are locked. To unlock other skills, users
have to complete all their available skills. The skills in a skill-tree are split into
check-points which represent for stages or the milestones of user’s study process.
Although there is no reward given when users reach each milestone, we could
see the expression of milestones in the skill-tree after users have completed the
number of skills.

3 Analyzing Gamification of Duolingo

This section presents a basic idea of game refinement theory and a game progress
model is figured out with a focus on the course’s structure of Duolingo to
quantify its attractiveness.

3.1 Assessment Method

We show a short sketch of the basic idea of game refinement theory [18]. The
‘game progress’ is twofold. One is a game speed or scoring rate, and another one is
game information progress with a focus on the game outcome. Game information
progress represents how certain is the result of the game in certain time or steps.
Having full information of the game progress, i.e. after its conclusion, game
progress x(t) will be given as a linear function of time t with 0 ≤ t ≤ tk and
0 ≤ x(t) ≤ x(tk ), as shown in Eq. (1).
x(tk )
x(t) = t (1)
tk
However, the game information progress given by Eq. (1) is unknown during
the in-game period. The presence of uncertainty during the game, often until
the final moments of a game, reasonably renders game progress as exponential.
Hence, a realistic model of game information progress is given by Eq. (2).
t n
x(t) = x(tk )( ) (2)
tk
In Eq. (2), n stands for a constant parameter which is given based on the perspec-
tive of an observer of the game. Then acceleration of game information progress
could be obtained by deriving Eq. (2) twice. Solving it at t = tk , we have Eq. (3).
x(tk ) x(tk )
x (tk ) = (tk )n−2 n(n − 1) = n(n − 1) (3)
(tk )n (tk )2
272 D. Huynh et al.

It is assumed in the current model that game information progress in any type of
game is encoded and transported in our brain. We do not yet know the mecha-
nism of information processing in the brain, but it is likely that the acceleration
of information progress is subject to the forces and laws of physics. Therefore,
we expect that the larger the value x(t k)
(tk )2 is, the more exciting the game becomes.
√
x(t )
Thus, we compute its root square, tk k , and use the result as a game refine-
ment measure for the considered game. We assign it as R-value, which is short
shown in Eq. (4). 
x(tk )
R= (4)
tk
The brief results of previous works [14] are shown in Table 1. According to
Table 1, we propose a suitable approach for various game types by using the
described model. We see that sophisticated games have R-value in the appropri-
ate range, 0.07–0.08. It has been proposed that information acceleration is one
of the direct factors of engagement for all game types.

Table 1. Measures of game refinement for boardgames and sports

Game R-value
Chess 0.074
Go 0.076
Basketball 0.073
Soccer 0.073

3.2 Game Progress Model in Duolingo

Duolingo is different from sports and boardgames. It is gamification product,
which means that they use game elements to create enjoyment points in learn-
ing environment. Therefore, in this study, we highlight a gamification structure
in each language course as the main aspect. As we mentioned in Sect. 2, the
structure of language course is constructed by some core elements such as lesson
and skill. Furthermore, following the basic idea of game refinement theory, game
progress is twofold, known as goal and time or steps to archive the goal in sports
and boardgames [17]. In a language course, the goal of learners is to complete
their study by getting all badges in the skill-tree. To archive a badge, users
must complete all lessons in a skill. Therefore, the game progress of Duolingo
language course can be measured by skills.
Let S and L be the average number of skills and the average number of lessons
in the same language courses, respectively. If one knows the game information
progress, for example after the game, the game progress x(t) is given by Eq. (5).

S
x(t) = t (5)
L
 Analyzing Gamification of “Duolingo” with Focus on Its Course Structure 273

However, the game information progress given by Eq. (5) is unknown during
the in-game period. Hence, a realistic model of game information progress of
Duolingo language course is given by Eq. (6).
t
x(t) = S( )n (6)
L
Then acceleration of game information progress is obtained by deriving Eq. (6)
twice. Solving it with t = L, we have Eq. (7).

Sn(n − 1) n−2 S
x (L) = L = 2 n(n − 1) (7)
Ln L
Therefore, the game refinement value of a language course in Duolingo can be
described as Eq. (8). √
S
R= (8)
L

4 Data Collection and Discussion

This section presents a gamification analysis of Duolingo using game refine-
ment measure and provides some discussions.

4.1 Data Collection

Duolingo has 120 million users around the world and currently teaches 19 dis-
tinct languages. The most popular courses are available for speakers of a variety
of languages: for example, we can learn English from 21 different languages [19].
In order to make the data more objective and reasonable, we collect the statistics
by observing a structure of each popular language courses. On the Duolingo’s
website, we go to collect the information of completed language courses and
group them by popular learning languages, as shown in Table 2. In each lan-
guage course, we could get the number of skills by counting a number of badges,

Table 2. Popular languages in Duolingo

Language Number of courses Total number of enrolment

English 21 181,412,000
Spanish 5 66,199,700
French 6 45,724,000
German 6 28,083,200
Italian 3 18,483,000
Portuguese 2 9,870,000
Others 1 11,532,000
274 D. Huynh et al.

which are displayed on a homepage. Besides, we also get the number of lessons
in a skill by adding all number of lessons under each badge icon.
After the data collection step, we calculate game refinement values. For
instance, to calculate R-value of English courses, we have to find the average
number of lesson L and the average number of skills S of 21 English courses. As
a result, R-value were calculated by Eq. (8), which is shown in Table 3.

Table 3. Measures of game refinement for Duolingo language course

Language S L R-value
English 55.619 291.958 0.0259
Spanish 64.142 319.571 0.0250
French 72.222 346.333 0.0243
German 89 381.25 0.0243
Portuguese 68 379 0.0204
Italian 66 385 0.0200

4.2 Discussion

In the previous studies (see Table 1), R-values of sophisticated games like sports
and boardgames often fall into the range between 0.07 and 0.08. However, we
noticed that the results of Duolingo show much lower. Here we take English
language as an example, there are 55.6 skills and 291.9 lessons on average, so
the R-value is 0.0259. To achieve 55.6 skills, learners must complete more than
291 lessons. Moreover, R-value of Portuguese is lower than English for the aver-
age number of skills and lessons are 68 and 379. According to Eq. (8), R-value
increases when the number of lessons decreases, which means that the goal is
easy to achieve. This is similar to increasing the number of skills. R-value falls
into the range between 0.020 and 0.025, which shows that the “game” in lan-
guage course will be too challenging. As we introduced, Duolingo is a learning
platform, which means that it is a serious environment and game elements are
used to increase motivation and engagement of learners, they are not used to
make a course become entertaining or relaxing as fun games. Hence, the R-value
range is reasonable.
However, with the degree of challenging which is indicated by R-value, the
“game” in language course only increases the motivation for advanced users or
who learned with a purpose. With novice users or nonnative-language learners,
they give up easily their study. There is a reason why Duolingo authors have
applied a creating milestones technique, as mentioned at the end of Sect. 2, to
respond learner’s efforts. Like as a game, the skills in course are structured so
that learners may have various “levels” of goals. Generally, the requirements of
each “level” of goal get increasingly harder from completing the initial tasks
until completing the course. This allows learners to learn and practice skills.
 Analyzing Gamification of “Duolingo” with Focus on Its Course Structure 275

We made a brief analysis of the most popular language course “English for
Spanish speakers” (EFSS) to see the obvious effect of creating “milestone” tech-
nique. We assume that each milestone in a course is a sub-game. Next, we calcu-
late R-value in each sub-game in a course, which is shown in Table 4. The R-value
of each milestone shows that the milestone is designed for various types of learn-
ers. For instance, in the first milestone, R-value is 0.081. This value is higher
than the results of the sophisticated sports and boardgames, which implies that
the “game” in the first milestone is so exciting and attractive for beginners. The
increase in the requirement is to give more challenging and exciting to learners
since their skill gets better at every milestone. To prevent the course from getting
bored, the number of lessons and skills should be lower, then the performance of
learners would be higher. Moreover, after going through many challenges, learn-
ers need enjoyments in order to avoid the drop-out from the course. Therefore,
in later milestones, the R-value tends to increase to be a higher value such as
0.094, which is shown in Table 4.

Table 4. Measures of game refinement for each milestone in EFSS course

Milestone S L R-value
1 10 39 0.081
2 12 56 0.062
3 15 97 0.039
4 14 79 0.047
5 8 30 0.094

Fig. 1. Game refinement trends when starting from Milestone 1 and Milestone 3

We collected the data of each milestone to quantify the R-value. In general,

users usually start from the first milestone and accomplish the final goal. Thus,
the trend of R-value for normal players is depicted in Fig. 1. We noticed that the
276 D. Huynh et al.

R-value goes down sharply and maintaining the low value after the 4th milestone.
However, for some advanced users, they can start from the third milestone by
ignoring the first and the second ones. In that case, we also calculate the R-
value for the players who start at the third milestone. An advanced user who
starts from the third milestone would not feel entertainment, for the game is
maintaining the low value.

5 Concluding Remarks
Duolingo is a huge system, we could not analyze all of elements at the same
time. In this study, we have focused on a “badge” element, which has been
used in the skill-tree to analyze the attractiveness. Badges is an important game
element, which is used to boot the motivation of learners when they study. As
a result, data analysis has shown that the game refinement value falls into the
range between 0.020 and 0.025, which is much lower than other sophisticated fun
games. It is reasonable because Duolingo is used in the serious environment
or non-game context. We also quantified the attractiveness of each milestone,
and made a comparison between the players who start at the first and third
milestone. It is assumed in this study that every milestone in a course is a sub-
game. After the brief analyzing, the game refinement value indicates that the
challenges in each milestone could adapt the advancement of learners’ skill. The
result shows that Duolingo is enjoyable for new comers who start from the
first milestone, however, less enjoyable for advanced users who start at the third
milestone. In this case, it is essential to increase the degree of gamification of
Duolingo by decreasing the number of lessons or increasing the number of skills
in the target courses.
Future works will focus on the analysis of the other game elements which have
been applied into Duolingo, and perform more experiments to understand the
effects of “milestone” technique. Activity theory-based model of serious game
(ATMSG) [3] is known as an analyzing framework, which helps users under-
stand in depth the roles of each piece in each happened actions of a game. We
will consider to use an ATMSG to figure out the used game elements in the
considered game, and understand their goals before measuring game refinement
values. Additionally, we will apply game refinement theory in more learning plat-
forms to find the most effective structure from the perspective of gamification.

Acknowledgments. The authors wish to thank the anonymous referees for their
constructive comments that helped to improve the article considerably. This research
is funded by a grant from the Japan Society for the Promotion of Science (JSPS),
within the framework of the Grant-in-Aid for Challenging Exploratory Research (grant
number 26540189).
 Analyzing Gamification of “Duolingo” with Focus on Its Course Structure 277

References
1. A brief history of gamification: part III the definitions. http://edulearning2.
blogspot.jp/2014/03/a-brief-history-of-gamification-part.html
2. Barata, G., Gama, S., Jorge, J., Gonalves, D.: Engaging engineering students with
gamification. In: 5th International Conference Games and Virtual Worlds for Seri-
ous Applications (VS-GAMES) (2013)
3. Carvalho, M.B., Bellotti, F., Berta, R., De Gloria, A., Islas Sedano, C., Baalsrud
Hauge, J., et al.: An activity theory-based model for serious games analysis and
conceptual design. Comput. Educ. 87, 166–181 (2015)
4. Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From game design elements to
gamefulness: defining “gamification”. In: 15th International Academic MindTrek
Conference: Envisioning Future Media Environments, pp. 9–15 (2011)
5. Duolingo. https://en.wikipedia.org/wiki/Duolingo
6. Gamification Teardown: Duolingo. https://blog.captainup.com/how-gamification-
makes-language-learning-even-more-awesome-on-duolingo/
7. Gee, J.P.: What Video Games Have to Teach About Learning and Literacy.
Palgrave Macmillan, New York (2003)
8. Hamari, J., Koivisto, J., Sarsa, H.: Does gamification work? A literature review
of empirical studies on gamification. In: 47th Hawaii International Conference on
System Sciences, Hawaii, USA (2014)
9. Iida, H., Takahara, K., Nagashima, J., Kajihara, Y., Hashimoto, T.: An appli-
cation of game-refinement theory to Mah Jong. In: Rauterberg, M. (ed.) ICEC
2004. LNCS, vol. 3166, pp. 333–338. Springer, Heidelberg (2004). doi:10.1007/
978-3-540-28643-1 41
10. Iida, H., Takeshita, N., Yoshimura, J.: A metric for entertainment of boardgames:
its implication for evolution of chess variants. In: Nakatsu, R., Hoshino, J. (eds.)
Entertainment Computing. ITIFIP, vol. 112, pp. 65–72. Springer, Heidelberg
(2003). doi:10.1007/978-0-387-35660-0 8
11. Kapp, K.M.: The Gamification of Learning and Instruction: Game-Based Methods
and Strategies for Training and Education. Pfeiffer, San Francisco (2012)
12. Klopfer, E.: Augmented Learning: Research and Design of Mobile Educational
Games. MIT Press, Cambridge (2008)
13. Muntean, C.I.: Raising engagement in e-learning through gamification. In: 6th
International Conference on Virtual Learning (ICVL2012), pp. 323–329 (2011)
14. Nossal, N., Iida, H.: Game refinement theory and its application to score limit
games. 2014 IEEE Games Media Entertain. (GEM), 1–3 (2014). doi:10.1109/GEM.
2014.7048120
15. Scepanovic, S., Zaric, N., Matijevic, T.: Gamification in higher education learning -
state of the art, challenges and opportunities. In: Serbia: International Conference
on e-Learning (2015)
16. Shaffer, D.W.: How Computer Games Help Children Learn. Palgrave Macmillan,
New York (2006)
17. Sutiono, A.P., Purwarianti, A., Iida, H.: A mathematical model of game refinement.
In: Reidsma, D., Choi, I., Bargar, R. (eds.) INTETAIN 2014. LNICST, vol. 136,
pp. 148–151. Springer, Heidelberg (2014). doi:10.1007/978-3-319-08189-2 22
18. Takeuchi, J., Ramadan, R., Iida, H.: Game refinement theory and its application
to Volleyball. Research report 2014-GI-31(3), Information Processing Society of
Japan, pp. 1–6 (2014)
19. Which countries study which languages, and what can we learn from it?
http://making.duolingo.com/which-countries-study-which-languages-and-what-
can-we-learn-from-it