Educational Games – Are They Worth The Effort?

A literature survey of the effectiveness of serious games

Per Backlund Maurice Hendrix

Interaction Lab Serious Games Institute
University of Skövde Coventry University
Skövde, Sweden Coventry, UK
[email protected] [email protected]

Abstract— Over the last decade educational games have evaluations in the field of serious games and the involvement
become more and more popular. There are many games of the evaluator as a stakeholder in the game development
specifically designed as educational games, as well as a number of process.
entertainment games that have been successfully used for
educational purposes. The EduGameLab project aims to This paper focuses on the sub-group of educational games
stimulate the use of games in the classroom. This paper presents in order to delimit our survey. As Tobias et al. [1] state, no
a meta-analysis of the effectiveness of game-based learning and survey of game research can list every study that has been
focuses specifically on empirical evidence on the effectiveness of made. The study was carried out in the frame of the
using games in education in the last decade. Moreover, the study EduGameLab, which aims to stimulate the use of serious
focuses on usage in formalized school contexts, i.e. pre-school, games in the classroom. Based on the focus of the project, we
elementary school, secondary school, high school and higher focus our study on studies published in the last decade
education. As secondary aims we also assess whether there are (between 2002 and 2012) and studies that have empirically
any clear methodological trends and whether a link could be evaluated the learning effect in some way. We also limit our
found between the outcome of empirical studies and the study to games used in formalized school contexts, i.e. pre-
evaluator being a stakeholder in the game development. school, elementary school, secondary school, high school and
higher education. In order to limit our study further, we have
Keywords—games; education; literature survey; effectiveness
decided to only include papers published in scientific journals.

I. INTRODUCTION Furthermore, Tobias et al. [1] argue that the superiority of

games for instruction has not been established. We attribute
In recent years, educational games have become more and this to different reasons. When dealing with educational games
more prevalent (e.g. [1]). There is an abundance of specifically from both a practitioner’s view point as well as from a
designed educational games [1], [2]. There are also many scholarly view point we find a number of challenges. One
entertainment games that have been used for training or major challenge is to prove their effectiveness and efficiency as
education [3]. Educational games are serious games educational tools. The issue of effectiveness refers to how well
specifically used for education. Serious games, in turn, is a we can isolate and measure the actual learning effect form
concept with numerous definitions. In a broad sense the term games. This is an important matter as we wish to better
refers to the idea of using games (specifically designed ones as understand if and how games are effective as pedagogical tools
well as entertainment games) for purposes beyond pure beyond accidental learning. However, even if we manage to
entertainment. identify a number of studies that show the effectiveness there is
The aim of this paper is to make a meta-analysis of still the issue of practical use in teaching. In this aspect there
scientific studies on the educational effectiveness of games, to are even further problems to be solved, spanning user
reveal what types of studies there are and what they conclude. acceptance (from teachers, students and parents), technological
With educational games being widely used, there is a clear restrictions, as well as questions concerning curricula and
need to analyse how the effectiveness of game-based learning content [3].
has been studied and how the results from such studies can
inform practitioners. From a practitioner’s point of view there II. BACKGROUND AND RELATED RESEARCH
is a need, not only to establish the usefulness and effectiveness
There are numerous definitions of computer games and
of educational games, but also for useful input for the effective
there are also widely used general conceptions. Salen and
development and use of educational games in practice. A
Zimmerman [4] summarize eight different definitions focusing
secondary aim is to provide an overview of what types of
on different elements and arrive at a quite condensed
studies there are that evaluate educational games and if there
definition: “A game is a system in which players engage in an
are any particular methodological trends in the area of serious
artificial conflict, defined by rules, that results in a quantifiable
games research. Finally our study identifies whether there is
outcome.” The definition is narrow, in particular with respect
any evidence of a link between the outcomes of empirical
to the restriction to quantifiable outcome. In many occasions a
This work was fully supported by the EduGameLab project which is
funded by the Life Long Learning programme by the European Commission.

978-1-4799-0965-0/13/$31.00 ©2013 IEEE

Authorized licensed use limited to: NATIONAL UNIVERSITY OF IRELAND GALWAY. Downloaded on October 22,2024 at 18:41:54 UTC from IEEE Xplore. Restrictions apply.
 more open characterization, such as the one provided by • Facilitating performance & learning in various topics.
Prensky [5], including the elements rules, goals and objectives,
outcomes and feedback, conflict, competition, challenge, • Uses of games in instructional situations.
opposition, interaction, and representation or story is useful. • Effects of playing games on school learning.
The concept of cooperation is also an important aspect of
games and playing games and indeed there are many games • Effects on aggression, hostility and motivation.
that do not focus on competitive elements or winning. Some
• Attitude change.
examples are simulation games such as The Sims
(thesims.com) or the open sandbox of Minecraft Tobias et al. [1] conclude that the research reviewed
(minecraft.net). indicate a promise for educational games. However, these
promises are deemed as tentative as they identified a number of
Serious games, which we perceive as an umbrella term,
areas where there is still a need for further research and
include different types of educational games as well as games
theoretical development. Tobias et al [1], conclude that “there
for other purposes such as: training, rehabilitation, marketing
is considerably more enthusiasm for describing the affordance
and social improvement. The term serious games has been
of games and their motivating properties than for conducting
defined by Zyda [6] as follows: “Serious game: a mental
research to demonstrate that these affordances are used to attain
contest, played with a computer in accordance with specific
instructional aims, or to resolve problems found in prior
rules, that uses entertainment to further government or
research.” One interpretation of this critique is the will within
corporate training, education, health, public policy, and
the community to develop and evaluate prototypes rather than
strategic communication objectives.” This definition is fairly
spending more efforts on the actual usage in teaching.
inclusive but its focus on entertainment sometimes clash with
what is marketed as serious games. Marsh [7] defines serious Egenfeldt-Nielsen [9] presents an overview of the
games in terms of a continuum between games for purpose and educational use of computer games by examining their
experiential environments for purpose. As an example, many underlying learning theories. There is a lack of knowledge
serious game applications are utilizing the technologies about the implications of using games in educational contexts
typically associated with computer games rather than the game as this particular area has its unique problems in terms of
play component. These applications are referred to as virtual methods, focus, and relevant research questions. Egenfeldt-
environments and digital media with no traditional gaming Nielsen [9] points out that the different learning approaches
characteristics [7]. For our work, we define serious games as (behaviourism, cognitivism, constructionism and socio-
games that engage the user, and contribute to the achievement cultural) all have something to offer in the realm of educational
of a defined purpose other than pure entertainment (irrespective game design and there are indeed titles to be found that
of whether the user is consciously aware of this). A game’s represent all of them. Hence, there is no silver bullet and the
purpose may be formulated by the user her/himself or by the field of educational games is not homogenous.
game’s designer, which means that also a commercial off-the-
shelf (COTS) game, used for non-entertainment purposes, may Hays [8] presents a review of 48 empirical research articles,
be considered a serious game. It is worth noticing that the published between 1982 and 2005, on the effectiveness of
utilization of game and visualization technologies, simulations instructional games and concludes that empirical studies of the
and virtual worlds for purposes beyond entertainment can be instructional effectiveness of games are fragmented and not
included under this definition as well. While this definition always methodologically sound. Furthermore, there is no
includes non-digital games, it should be noticed that most evidence that games are the preferred instructional method in
references to serious games are in fact to digital serious games. all situations. This implies that the teaching context and the
pedagogical activities surrounding the game are important.
A number of recent reviews of computer game research
have been carried out. As the topic has become more and more Vogel et al. [2] present a meta-analysis of computer games
popular the number of studies is increasing and as indicated by and simulations for learning. They claim that it is difficult to
Tobias et al. [1] it is no longer possible to carry out an determine the nature of the relationship between games and
extensive review covering the whole area. By necessity, learning as there is no common ground with respect to what
reviews will have to focus on different topics and employ skills and domains are analysed. To remedy this, they focus
inclusion criteria. We have identified some recent reviews by their review on studies on the gain of cognitive skills or
Hays [8], Vogel et al. [2], Egenfeldt-Nielsen [9], Ke [10] and attitudinal change. Furthermore, they only include studies that
Tobias et al. [1] which we shortly introduce below. report statistics, assessing the difference between traditional
classroom teaching and computer gaming or interactive
Tobias et al. [1] present an extensive review of research on simulation teaching. This shows the diversity of the field.
computer games, which examines research evidence for the
effectiveness of games for instruction. 95 studies are According to Ke [10] empirical research on instructional
categorized into groups with respect to their intention and games is fragmented. Furthermore, Ke [10], referring to
knowledge claim. The following categories are identified: Dempsey et al. [11] indicates that much of the evaluation of
games has been anecdotal, descriptive, or judgmental. Ke [10]
• Transfer of knowledge, skills and attitudes from presents an analysis of which methods have been used and
games to real world tasks. what the results of these empirical studies are. Interestingly, no
analysis is made of the evaluator, i.e. whether the evaluator is
• Effects on cognitive processes such as visual independent or a stakeholder in the development of the game.
attention, spatial visualization and problem solving.

Authorized licensed use limited to: NATIONAL UNIVERSITY OF IRELAND GALWAY. Downloaded on October 22,2024 at 18:41:54 UTC from IEEE Xplore. Restrictions apply.
 III. METHOD relationship between video games, violence, aggression, and
No survey of game research can list every study that has social behaviour. Finally, the papers selected were divided
been made as they have increased dramatically over the last between the authors to be summarized.
couple of years and the field has evolved rapidly. Hence, in
order to give an accurate overview of the current state of the IV. RESULTS OF THE LITERATURE SURVEY
research, we limit our survey with respect to a number of The final selection of papers included 40 papers covering
criteria. This review has three major limitations: the following topics: mathematics (including conceptual
1. It covers the time 2002 – first half of 2012. learning), cancer treatment, computer science, , bullying,
engineering, fire fighting, language, geography, history,
2. It focuses on studies that have empirically evaluated health, natural sciences, nutrition, physics, problem solving,
the learning effect in some way. social sciences, software development and surgery. The
3. It only includes papers published in scientific journals, classification is summarized in Table 1. The table also
hence excluding conferences. indicates the educational context, whether the evaluator was
independent or a stakeholder in the development of the game,
The literature search was carried out in May and June 2012 the evaluation method used and the result of the evaluation.
and the following computerized bibliographic databases and
search engines were selected as the most appropriate for the Mathematics lends itself well to game-based learning, at
topic: Academic search elite (Ebscohost), arXiv, LibHub, least if we judge by the number of studies (13/40 studies
Inspec, ScienceDirect, ERIC, PsyhINFO, SAGE Online, selected) in that particular field. However, the survey shows
Emerald, ACM Portal, SocINDEX, Google Scholar, Springer somewhat mixed results in terms of learning effect. 7/13
Link, IEEEXplore, CiteSeer and Scopus. evaluations indicate positive results for learning effect; 5/13 are
neutral and 1/13 is negative (at least for some groups). In
Only peer reviewed journals were included in the results. general, positive effects are found for motivational aspects.
We excluded conferences and unpublished reports in order to There seem to be a fondness for classroom experiments and
focus on empirical studies with well documented research trials in the realm of mathematics games, seven of the studies
approaches which we expect to find in journals. Furthermore, selected carried out some form of controlled experiment. One
we expect that studies of high quality that have been presented interesting exception, which tells us a bit more about the
at conferences will appear in elaborated versions in journals. broadness of the application of games and simulations in
The database searches were carried out using the following key mathematics teaching, is [12]. Kim and Chang [12] analysed
words, and combinations of them: empirical data from the US database called National
Assessment of Educational Progress (NAEP) on the effects of
• Video game
playing computer games on mathematics achievements for 4th
• Computer game graders. This database contains data about school achievements
in various subjects and stores data about study results as well as
• Education the frequency of computer game use in class as indicated by
• Training the teachers. The results are somewhat mixed with respect to
background and gender. English-speaking students who played
• Evaluation computer mathematics games in school every day displayed
significantly lower mathematics achievement than those who
• Empirical evaluation never played. On the other hand, positive effects of daily
• Effectiveness and Game based learning/training. computer use were noted among male students whose first
language was not English. Male language minority students
Key words were combined in order to reduce the number of who daily played computer games in mathematics
relevant hits. In particular, we looked for empirical studies with demonstrated higher mathematics performance compared to
some kind of effect measurements. The data bases were male English-speaking students who never played.
divided between the two authors and the searches yielded 120
hits that were deemed relevant based on an inspection of titles, Kim and Chang [12] conclude that the amount of time
keywords and abstracts. After sorting out duplicates between spent on playing mathematics games in class may be an
the different databases, 99 papers remained to be more important variable to consider. When male students played
carefully inspected. Each of these was independently inspected mathematics games every day they showed low mathematics
by the two authors, which resulted in the final selection of 40 performance. In contrast, when they played computer games
papers which the authors could agree on, i.e. both authors had with lower frequency in mathematics class, they outperformed
identified them as relevant to the review. The final selection male students who did not play computer games at all.
only included papers that presented empirical data on the Furthermore, according to Kim and Chang [12] the results
learning effect of computer games in formalized school should be interpreted with care as the study used a secondary
contexts, i.e. pre-school, elementary school, secondary school, database that records no details about which games were used.
high school and higher education. We excluded the military, Language learning is another popular topic taught with
business and vocational training domains as the many games in serious games with four out of 40 studies selected. Three of the
these fields would distract from our focus on formalised selected studies employ specifically designed language
classroom education. We also excluded studies about the learning games, whereas the last one utilises a

Authorized licensed use limited to: NATIONAL UNIVERSITY OF IRELAND GALWAY. Downloaded on October 22,2024 at 18:41:54 UTC from IEEE Xplore. Restrictions apply.
 TABLE I. SUMMARY OF PAPERS INCLUDED IN THE SURVEY

Author Educational context Evaluator Method Result Topic

Rub11 [13] Elementary Developer Mixed-method Positive Bullying
Kato08 [14] General Independent Experiment Positive Cancer treatment
Pap09 [15] Secondary School Developer Experiment Positive Computer Science
Sind09 [16] Higher Education Developer Experiment Neutral Computer Science
Ebn07 [18] Higher Education Developer Experiment Positive Engineering
Chu07 [19] Elementary Independent Experiment Positive Fire fighting
Vos11 [20] Elementary Independent Experiment Positive First language
Asa12 [21] Independent Experiment Positive Geography
Tüz09 [22] Elementary Independent Mixed-method Positive Geography
Vir05 [23] Elementary Developer Experiment Positive Geography
Tüz07 [24] Elementary Developer Mixed-method Unclear Health
Hui09 [25] Elementary Independent Quasi-experimental Positive History
Kenn11 [26] Higher Education Independent Single instance trial Positive History
Conn11 [27] Secondary School Developer Experiment Negative Language
Rou06 [17] Elementary Unclear Experiment Neutral Mathematics/conceptual
Cho11 [28] Higher Education Independent Case study Positive Mathematics
Kim10 [12] Elementary Independent Survey Negative Mathematics
Kab10 [29] Higher Education Developer Experiment Neutral Mathematics
Ke07 [30] Elementary Independent Experiment Positive Mathematics
Ke08 [31] Elementary Independent Mixed-method Neutral Mathematics
Kord11 [32] Elementary Developer Pilot-study Positive Mathematics
Lia11 [33] Elementary Developer Pilot-study Positive Mathematics
Main11 [34] Elementary Independent Pilot-study Positive Mathematics
Pan12 [35] Elementary Independent Experiment Neutral Mathematics
Sung08 [36] Pre-school Developer Experiment Positive Mathematics
Ros03 [37] Elementary Developer Experiment Neutral Mathematics
Wil06 [38] Elementary Developer Trial Positive Mathematics
Liu09 [39] Elementary Developer Quasi-xperimental Positive Natural Sciences
Wang08 [40] Elementary Developer Experiment Positive Natural Sciences
Mun08 [41] Elementary Developer Mixed-method Positive Nutrition
Rav02 [42] Secondary School Unclear Mixed-method Positive Physics
Hua10 [43] High School Developer Quasi-experimental Mixed Problem solving
Liu10 [44] Elementary Developer Quasi-experimental Positive Second language
Piir09 [45] Unclear Independent Qualitative Positive Second language
Yang12 [46] Unclear Independent Quasi-experimental Positive Social Sciences
Hain11 [27] Higher Education Developer Experiment Positive Software development
Wangen09 [47] Higher Education Developer Experiment Neutral Software development
Gom07 [48] Higher Education Independent Experiment Positive Surgery
Gom08 [49] Higher Education Independent Experiment Positive Surgery
Qin10 [50] Higher Education Developer Pilot-study Positive Surgery

Authorized licensed use limited to: NATIONAL UNIVERSITY OF IRELAND GALWAY. Downloaded on October 22,2024 at 18:41:54 UTC from IEEE Xplore. Restrictions apply.
 commercial entertainment game. Perhaps the popularity of is less clear. Students that were taught using the game
games for language learning can be explained by the fact that generated more and more valid ideas, but comparing test-
learning language requires being able to read and write but also scores reveals that the control group actually learnt more [43].
being able to listen and talk. Teaching listening and talking via
written texts is very difficult and therefore the use of audio and Yang [46] studied the difference in effectiveness between a
video is very popular. Games are more active than videos and game-based learning approach and traditional learning in a
can require both passive and active command of the language, quasi-experiment carried out over a full semester (23 weeks) in
making them well suited to language learning. Furthermore, two ninth-grade Civic and Society classes (44 students, ages
any game may provide (first and second) language learning 15-16). The aim of the study was to compare problem solving
opportunity as many are published internationally in English. skills, motivation and academic achievement between an
Piirainen-Marsh and Tainio [45] present and an interesting experiment group (using digital game based learning) and a
example, both methodologically and as an example of control group (using traditional learning). The control and
spontaneous learning. They report on a qualitative study which experiment groups received the same instruction and learning
uses a social-interactional to analyse learning from playing materials during the first two classes of the week, but different
games. They analyse two 13-year-old Finnish boys playing a instruction for the last class of the week, which is referred to as
the intervention time. During the intervention time the control
fantasy role-playing game. Data was drawn from 13 hours of
interactions from teenagers playing computer games. The study group received lectures for 50% of the time and spent the rest
does not attempt to “measure” the learning effect but rather of the time asking questions, completing hand-outs, reporting
provides deep analyses of how the participants engage and results, and receiving feedback. The experiment group spent
interact when playing the game. These activities and 50% of the intervention time playing digital games. Some
interactions provide many opportunities to both read and use examples of games played are: Tycoon City: New York and
English and it is these learning situations that are analysed. SimCity Societies. Teaching was organized so that the
Transcripts of the dialogues between players were analysed to instructor began by clarifying the gaming tasks and provided
study how they used the English language. time for students to design their strategies before playing.

The results show how the players frequently repeated Problem solving abilities were evaluated using a
voice-overs and texts aloud as well as borrowed terms and standardized test examining the ability to find causes, find
concepts from the game while discussing game events with solutions, and avoid problems. The game based strategy was
each other. Another interesting observation is how the players clearly effective in promoting students’ problem solving skills,
anticipate the dialogue of the game and co-construct their own while the control group showed no improvement. Furthermore,
versions as the play. To summarise, Piirainen-Marsh and the game based learning approach resulted in better learning
Tainio [45] conclude that they demonstrate how a wide range motivation for students in the experimental group. Finally, the
of interactional opportunities for using English are available in academic achievement tests showed no statistically significant
the game. The analysis shows how players pay detailed difference between the two groups. To conclude, Yang [46]
attention to the textual and vocal resources in the game and showed that the quantitative improvement in problem-solving
how they adopt gaming vocabulary when they memorize and learning motivation suggest that digital game based
chunks of game dialogue and reproduce or adapt these in learning may be a useful and productive tool to support
appropriate contexts. However, we can’t draw any evidence of students in effective learning.
longitudinal changes or development of language expertise In addition to increasing knowledge, serious games are
from this data. increasingly being used with the intention to change behaviour
Several studies look at using game-based learning for (three studies). The topics being taught are usually not directly
teaching higher cognitive skills such as collaboration, related to the school curriculum, but centre around problem
argumentation and problem solving (three studies) and behaviours such as bullying, medication adherence, emergency
behavioural change (three studies). Huang et al. [43] developed evacuation and health related behaviour. Munguba et al. [41]
a system called the Idea Storming Cube (ISC), which aims to carried out a study to evaluate the effect of using interactive
support and engage students in divergent thinking in the games in an occupational therapy nutrition education
programme for obese children. The goal was to compare a
problem-solving process. Specifically, the system was
developed to support problem solving in relation to debris flow video game and a board game, both specifically designed to
problems, which is considered an urgent problem as they occur improve the subjects’ eating habits. Both quantitative and
frequently in Taiwan. The system is inspired by a Rubrik's qualitative techniques were used to study learning of concepts
cube, used to combine and associate ideas. As the activity in nutritional education. The results indicate that play activities
continues the participants will generate and exchange ideas. promoted learning of nutritional concepts. The extent to which
The system asks them to generate ideas and describe them in eating behaviour was actually changed is unclear even though
natural language. The ideas are validated by the system and all the secondary goal, after learning about nutrition, is to change
new and unique ideas will generate points and as a return, the eating behaviour. This may be attributed to the fact that
users can rotate the cube and see ideas from their peers. The changing eating habits is a long term commitment and thus
results are positive however, while game-based training seems hard to evaluate unless using a longitudinal study. Another
to have a positive impact on collaborative argumentation and factor may be the influence of parents over the eating habits of
problem solving in relation to civic and society based topics, especially younger children.
for problem solving with regards to debris flow problems this

Authorized licensed use limited to: NATIONAL UNIVERSITY OF IRELAND GALWAY. Downloaded on October 22,2024 at 18:41:54 UTC from IEEE Xplore. Restrictions apply.
 Computing is also a topic that is popular with serious game effective with students who are struggling with traditional
developers (four studies selected). This may be partially due to methods of teaching geography.
the fact that many serious game developers themselves are
computing teachers of practitioners. It is also a very exact topic In addition to the categories presented above, there are
making assessment within a game easy. The studies selected several studies of games teaching a variety of topics such as
are mainly positive. However the approach seems more occupational therapy, orienteering, civil engineering, fire-
suitable to university students than school students and one of fighting, first-aid, the history of Amsterdam and historical
the studies presented concludes that there is a positive effect disease epidemics. We included five such studies as they fit our
based on student’s opinions despite the scores not showing a selection criteria.
significant difference. Interestingly, we note that teaching
computing via games includes both ‘hard’ and ‘soft’ topics V. CONCLUSION AND DISCUSSION
from computer fundaments and software measurements to In this report, we have summarized the current state of the
relatively more fuzzy topics such as requirements collection research on the effect and effectiveness of serious games.
and analysis. Hainey et al. [51] argue that traditional Using a structured method we identified papers from a number
educational techniques such as role-play, live-through case of databases. Papers were selected based on a number of
studies and paper-based case studies are insufficient and that criteria. As figure 1 shows, the identified research shows a fair
other approaches are required. One proposed approach is game amount of evidence that serious games have a positive effect
based learning and the authors present a prototype game which on learning. 29 out of our selected 40 studies show positive
teaches the different roles in a software engineering project, results, seven out of 40 neutral and only two out of 40 negative
such as project manager, systems analyst, systems designer or results. For two studies the results are somewhat unclear. From
team leader. The general idea of the game is that the team this we can conclude that if they are not always superior to
(played by one or more players) should manage and deliver other types of learning material, the evidence that serious
different software development projects. Players can interact games can be effective learning materials in their own right is
with non-player characters (NPC) and with each other through quite strong.
a text-based interface. Answers from NPCs are in the form of
written transcripts which contains general background
information as well as requirements which the player has to
identify and document. At some stage the analyst has to send
the list of requirements to the designer to produce an outline
high-level design.
Surgery is a different type of topic to most other topics
being taught with serious games in that it is only taught to
university students. Even so, three studies were included in this
survey as we perceive medical training of particular interest.
Medical training is quite unique in that surgical practice is very
costly and carries a risk to human health. Therefore successful Fig. 1. Evaluation results of identified emperical studies
simulation and game-based technique can save significant
amounts of money and reduce patient risk. This means that the We notice some interesting examples of using commercial
emphasis lies not on comparing these simulations and games entertainment games in teaching. Even though the major
with the traditional methods, but more on assessing whether reference to serious games refers to specifically designed
they are useful training tools in their own right and whether games, there seems to be a potential for using entertainment
performance on these tools correlates with real world games in education. Piirainen-Marsh and Tainio [45] provide
performance. The latter means that the tools are useful in an interesting example of language learning, even though not in
assessing whether a student can safely progress to real patients. a formal teaching situation. Yang [46] showed that a game
Natural sciences teaching has seen an increase in the use of based strategy using commercial entertainment games was
technology either mobile out in the field, or behind the clearly effective in promoting students’ problem solving skills,
computer, even though only two studies were selected for this whereas the control group showed no improvement.
survey. These environments offer the opportunity for students Furthermore, the game based learning approach resulted in
to explore environments and the selected studies about teaching better learning motivation for students in the experimental
natural sciences show an increase in motivation and learning group. Finally, the academic achievement tests showed no
gains. The use of digital technology in geography is popular, statistically significant difference between the two groups.
and is expected to increase with devices such as GPSs and These results are interesting in that they suggest some benefits
smart phones with maps. For this survey three studies that from using entertainment games without any detriment in
represent more traditional 3D games for learning were selected academic achievement.
and they show that geography is a popular topic for the In 21 cases the developer was also one of the main
application of game-based learning. The papers on this topic evaluators. 14 out of these 21 studies show a positive result,
found in this survey, show that serious games for teaching four studies show a neutral result, one is negative and two are
geography can be very effective and they are especially unclear. In 17 studies the evaluation is carried out by
independent evaluators. 14 of these studies show a positive

Authorized licensed use limited to: NATIONAL UNIVERSITY OF IRELAND GALWAY. Downloaded on October 22,2024 at 18:41:54 UTC from IEEE Xplore. Restrictions apply.
 result, one a negative result and two a neutral result. It seems ACKNOWLEDGMENT
that there is a tendency towards more positive evaluation This paper is has been fully supported by the European
results overall. There is also a trend towards evaluations by a Commission under Life Long Learning program project
stakeholder in development of the game. This trend can be EduGameLab (grant 511640-LLP-1-2010-1-FR-KA3-KA3MP)
compared to Garg et al. [52] who analysed the development
and evaluation of clinical decision support systems and found a
strong trend towards positive evaluation results when the
developer was also the evaluator, compared with studies in REFERENCES
which the authors were not the developers (74% success vs. [1] S. Tobias, J. D. Fletcher, D. Y. Dai, and A. P. Wind, ‘Review of
28%; respectively). Even though there are some signs of a research on computer games’, Computer games and instruction, pp.
slight bias in our survey, the effect is not as strong as in other 127–222, 2011.
[2] J. J. Vogel, D. S. Vogel, J. Cannon-Bowers, C. A. Bowers, K. Muse,
fields [52]. Interestingly, the presence of studies with unclear and M. Wright, ‘Computer gaming and interactive simulations for
findings increases when the evaluator is a stakeholder in the learning: A meta-analysis’, Journal of Educational Computing
game development. It is not clear why this is the case, but it Research, vol. 34, no. 3, pp. 229–243, 2006.
could be due to over ambitious evaluation setups of [3] J. Kirriemuir and A. McFarlane, ‘Literature review in games and
stakeholders aiming to provide a much proof as possible for learning’, 2004.
[4] K. Salen and E. Zimmerman, Rules of play: Game design fundamentals.
their game.
MIT press, 2003.
Notably, not much research has been done on how games [5] M. Prensky, ‘Digital natives, digital immigrants Part 1’, On the horizon,
vol. 9, no. 5, pp. 1–6, 2001.
are actually used in teaching. This includes user acceptance [6] M. Zyda, ‘From visual simulation to virtual reality to games’,
(teachers, students and parents), technological restrictions as Computer, vol. 38, no. 9, pp. 25–32, 2005.
well as questions concerning curricula and content (see e.g. [7] T. Marsh, ‘Serious games continuum: Between games for purpose and
[53], [3]). According to Egenfeldt-Nielsen [9], other barriers experiential environments for purpose’, Entertainment Computing, vol.
include the fact that the educational setting uses short lessons 2, no. 2, pp. 61–68, 2011.
in a particular physical space, variations in gaming skills [8] R. T. Hays, ‘The effectiveness of instructional games: A literature
review and discussion’, DTIC Document, 2005.
between students and practical issues such as installation and [9] S. Egenfeldt-Nielsen, ‘Overview of research on the educational use of
teacher preparation costs. With regards to user acceptance video games’, Digital kompetanse, vol. 1, no. 3, pp. 184–213, 2006.
Egenfeldt-Nielsen [9] notes a certain amount of scepticism [10] F. Ke, ‘A qualitative meta-analysis of computer games as learning
among students and teachers. There are also reports of practical tools’, Handbook of research on effective electronic gaming in
problems and according to Kirriemuir and McFarlane [3] it is education, vol. 1, pp. 1–32, 2009.
[11] J. V. Dempsey, K. Rasmussen, and B. Lucassen, ‘Instructional gaming:
unlikely that entertainment games will be integrated into the Implications for instructional technology.’, presented at the Annual
curriculum due to a number of reasons: it is hard for a teacher Meeting of the Association for Educational Communications and
to identify how a particular game might be relevant with Technology, Nashville, TN, 1996.
respect to the curriculum; the difficulty to persuade other [12] S. Kim and M. Chang, ‘Computer games for the math achievement of
school stakeholders of the benefits of using games as diverse students’, Educational Technology & Society, vol. 13, no. 3, pp.
224–232, 2010.
educational tools, especially entertainment games; the lack of [13] A. Rubin-Vaughan, D. Pepler, S. Brown, and W. Craig, ‘Quest for the
time for teachers to learn how to use games as teaching tools; Golden Rule: An effective social skills promotion and bullying
the high amount of irrelevant content and functionality (with prevention program’, Computers & Education, vol. 56, no. 1, pp. 166–
respect to the teaching situation) in entertainment games; 175, 2011.
technology barriers in the IT equipment in many schools. We [14] P. M. Kato, S. W. Cole, A. S. Bradlyn, and B. H. Pollock, ‘A video
find that practical pedagogical problems like these are the next game improves behavioral outcomes in adolescents and young adults
with cancer: A randomized trial’, Pediatrics, vol. 122, no. 2, pp. e305–
step to be addressed in research once we have established the e317, 2008.
potential of using games in teaching. This is, however, a [15] M. Papastergiou, ‘Digital Game-Based Learning in high school
different strand of research that calls for longitudinal studies in Computer Science education: Impact on educational effectiveness and
actual teaching situations, which has both practical and ethical student motivation’, Computers & Education, vol. 52, no. 1, pp. 1–12,
implications beyond what we have experienced so far. Jan. 2009.
[16] G. Sindre, L. Natvig, and M. Jahre, ‘Experimental validation of the
The diversity of the field, as indicated by previous reviews learning effect for a pedagogical game on computer fundamentals’,
[2], [9], [10] probably calls for a diversity of scientific methods Education, IEEE Transactions on, vol. 52, no. 1, pp. 10–18, 2009.
[17] M. Roussou, M. Oliver, and M. Slater, ‘The virtual playground: an
to study it. However, within the body of evidence identified in educational virtual reality environment for evaluating interactivity and
this study, there is a distinct lack of so called longitudinal conceptual learning’, Virtual Reality, vol. 10, no. 3, pp. 227–240, 2006.
empirical studies of actual use of games for learning. These are [18] M. Ebner and A. Holzinger, ‘Successful implementation of user-
studies that assess the effect over time. Hence does learning centered game based learning in higher education: An example from
with a game have any medium to long term positive effects on civil engineering’, Computers & Education, vol. 49, no. 3, pp. 873–890,
2007.
students? Can games be effective teaching tools in schools over [19] T.-Y. Chuang, ‘Effect of computer-based video games on children: An
a longer period of time? If so, how should teaching with games experimental study’, in Digital Game and Intelligent Toy Enhanced
be organized? And do benefits of learning with a game remain Learning, 2007. DIGI℡’07. The First IEEE International Workshop on,
if the game becomes an established form for teaching certain 2007, pp. 114–118.
parts of the curriculum or is some of its effect due to students [20] N. Vos, H. van der Meijden, and E. Denessen, ‘Effects of constructing
(and perhaps teachers) reacting to the novelty factor? versus playing an educational game on student motivation and deep
learning strategy use’, Computers & Education, vol. 56, no. 1, pp. 127–
137, 2011.

Authorized licensed use limited to: NATIONAL UNIVERSITY OF IRELAND GALWAY. Downloaded on October 22,2024 at 18:41:54 UTC from IEEE Xplore. Restrictions apply.
 [21] O. S. Asaolu, ‘International Evaluation of a Localized Geography grade students’, Computers & Education, vol. 40, no. 1, pp. 71–94,
Educational Software’, African Journal of Computing & ICT, vol. 5, no. 2003.
4, 2012. [38] A. Wilson, S. Dehaene, P. Pinel, S. Revkin, L. Cohen, and D. Cohen,
[22] H. Tüzün, M. Yılmaz-Soylu, T. Karakuş, Y. İnal, and G. Kızılkaya, ‘Principles underlying the design of “The Number Race”, an adaptive
‘The effects of computer games on primary school students’ computer game for remediation of dyscalculia’, Behavioral and Brain
achievement and motivation in geography learning’, Computers & Functions, vol. 2, no. 1, pp. 1–14, 2006.
Education, vol. 52, no. 1, pp. 68–77, 2009. [39] T.-Y. Liu, T.-H. Tan, and Y.-L. Chu, ‘Outdoor natural science learning
[23] M. Virvou, G. Katsionis, and K. Manos, ‘Combining software games with an RFID-supported immersive ubiquitous learning environment’,
with education: Evaluation of its educational effectiveness’, Educational Technology & Society, vol. 12, no. 4, pp. 161–175, 2009.
Educational Technology & Society, vol. 8, pp. 54–65, 2005. [40] T.-H. Wang, ‘Web-based quiz-game-like formative assessment:
[24] H. Tüzün, ‘Blending video games with learning: Issues and challenges Development and evaluation’, Computers & Education, vol. 51, no. 3,
with classroom implementations in the Turkish context’, British Journal pp. 1247–1263, 2008.
of Educational Technology, vol. 38, no. 3, pp. 465–477, 2007. [41] M. C. Munguba, M. T. M. Valdes, and C. A. B. da Silva, ‘The
[25] J. Huizenga, W. Admiraal, S. Akkerman, and G. ten Dam, ‘Mobile application of an occupational therapy nutrition education programme
game-based learning in secondary education: engagement, motivation for children who are obese’, Occupational Therapy International, vol.
and learning in a mobile city game’, Journal of Computer Assisted 15, no. 1, pp. 56–70, 2008.
Learning, vol. 25, no. 4, pp. 332–344, 2009. [42] A. Ravenscroft and M. P. Matheson, ‘Developing and evaluating
[26] S. Kennedy-Clark and K. Thompson, ‘What Do Students Learn When dialogue games for collaborative e–learning’, Journal of Computer
Collaboratively Using A Computer Game in the Study of Historical Assisted Learning, vol. 18, no. 1, pp. 93–101, 2002.
Disease Epidemics, and Why?’, Games and Culture, vol. 6, no. 6, pp. [43] C.-C. Huang, T.-K. Yeh, T.-Y. Li, and C.-Y. Chang, ‘The idea storming
513–537, 2011. cube: Evaluating the effects of using game and computer agent to
[27] T. M. Connolly, M. Stansfield, and T. Hainey, ‘An alternate reality support divergent thinking’, Educational Technology & Society, vol. 13,
game for language learning: ARGuing for multilingual motivation’, no. 4, pp. 180–191, 2010.
Computers & Education, vol. 57, no. 1, pp. 1389–1415, 2011. [44] T. Y. Liu and Y. L. Chu, ‘Using ubiquitous games in an English
[28] A. F. Chow, K. C. Woodford, and J. Maes, ‘Deal or No Deal: using listening and speaking course: Impact on learning outcomes and
games to improve student learning, retention and decision-making’, motivation’, Computers & Education, vol. 55, no. 2, pp. 630–643, 2010.
International journal of mathematical education in science and [45] A. Piirainen–Marsh and L. Tainio, ‘Collaborative Game-play as a Site
technology, vol. 42, no. 2, pp. 259–264, 2011. for Participation and Situated Learning of a Second Language’,
[29] Z. Kablan, ‘The Effect of Using Exercise-Based Computer Games Scandinavian Journal of Educational Research, vol. 53, no. 2, pp. 167–
during the Process of Learning on Academic Achievement among 183, 2009.
Education Majors.’, Educational Sciences: Theory and Practice, vol. [46] Y.-T. C. Yang, ‘Building virtual cities, inspiring intelligent citizens:
10, no. 1, pp. 351–364, 2010. Digital games for developing students’ problem solving and learning
[30] F. Ke and B. Grabowski, ‘Gameplaying for maths learning: cooperative motivation’, Computers & Education, vol. 59, no. 2, pp. 365–377, 2012.
or not?’, British Journal of Educational Technology, vol. 38, no. 2, pp. [47] C. Gresse von Wangenheim, M. Thiry, and D. Kochanski, ‘Empirical
249–259, 2007. evaluation of an educational game on software measurement’, Empirical
[31] F. Ke, ‘Computer games application within alternative classroom goal Software Engineering, vol. 14, no. 4, pp. 418–452, 2009.
structures: cognitive, metacognitive, and affective evaluation’, [48] A. H. Gomoll, R. V. O’Toole, J. Czarnecki, and J. J. Warner, ‘Surgical
Educational Technology Research and Development, vol. 56, no. 5, pp. experience correlates with performance on a virtual reality simulator for
539–556, 2008. shoulder arthroscopy’, The American Journal of Sports Medicine, vol.
[32] M. Kordaki, ‘A computer card game for the learning of basic aspects of 35, no. 6, pp. 883–888, 2007.
the binary system in primary education: Design and pilot evaluation’, [49] A. H. Gomoll, G. Pappas, B. Forsythe, and J. J. Warner, ‘Individual
Education and Information Technologies, vol. 16, no. 4, pp. 395–421, Skill Progression on a Virtual Reality Simulator for Shoulder
2011. Arthroscopy A 3-Year Follow-up Study’, The American Journal of
[33] C. C. Y. Liao, Z.-H. Chen, H. N. H. Cheng, F.-C. Chen, and T.-W. Sports Medicine, vol. 36, no. 6, pp. 1139–1142, 2008.
Chan, ‘My-Mini-Pet: a handheld pet-nurturing game to engage students [50] J. Qin, Y.-P. Chui, W.-M. Pang, K.-S. Choi, and P.-A. Heng, ‘Learning
in arithmetic practices’, Journal of Computer Assisted Learning, vol. 27, blood management in orthopedic surgery through gameplay’, Computer
no. 1, pp. 76–89, 2011. Graphics and Applications, IEEE, vol. 30, no. 2, pp. 45–57, 2010.
[34] S. Main and J. O’Rourke, ‘“New Directions for Traditional Lessons”: [51] T. Hainey, T. M. Connolly, M. Stansfield, and E. A. Boyle, ‘Evaluation
Can Handheld Game Consoles Enhance Mental Mathematics Skills?’, of a game to teach requirements collection and analysis in software
Australian Journal of Teacher Education, vol. 36, no. 2, p. 4, 2011. engineering at tertiary education level’, Computers & Education, vol.
[35] H. Panoutsopoulos and D. G. Sampson, ‘A Study on Exploiting 56, no. 1, pp. 21–35, 2011.
Commercial Digital Games into School Context’, Educational [52] A. X. Garg, N. K. Adhikari, H. McDonald, M. P. Rosas-Arellano, P. J.
Technology & Society, vol. 15, no. 1, pp. 15–27, 2012. Devereaux, J. Beyene, J. Sam, and R. B. Haynes, ‘Effects of
[36] Y.-T. Sung, K.-E. Chang, and M.-D. Lee, ‘Designing multimedia games computerized clinical decision support systems on practitioner
for young children’s taxonomic concept development’, Computers & performance and patient outcomes’, JAMA: the journal of the American
Education, vol. 50, no. 3, pp. 1037–1051, 2008. Medical Association, vol. 293, no. 10, pp. 1223–1238, 2005.
[37] R. Rosas, M. Nussbaum, P. Cumsille, V. Marianov, M. Correa, P. [53] R. Sandford, M. Ulicsak, and K. Facer, ‘Teaching with Games: using
Flores, V. Grau, F. Lagos, X. López, and V. López, ‘Beyond Nintendo: computer games in formal education’, Futurelab, Bristol, 2006.
design and assessment of educational video games for first and second

Authorized licensed use limited to: NATIONAL UNIVERSITY OF IRELAND GALWAY. Downloaded on October 22,2024 at 18:41:54 UTC from IEEE Xplore. Restrictions apply.