Delft University of Technology

Pragmatic software testing education

Aniche, Maurício; Hermans, Félienne; van Deursen, Arie

DOI
10.1145/3287324.3287461
Publication date
2019
Document Version
Accepted author manuscript
Published in
SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education

Citation (APA)
Aniche, M., Hermans, F., & van Deursen, A. (2019). Pragmatic software testing education. In SIGCSE 2019
- Proceedings of the 50th ACM Technical Symposium on Computer Science Education (pp. 414-420).
Association for Computing Machinery (ACM). [Link]

Important note
To cite this publication, please use the final published version (if applicable).
Please check the document version above.

Copyright
Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent
of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.
Takedown policy
Please contact us and provide details if you believe this document breaches copyrights.
We will remove access to the work immediately and investigate your claim.

This work is downloaded from Delft University of Technology.

For technical reasons the number of authors shown on this cover page is limited to a maximum of 10.
 Pragmatic Software Testing Education
Maurício Aniche Felienne Hermans Arie van Deursen
Delft University of Technology Delft University of Technology Delft University of Technology
The Netherlands The Netherlands The Netherlands
[Link]@[Link] [Link]@[Link] [Link]@[Link]
ABSTRACT moved away from being considered the activity that ‘less skilled’
Software testing is an important topic in software engineering edu- software engineers do to one of the most important skills an engi-
cation, and yet highly challenging from an educational perspective: neer should have.
students are required to learn several testing techniques, to be able The act of inspecting large and complex code bases to find bugs
to distinguish the right technique to apply, to evaluate the qual- is not a trivial task in the real world: engineers need to have a broad
ity of their test suites, and to write maintainable test code. In this understanding of different practices that vary from simple manual
paper, we describe how we have been adding a pragmatic perspec- exploratory testing, where a human tries to find bugs manually by
tive to our software testing course, and explore students’ common interacting with the system, to advanced bleeding-edge testing tech-
mistakes, hard topics to learn, favourite learning activities, and niques, such as automated testing and automated test generation,
challenges they face. To that aim, we analyze the feedback reports where engineers program machines to test their system.
that our team of Teaching Assistants gave to the 230 students of our Companies such as Facebook [12], Google [41], and Mi-
2016-2017 software testing course at Delft University of Technology. crosoft [35] take testing seriously and require their engineers to
We also survey 84 students and seven of our teaching assistants on master such techniques. Surveys have shown that developers un-
their perceptions. Our results help educators not only to propose derstand the importance of testing-related training [15] and yet
pragmatic software testing courses in their faculties, but also bring many of them still lack formal testing education [6, 34].
understanding on the challenges that software testing students face Indeed, educating a student in the art of software testing is
when taking software testing courses. challenging, for both students and educators. From the educator’s
perspective, it is hard to keep a testing course up-to-date with the
CCS CONCEPTS novelties of the field as well as to come up with exercises that
are realistic [14]. Due to the importance of the topic, educators
• Applied computing → Education; • Software and its engi-
have been experimenting with the introduction of testing earlier
neering → Software verification and validation;
in Computer Science programs [17, 19–21, 23, 27], introducing
a test-first approach in CS courses [9, 10, 22], developing tools
KEYWORDS focused on software testing education [11, 38], and proposing more
software testing education, software engineering education, com- complete postgraduate courses focused on testing [39]. Educators
puter science education. also face the fact that some testing topics are not conceptually
ACM Reference Format: straightforward, not easy to demonstrate and generalize, and are
Maurício Aniche, Felienne Hermans, and Arie van Deursen. 2019. Pragmatic not all available in a single textbook [40].
Software Testing Education. In SIGCSE ’19: 50th ACM Technical Symposium This paper has a twofold goal. First, to present how we have been
on Computer Science Education, February 27–March 2, 2019, Minneapolis, MN, teaching pragmatic software testing to the first year CS students
USA. ACM, New York, NY, USA, 7 pages. [Link] at Delft University of Technology. Second, we explore students’
3287461 common mistakes, hard topics to learn, favourite learning activities,
and challenges they face when learning pragmatic software testing.
1 INTRODUCTION To this aim, we analyzed the 1,993 quotes from the feedback
Every software developer should be aware of the (high) impact report that we, as teachers and teaching assistants, gave to each of
that malfunctioning software can have in our society. We have the 230 students of the 2017 edition of the Software Quality and
seen huge losses in the financial market [30], and even researchers Testing course, which is taught at the first year of our Computer Sci-
withdrawing their papers [33]; all of them caused by software bugs. ence bachelor. In addition, we performed a survey with 84 students,
Making sure software works is maybe the greatest responsibility which we augmented by also surveying seven of our TAs.
of a software developer. Luckily, over the years, software testing The main contributions of this paper are:
• A proposal for a pragmatic software testing course based on
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
nine key principles that can be taught for computer science
for profit or commercial advantage and that copies bear this notice and the full citation students, including building a test mindset and interaction
on the first page. Copyrights for components of this work owned by others than the with practitioners (Section 3).
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission • An empirical analysis of the students’ most common mis-
and/or a fee. Request permissions from permissions@[Link]. takes (Section 6.1), their perceptions on the most difficult
SIGCSE’19, February 27–March 2, 2019, Minneapolis, MN, USA topics in software testing (Section 6.2), and the importance
© 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-5890-3/19/2. . . $15.00 of different teaching activities (Section 6.3) when learning
[Link] pragmatic software testing.
SIGCSE’19, February 27–March 2, 2019, Minneapolis, MN, USA Maurício Aniche, Felienne Hermans, and Arie van Deursen

2 RELATED WORK The teaching team is currently composed of two teachers and
Software Testing is an important part of any Software Engineering teaching assistants (TAs). The number of TAs vary as our university
program [2, 8, 26, 42], and by itself poses several other challenges to has a policy of 1 TA per 30 students. Teachers are responsible for the
educators. Unfortunately, the topic still does not receive its deserved course design, lectures, creating and assessing multiple choice ex-
attention in several CS programs. Wong [42] argues that many ams, and they have the overall responsibility of the course. TAs are
engineers are not well trained in software testing because most CS responsible for helping students, grading all labwork deliverables,
programs offer ST as an elective course. Clarke et al. [8] also points and for giving concrete and specific feedback on what students can
to the fact that due to the large number of topics to be covered in a improve.
Software Engineering program, little attention is given to Software Learning goals. At the end of the course, students (1) are able to
Testing. Astigarraga et al. [2] show that most CS programs tend create unit, integration, and system tests using current existing
to emphasize development at the expense of testing as a formal tools (e.g., JUnit, Mockito) that successfully test complex software
engineering discipline. Lemos et al. [26] show that software testing systems, (2) are able to derive test cases that deal with exceptional,
education can improve code reliability in terms of correctness; corner, and bad weather cases by performing several different tech-
however, authors also argue that university instructors tend to niques (i.e., boundary analysis, state-based testing, decision tables),
lack the same knowledge that would help students increase their (3) are able to measure and reflect on the effectiveness of the de-
programming skills toward more reliable code. veloped test suites by means of different test adequacy metrics
Educators have been suggesting different approaches on how to (e.g., line and branch code coverage, MC/DC), (4) are able to reflect
introduce testing in a CS curriculum: from students submitting their on limitations of current testing techniques, when and when not
assignments together with test plans or sets [16, 17, 21], performing to apply them in a given context, and to design testable software
black-box testing in a software seeded with errors [21, 24, 31], systems, (5) Participants are able to write maintainable test code
students testing each others’ programs [36], to suggesting students by avoiding well-known test code smells (e.g., Assertion Roulette,
to use a test-first approach at the very beginning of the program [9, Slow or Obscure Tests).
10, 22, 27]. Many of these authors even suggest that tests should be Program. The course covers software quality attributes, maintain-
incorporated to the Computer Science and Software Engineering ability and testability, manual and exploratory testing, automated
curricula, not only as an elective discipline, but throughout the testing, devops, test adequacy, model-based testing, state-based test-
curriculum. More specifically, Jones [23] suggests that students need ing, decision tables, reviews and inspections, design-by-contract,
to see the practice of software testing as part of the educational embedded system testing, test-driven design, unit versus integra-
experience and that each core course in the curriculum should tion testing, mocks and stubs. More specifically:
impart one or more testing experiences.
In addition, educators have proposed tools that are solely fo- • Week 1: Introduction to software testing, fault vs failure, princi-
cused on software testing education. Elbaum et al. [11] propose ples of testing, (un)decidabilitily, introduction to JUnit, introduc-
BugHunt. BugHunt is a tool that contains four different lessons tion to labwork.
on software testing (terminology, black box, white box, efficiency • Week 2: Life cycle, validation vs verification, V-model, code re-
in testing). 79% of the students in their experiment agreed that views. Functional testing, partition testing, boundary testing, and
BugHunt added significant value to the material presented in the domain testing.
lecture(s) on software testing, and 61% of the students agreed that • Week 3: Structural testing, adequacy criteria, code coverage. Unit
BugHunt could replace the classes on testing. Spacco and Pugh vs integration vs system testing, mock objects, and test-driven
propose Marmoset [38], a tool to help incentivize students to test development.
their software. Marmoset’s innovative element is that if a submis- • Week 4: State-based testing, model-based testing, and decision
sion passes all of the public test cases, then students are given the tables.
opportunity to test their code against a test suite that is not publicly • Week 5: Test code quality, test code smells. Design for testability.
disclosed. Design-by-contracts.
• Week 6: Security testing. Search-based software testing.
• Week 7: Guest lectures from industry.
3 PRAGMATIC SOFTWARE TESTING
EDUCATION Key elements. To achieve a pragmatic software testing course, we
The Software Testing and Quality Engineering at Delft University have devised and currently follow some key elements:
of Technology is a course that covers several different aspects of Theory applied in the lecture. We put our efforts into developing lec-
software testing, ranging from topics in the ISTQB industry certifi- tures where students can see theory being applied to practice. Our
cation [5] to software testing automation, as well as the future of lectures often have the following structure: we present a (buggy)
testing by means of selected research papers. code implementation (initially on slides, and later in the IDE), we
The course is currently a compulsory part of the 4th quarter discuss where the bug is, we explore, at a conceptual level, a sys-
of the first year in the Computer Science bachelor. The course tematic approach to detect the bug, we apply the approach into a
corresponds to 5 ECTS (140 hours). Students have two lectures set of concrete examples. In other words, we do not only focus on
of 1.5 hours plus 4 hours of labwork a week. As a pre-requisite, explaining abstract ideas, but on concretely showing how to apply
students should have at least basic knowledge on Java programming them on different real world problems, using real-world tools, like
language. JUnit, Mockito, and Cucumber.
Pragmatic Software Testing Education SIGCSE’19, February 27–March 2, 2019, Minneapolis, MN, USA

Real-world pragmatic discussions. Software testing is a challenging Empirical research has indeed shown that test code smells often
activity to be done in practice. This means that developers often happen in software systems, and that their presence has a strong
make trade-offs in deciding what and how much to test. Engineering negative impact on the maintainability of the affected classes [3].
questions that arise when complex software systems are being We often reinforce the importance of refactoring test code and
tested, such as “how much should I test?”, “how should I test a make sure they are free of smells. To any test code we write during
mobile application that communicates with a web server?”, and live coding sessions, we make sure that they are as free of smells
“should I use mocks to test this application?” are often discussed in as possible. Test smell catalogues such as the ones proposed by
classroom so that students see how to extrapolate from our often Meszaros [32] are deeply discussed in a dedicated lecture.
small exercises to their future real lifes as developers. Design systems for testability. Designing software in such a way
Build a testing mindset. Software testing is not seen as an important that it eases testability is a common practice among practition-
task by many students. A software testing course should inspire ers [13, 18, 29]. This requires us to not only discuss software testing
students to think about testing whenever they implement any piece in our course, but software architecture and design principles of
of code. In our testing course, we aim to achieve such a testing testable software systems, such as dependency inversion [28], ob-
mindset by (1) showing how testing can be a creative activity, re- servability and controllability, in an entire dedicated lecture for the
quiring strong developers, by means of several live coding sessions topic. Questions like “Do I need to test this behavior via an unit
and rich pragmatic discussions, (2) demonstrating not only the use- or a system test?”, “How can I test my mobile application?” are
fulness of any testing technique we teach, but also how they are extensively discussed not only through the eyes of software testing,
applied, as well as what trade-offs such techniques have in the real– but also to the eyes of software design.
world, (3) bringing guest lecturers who talk about the importance Mixture of pragmatic and theoretical books. The two books we use
of software testing for their companies. as textbooks in the course are the “Foundations of software testing:
Software testing automation. The software engineering industry ISTQB certification” [5], which gives students a solid foundation
has long been advocating the automation of any software testing about testing theory, and the “Pragmatic Unit Testing in Java 8 with
activity [12, 35, 41]. However, some software testing courses still JUnit” [25], which gives students concrete and practical examples
focus on writing test case specifications solely as documents, and on how to use testing tools, like JUnit. We believe both complement
do not discuss how to automate them. In our course, to all the each other and both are important for students who will soon
theoretical and systematic test design techniques we present, from become a software tester.
functional testing to structural testing, from unit to system-level Interaction with practitioners. We strongly encourage their inter-
tests, students later write them in a form of an automated test. action with practitioners throughout our course. Having guest
Mastering tools such as JUnit and Mockito, standard tools for test lectures from industry practitioners helps us to show the pragmatic
automation in Java, is a clear learning goal of our course. The side of software testing. Guests focus their lectures on how they
importance of automation also strongly appears in our labwork, apply software testing at their companies, tools they use, their pros
which we discuss next. and cons, and on the mistakes and challenges they face. In the
A hands-on labwork. We see the labwork as an important learning 2017 edition, we also experimented with Ask-Me-Anything (AMA)
method. In our course, by means of a practical labwork assignment, sessions, where we called experts from all over the world via Skype
students apply a selection of techniques to a 3k lines of code game and students had 15 minutes to ask any software-testing related
written in Java, namely, JPacMan. The labwork contains a set of 50 questions.
exercises in which students are able to exercise all the techniques Grading. We currently use the following formula to grade our
we teach. It is important to notice that students not only generate students: 0.25 * labwork + 0.75 * exam. The labwork (as
test cases on the paper, but also automate them. A great amount of we explain below) is composed of 4 deliverables, each graded by
their work is in actually producing automated JUnit test cases. our TAs in a range of [0..10]. We later average the grades of four
In the following, we present the main deliverables of our labwork. deliverables, which compose the labwork component of the grade.
The complete assignment can be found in our online appendix [1]. At the end of the course, we propose a 40-question multiple choice
• Part 0 (Pre-requisites). Clone the project from Github, configure exam. Students may take a resit 6 weeks later if they did not pass in
the project in your IDE, write your first JUnit test, run coverage the first time. We also offer an optional midterm exam for students
analysis. who want to practice beforehand.
• Part 1. Write a smoke test, functional black-box testing, boundary
tests, reflect on test understandability and best practices. 4 RESEARCH METHODOLOGY
• Part 2. White-box testing, mock objects, calculate code coverage The goal of this study is to provide a better understanding of the dif-
and apply structural testing, use decision tables for complex ficulties and challenges that students face when learning pragmatic
scenarios, reflect on how to reduce test complexity and how to software testing.
avoid flaky tests. To that aim, we analyze the data from 230 students of the 2016-
• Part 3. Apply state-based testing, test reusability, refactor and 2017 edition of our software testing course. We propose three re-
reflect on test smells. search questions:
Test code quality matters. Due to the importance of automated test- RQ1 : What common mistakes do students make when learning
ing activities, software testers will deal with large test codebases. software testing?
SIGCSE’19, February 27–March 2, 2019, Minneapolis, MN, USA Maurício Aniche, Felienne Hermans, and Arie van Deursen

RQ2 : Which software testing topics do students find hardest to

30
learn?
RQ3 : Which teaching methods do students find most helpful?

Number of students
To answer our research questions, we collect and analyze data
20
from three different sources: the feedback reports that TAs give to
when
students throughout the course, a survey with students, and a sur- after
vey with the TAs, both performed after the course. We characterize before

the participants in Section 5. In the following, we detail the three 10

parts of our methodology.
Manual content analysis on the feedback. As we explain in
Section 3, students work on and produce four deliverables during 0
the course. After each deliverable, our team of TAs manually reads
2.5 5.0 7.5 10.0
students’ reports, source code, and tests, and with the help of a
rubric, provides them with rich qualitative feedback.
This feedback usually contains several quotes that touch on a Figure 1: Histogram of the students’ (X axis) perceptions in
mix of different topics, such as mistakes they made in the exercises, terms of their knowledge (Y axis) on software testing before
tips on how to improve their existing work, issues on the written and after the course. Scale: from 1 to 10.
report, and even compliments for their good work. The language
of such feedback reports is usually informal, as we do not give
constraints to TAs on how the feedback should be.
We analyze the content of all feedback reports. To that aim, we Survey with Teaching Assistants. Our TAs support students
first filter out any feedback that is not directly related to software throughout the course, by answering their questions, supporting
testing (e.g., comments on exercises that were not done, or compli- their work during the lab, and by grading their assignments. As a
ments). We then follow an iterative process, derived from standard consequence of such intense contact with students, TAs obtain a
qualitative data analysis procedures [37]: (1) we assign a code for good perspective on the challenges of teaching software testing.
each quote in the feedback; the code summarizes the essence of We also performed a similar survey with TAs, focusing on what
the quote, (2) if a quote does not belong to any existing codes, we they perceive as challenges for students. The survey contained the
introduce a new code, (3) each quote has just a single code; if a same two closed questions from the students’ survey (challenges
quote tackles two different problems, we split the original quote when applying software testing, and the importance of the differ-
into two quotes, (4) to assign the correct code to a quote, we used ent activities). In the open questions, we focused on asking about
our knowledge of the testing course, labwork, and the existing the common mistakes students do during the lab, as well as their
rubrics. We assigned 40 different codes to a total of 1,993 quotes. As perceptions on the challenges that students face.
a next step, we started an iterative merging process to derive the We shared the survey internally at the end of our course. We
final themes, by grouping similar codes into higher-level themes, also did not make answering the survey compulsory for TAs. At
e.g., the theme “maintainability of test code” contains quotes from the end, we received 7 complete answers out of the 10 TAs.
the “test quality”, and “test duplication” codes. We ended up with
eight themes that we present in the Results (Section 6).
5 CHARACTERIZATION OF THE
Survey with students. With the goal of capturing their percep-
PARTICIPANTS
tions on learning software testing, we asked students to answer a
questionnaire that contained both open and closed questions at the Students. 66 students identify themselves as male, 8 as female, and
end of the course. 10 preferred not to answer. 89.3% of the students are between 18 to
The survey contains a total of 18 questions, none of which are 24 years, five are between 25 and 34, and four are 17 or younger.
required. The two closed questions of the survey asked students Only three students were international students. In terms of Java
about the difficulty of learning and putting into practice the con- knowledge, in a scale from 1 to 10, 9.5% of students consider their
cepts and techniques we taught, and about the importance of the knowledge between 9 and 10, and 72% of them consider themselves
different activities we used throughout the course. In these ques- between 7 and 8. Only 4 students consider themselves 5 or below.
tions, students had to choose from a five point Likert-scale, ranging Thanks to the introduction to JUnit that students receive during
from strongly disagree to strongly agree (see Figures 2 and 3). The their very first course on programming, most of them already had
open questions were mostly focused on understanding the students’ some knowledge on software testing prior to our course. In fact, as
main challenges, difficulties, and suggestions of improvements for we show in Figure 1, before the course starts, in a scale from 1 to 10,
our testing course. We apply qualitative techniques to analyze the 39% of them consider themselves between 6 and 8, 44% between 4
results of each open question individually, similarly to our analysis and 5, and only 16% between 1 and 3. No student considered herself
of the feedback reports. The full survey as well the full code book a 9 or 10. Students considered that their knowledge increased after
can be found in our online appendix [1]. the course. All of them considered their knowledge after the course
We did not make answering the survey compulsory for the stu- as 6 or greater; 39% of them ranked themselves with a 8, and 14.6%
dents. We received 84 complete answers out of the 230 students. with a 9. Two students ranked themselves with a 10.
Pragmatic Software Testing Education SIGCSE’19, February 27–March 2, 2019, Minneapolis, MN, USA

Usage of JUnit Q1 76% 23% 1%

State-based testing (247 times, 12.39%). When it comes to state-based
AAA pattern Q2 73% 25% 2%
Choose the test level Q3 29% 50% 21%
testing, students often miss or create wrong states or events (56) and
Mock Objects Q4 42% 31% 27% transitions (72), or develop non-clear or not legible state machines
Boundary Testing Q5 54% 32% 14%
(68).
Structural testing Q6 44% 40% 16%
Apply MC/DC Q7 17% 34% 49% Assertions (158 times, 7.93%). Most feedback related to assertions
State−based testing Q8 30% 37% 33%
focus on missing assertions, i.e., the student forgot to assert one or
Best practices Q9 46% 41% 14%
Testability Q10 35% 41% 25%
more expected result, and on assertions that are wrong or should
TDD Q11 31% 37% 32% not exist in that test.
Design by contracts Q12 30% 54% 16%
Acceptance tests Q13 46% 35% 20%
Mock Objects (117 times, 5.87%). Students required some feedback
How much to test Q14 19% 39% 42% on how to use mock objects. More specifically, on how to prop-
Defensive programming Q15 27% 48% 25%
erly verify interactions with mock objects (i.e., Mockito’s ‘verify’
Exploratory Testing Q16 36% 46% 18%
Avoiding flaky tests Q17 56% 26% 19%
method) and to explain when one should mock an object.
Minimum set of tests Q18 35% 40% 25%
Tools (84 times, 4.21%). Students sometimes do not use the tools
100 50 0 50 100
properly. More specifically to our course, students commonly use
Figure 2: Students’ perception on the difficulty of each test- JUnit 4 features instead of JUnit 5, do not correctly use AssertJ’s
ing topic. Scale: very easy, easy, neutral, hard, very hard. The fluent API, and make wrong use of Cucumber features.
full questionnaire can be found in our appendix [1]. TAs perspective. Overall, the observations of TAs match with
what we observed in the labwork analysis. In terms of testing best
Teaching Assistants. All TAs are between 18 and 24 years-old, practices, TAs mentioned to help students in writing maintainable
one of them being female. They all ranked their Java knowledge test code. According to one TA, students often write tests that
between 8 and 10, and software testing knowledge between 7 and 8. contain unnecessary code and weird interactions with the class
Four of them are TAs for the first time in our software course; the under test. In addition, according to one TA, students do not clearly
other three TAs are performing this role for the third year in a row. see how to reuse test code. Another TA mentioned that a common
question is on how to properly test exceptions. Finally, a TA also
6 RESULTS observed that students often write tests that actually do not exercise
any production code (in this case, JUnit still shows a green bar,
6.1 RQ1 : What common mistakes do students
giving a false impression of success to the student).
make when learning software testing?
We characterize the labwork feedback in eight different themes
6.2 RQ2 : Which software testing topics do
(ordered by their frequency): test coverage, maintainability of test
code, understanding testing concepts, boundary testing, state-based students find hardest to learn?
testing, assertions, mock objects, and tools. In Figure 2, we show, based on the survey data, how students and
Test coverage (416 times, 20.87%). Students commonly either miss TAs perceive the difficulty of each of the topics we teach.
tests, i.e., they do not provide all the expected tests for a given piece Most students consider using the JUnit framework (Q1) as well
of code, or they write tests that are not totally correct, e.g., the test as to think about the Act-Arrange-Assert pattern that composes
does not actually test the piece of code, or the test exercises the any unit test (Q2) easy to learn. In fact, 76% and 73% of students
wrong class. In addition, we also observed cases (14) where the consider it easy or very easy to learn JUnit and to use the AAA
student actually “overtested” (i.e., wrote tests for more cases than pattern, respectively. These perceptions are also shared by TAs, and
required). matches with RQ1 results, as the number of feedback related to bad
tool usage is small (4.21%).
Maintainability of test code (407 times, 20.42%). Students often need Interestingly, applying MC/DC (Modified Condition/Decision
advice on how to write maintainable test code. More specifically, Coverage) [7] criteria to test complicated conditions (Q7) was con-
test quality advices in general, such as better naming and excessive sidered hard or very hard by 49% of the students; this is the hardest
complexity (247), code duplication and lack of reusability (69), tests topic among all of them. However, it seems that other coverage crite-
that could be split in two (31), better usage of test cleanup features, ria are easier to learn, as only 16% of students considered structural
such as JUnit’s Before and After (47). testing hard (Q6).
Understanding testing concepts (306 times, 15.35%). Students provide Applying software testing in a pragmatic way, as expected, was
incomplete answers or have difficulties when it comes to questions considered hard for students. Deciding how much testing is enough
that involve testing concepts and ideas, such as what flaky tests are (Q14) is also considered a hard topic by 42% of students (the second
about, advantages and disadvantages of unit and system tests, and most hard topic). TAs agree and even perceive this topic harder
the importance of removing test smells. than the students. This result also matches with our findings on
Boundary testing (258 times, 12.95%). Students often miss all the RQ1 , where test coverage is the most prominent topic in feedback.
tests required to cover a boundary (142). As we also ask them to In addition, writing the mininum set of tests that gives confidence
first build a decision table and then derive the tests, we also see (Q18) is considered hard for 25% of students and neutral for 40%.
that they often miss elements in the table (50) and generate tables Choosing the right level of testing (e.g., unit, integration, or system
that are not fully correct (46). tests) is not considered easy to all of them; 29% consider it easy,
SIGCSE’19, February 27–March 2, 2019, Minneapolis, MN, USA Maurício Aniche, Felienne Hermans, and Arie van Deursen

Lectures Q1 0% 7% 93% and discussions and interactions during the lecture (Q4) are con-
Guest lectures Q2 10% 18% 72%
sidered important by 75% and 65% of students, respectively. We
conjecture that discussions and live coding are moments in which
Live coding Q3 6% 19% 75%
students have the opportunity to discuss the topics they consider
Live discussions Q4 7% 28% 65%
hard, such as how much testing is enough, which test level to use,
PragProg book Q5 29% 51% 20% and test code best practices (as seen in RQ1 and RQ2 ).
ISTQB book Q6 31% 36% 33% On the other hand, the two books we use as textbooks in the
course are not considered fundamental for students. More specif-
Labwork Q7 1% 6% 93%
ically, 31% of students find the ISTQB [5] not important and 36%
Support from TAs Q8 7% 12% 80%
are neutral (Q6), whereas 29% of them find the PragProg [25] not
Related papers Q9 35% 34% 30% important and 51% are neutral (Q5). Reading related papers (Q9) is
AMA sessions Q10 30% 38% 32% also considered not important for 35% of them.
Midterm exam Q11 9% 19% 73%

100 50 0 50 100
6.4 Limitations of our study
The qualitative analysis of the open questions in the survey was
Figure 3: Importance of different activities in software manually conducted by the first author of this paper. The analysis,
testing learning. Scale=strongly disagree, disagree, neutral, therefore, could be biased towards the views of the authors. To
agree, strongly agree. The full questionnaire can be found mitigate the threat, we make all the data available for inspection in
in our appendix [1]. our online appendix [1],
TAs were responsible for giving feedback to students throughout
the study. Although we instruct all TAs on how to grade and what
50% of the students are neutral, and 21% perceive it as a hard topic kind of feedback to give (they all follow the same rubrics), different
(Q3). Not a single TA perceived this topic as easy. We believe these TAs have different personalities. In practice, we observed that some
findings highlight even more the importance of discussing even TAs provided more feedback than other TAs. While we believe this
more the pragmatic side of software testing. could have little impact on the percentages of each theme in RQ1 ,
When it comes to testing code best practices, students had a we do not expect any other theme to emerge,
contradicting perceptions. The usage of mocks to simulate a de- In terms of generalizability, although we analyzed the behavior
pendency (Q4) and writing fast, reproducible, and non-flaky tests of 230 students, we do not claim that our results are complete
(Q17) were considered easy topics to be learned by 42% and 56% of and/or generalizable. Furthermore, most students were dutch (we
students, respectively. TAs agree that students learn these topics only had 3 international students answering our survey), which
with less difficulties. However, when it comes to following testing may introduce cultural bias to our results. We urge researchers
best practices (Q9), 46% of students perceive it as an easy topic, to perform replications of this study in different countries and
while 71% of TAs perceive it as a hard topic for students. The stu- universities.
dents’ perceptions also contradicts the results of RQ1 , where we
observe a large number of feedback focused on best practices in 7 CONCLUSIONS
their assignments. Software testing is a vital discipline in any Software Engineering
Finally, testability seems less challenging for students than for curriculum. However, the topic poses several challenges to edu-
TAs. While students perceive optimizing code for testability (Q10) cators and to students. In this paper, we proposed a pragmatic
as just somewhat challenging (35% find it easy, 41% are neutral, and software testing curriculum and explored students’ common mis-
25% find it hard), 67% of TAs belive that testability is a hard topic for takes, hard topics to learn, favourite learning activities, important
students. As we conjecture that TAs have a better understanding learning outcomes, and challenges they face when studying soft-
of testability than the students, these findings suggest that the ware testing.
students are not sufficiently aware of the difficulty of testability. Researchers and educators agree that software testing education
is fundamental not only to industry, but also to research. We hope
6.3 RQ3 : Which teaching methods do students this paper helps the community to improve even more the quality of
find most helpful? their software testing courses. As Bertolino [4] states in her paper
on the achievements, challenges, and dreams on software testing
In Figure 3, we show how students perceive the importance of each
research: “While it is research that can advance the state of the art, it
learning activity we have in our software testing course.
is only by awareness and adoption of those results by the next-coming
Students perceive activities that involve practitioners as highly
generation of testers that we can also advance the state of practice.
important. More specifically, guest lectures from industry (Q2) were
Education must be continuing, to keep the pace with the advances in
considered important by 72% of participants. The Ask-me-Anything
testing technology”.
sessions (Q10), on the other hand, was considered important by
only 32% of participants; 38% are neutral, and 30% do not consider
ACKNOWLEDGMENTS
them important.
Moreover, different interactions during the lecture are also con- We thank all the students and teaching assistants that followed our
sidered important for students. Teachers performing live code (Q3) course in the last years.
Pragmatic Software Testing Education SIGCSE’19, February 27–March 2, 2019, Minneapolis, MN, USA

REFERENCES undergraduate computer science curriculum. In ACM SIGCSE Bulletin, Vol. 29.
[1] Maurício Aniche, Felienne Hermans, and Arie van Deursen. 2018. Pragmatic ACM, 360–364.
Software Testing Education: Appendix. (2018). [Link] [22] David Janzen and Hossein Saiedian. 2008. Test-driven learning in early program-
1459654. ming courses. In ACM SIGCSE Bulletin, Vol. 40. ACM, 532–536.
[2] Tara Astigarraga, Eli M Dow, Christina Lara, Richard Prewitt, and Maria R Ward. [23] Edward L Jones. 2001. An experiential approach to incorporating software testing
2010. The emerging role of software testing in curricula. In Transforming Engineer- into the computer science curriculum. In Frontiers in Education Conference, 2001.
ing Education: Creating Interdisciplinary Skills for Complex Global Environments, 31st Annual, Vol. 2. IEEE, F3D–7.
2010 IEEE. IEEE, 1–26. [24] Edward L Jones. 2001. Integrating testing into the curriculum—arsenic in small
[3] Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and Dave doses. ACM SIGCSE Bulletin 33, 1 (2001), 337–341.
Binkley. 2015. Are test smells really harmful? An empirical study. Empirical [25] Jeff Langr, Andy Hunt, and Dave Thomas. 2015. Pragmatic unit testing in Java 8
Software Engineering 20, 4 (2015), 1052–1094. with JUnit. The Pragmatic Bookshelf.
[4] Antonia Bertolino. 2007. Software testing research: Achievements, challenges, [26] Otávio Augusto Lazzarini Lemos, Fábio Fagundes Silveira, Fabiano Cutigi Ferrari,
dreams. In 2007 Future of Software Engineering. IEEE Computer Society, 85–103. and Alessandro Garcia. 2017. The impact of Software Testing education on code
[5] Rex Black, Erik Van Veenendaal, and Dorothy Graham. 2012. Foundations of reliability: An empirical assessment. Journal of Systems and Software (2017).
software testing: ISTQB certification. Cengage Learning. [27] Will Marrero and Amber Settle. 2005. Testing first: emphasizing testing in early
[6] FT Chan, TH Tse, WH Tang, and TY Chen. 2005. Software testing education and programming courses. In ACM SIGCSE Bulletin, Vol. 37. ACM, 4–8.
[28] Robert C Martin. 2002. Agile software development: principles, patterns, and
training in Hong Kong. In Quality Software, 2005.(QSIC 2005). Fifth International practices. Prentice Hall.
Conference on. IEEE, 313–316. [29] Robert C Martin. 2017. Clean architecture: a craftsman’s guide to software structure
[7] John Joseph Chilenski and Steven P Miller. 1994. Applicability of modified and design. Prentice Hall Press.
condition/decision coverage to software testing. Software Engineering Journal 9, [30] Scott Matteson. [n. d.]. Report: Software failure caused 1.7 trillion
5 (1994), 193–200. in financial losses in 2017. [Link]
[8] Peter J Clarke, Debra Davis, Tariq M King, Jairo Pava, and Edward L Jones. 2014. report-software-failure-caused-1-7-trillion-in-financial-losses-in-2017/.
Integrating testing into software engineering courses supported by a collaborative ([n. d.]).
learning environment. ACM Transactions on Computing Education (TOCE) 14, 3 [31] Renée McCauley and Ursula Jackson. 1999. Teaching software engineering early:
(2014), 18. experiences and results. ACM SIGCSE Bulletin 31, 2 (1999), 86–91.
[9] Stephen H Edwards. 2003. Rethinking computer science education from a test- [32] Gerard Meszaros. 2007. xUnit test patterns: Refactoring test code. Pearson Educa-
first perspective. In Companion of the 18th annual ACM SIGPLAN conference on tion.
Object-oriented programming, systems, languages, and applications. ACM, 148– [33] G. Miller. [n. d.]. A Scientist’s Nightmare: Software Problem Leads to Five
155. Retractions. [Link] ([n. d.]).
[10] Stephen H Edwards. 2004. Using software testing to move students from trial- Last visited in October, 2017.
and-error to reflection-in-action. ACM SIGCSE Bulletin 36, 1 (2004), 26–30. [34] SP Ng, Tafline Murnane, Karl Reed, D Grant, and TY Chen. 2004. A prelimi-
[11] Sebastian Elbaum, Suzette Person, Jon Dokulil, and Matt Jorde. 2007. Bug hunt: nary survey on software testing practices in Australia. In Software Engineering
Making early software testing lessons engaging and affordable. In Proceedings of Conference, 2004. Proceedings. 2004 Australian. IEEE, 116–125.
the 29th international conference on Software Engineering. IEEE Computer Society, [35] Alan Page, Ken Johnston, and Bj Rollison. 2008. How we test software at Microsoft.
688–697. Microsoft Press.
[12] Facebook. [n. d.]. Building and Testing at Facebook. [Link] [36] James Robergé and Candice Suriano. 1994. Using laboratories to teach software
[Link]/notes/facebook-engineering/building-and-testing-at-facebook/ engineering principles in the introductory computer science curriculum. In ACM
10151004157328920/. ([n. d.]). Last visited in October, 2017. SIGCSE Bulletin, Vol. 26. ACM, 106–110.
[13] Steve Freeman and Nat Pryce. 2009. Growing object-oriented software, guided by [37] Johnny Saldaña. 2015. The coding manual for qualitative researchers. Sage.
tests. Pearson Education. [38] Jaime Spacco and William Pugh. 2006. Helping students appreciate test-driven
[14] Vahid Garousi and Aditya Mathur. 2010. Current state of the software testing development (TDD). In Companion to the 21st ACM SIGPLAN symposium on
education in north american academia and some recommendations for the new Object-oriented programming systems, languages, and applications. ACM, 907–
educators. In Software Engineering Education and Training (CSEE&T), 2010 23rd 913.
IEEE Conference on. IEEE, 89–96. [39] Muhammad Dhiauddin Mohamed Suffian, Suhaimi Ibrahim, and Mo-
[15] Vahid Garousi and Junji Zhi. 2013. A survey of software testing practices in hamed Redzuan Abdullah. 2014. A proposal of postgraduate programme for
Canada. Journal of Systems and Software 86, 5 (2013), 1354–1376. software testing specialization. In Software Engineering Conference (MySEC), 2014
[16] Judith L Gersting. 1994. A software engineering “frosting” on a traditional CS-1 8th Malaysian. IEEE, 342–347.
course. In ACM SIGCSE Bulletin, Vol. 26. ACM, 233–237. [40] Joseph Timoney, Stephen Brown, and Deshi Ye. 2008. Experiences in software
[17] Michael H Goldwasser. 2002. A gimmick to integrate software testing throughout testing education: some observations from an international cooperation. In Young
the curriculum. In ACM SIGCSE Bulletin, Vol. 34. ACM, 271–275. Computer Scientists, 2008. ICYCS 2008. The 9th International Conference for. IEEE,
[18] Misko Hevery. 2008. Testability explorer: using byte-code analysis to engineer 2686–2691.
lasting social changes in an organization’s software development process.. In [41] James A Whittaker, Jason Arbon, and Jeff Carollo. 2012. How Google tests software.
Companion to the 23rd ACM SIGPLAN conference on Object-oriented programming Addison-Wesley.
systems languages and applications. ACM, 747–748. [42] Eric Wong. 2012. Improving the state of undergraduate software testing education.
[19] Thomas B Hilburn. 1996. Software engineering-from the beginning. In Software In American Society for Engineering Education. American Society for Engineering
Engineering Education, 1996. Proceedings., Ninth Conference on. IEEE, 29–39. Education.
[20] Thomas B Hilburn and Massood Townhidnejad. 2000. Software quality: a cur-
riculum postscript?. In ACM SIGCSE Bulletin, Vol. 32. ACM, 167–171.
[21] Ursula Jackson, Bill Z Manaris, and Renée A McCauley. 1997. Strategies for
effective integration of software engineering concepts and techniques into the