Creating Your Own In-House

Software Testing Course

Cem Kaner, J.D., Ph.D.

Tutorial at the
Conference of the
Association for Software Testing
June 5, 2006
Copyright (c) Cem Kaner 2006. This work is licensed under the Creative Commons Attribution-ShareAlike License. To view a
copy of this license, visit http://creativecommons.org/licenses/by-sa/2.0/ or send a letter to Creative Commons, 559 Nathan
Abbott Way, Stanford, California 94305, USA.
These notes are partially based on research that was supported by NSF Grant EIA-0113539 ITR/SY+PE: "Improving the
Education of Software Testers." Any opinions, findings and conclusions or recommendations expressed in this material are
those of the author and do not necessarily reflect the views of the National Science Foundation.
 Source Materials on the Disk
• Video lectures
• Activities
• Assignments
• Sample exam questions
• Some readings
• A little additional instructional support material

• James Bach, Scott Barber, Tim Coulter and I are creating a site
like this at www.satisfice.com (“Satisfice University”) that will
host supervised courses. (Some will cost money, others will be
free.)
• I have a closed moodle site at Florida Tech but expect to open a
public one in mid-summer.
• Both of these will provide access to reusable content.
 Today’s Key Assertions
1. Testing is evolving slowly because there is so little educational
support for it.
2. University support will continue to be inadequate for the
foreseeable future. Companies will therefore have to develop
their own training strategies.
3. Commercial short courses are often ineffective because they
– try to cover too much,
– at too shallow a level,
– without application to the learner’s specific situation,
– with too little opportunity for practice,
– and less opportunity for assessment and feedback.
• You can develop a suitable training program that integrates into
your work environment at a low net cost.
• The full curriculum might be broader than you first suspect.
 Meet & Greet
• Who are you?
• What are the most important three things we should
know about your background?
• Why are you here?
• What do you hope to leave with?
 Initial task: Connect to Moodle
• Your card has your user ID and password
• Router is bedbug
• Encryption is WEP:
603a96b7e7
• Please update your profile by personalizing it in some
way.
• This task is complete when you have:
– Updated your profile

– Downloaded and viewed one piece of content from

any topic in the testing course
• I’ll do a quick tour while you’re setting up.
 Group Assignment
• Work in N (TBD) groups on the following questions:
– What have been the big new ideas in testing in the past N
years?
– Have they really been new or are we looking at slow
propagation?
– How does this compare to the rate of change in

ƒ programming?
ƒ technical support?
ƒ technical training?
ƒ project management?
– Why do you think this is?

• Please report your conclusions back to the main group.

Content Discussion of the Assignment
• Consider the answers and their implications for
training
Process Discussion of the Assignment
• What potential objectives did this group activity serve?
• What potential objectives did this activity not serve?

• In the classroom, we have a vast range of available

activities.
– The best choice is the one that serves the educational
objectives at hand.
 A degree program in testing?
• We considered this at Florida Tech
– Certainly a rich enough intellectual problem to deserve a degree.
(We’ll brainstorm the range later.)
– Serious risk of career stereotyping and lock-in (exactly the
opposite to objectives of university-level education)
– Abandoned in favor of a software engineering degree that offers
ƒ black box software testing
ƒ programmer testing
ƒ additional test-relevant courses (e.g. human factors) and
testing options.
– This is better than most other places but it still just scratches the
surface.
(By the way, with pull of U.S. national research funding
toward security and away from commercial software, it’s not
clear that Florida Tech is near a steady state)
 Summary so far
• Testing has evolved slowly compared to related fields
• Educational opportunities for testing are limited
• University-level educational opportunities will
continue to be limited

• Conclusion:
• You want a good tester?
– Plan A: Hire experienced staff who can demonstrate
relevant knowledge, skill and attitudes
– Plan B: Train your own staff

• This course is about Plan B.

 Teaching Goals Inventory
• Take the inventory

• If you finish early, start brainstorming an answer to

the following questions:
– What courses should testers “have to” take?

– For each course:

ƒ What are its key objectives?

ƒ What is its relevance to software testing?
• Over the course of the day, please post your answers to
these to the flipcharts
 Teaching Goals Inventory
It is useful to prioritize among goals that fit within 6 categories
• Discipline-specific knowledge and skill
e.g. Develop skill in using materials, tools, technology central to this subject
• Basic academic success skills
e.g. develop listening, reading, and speaking skills; develop appropriate study
skills, strategies & habits
• Higher order thinking skills
e.g. develop problem-solving skills
• Liberal arts and academic values
e.g. develop an openness to new ideas; develop an informed historical perspective
• Work and career preparation
e.g. develop ability to work productively with others; improve ability to organize
and use time effectively.
• Personal development
e.g. develop a sense of responsibility for one’s own behavior
See Angelo & Cross, Classroom Assessment Techniques, 1993.
 Teaching Goals Inventory
Cluster Goals in Percent rated Mean
cluster “essential” rating

1. Higher order thinking skills 1 – 8 (8)

2. Basic academic success skills 9 – 17 (9)

3. Discipline-specific knowledge & skills 18 – 25 (8)

4. Liberal arts & academic values 26 – 35 (10)

5. Work & career preparation 36 – 43 (8)

6. Personal development 44 – 52 (9)

 Teaching Goals Inventory
• How many did you rate essential?
• How are you going to find time to cover all of those?
• How well do you have to teach them to cover them well
enough?

• Commercial training appears to cover massive amounts

of material. Unfortunately, we are confounding
quantity with quality.

• Here is my experience report as a teacher (1983-2006)

The instructional challenge,
as I see it

Software testing
is cognitively complex,
requires critical thinking,
effective communication, and
rapid self-directed learning.
 Software testing is a
process of empirical,
technical investigation
of the product under
test conducted to
provide stakeholders
with quality-related
information.
http://www.testingeducation.org/BBST/BBST--IntroductiontoTestDesign.html
http://www.testingeducation.org/BBST/BBST--IntroductiontoTestDesign.html
 Contexts Vary Across Projects
Testers must learn, for each new product:
– What are the goals and quality criteria for the project
– What skills and resources are available to the project
– What is in the product
– How it could fail
– What the consequences of potential failures could be
– Who might care about which consequence of what failure
– How to trigger a fault that generates the failure we're seeking
– How to recognize failure
– How to decide what result variables to pay attention to
– How to decide what other result variables to pay attention to in the event of
intermittent failure
– How to troubleshoot and simplify a failure, so as to better
(a) motivate a stakeholder who might advocate for a fix
(b) enable a fixer to identify and stomp the bug more quickly
– How to expose, and who to expose to, undelivered benefits, unsatisfied
implications, traps, and missed opportunities.
It's kind of like CSI
MANY tools, procedures,
sources of evidence.
• Tools and procedures don't define
an investigation or its goals.
• There is too much evidence to test,
tools are often expensive, so
investigators must exercise
judgment.
• The investigator must pick what to
study, and how, in order to reveal
the most needed information.
 Commercial Teaching Style
• Primary communication style
was lecture
– Real-life examples

ƒ Motivating
ƒ Memorable
ƒ Illustrate applications
ƒ Illustrate complexity
• Lectures can be excellent for
conveying basic knowledge,
but they are weak for
developing higher order
cognitive skills
 Example Problem: Domain Testing

• Most widely taught testing technique

– For details, see
http://www.testingeducation.org/BBST/Domain.html
– Easy to explain the basic concepts

– Classic examples widely taught

– Students quickly signal that they understand it

– But when you give them exercises under slightly new

circumstances
ƒ They blow it
– And then they blow the next one
o And the next one . . .
 Brilliant (?) idea
• Lots of practice
exercises
• Like we used to do as
math students
 I Tried This With Commercial Students
• Many (often, most) of them needed a lot of practice
under changing circumstances
• But the perceived slow pace of the course made them
anxious
• And the shorter topic checklist created a marketing
disadvantage for my courses.
 So I Became a University Instructor
• Big pay cut, but a clear goal:

\ÅÑÜÉäx à{x áàtàx Éy à{x ÑÜtvà|vx |Ç

áÉyàãtÜx àxáà|Çz uç |ÅÑÜÉä|Çz à{x àxáà@
ÜxÄtàxw xwâvtà|ÉÇ Éy áÉyàãtÜx àxáàxÜá
• This responds to my belief that the most significant cause
of the slow pace of change in testing, compared to
programming, is the weak educational support for theory,
practice, and experimentation in testing.
 Commercial vs. Academic
• Drive-by teaching • Local teaching
– 2-5 days, rapid-fire ideas, – Several months, a few
visiting instructor hours per week, students
get to know instructor
• Broad, shallow coverage • Deeper coverage
• Time constraints limit • Activities expected to
activities develop skills
• No time for homework • Extensive homework
• No tests • Assessment expected
• Familiarity • Mastery
• Work experience helps to • Students have no work
bring home concepts experience, need context
• Objective: one applicable • Expect mastery of several
new idea per day concepts and skills
 The idea is to develop
courses in an academic
environment (where I can
learn more about what
works and why), with the
goal of providing an
alternative model for
commercial (in-house)
training and professional
self-study
 Back to that
Brilliant (?) idea
• Lots of practice
exercises
• Like we used to do as
math students

• It was impractical in
commercial training
• Now, at last, we can
try it on university
students.
 Padmanabhan's Thesis:
Practice on Domain Testing
• 15 classroom hours of lecture plus examples plus practice, practice,
practice. Lots of procedural instruction and drill
• Students mastered every procedure
• Final exam
– Applied what they knew to similar questions (near transfer)

ƒ They aced them

– Applied what they knew to a problem that was beyond their
practice (not beyond the lecture) (a little bit farther transfer)
ƒ They all failed miserably
• Successful transfer of learning requires more than procedural
training and practice (This is a what-else-is-new result in science
education.)
Dealing With the Transfer Problem
• In science / math
education, the
transfer problem
is driving
fundamental
change in the
classroom
• Students learn
(and transfer)
better when they
discover
concepts, rather
than by being told
them
 Andragogy
• Pedagogy: study of
teaching / learning of
children
• Andragogy: study of
teaching / learning of adults
• University undergrads are
in a middle ground
between the teacher-
directed child and the fully-
self-directed adult
• Both groups, but especially
adults, benefit from activity-
based and discovery-based
styles
Characterizing Cognitive Complexity
• Anderson & Krathwohl
(2001) provide a modern
update to Bloom's (1956)
taxonomy
Characterizing Cognitive Complexity
Cognitive Process Dimension

Remember Understand Apply Analyze Evaluate Create

Knowledge Dimension

Factual lecture lecture

Conceptual lecture lecture

Procedural lecture lecture

Meta-
Cognitive

Anderson & Krathwohl, 2001

We Start Over
• Cognitively complex
material
• We need to develop skill,
judgment, and attitudes,
not just knowledge of facts
and definitions
• We face the usual (for
science education)
transfer problems
• Set a few explicit learning
objectives
• And assess against them
 My Learning Objectives
• Learn many test techniques well enough to know how,
when, and why to use them
• Foster strategic thinking--prioritization, designing
tests/reports for specific audiences , assess the
requirements for complex testing tasks (such as test
automation, test documentation)
• Apply (and further develop) communication skills (e.g. for
bug reporting, status reporting, specification analysis)
• Improve and apply teamwork skills (peer reviews, paired
testing, shared analysis of challenging problems)
• Gain (and document) experiences that can improve the
student's chances of getting a job in testing
 Here's Where We Are Now
• We created a variety of out-of-classroom activities, such
as homework (with application to real products) and
group study sessions
• Students praised the (by now, well polished) lectures
• But they often told us that they learned the most from the
out-of-class activities
• In many cases, the most effective (our subjective
assessment) student-and-instructor interactions happened
out of class, such as discussions at the local cafe.
• So we turned the class inside out
– Lectures out of the classroom

– Activities (including discussion) in the classroom

 Tour de Course
• The BBST main page
• The first instructional segment (later ones are
similarly structured):
– Video lecture

– Lecture slides

– Worked examples such as this one

– Review quizzes

– Activities

ƒ Soon to be applied to a product under test

– A set of study guide questions

ƒ From a much larger pool

 Lectures On-Line
• http://www.testingeducation.org/BBST
• Video lectures
– Students watch them before class
– Take simple quiz that checks that they watched the
video and paid attention
– Then we do in-class activities
• The results seem good
– Good student satisfaction
– Exam results aren’t as different as I expected
– Not enough time for the activities
ƒ In an in-house course, time is not constrained by
the same type of schedule. It is constrained by
value to the project and the staff.
 Application Under Test
• Like service learning, but not as heavy a commitment
for the students or for me
– Facilitates student learning (application level and
above)
– Facilitates student transfer of skills / knowledge to
the workplace
– We pick a well-known product

– Students apply what they learn to that product

– Typically, I use an open source product because it

avoids NDA problems, students can show their
work at interviews
• In an in-house course, the AUT is your product
 Study Guides
• http://www.testingeducation.org/k04/BBSTreviewf
all2005.htm
• 100 questions, include all candidates for mid-term and
final exam
• Students prepare answers together, assess each other's
work
• I can require well-organized, thoughtful answers
• Fosters strategic preparation
• Reduces disadvantage of students whose native
language is not English
• Creates cooperative learning tasks that should help
limited-English-proficiency students improve language
skills
 Study Guides
• Study guide results
– Students inexperienced with these, often blow the first test

– Make-up mid-terms

ƒ Replace grade, not average, not best 1 of 2 results

ƒ Students who take it improve more (1st test compared to
final exam) than students who did not take it
– Practice effect, motivation confound

– Writing is better, answers are better, I have greater

freedom to grade less forgivingly
– Many students told me this was the most valuable learning
experience in the course, and the most time-consuming
 Assessing student reaction
• Chose the Student Assessment of Learning Gains
http://www.flaguide.org/cat/salg/
• Measures student perceptions of their 'gains' in
learning
• Customizable
• Administered online
• FREE
• Beats the standard course evaluation form!
• Students each spent over an hour providing their
evaluation.
 Let’s play with Moodle
• Multiple choice review questions
• Submitting assignments
• Peer review of assignments
– Forums

– Wikis

– Workshops

Example of a useful group activity:

• Heuristic test strategy model applied to Firefox
specification
– Compared student results

– Compared results of using different tools

 More playing with Moodle
• Creating clone courses
• Using Moodle as a broader purpose, password-
protected, content management system
• Q & A freeform
How would you structure your course?
• One vision of the in-house course
– Meet weekly for a year

– Watch 10-25 minutes of video in advance

– Discuss the lesson and its applicability

– Over the next week, try to apply it on the job to the

current project(s) in test
– Discuss the application results in the next week or
move to the next segment.
• At the end of the course, students know how things fit
into their environment, and have multiple examples
(from multiple student colleagues).
 Discussion in groups
• Develop at least two additional course models that
would work in your environment
 If we have time …
• Let’s come back to curriculum
• What are the key topics
– And what courses do we need to develop to meet
them?
• How can we create such a series of courses?