CSc 120

Introduction to Computer Programming II

08: Testing
Why test?
• Mars Climate Orbiter
‒ Purpose: to study the Martian
climate and to serve as a relay for
the Mars Polar Lander

‒ Disaster: Bad trajectory caused it to

disintegrate in the upper
atmosphere of Mars

‒ Why: Software bug - failure to

convert English units to metric
values (pound-seconds vs. newton-
seconds) as specified in the contract

2
Why test?
• THERAC-25 Radiation Therapy
‒ 1985 to 1987: two cancer patients at the East Texas
Cancer Center in Tyler received fatal radiation overdose
(a total of 6 accidents) – massive overdose
‒ Why: Software bug - mishandled race condition (i.e.,
miscoordination between concurrent tasks)

3
Why test?
• Hive Thermostat
• February, 2016: customers were roasting at home
• the thermostat mysteriously began setting the
temperature to 90 degrees F (32 C)

• Hive:
‒ “We are aware of a temporary glitch…where a certain
sequence of commands in the Hive iOS app can cause
thermostat temperature to rise to 90 degrees F.”

4
Why test?
• Hive user:

5
Purpose of testing
• Every piece of software is written with some
functionality in mind

• Testing aims to identify whether the program

meets its intended functionality
_______ of bugs, not their
‒ "testing can only prove the presence
absence"
‒ the more thoroughly your software is tested, the more
confidence you can have about its correctness

‒ "Test until fear turns into boredom." – Kent Beck

6
Testing and test cases

"thoroughly" ≠ lots of test cases

def main():
x = input()
if x %2 == 1: # x is odd
do_useful_computation() It isn’t enough to simply
have a lot of test cases.
else: They have to “cover” the
program adequately.
delete_all_files()
send_rude_email_to_boss()
crash_computer()

7
Approaches to testing
Black-box testing White-box testing
• Focuses only on • Focuses on the code
functionality ‒ examines the code to
‒ does not look at how figure out what tests to
the code actually works use

• Good for identifying • Good for identifying

missing features, bugs and programming
misunderstandings of
the problem spec errors

8
black-box testing

9
Black-box testing: what to test?
• Based purely on the desired functionality
‒ shouldn’t be influenced by the particular code you wrote
(that’s white-box testing)

• Aspects to consider:
‒ expected outcome
o normal vs error
‒ characterizing values
o edge cases vs “regular” values

10
Black-box testing: Outcomes
• Choose tests for both normal and error behaviors
‒ assumes that we know what the error situations are

• Desired program behavior:

‒ on normal inputs: produce the expected behavior
‒ on error inputs:
o detect and indicate that an error occurred
o then behave appropriately as required by the problem spec

• Passing a test:
‒ the program passes a test if it shows the desired
behavior for that test

11
Black-box testing: Values
• Edge cases:
‒ at or near the end(s) of the range of a value the program
is supposed to operate on
‒ Examples:
o “zero-related” : 0, [ ], empty string, empty file, ...
o “one-related” : 1, ‒1, list with one element, file with one line,
...
o (maybe) large values

• “Regular” values:
‒ not edge cases

12
Example:
“Read a file containing integers and print the sum of
the numbers that occur on odd-numbered lines.”

Sample input file:

13
Example “Read a file containing integers and print the sum of
the numbers that occur on odd-numbered lines.”

Testing for outcome (legal vs. error):

Normal behavior Error behavior
• no. of numbers = 1 • input file does not exist
‒ 0 adds (or is unreadable)
• no. of numbers = 3 • file has non-numeric
‒ 1 add; 1 skip in-between characters
• no. of numbers = 4 • empty line
‒ 1 add; 1 skip at end • more than one number
• > 4 numbers on a line
‒ several add operations
14
Example “Read a file containing integers and print the sum of
the numbers that occur on odd-numbered lines.”

Testing for values (edge cases vs. regular values):

Edge cases Regular values
• empty file • a file with several
• file with one number numbers, one per line

15
 Example “Read a file containing integers and print the sum of
the numbers that occur on odd-numbered lines.”

Putting these together:

Normal behavior Error behavior

edge
• empty file • input file does not exist (or
is unreadable)
• file with one number
• file has non-numeric
• a file with several characters
regular numbers, one per line
• empty line
• more than one number on
a line

16
 EXERCISE/ICA27
Do problems 1 and 2.

17
 EXERCISE-ICA27-p.1
Consider this program specification:

Write a program that reads a file name and computes (and

prints out) the length of the longest line in that file.

Specify input files that exemplify each of the following:

a) two error cases

b) two edge cases

c) one regular (normal) case 18

 EXERCISE/ICA27-p.1
Consider this program specification:

Write a program that reads a file name and computes (and prints
out) the length of the longest line in that file.

Specify input files that exemplify each of the following:

a) two error cases
the file does not exist
the file is readable but not organized into lines (it’s a JPEG,…)
a) two edge cases
the file has one line
the file is empty
a) one regular (normal) case
the file has many lines, each line containing readable values 19
 EXERCISE/ICA27-p.2
Consider this program specification:

Write a program that reads a (possibly empty) file containing only

numbers (and whitespace) and prints out the difference between the
smallest and largest numbers. An empty input file should generate no
output.

Specify input files that exemplify each of the following:

a) two error cases

a) two edge cases

b) one regular (normal) case

20
 EXERCISE
Consider the rhyming words assignment.
Specify input files that exemplify each of the following:

a) one error cases

b) one edge cases

c) one regular (normal) case

21
 EXERCISE-sol
Consider the rhyming words assignment.
Specify input files that exemplify each of the following:

a) two error cases

An input file that has more than one pronunciation per line
An empty input file

b) one edge case

A small input file with correct pronunciations, but no two
words in the file rhyme.

c) one regular (normal) case

An input file that has two words, where the pronunciation of
each has phonemes meet the rules for rhyming 22
white-box testing

24
White-box testing: what to test?
• Ideally, that every path through the code works
correctly
‒ but this can be prohibitively difficult and expensive

unit testing
• Instead, what we often do is:
‒ check that the individual pieces of the program work
properly
‒ verify expected inputs/outputs of individual functions
‒ use assert statements to check those assumptions (called
invariants)

25
Invariants and assertions
• Invariant: an expression at a program point that
always evaluates to True when execution reaches
that point

• Assertion: a statement that some expression E is an

invariant at some point in a program
‒ Python syntax:
assert E
assert E, ″error message″

26
Invariants and assertions
• Assertion: a statement that some expression E is an
invariant at some point in a program
o Python syntax:
assert E
assert E, ″error message″

• assert:
o E evaluates to True or False
o If E evaluates to True, program execution continues
o otherwise, the error message is printed and execution halts with
an AssertionError

27
Invariants and assertions

Assertion: a statement that some expression E is an

invariant at some point in a program
‒ Python syntax:
assert E
assert E, ″error message″
Example:
def sum_evens(nums):
assert len(nums) > 0, ”nums is empty”
...

28
Using asserts
• checking arguments to functions
‒ e.g., if an argument's value has to be positive
‒ Precondition(s) of the function
• checking data structure invariants
‒ e.g., i >= 0 and i < len(name)
• checking "can't happen" situations
‒ this also serves as documentation that the situation
can't happen
• after calling a function, to make sure its return
value is reasonable
‒ Postcondition(s) of the function

29
Using asserts
• Some invariants are complex:
‒ numlist has at least one even number
‒ arglist consists of strings that contain at least one
vowel
• You can write your own functions that can be used
in assert statements

30
 EXERCISE/ICA27
Do problems 3, 4, and 5.

31
 Whiteboard -REVIEW
• In black-box testing, what does the tester know about the code
being tested?

______________________________

• When black-box testing, what are some of the kinds of cases we

should test?

o _________
o _________
o _________

• How does white-box testing differ from black-box testing?

__________________________________________
32
Unit testing
• Tests individual units of code, e.g., functions,
methods, or classes
‒ e.g.: given specific test inputs, does the function behave
correctly?
o CloudCoder!
‒ useful for making programmers focus on the exact
behavior of the function being tested
o e.g., preconditions, postconditions, invariants
‒ helps find problems early
• Isolate a unit and validate its correctness
• Often automated, but can be done manually

33
 Unit testing
# grid_is_square(arglist) – returns True if arglist
# has the shape of a square grid, i.e.,
# the length of each element ("row") of arglist is
# equal to the number of rows of arglist
def grid_is_square(arglist):
num_rows = len(arglist)
for row in arglist:
if len(row) != num_rows:
return False
return True

34
 Unit testing
# grid_is_square(arglist) – returns True if arglist
# has the shape of a square grid, i.e.,
# the length of each element ("row") of arglist is
# equal to the number of rows of arglist
def grid_is_square(arglist):
num_rows = len(arglist)
for row in arglist:
if len(row) != num_rows:
return False
return True • Write three white box test cases (inputs) for
this. (Use a whiteboard.)
• (I.e., give the specific arglist that would be
passed in to test the function.)
35
Code coverage
• Code coverage refers to how much of the code is
executed ("covered") by a set of tests
‒ want to be at (or close to) 100%
‒ coverage tools report which parts of the program were
executed, and how much
o e.g., Coberta.py, CodeCover, Coverage.py (Python), etc.
• Figuring out how to increase coverage often leads
to testing edge cases

36
Unit testing: practical heuristics
• Check both normal and error behaviors
• edge-case inputs:
‒ zero values (0, empty list/string/tuple/file, …)
‒ singleton values (1, list/string/tuple/file of length 1, …)
‒ large values
• if statements: make sure each outcome (True/False)
is taken
• Loops: test 0, 1, >1 iterations

37
Unit testing: what to check?
• Not just “output is what we expect”
‒ very often, this is the only thing that programmers
check
‒ not enough:
o a program can produce the expected output “accidentally”

38
Passing test cases "accidentally"
• Problem spec:
‒ "Write a function grid_is_square(arglist) that returns
True if arglist is a square grid, i.e., its no. of rows equals
its no. of columns."

• Submitted "solution": Passes half the

test cases …
def grid_is_square(arglist):
return True
… but is wrong!

39
Unit testing: what to check?
• Not just “output is what we expect”
‒ very often, this is the only thing that programmers
check
‒ not enough:
o a program can produce the expected output “accidentally”

• Check that invariants hold at key points

40
Unit testing: what to check?
• Not just “output is what we expect”
‒ remember “accidental” success
• Check that invariants hold at key points

if

CHECK CHECK

CHECK

41
Unit testing: what to check?
• Check that invariants hold at key points

loop

CHECK

CHECK

42
Unit testing: what to check?
• Check that invariants hold at key points
①Check that nothing breaks if the loop
does not execute at all

loop

CHECK

CHECK

43
Unit testing: what to check?
• Check that invariants hold at key points
①Check that nothing breaks if the loop
does not execute at all

②Check that everything is initialized

loop
properly when the loop is first entered
CHECK

CHECK

44
Unit testing: what to check?
• Check that invariants hold at key points
①Check that nothing breaks if the loop
does not execute at all

②Check that everything is initialized

loop
properly when the loop is first entered
CHECK
③Check that everything is OK after going
around the loop
CHECK

45
Unit testing: summary
• Test normal (include edge cases) and error values
• If statements: test all branches (if/elif/else)

• Loops: check invariants for:

‒ 0 iterations
‒ 1 iteration
‒ >1 iteration

• Functions:
‒ check return values

46
Example: buggy list-lookup
# lookup(string, lst) -- returns the
# position where the given string
# occurs in lst.
def lookup(string, lst):
for i in range(len(lst)):
if string == lst[i]:
return i

47
Example: buggy list-lookup
# lookup(string, lst) -- returns the
# position where the given string
# occurs in lst. 0, 1, >1 iterations Þ lists
of length 0, 1, 2
def lookup(string, lst):
for i in range(len(lst)):
if string == lst[i]:
return i

48
Example: (buggy) list-lookup
# lookup(string, lst) -- returns the
# position where the given string
# occurs in lst. 0, 1, >1 iterations Þ lists
of length 0, 1, 2
def lookup(string, lst): both branches taken Þ
for i in range(len(lst)): string is at positions 0, 1
if string == lst[i]:
return i

49
Example: (buggy) list-lookup
# lookup(string, lst) -- returns the
# position where the given string
# occurs in lst. 0, 1, >1 iterations Þ lists
of length 0, 1, 2
def lookup(string, lst): both branches taken Þ
for i in range(len(lst)): string is at positions 0, 1
if string == lst[i]: some possible test inputs:
return i 'a', []
'a', ['a’]
'a', ['b','a’]

50
Example: (buggy) list-lookup
# lookup(string, lst) -- returns the
# position where the given string
# occurs in lst. 0, 1, >1 iterations Þ lists
of length 0, 1, 2
def lookup(string, lst): both branches taken Þ
for i in range(len(lst)): string is at positions 0, 1
if string == lst[i]: some possible test inputs:
return i 'a', []
Note: this will 'a', ['a’]
catch the no- 'a', ['b','a’]
return-value bug

51
 EXERCISE/ICA28
Do problems 1 and 2.

52
 EXERCISE-ICA28-p1
Write four unit tests for the function below:

# average(lst) -- returns the

# average of the numbers in lst.
def average(lst):
sum = 0
for i in range(len(lst)):
sum += lst[i]
return sum/len(lst)

53
 EXERCISE-ICA28-p1
# average(lst) -- returns the
# average of the numbers in lst.
0, 1, >1 iterations Þ lists
def average(lst): of length 0, 1, 2
sum = 0
for i in range(len(lst)):
sum += lst[i]
return sum/len(lst)

54
 EXERCISE-ICA28-p1
# average(lst) -- returns the
# average of the numbers in lst.
0, 1, >1 iterations Þ lists
def average(lst): of length 0, 1, 2
sum = 0
some possible test inputs:
for i in range(len(lst)): []
sum += lst[i] [17]
[5, 12]
return sum/len(lst)

55
 EXERCISE-ICA28-p1
Write four unit tests for the function below:

# average(lst) -- returns the

# average of the numbers in lst. 0, 1, >1 iterations Þ lists
of length 0, 1, 2
def average(lst):
some possible test inputs:
sum = 0 []
for i in range(len(lst)): [17]
sum += lst[i] [5, 12]

return sum/len(lst)
Note: this will catch the
divide-by-zero on empty list
bug
56
 EXERCISE-ICA28-p2
Write four unit tests for the function below:

# Returns a list consisting of the strings in wordlist

# that end with tail.
def words_ending_with(wordlist, tail):
outlist = []
for item in wordlist:
if item.endswith(tail):
outlist.append(item)
return outlist

57
Testing strategy
• Test as a part of program development
‒ try out small tests even when the code is only
partially developed (i.e., lots of stubs)
o helps catch problems at function boundaries, e.g., number
and types of arguments
o can help identify bugs in the design, e.g., missing pieces

• Start with tiny test inputs (work your way up to

small, then medium, then large)
‒ problems found on tiny inputs are usually easier to
debug

58
 EXERCISE/ICA28
Do problems 3 and 4.

59