Software Testing and Quality Assurance

Theory and Practice

Chapter 3
Unit Testing

1
 Chapter 1 - Review
1. What is total quality management (TQM)? What is the difference between TQM and TQC?
2. Explain the differences between validation and verification.
3. Explain the differences between failure, error, and fault .
4. What is a test case? What are the objectives of testing?
5. Explain the concepts of unit, integration, system, acceptance, and regression testing.
6. What are the different sources from which test cases can be selected?
7. What is the difference between fault injection and fault simulation?
8. Explain the differences between structural and functional testing.
9. What are the strengths and weaknesses of automated testing and manual testing?
10. What the difference between white box and black box testing
11. What are the broad categories of Text Execution Metrics

2
 • Concept of Unit Testing
• Static Unit Testing
• Defect Prevention
• Dynamic Unit Testing
• Mutation Testing
• Debugging
3

• Unit Testing in eXtreme Programming

• Tools For Unit Testing
Outline of the Chapter
 • Static Unit Testing
– Code is examined over all possible behaviors that
might arise during run time
– Code of each unit is validated against
requirements of the unit by reviewing the code
• Dynamic Unit Testing
4

– A program unit is actually executed and its

outcomes are observed
– One observe some representative program
behavior, and reach conclusion about the quality
of the system
Concept of Unit Testing

• Static unit testing is not an alternative to

dynamic unit testing
• Static and Dynamic analysis are
complementary in nature
• In practice, partial dynamic unit testing is
performed concurrently with static unit
testing
• It is recommended that static unit testing be
performed prior to the dynamic unit testing
 • In static unit testing code is reviewed by
applying techniques:
– Inspection: step by step peer group review of
a work product, with each step checked against
pre-determined criteria
– Walkthrough: the author leads the team
through a manual or simulated executed of the
product using pre-defined scenarios
5

• The idea here is to examine source code in

detail in a systematic manner
• The objective of code review is to review
the code, and not to evaluate the author of
the code
Static Unit Testing

• Code review must be planned and

managed in a professional manner
• The key to the success of code is to divide
and conquer
– An examiner inspect small parts of the unit in
isolation
• nothing is overlooked
• the correctness of all examined parts of
the module implies the correctness of the
whole module
 Figure 3.1: Steps in the code review process
6
Performing Code Review

• Step 1: Readiness
– Criteria
• Completeness – must be available
• Minimal functionality – must compile and link
• Readability -
• Sufficient Complexity to warrant group review
• Requirements and design documents –
all should be available
– Roles
• Moderator
• Author
• Presenter
• Record keeper
• Reviewers
• Observer
• Step 2: Preparation
– List of questions
– Potential Change Request (CR)
– Suggested improvement opportunities
 7
Static Unit Testing (Code Review)

• Step 3: Examination
– The author makes a presentation
– The presenter reads the code
– The record keeper documents the Change Requests
(CR)
– Moderator ensures the review is on track
• Step 4: Re-work
– Make the list of all the Change Requests
– Make a list of improvements
– Record the minutes meeting
– Author works on the CRs to fix the issue
• Step 5: Validation
– CRs are independently validated
• Step 6: Exit
– A summary report of the meeting minutes is
distributes
 8
Static Unit Testing (Code Review)

A Change Request (CR) includes the following

details:
– Give a brief description of the issue
– Assign a priority level (major or minor) to a CR
– Assign a person to follow it up
– Set a deadline for addressing a CR
 Code Review Checklist
• Does the code do what has been specified in • Can each atomic function be reviewed
the design specification? and understood in 10–15 minutes? If not,
• Does the procedure used in the module it is considered to be too complex.
solve the problem correctly? • Has the code been adequately commented
• Does a software module duplicate another upon?
existing module which could be reused? • Have all the variables and constants been
• If library modules are being used, are the correctly initialized? Have correct types
right libraries and the right versions of the and scopes been checked?
libraries being used? • Are the global or shared variables, if there
• Does each module have a single entry point are any, carefully controlled?
and a single exit point? Multiple exit and • Are there data values hard coded in the
entry point programs are harder to test. program? Rather, these should be
• Is the cyclomatic complexity of the module declared as variables.
more than 10? If yes, then it is extremely • Are the pointers being used correctly?
difficult to adequately test the module. • Etc – see book for more questions
9
 • The number of lines of code
(LOC) reviewed per hour
• The number of CRs generated per
thousand lines of code (KLOC)
• The number of CRs generated per
hour
10

• The total number of hours spend

on code review process
The following metrics can be collected from a code review:
Static Unit Testing (Code Review)
 • The code review
methodology can be
applicable to review other
documents
• Five different types of
system documents are
11

generated by engineering
department
– Requirement
Static Unit Testing (Code Review)

– Functional Specification
– High-level Design
– Low-level Design
– code
• In addition installation,
user, and trouble shooting
guides are developed by Table 3.1: System documents
technical documentation
group
 • Goal is to reduce the number of CRs
generated during code review
– Build [internal diagnostic
tools]instrumentation code into the code
– Use standard control to detect possible
occurrences of error conditions –e.g divides by
zero
12

– Ensure that code exists for all return values

– Ensure that counter data fields and buffer
overflow/underflow are appropriately handled
– Provide error messages and help texts from a
common source
– Validate input data [e.g., arguments passed
onto a function]
Defect Prevention

– Use assertions to detect impossible conditions

– Fully document the assertions that appears to
be unclear
– After every major computation reverse-
compute the input(s) from the results in the
code itself [ e.g., computing the square root
and square of a number]
– Include a loop counter within each loop
– Define a variable to indicate the branch of
decision logic that will be taken. Check this
value after the decision has been made and the
right branch has supposedly been taken.
 • Execution-based unit testing, Unit tested in
isolation
• Execution differs from ordinary execution in
the following way:
– A unit under test is taken out of its actual
execution environment.
13

– The actual execution environment is emulated by

writing more code so that the unit and the
emulated environment can be compiled together.
– The above compiled aggregate is executed with
selected inputs.
Dynamic Unit Testing

• Outcome collected in a variety of ways, such

as on a screen, logging on files, and software
instrumentation of the code to reveal run time
behavior.
– The result is compared with the expected outcome.
• The caller unit is known as test driver
– A test driver is a program that invokes the unit under
test (UUT)
– It provides input data to unit under test and report the
test result
• The emulation of the units called by the UUT are
called stubs
– It is a dummy program
• The test driver and the stubs are together called
scaffolding
• how to determine input test data?
Dynamic Unit Testing

14
 • Control flow testing (chapter 4)
– Draw a control flow graph (CFG) from a
program unit
– Select a few control flow testing criteria
– Identify a path in the CFG to satisfy the
selection criteria
Selection of test data is broadly based on the following techniques:

– Derive the path predicate expression from

15

the selection paths

– By solving the path predicate expression
for a path, one can generate the data
• Data flow testing (chapter 5)
– Draw a data flow graph (DFG) from a
Dynamic Unit Testing

program unit and then follow the

procedure described CFG
• Domain testing (chapter 6)
– Domain errors are defined and then test
data are selected to catch those faults
• Functional program testing (chapter 9)
– Input/output domains are defined to
compute the input values that will cause the
unit to produce expected output values
 • A mutation of a program is a modification
of the program created by introducing a
single, small, legal syntactic change in the
code
• A modified program is called mutant
• A mutant is said to be killed when the
execution of test case cause it to fail. The
16

mutant is considered to be dead

• A mutant is an equivalent to the given
program if it always produce the same
output as the original program
• A mutant is called killable or stubborn, if
Mutation Testing

the existing set of test cases is insufficient

to kill it
• A mutation score for a set of test cases is
the percentage of non-equivalent mutants
killed by the test suite
• The test suite is said to be mutation-
adequate if its mutation score is 100%
 Mutation testing

•
8. exit(0); }
rank is %d \n”, r);
7. printf(“Value of the

atoi(argv[r]))
6. if (atoi(argv[i]) >
5. for i = 2 to 3 do
4. {r = 1;
3. char *argv[];
2. int argc, r, i;
1. main(argc,argv)

Consider the following program

rank is 1

Test Case 3:
rank is 2

Test Case 2:
is 3

Test Case 1:
P

output: Value of the rank

input: 1 2 3
output: Value of the
input: 3 1 2

output: Values of the

input: 1 2 1
r = i;
Mutant 1: Change line 5 to for i = 1 to 3 do
Mutant 2: Change line 6 to if (i > atoi(argv[r])) r = i;
Mutant 3: Change line 6 to if (atoi(argv[i]) >= atoi(argv[r])) r = i;
Mutant 4: Change line 6 to if (atoi(argv[r]) > atoi(argv[r])) r = i;
Execute modified programs against the test suite, you will get the results:
Mutants 1 & 3: Programs will pass the test suite, i.e., mutants 1 & 3 are not killable
Mutant 2: Program will fail test cases 2
Mutant 4: Program will fail test case 1 and test cases 2
Mutation score is 50%, assuming mutants 1 & 3 non-equivalent
17
 Mutation testing
• The score is found to be low because we assumed mutants 1 & 3 are nonequivalent
• We need to show that mutants 1 and 3 are equivalent mutants or those are killable
• To show that those are killable, we need to add new test cases to kill these two mutants
• First, let us analyze mutant 1 in order to derive a “killer” test. The difference between P and
mutant 1 is the starting point
• Mutant 1 starts with i = 1, whereas P starts with i = 2. There is no impact on the result r.
Therefore, we conclude that mutant 1 is an equivalent mutant
• Second, if we add a fourth test case as follows:
• Test Case 4:
• input: 2 2 1
• Program P will produce the output “Value of the rank is 1” and mutant 3 will produce the
output “Value of the rank is 2”
• Thus, this test data kills mutant 3, which give us a mutation score 100%
 Mutation testing makes two major
assumptions:
• Competent Programmer hypothesis
– Programmers are generally competent and
they do not create random programs,
except for simple errors.
19

– In other words, the mutants to be

considered are the ones falling within a
small deviation from the original program.
– mutation operators are expected to
model programming errors made by
Mutation Testing

programmers.
• Coupling effects
– Complex faults are coupled to simple faults in
such a way that a test suite detecting simple
faults in a program will detect most of the
complex faults
– If the software contains a fault, there will usually
be a set of mutants that can only be killed by a
test case that also detect that fault.
 • The process of determining the cause of a
failure is known as debugging
• The purpose is to isolate and determine its
specific cause, given a symptom of a
problem
• There are three approaches to debugging
20

– Brute force - Print statements are scattered

throughout the source code,
• dynamic debugger my be used allows the
software engineer to navigate by stepping
through the code, observe which paths
have executed, and observe how values of
variables change during execution
– Cause elimination
• Induction
Debugging

– collect data, what happened and the

symptoms, collected data is organized in
terms of behavior and symptoms, and their
relationship is studied to find a pattern to
isolate the causes.
– A cause hypothesis is devised, and the
above data are used to prove or disprove
the hypothesis
• Deduction
– all possible causes are developed in order
of their likelihoods, and tests are
conducted to eliminate or substantiate
each cause in decreasing order of their
likelihoods.
• Backtracking
– starts at a point in the code where a failure
was observed and traces back the
execution to the point where it occurred
 A Debugging Heuristic
• Reproduce the symptom(s).
– Gather all the information and conduct causal analysis The goal of causal analysis is to identify the root
cause of the problem and initiate actions so that the source of defects is eliminated.
• Formulate some likely hypotheses for the cause of the problem based on the causal analysis
• Develop a test scenario for each hypothesis to be proved or disproved
• Prioritize the execution of test cases
– highly probable hypotheses are executed first
• Execute the test cases in order to find the cause of a symptom
• Fix the problem.
• Document the changes which have been made.
– Document the changes in the source code itself to reflect the change.
– Update the overall system documentation.
– Changes to the dynamic unit test cases.
– File a defect in the defect tracking database if the problem was found after the code was checked in to the
version control system.
21
 Unit testing in extreme programming
• A Test Driven Development (TDD) approach to code development is used in the XP
methodology
– In XP, a few unit tests are coded first, then a simple, partial system is implemented to pass the tests
– programmer writes low-level tests before writing production code – referred to as test first
– Then, one more new unit test is created, and additional code is written to pass the new test, but not more,
until a new unit test is created. The process is continued until nothing is left to test

22
 Figure 3.3: Test-first process in XP

23
Unit Testing in eXtreme Programming

1. Pick a requirement, i.e., a story

2. Write a test case that will verify a
small part of the story and assign a
fail verdict to it
3. Write the code that implement
particular part of the story to pass
the test
4. Execute all test
5. Rework on the code, and test the
code until all tests pass
6. Repeat step 2 to step 5 until the
story is fully implemented
 • Three laws of Test Driven development
(TDD)
– Create tests first - Code is added only when a
tests breaks
– One may not write more of a unit test than is
sufficient to fail
– One may not write more production code than
24

is sufficient to make the failing unit test pass

Unit Testing in eXtreme Programming

• Creating unit tests helps a developer focus

on what needs to be done.
• Requirements, that is, user stories, are
nailed down firmly by unit tests.
– Tests before code – solidifies requirements
• Creating unit tests independently before
coding sets up checks and balances and
improves the chances of getting the
system right the first time
• Unit tests are released into the code
repository along with the code they test.
(code without one may not be released)
• Upon discovering a missing unit test it
must be created at that time
 • JUnit: It is a framework for performing
unit testing of Java programs.
– Other frameworks: NUnit (C#), CPPUnit (C+
+), fUnit (Fortran), unittest (python)
• Intuitive steps to test a method in Java
(Ex. Move() method of PlanetClass)
– Create an object instance of PlanetClass. Call
25

it Mars.
JUnit – A Framework for Unit Testing

– Select values of all input parameters of

Move().
– Compute the expected value to be returned by
Move(). Let it be y.
– Execute method Move() on Mars with the
selected input values.
• Let Move() return a value called z.
– Compare the actual output (z) returned by
Move() with the expected value (y).
• If (z == y), Move() passes the test;
otherwise it fails. Report the result.


• JUnit makes writing of test cases easier.

Next slide …

 • JUnit provides a basic class called
TestCase.
• The tester
– Extends the TestCase class for each test case.
10 extensions for 10 test cases.
– Alternatively, extend TestCase to have 10
methods for 10 test cases.
26

• The TestCase class provides methods to

JUnit – A Framework for Unit Testing

make assertions.
– assertTrue(Boolean condition)
– assertFalse(Boolean condition)
– assertEquals(Object expected, Object actual)
– assertEquals(int expected, int actual)
– assertEquals(double expected, double actual,
double tolerance)
– assertSame(Object expected, Object actual)
– assertNull(Object testobject)
– …
• The tester can have her own assertions.
 import TestMe; // TestMe is the class whose methods are going to
be tested.
import junit.framework.*; // This contains the TestCase class.
public class MyTestSuite extends TestCase { // Create a subclass
of TestCase
public void MyTest1() { // This method is the first test case
TestMe object1 = new TestMe( ... ); // Create an instance of
TestMe with desired params
int x = object1.Method1(...); // invoke Method1 on object1
27

assertEquals(365, x); // 365 and x are expected and actual

values, respectively.
JUnit – A Framework for Unit Testing

}
public void MyTest2() { // This method is the second test case
TestMe object2 = new TestMe( ... ); // Create another
instance of
// TestMe with desired parameters
double y = object2.Method2(...); // invoke Method2 on
object2
assertEquals(2.99, y, 0.0001d); // 2.99 is the expected
value;
// y is the actual value;
// 0.0001 is tolerance level
}
}
Figure 3.5: An example test suite
 • Code auditor
– This tool is used to check the quality of the
software to ensure that it meets some
minimum coding standard
• Bound checker
– This tool can check for accidental writes into
the instruction areas of memory, or to other
28

memory location outside the data storage area

of the application
• Documenters
– These tools read the source code and
automatically generate descriptions and
Tools For Unit Testing

caller/callee tree diagram or data model from

the source code
• Interactive debuggers
– These tools assist software developers in
implementing different debugging techniques
Examples: Breakpoint and Omniscient
debuggers
• In-circuit emulators
– It provides a high-speed Ethernet connection
between a host debugger and a target
microprocessor, enabling developers to
perform source-level debugging
 • Memory leak
detectors
– These tools
test the
allocation of
memory to
an
29

application
which
request for
memory and
fail to de-
allocate
memory
Tools for Unit Testing

• Static code
(path) analyzer
– These tool
identify paths
to test based
on the
structure of
code such as
McCabe’s
cyclomatic
complexity
measure
Table 3.3: McCabe complexity
measure
 • Software inspection support
– Tools can help schedule group inspection
• Test coverage analyzer
– These tools measure internal test coverage,
often expressed in terms of control structure of
the test object, and report the coverage metric
• Test data generator
30

– These tools assist programmers in selecting

test data that cause program to behave in a
desired manner
• Test harness
– This class of tools support the execution of
Tools for Unit Testing

dynamic unit tests

• Performance monitors
– The timing characteristics of the software
components be monitored and evaluate by
these tools
• Network analyzers
– These tools have the ability to analyze the
traffic and identify problem areas
 • Simulators and emulators
– These tools are used to replace the real
software and hardware that are not currently
available. Both the kinds of tools are used for
training, safety, and economy purpose
• Traffic generators
– These produces streams of transactions or data
31

packets.
• Version control
– A version control system provides
functionalities to store a sequence of revisions
of the software and associated information
files under development
Tools for Unit Testing