Why, what, where?

• Zero defect software practically impossible

to achieve
Verification
• Careful and continuous verification needed
• Everything must be verified (spec.
(Note: verification used in a very documents, design documents, test data, …)
general sense, including validation) – even the verification must be verified!
• Verification along all development process,
not just at the end

Carlo Ghezzi--SE-Verif 1 Carlo Ghezzi--SE-Verif 2

Software engineering
Verification in engineering
• Programs do not display a “continuous”
• Example of bridge design behavior
• One test assures infinite correct situations • Verifying the function in one point does not
tell us anything about other points
– Example 1
...
a = … / (x +20) ...
...
Any value of x is ok, except for x = -20!

Carlo Ghezzi--SE-Verif 3 Carlo Ghezzi--SE-Verif 4

 Difficulties Main approaches
• For many qualities, it is not a binary • TESTING
(yes/no) property – experimenting with behavior of the products
• Many properties are subjective – sampling behaviors
• GOAL: find “counterexamples”
• Some are even implicitly stated
– is a dynamic technique
• ANALYSIS
– analytic study of properties
– is a static technique
Carlo Ghezzi--SE-Verif 5 Carlo Ghezzi--SE-Verif 6

Testing Testing and debugging

• Program testing can be used to show the • Testing should identify the presence of
presence of bugs, but never to show their errors
absence. (Dijkstra 1972) • Debugging must locate and eliminate errors
– random testing is not effective to find flaws (it • Then a test may be repeated to see if the
is “blind”, it does not “look for bugs”) error has been eliminated
– repeatability is essential!
– For concurrent /real-time systems it may be
difficult to achieve
Carlo Ghezzi--SE-Verif 7 Carlo Ghezzi--SE-Verif 8
 Definitions
Definitions • FAILURE
– P(d) is not correct
• may be undefined (error state) or may be the wrong
• P (program), D (input domain), R (output result
domain)
• ERROR
– P: D → R (may be partial)
– anything that causes a failure
• correctness defined by OR ⊆ D × R • typing mistake
– P(d) correct if <d, P(d)> ∈ OR • programmer forgot to test “x = 0”
– P correct if all P(d) are correct • FAULT
– incorrect intermediate state entered by program
Warning: definitions are not standardized!
Carlo Ghezzi--SE-Verif 9 Carlo Ghezzi--SE-Verif 10

Goals Definitions
• TESTING • Test case t
– Try to increase the probability that the presence – an element of D
of an error results in a failure • Test set T
• DEBUGGING – a finite subset of D
– Try to identify faults as early as possible • Test is successful if P(t) is correct
• Test set successful if P correct for all t in T
Warning: definitions are not standardized!

Carlo Ghezzi--SE-Verif 11 Carlo Ghezzi--SE-Verif 12

 Test criteria
Definition
• A criterion C defines finite subsets of D (test
• Ideal test set T sets)
– if P is incorrect, there is an element of T such – C⊆P (D)
that P(d) is incorrect
• A test set T satisfies C if is an element of C
• if an ideal test set exists for any program, – Example
we could prove program correctness by C = {<x1, x2,..., xn> | n ≥ 3 ∧ ∃ i, j, k, ( xi<0 ∧ xj=0 ∧ xk>0)}
testing <-5, 0, 22> is a test set that satisfies C
<-10, 2, 8, 33, 0, -19> also does
<1, 3, 99> does not

Carlo Ghezzi--SE-Verif 13 Carlo Ghezzi--SE-Verif 14

Properties of criteria (1)

• C is consistent Properties of criteria (2)
– for any pairs T1, T2 satisfying C T1 is
successful iff T2 is successful • C1 is finer than C2
• so any of them provides the “same” information – for any program P
• C is complete • for any T1 satisfying C1 there is a subset T2 of T1
which satisfies C2
– if P is incorrect, there is a test set T of C that is
not successful
• C is complete and consistent
– identifies an ideal test set
– allows correctness to be proved!
Carlo Ghezzi--SE-Verif 15 Carlo Ghezzi--SE-Verif 16
 Empirical testing criteria
Properties of properties
• Attempted compromise between the
• None is effective, i.e., no algorithms exist to impossible and the inadequate
state if a program, test set, or criterion has • Goal: try to divide D into subdomains D1,
that property D2, …, Dn where any element of each Di is
• In particular, there is no algorithm to derive likely to have similar behavior
a test set that would prove program – D = D1 ∪ D2 ∪… ∪ Dn
correctness • Select one test for each subdomain
– there is no constructive criterion that is • If Dj ∩ Dk ≠ ∅ then use an element in D’=
consistent and complete Dj ∩ Dk to reduce tests
Carlo Ghezzi--SE-Verif 17 Carlo Ghezzi--SE-Verif 18

Empirical testing criteria Testing in the small

We try to achieve a “complete coverage principle”:
If D is partitioned into D1, D2, …, Dn, the test set T We test individual modules
contains at least one element from each Di • BLACK BOX (functional) testing
– partitioning criteria based on the module’s
specification
• WHITE BOX (structural) testing
– partitioning criteria based on module’s internal
code

Carlo Ghezzi--SE-Verif 19 Carlo Ghezzi--SE-Verif 20

 Structural Coverage Testing
• (In)adequacy criteria
– If significant parts of program structure are not
tested, testing is inadequate
White box testing • Control flow coverage criteria
– Statement coverage
derive specifications from program code – Edge coverage
– Condition coverage
– Path coverage
– Data flow (syntactic dependency) coverage

Carlo Ghezzi--SE-Verif 21 Carlo Ghezzi--SE-Verif 22

Statement Coverage
Statement coverage
int select(int A[], int N, int X) i=0
{
• Select a test set T such that every int i=0;
while (i<N and A[i] <X)
statement in P is executed at least once by {
i<N and A[i] <X
True
some d in T if (A[i]<0) False
A[i]<0
A[i] = - A[i]; True
– each Di is the set of data that execute i++; False
A[i] = - A[i];
}
statement i return(1); return(1)
– we should try to minimize the number of Di } i++
to make a partition One test datum (N=1, A[0]=-7, X=9) is enough to guarantee statement
coverage of function select
Faults in handling positive values of A[i] would not be revealed

Carlo Ghezzi--SE-Verif 23 Carlo Ghezzi--SE-Verif 24

 Edge Coverage
Edge coverage criterion
int select(int A[], int N, int X) i=0
{
• Select a test set T such that every edge int i=0;
while (i<N and A[i] <X) i<N and A[i] <X
(branch) of the control flow is traversed at { True
False
least once by some d in T if (A[i]<0) A[i]<0
True
A[i] = - A[i];
i++; False
A[i] = - A[i];
}
return(1)
return(1);
} i++

We must add a test datum (N=1, A[0]=7, X=9) to cover branch False of
the if statement. Faults in handling positive values of A[i] would be
revealed. Faults in exiting the loop with condition A[i] <X would not be
revealed
Carlo Ghezzi--SE-Verif 25 Carlo Ghezzi--SE-Verif 26

Condition Coverage
Condition coverage criterion
int select(int A[], int N, int X) i=0
{
• Select a test set T such that all possible int i=0;
while (i<N and A[i] <X) i<N and A[i] <X
values of the constituents of compound { True
False
conditions are exercised at least once if (A[i]<0) A[i]<0
True
A[i] = - A[i];
i++; False
A[i] = - A[i];
}
return(1)
return(1);
} i++

Both conditions (i<N), (A[i]<X) must be false and true for different tests.
In this case, we must add tests that cause the while loop to exit for a
value greater than X. Faults that arise after several iterations of the loop
would not be revealed.
Carlo Ghezzi--SE-Verif 27 Carlo Ghezzi--SE-Verif 28
 Path Coverage
Path coverage criterion
int select(int A[], int N, int X) i=0
{
• Select a test set T which traverses all paths int i=0;
while (i<N and A[i] <X)
from the initial to the final node of P’s {
i<N and A[i] <X
True
control flow if (A[i]<0) False
A[i]<0
A[i] = - A[i]; True
– all paths is unfeasible, must specify conditions i++; False
} A[i] = - A[i];
return(1); return(1)
} i++;

The loop must be iterated given number of times.

PROBLEM: uncontrolled growth of test sets. We need to select a
significant subset of test cases.
Carlo Ghezzi--SE-Verif 29 Carlo Ghezzi--SE-Verif 30

Coverage-based test evaluation Data-flow based coverage criteria

Number of items covered • Evaluate where a variable is defined
×100
Total number of items a = 100;
• where a variable is used
printf (a);
Ideally it should be 1
but can be impossible
to achieve a = b + 1;
a is defined; b is used

Carlo Ghezzi--SE-Verif 31 Carlo Ghezzi--SE-Verif 32

 The infeasibility problem
Data Flow Coverage • Syntactically indicated behaviors
int select(int A[], int N, int X) DEF={i} (statements, edges, paths, data flows, etc.)
{
int i=0;
are often impossible
while (i<N and A[i] <X) USE={i,N,A,X} – unreachable code, unfeasible edges, paths, etc.
{ True
if (A[i]<0) False
USE={A,i} True
• Adequacy criteria may be impossible to
A[i] = - A[i];
i++; False
USE={A,i}
satisfy
}
return(1);
DEF{A,i} – Manual justification for omitting each
} impossible test case
USE={i}
DEF={i} – Adequacy “scores” based on coverage
• example: 95% statement coverage, 80% def-use
Exercise Def-Use paths: selects paths based on effects on the coverage
variables, rather than number of iteration of loops
Carlo Ghezzi--SE-Verif 33 Carlo Ghezzi--SE-Verif 34

Symbolic execution
• Propagates the symbolic values of vars over
each execution paths Black box testing
• Symbolic state:
<path-condition, symbolic bindings>
derive test sets from specifications
• Synthesizing a patch condition means
finding values which would imply
executing that path

Carlo Ghezzi--SE-Verif 35 Carlo Ghezzi--SE-Verif 36

 Specifications in natural language
• Derivation of test set cannot be automated Exercise
• some structure (e.g., organization standards)
can help An event queue in a simulation system is a
priority queue where events are extracted
• guidelines to increase confidence level and
according to their time-stamps (earliest
reduce discretionality: at least on test case for
time-stamps first, last-in among events with
each
the same time-stamp)
– subsets of “valid” homogeneous data
– “non valid” (combination of) data
– boundary data • Devise a set of functional test cases for an
– specific data (treated independently, error prone,...) event queue
Carlo Ghezzi--SE-Verif 37 Carlo Ghezzi--SE-Verif 38

A possible solution
• Did you consider at least the following cases: A possible solution (cont.)
• extracting an event before any insertion
• inserting only events with different time-stamps out
• extracting more events than previously inserted
of time-stamp order
• extracting the only event in a queue
• inserting at least two events with the same time-
• many insertions followed by exactly the same number stamp without interleaving events
of extractions
• inserting at least two events with the same time-
• many alternate insertions/extractions stamp with interleaving events
• inserting as many events as the capacity (if defined) • insert/extract in/from a non initialized queue (in
• inserting more events than the capacity (if defined) some languages, e.g., Java, may not be possible)
• inserting only events with different time-stamps in • ....
time-stamp order

Carlo Ghezzi--SE-Verif 39 Carlo Ghezzi--SE-Verif 40

 Decision table based testing P *

B * *

“The word-processor may present portions of text in I * *

three different formats: plain text (p), boldface (b), E * *

italics (i). The following commands may be applied to
SE * * *
each portion of text: make text plain (P), make boldface
(B), make italics (I), emphasize (E), superemphasize E=B *

(SE). Commands are available to dynamically set E to E=I *

mean either B or I (we denote such commands as E=B SE = B

*
and E=I, respectively.) Similarly, SE can be SE = I
*
dynamically set to mean either B (command SE=B) or I SE = B + I
(command SE=I), or B and I (command SE=B+I.)” *

action
p b i b i b i b,i b,i

Carlo Ghezzi--SE-Verif 41 Carlo Ghezzi--SE-Verif 42

Cause effect graphs Further constraints

B “Both B and I exclude P (i.e., one cannot ask both for
I O b
plain text and, say, italics for the same portion of text.)
P
R
E and SE are mutually exclusive.”
E A
N
a at least a
E=B D
i
at most one
SE one e b i b

E=I
A p
SE = B N c c
O D
SE = I R
SE = B + I requires
one and only a masks
one a a

The AND/OR graph represents the correspondence between o b r m

causes and effects b b

Carlo Ghezzi--SE-Verif 43 Carlo Ghezzi--SE-Verif 44

 B
m
I O
Coverage criterion
m b
R
P

E A
• Generate all possible input combinations
E=B
N
D
i and check outputs
SE
m
• May reduce the number by going
m
E=I backwards from outputs
A p
SE = B N – OR node with true output:
O D
SE = I R • use input combinations with only one true input
SE = B + I – AND node with false output:
X m Y = X implies not Y • use input combinations with only one false input

Carlo Ghezzi--SE-Verif 45 Carlo Ghezzi--SE-Verif 46

White vs black box

Syntax driven testing
Black box White box
• Given the syntactic structure of input, • it depends on the • it is based on control or
specification notation data flow coverage
generate tests that cover all rules • it scales up • it does not scale up (mostly
<expression> ::= <expression> + <term> | (different techniques at applicable at unit and
<expression> - <term> | <term> different granularity integration testing level)
<term> ::= <term> * <factor> levels) • it cannot reveal missing
| <term> / <factor> | <factor> • it cannot distinguish path errors
<factor> ::= ident | ( <expression>) between different (part of the specification
implementations (same that is not implemented)
• The specification is formal, and test specification
generation can be automated implemented with
different modules)
Carlo Ghezzi--SE-Verif 47 Carlo Ghezzi--SE-Verif 48
 The budget coverage criterion The oracle problem
How to inspect the results of executing test
• Industry’s answer to “when is testing done”
and reveal failures
– When the money is used up
– When the deadline is reached
• Oracles are required at each stage of testing
• This is sometimes a rational approach! • Automated test oracles are required for
running large amounts of tests
– Implication 1: Adequacy criteria answer the
wrong question. Selection is more important. • Oracles are difficult to design - no universal
– Implication 2: Practical comparison of recipe
approaches must consider the cost of test case
selection
Carlo Ghezzi--SE-Verif 49 Carlo Ghezzi--SE-Verif 50

Module testing in context Create Scaffolding

• Must create the context for executing the Goal
unit To setup the environment for executing the
D test
R initialization of non-local variables
• DRIVER I initialization of parameters ORACLE
V
– must initialize parameters and non local E activation of the unit check the
R
variables correspondence
between the
– must call the unit PROGRAM UNIT
produced and
• STUB “templates” of modules used by the the expected
S result
– must provide some replacements for the called T unit (functions called by the unit)
U
units B “templates” of any other entity used
by the unit
Carlo Ghezzi--SE-Verif 51 Carlo Ghezzi--SE-Verif 52
 Integration testing and USES
Regression Testing
relation Testing a new version (release): how can we
A minimize effort using results of testing of
If integration and test previous versions?
proceeds bottom up
B C only need drivers • On a previous release:
– save scaffoldings (drivers,stubs,oracles)
D E – record test cases (<inputs,outputs>)
• On the new release:
– keep track of changes
– evaluate impact of changes
Carlo Ghezzi--SE-Verif 53 Carlo Ghezzi--SE-Verif 54

Test Documentation (cont....)

Test Documentation • Documentation of test suites:
– software tested
– version
• Must be an organization standard – goal
• Depends on – overall results
– author
– organization (size,turnover,..),
– type of software (criticality, average life, • Documentation of Test Cases:
– goal
complexity, number of versions,...) – “environment” (driver,stub,oracle)
• It must include at least: – input
– test suite documentation – expected output
– actual output
– test case documentation – result
– observations
Carlo Ghezzi--SE-Verif 55 Carlo Ghezzi--SE-Verif 56
 Tool Support
• Planning and Management
Tool Support (cont....)
• activity planning
• Coverage Analysis
• error tracking – simple structural coverage
• repository management (statement|branch|path)
• report generation – graphical (incremental) visualization
• Test execution – report generation
• (code analysis) • Regression testing
• (partial) generation of scaffoldings – re-execution of test suites
• capture and playback
– comparison with previous runs
• record of results of test runs
– report generation
Carlo Ghezzi--SE-Verif 57 Carlo Ghezzi--SE-Verif 58

Software Inspection:
Low tech but effective
• Fagan Code Inspections
– One of many “walk-through” and inspection
techniques; among the most successful
Software Inspections • More formal and well-defined than “structured
walk-throughs” etc.

Fagan code inspections – Has been extended to designs, requirements,

etc. with similar organizing principles
Related formal review techniques – A completely manual technique for finding and
Lessons correcting errors

Carlo Ghezzi--SE-Verif 59 Carlo Ghezzi--SE-Verif 60

 Software Inspection Roles Software Inspection Process
• Planning
• Moderator:
• Moderator checks entry criteria, choose participants,
– Typically borrowed from another project. schedule meeting
Chairs meeting, chooses participants, controls • Overview
process
• Provide background education, assign roles
• Readers, Testers: • Preparation
– Read code to group, look for flaws
• Inspection (see ahead)
• Author:
• Rework
– Passive participant; answer questions when
asked • Follow-up (& possible re-inspection)
Carlo Ghezzi--SE-Verif 61 Carlo Ghezzi--SE-Verif 62

Checklists — NASA example

In the Meeting
From
From“Software
“SoftwareFormal
FormalInspections
InspectionsGuidebook,”
Guidebook,”
Office
Officeof
ofSafety
Safetyand
andMission
MissionAssurance,
Assurance,NASA-
NASA-
GB-A302
GB-A302approved
approvedAugust
August1993
1993
• Goal: Find as many faults as possible • About 2.5 pages for C code, 4 for FORTRAN
– max 2 x 2 hour sessions per day • Divided into: Functionality, Data Usage, Control, Linkage,
Computation, Maintenance, Clarity
– approx. 150 source lines/hour • Examples:
• Approach: Line-by-line paraphrasing – Does each module have a single function?
– Does the code match the Detailed Design?
– Reconstruct intent of code from source – Are all constant names upper case?
– May also “hand test” – Are pointers not typecast (except assignment of NULL)?
– Are nested “INCLUDE” files avoided?
• Find and log defects, but don’t fix them – Are non-standard usages isolated in subroutines and well
– Moderator responsible for staying on track documented?
– Are there sufficient comments to understand the code?

Carlo Ghezzi--SE-Verif 63 Carlo Ghezzi--SE-Verif 64

 Why does inspection work?
Incentive Structure
from [Fagan 86] • The evidence says it is cost-effective.
Why?
• Faults found in inspection are not used in – Detailed, formal process, with record keeping
personnel evaluation – Check-lists; self-improving process
– Programmer has no incentive to hide faults
– Social aspects of process, esp. for author
• Faults found in testing (after inspection) are – Consideration of whole input space
used in personnel evaluation – Applies to incomplete programs
– Programmer has incentive to find faults in
• Limitations
inspection, but not by inserting more
– Scale: Inherently a unit-level technique
– Non-incremental; what about evolution?
Carlo Ghezzi--SE-Verif 65 Carlo Ghezzi--SE-Verif 66