A Framework for Testing and Analysis

Learning objectives
• Introduce dimensions and tradeoff between
test and analysis activities
• Distinguish validation from verification
activities
• Understand limitations and possibilities of test
and analysis

Based on slides by Mauro Pezzè & Michal Young Ch 2, slide 2

 Verification and validation
• Validation:
does the software system meet the user's real
needs?
are we building the right software?

• Verification:
does the software system meets the
requirements specifications?
are we building the software right?

Based on slides by Mauro Pezzè & Michal Young Ch 2, slide 3

Based on slides by Mauro Pezzè & Michal Young Ch 2, slide 4
 Validation and Verification

SW
Actual Specs
Requirements System

Validation Verification
Includes usability Includes testing,
testing, user inspections, static
feedback analysis

Based on slides by Mauro Pezzè & Michal Young Ch 2, slide 5

 Verification or validation depends on
the specification
12345678

Example: elevator response

Unverifiable (but validatable) spec: ... if a user

presses a request button at floor i, an available
elevator must arrive at floor i soon...
Verifiable spec: ... if a user presses a request
button at floor i, an available elevator must
arrive at floor i within 30 seconds...

Based on slides by Mauro Pezzè & Michal Young Ch 2, slide 6

 Quiz
• Which of the following are good requirement
specifications?
A. The system should be reliable.
B. The start-up time of the system shall not be
annoying to users.
C. The system should allow the user to check their
bank account balance.
D. The system should solve all the management
problems for the university.
• [Follow-up] Can you change the “bad” ones to
make them good requirement specs?
Based on slides by Mauro Pezzè & Michal Young 7
 Validation and Verification Activities

Actual Needs and Delivered

Constraints User Acceptance (alpha, beta test)
Package
Review

System System Test System

Specifications Integration
Analysis /
Review

Subsystem Integration Test Subsystem

Design/Specs
Analysis /
Review
validation
Unit/
Unit/
Component Module Test Components
Specs

User review of external behavior as it is verification

determined or becomes visible

Based on slides by Mauro Pezzè & Michal Young Ch 2, slide 8

 ever
You can’t always get what you want

Property
Decision Pass/Fail
Program Procedure

Correctness properties are undecidable

the halting problem can be embedded in almost
every property of interest

Based on slides by Mauro Pezzè & Michal Young Ch 2, slide 9

 Getting what you need ...
Theorem proving: Perfect verification of
Unbounded effort to
verify general
arbitrary properties by • optimistic inaccuracy: we
logical proof or exhaustive
properties. testing (Infinite effort) may accept some programs
Model checking:
that do not possess the
Decidable but possibly
intractable checking of
property (i.e., it may not
simple temporal detect all violations).
properties.
Data flow – testing
analysis
• pessimistic inaccuracy: it is
Typical testing not guaranteed to accept a
Precise analysis of
simple syntactic
techniques program even if the program
properties. does possess the property
being analyzed
– automated program analysis
techniques
Simplified
properties
Optimistic
inaccuracy
• simplified properties: reduce
Pessimistic
the degree of freedom for
inaccuracy simplifying the property to
check
Based on slides by Mauro Pezzè & Michal Young Ch 2, slide 10
 Example of simplified property:
Unmatched Semaphore Operations
original problem simplified property
Java prescribes a
if ( .... ) { more restrictive, but
... statically checkable
lock(S); construct.
}
Static
...
checking for
if ( ... ) {
match is synchronized(S) {
...
necessarily ...
unlock(S);
inaccurate ... ...
}
}

Based on slides by Mauro Pezzè & Michal Young Ch 2, slide 11

 Some Terminology
• Safe: A safe analysis has no optimistic inaccuracy,
i.e., it accepts only correct programs.
• Sound: An analysis of a program P with respect to a
formula F is sound if the analysis returns true only
when the program does satisfy the formula.
– How does this relate to optimistic inaccuracy?
• Complete: An analysis of a program P with respect
to a formula F is complete if the analysis always
returns true when the program actually satisfies
the formula.
– How does it relate to optimistic inaccuracy?

Based on slides by Mauro Pezzè & Michal Young Ch 2, slide 12

 Summary
• Most interesting properties are undecidable,
thus in general we cannot count on tools that
work without human intevention
• Assessing program qualities comprises two
complementary sets of activities: validation
(daes the software do what it is supposed to
do?) and verification (does the system behave
as specificed?)
• There is no single technique for all purposes:
test designers need to select a suitable
combination of techniques

Based on slides by Mauro Pezzè & Michal Young Ch 2, slide 13