ECE 453 – CS 447 – SE 465

Software Testing &

Quality Assurance
Instructor
Kostas Kontogiannis

1
 Overview
System Testing
General - Introduction
Threads
Basis Concepts for Requirements Specification
Finding Threads
Structural Strategies for Thread Testing
Functional Strategies for Thread Testing
System Testing Guidelines

Ref: “Software Testing A Craftsman's Approach” 2nd edition, Paul C. Jorgensen 2

 System Testing
• Of the three levels of testing, the system level testing is closest to
everyday experience, that is we evaluate a product with respect to our
expectations

• Concerns with the app’s externals and the goal in not to find faults but
to demonstrate performance

• We tend to approach system testing from a functional standpoint

rather than from a structural one

• However, still much more than functional

– Load/stress testing
– Usability testing
– Performance testing
– Resource testing
3
 System Testing
• Functional testing

– Objective: Assess whether the app does what it is

supposed to do

– Basis: Behavioral/functional specification

– Test case: A sequence of ASFs (thread)

4
 System Testing
• Functional testing: coverage

• Event-based coverage
– PI1: each port input event occurs
– PI2: common sequences of port input event occurs
– PI3: each port input in every relevant data context
– PI4: for a given context, all possible input events
– PO1: each port output event
– PO2: each port output event occurs for each cause
• Data-based
– DM1: Exercise cardinality of every relationship
– DM2: Exercise (functional) dependencies among relationships

5
 System Testing
• Stress testing: push it to its limit + beyond

Volume

Users
Application response
: (System)
rate

Resources: phy. + logical

6
 System Testing
• Performance testing
– Performance seen by
• users: delay, throughput
• System owner: memory, CPU, comm
– Performance
• Explicitly specified or expected to do well
• Unspecified  find the limit

• Usability testing
– Human element in system operation
• GUI, messages, reports, …

7
 Threads

• We view system testing in terms of threads of system level behavior.

• Many possible views of a thread:

– a scenario of normal usage
– a system level test case
– a stimulus/response pair
– behavior that results from a sequence of system level inputs
– an interleaved sequence of port input and output events
– a sequence of transitions in a state machine description of a system
– an interleaved sequence of object messages and method executions
– a sequence of machine instructions
– a sequence of source instructions
– a sequence of atomic system functions

8
 Thread Levels
• Threads have distinct levels:

– Unit level thread is understood as an execution-time path of instructions

or some path on a flow graph.
– Integration level thread is a sequence of MM-paths that implement some
atomic function. Denoted perhaps as a sequence of module executions and
messages.
– System level thread is a sequence of atomic system functions.

• Since ASFs have port events as their inputs and outputs, the sequence
of ASFs implies an interleaved sequence of port input/port output
events.

– Threads provide a unifying view of the three levels of testing:

• Unit testing tests individual functions
• Integration tests examine interaction among units
• System testing examines interactions among ASFs.
9
 Thread Definitions (1)
• Unit thread: is a path in the program graph of a unit.
– Two levels of threads in integration testing: MM-Paths and
ASFs.

• MM-Path: is a path in the MM-Path graph of a set of units.

– directed graph in which module execution paths are nodes and
edges show execution time sequence

• ASF Graph: (for a system defines in terms of ASFs) is the

directed graph in which nodes are ASFs and edges represent
sequential flow.

10
 Thread Definitions (2)
• Source ASF: an ASF that appears as a source node in the ASF graph
of a system,

• Sink ASF: is an ASF that appears as sink node in the ASF graph.

• System thread: a path from a source ASF to a sink ASF in the ASF
graph of a system.

• Thread graph: (for a system defined in terms of system threads) is

directed graph in which nodes are system threads and edges represent
sequential execution of individual threads.

• The above definitions provide a coherent set of increasing broader

views of threads. 11
Basis Concepts for Requirements
Specification
• The objective is to discuss system testing with respect to a basis
set of requirements specification constructs

• Consider the following requirements specification constructs:

– data,
– actions,
– ports,
– events,
– threads.

• Every system can be specified in terms of these basic concepts.

12
 Data-Centric Thread Identification
• For a system that is described in terms of its data,
– the focus is on the information used/created by the system (described in terms of
variables, data structures, fields, records, data stores, and files)
– for example ER models are useful at the highest level.

• The data centered view is also starting point for many OO analysis methods.

• Data refers to information that is either initialized, stored, updated or possibly

destroyed.

• Data-centric systems are often specified in terms of CRUD actions (Create,

Retrieve, Update, Delete)

13
 Data-Centric Thread Identification
• Often, threads can be identified directly from the data
model.
– Relationships can be 1:1, n:1, etc. and these distinctions have
implications for threads that process the data.

• Also possible to have read-only data (i.e. expected PIN

pairs, etc.)
– this must be part of system initialization process
• if not, then there must be threads that create the data.
• Hence read-only data is an indicator of source ASFs.

14
Action-Centric Thread Identification
• Action centered modeling is most common requirements
specification form.
– Actions have inputs and outputs and these can be either data or
port events.
– Actions can also be decomposed in to lower level actions (i.e.
typical data flow diagrams).

• The input/output view of actions is the basis of functional

testing and,
– the decomposition [and eventual implementation] of actions is
the basis of structural testing.

15
 Port-Centric Thread Identification

• Every system has ports (and port devices):

– sources and destination of system level inputs and outputs.
– distinction between port device which is connected to a
system port

• If no physical port devices in system, much of

system testing can be accomplished by moving the
port boundary inward to the logical instances of port
events.

16
Event-Centric Thread Identification
• Have characteristics of data and some of actions
– a system level input which occurs at a port.

• Events can be inputs to or outputs of actions:

– Can be either discrete (more generally) or continuous
(temperature, altitude, etc.).
– Discrete events have a time duration and this can be critical
in real-time systems.
– Events have characteristics of data and some of actions
• i.e. a system level input which occurs at a port.

17
 Threads
• Threads are the least frequently used of the
fundamental constructs.
– Since threads are tested, it is up to tester to find them in the
interactions of the data, events, and actions.
– Usually easy to find threads in a control model of the system.

• Finding Threads
– A finite state machine model of the system is a good starting
point to find threads since the paths are easily converted to
threads.

18
E/R Model of Basis Concepts

Data Action
n n
n

Event
n

n n

Port Thread

19
 Modeling Relationships

Data

Structural
Model Event

Behavior
Action Model

Context
Thread
Model

Port 20
 Finding Threads (1)
– Usually, one deals with a hierarchy of state machines
i.e. the card entry state of an ATM may be decomposed
 
into lower levels that deal with details like:
• jammed cards,
• cards that are upside-down,
• checking the card against the list of cards for which service is
offered, etc.).
wrongCard/ScreenS1, eject card

fail PIN/ screen S4

1. Card 2. PIN 3.wait
Entry Entry Tr. Choice

goodCard/screen S2 good PIN/screen S5

21
Figure 1.
 Finding Threads (2)
• At this level, states correspond to states
of processing, and transitions are
caused by logical (rather than port) 1. Card
events. Entry

2.1 First
• Once the details of a macro-state are PIN try
tested,
2.2 2nd
– We then use an easy thread to get to PIN Try
the next macro-state.
– Within the decomposition of the macro 2.3 3rd
state, PIN try

• identify the port input and port output 3. Await

TR. Chc
events.[ i.e. within 2.1 on Figure 2,
digit and cancel key port input occur]
Figure 2.
22
 Finding Threads (3)
• the hierarchy of finite state machines multiplies
the number of threads.

• ideal to reach a state machine in which transitions

are caused by actual port input events, and the
actions on transitions are port output events
– then generating the test cases for these threads is
mechanical
– just follow a path of transitions,
• and note the inputs and outputs as they occur along the path.
23
 Structural Strategies for Thread
Testing
• Generating the threads may be easy, but to decide which
one to test is complex.

• Encounter the same path explosion problem at system

level as at unit level.

• Bottom Up Threads
– When state machines are organized in a hierarchy, it is possible to
work bottom up. i.e. in Fig 2, the “PIN Try” state machine might
consist of 6 paths.
• Traverse these paths and then go up one level to the “PIN Entry” [fig.
3] machine which has 4 basic paths and so on up the hierarchy

24
 Structural Strategies for Thread
Testing
• As seen in unit testing, structural testing can be
misleading.
– The assumption is that path traversal uncovers faults
and traversing a variety of paths reduces redundancy.

• A more serious flaw with these threads is that it is

not really possible to execute them “by
themselves” due to the hierarchical state machines.

25
Node and Edge Coverage Metrics
• Since FSMs are directed graphs, use same test coverage metrics as at
the unit level.

• The hierarchical relationship indicates that the upper level machine

treats the lower level machine as a procedure that is entered and
returned from.

• Two fundamental choices are node coverage and edge coverage

– node coverage is similar to statement coverage at unit level: bare
minimum .
– edge (state transition) coverage is more acceptable. If the state machines
are ‘well formed’ [transitions in terms of port events], then edge coverage
also guarantees port event coverage.
26