Software Testing ISEB Foundation Certificate Course

1 Principles

2 Lifecycle

3 Static testing 6 Tools

4 Dynamic test 5 Management techniques

Dynamic Testing Techniques

1 4

2 5

3 6

ISEB Foundation Certificate Course

Dynamic Testing Techniques

Contents
What is a testing technique? Black and White box testing Black box test techniques White box test techniques Error Guessing

Why dynamic test techniques?

Exhaustive testing (use of all possible inputs and conditions) is impractical - must use a subset of all possible test cases - must have high probability of detecting faults Need thought processes that help us select test cases more intelligently - test case design techniques are such thought processes

What is a testing technique?

a procedure for selecting or designing tests based on a structural or functional model of the software successful at finding faults 'best' practice a way of deriving good test cases a way of objectively measuring a test effort
Testing should be rigorous, thorough and systematic

Advantages of techniques

Different people: similar probability find faults - gain some independence of thought Effective testing: find more faults - focus attention on specific types of fault - know you're testing the right thing Efficient testing: find faults with less effort - avoid duplication - systematic techniques are measurable
Using techniques makes testing much more effective

Measurement

Objective assessment of thoroughness of testing (with respect to use of each technique) - useful for comparison of one test effort to another E.g.
Project A 60% Equivalence partitions 50% Boundaries 75% Branches Project B 40% Equivalence partitions 45% Boundaries 60% Branches

1 4

2 5

3 6

ISEB Foundation Certificate Course

Dynamic Testing Techniques

Contents
What is a testing technique? Black and White box testing Black box test techniques White box test techniques Error Guessing

Three types of systematic technique

Static (non-execution)
examination of documentation, source code listings, etc. Functional (Black Box) based on behaviour / functionality of software Structural (White Box) based on structure of software

Some test techniques

Static
Reviews
Inspection Walkthroughs Desk-checking Data Flow Control Flow etc. Static Analysis

Dynamic Behavioural
Non-functional etc. Usability Performance etc. Arcs LCSAJ Functional etc. Equivalence Partitioning Boundary Value Analysis Cause-Effect Graphing

Structural

etc. Symbolic Execution

Statement
Branch/Decision Branch Condition

Random
State Transition

Definition -Use

Branch Condition Combination

Black box versus white box?

Black box appropriate at all levels but dominates higher levels of testing
Acceptance

System

White box used predominately at lower levels to compliment black box

Integration

Component

1 4

2 5

3 6

ISEB Foundation Certificate Course

Dynamic Testing Techniques

Contents
What is a testing technique? Black and White box testing Black box test techniques White box test techniques Error Guessing

Black Box test design and measurement techniques

Techniques defined in BS 7925-2 - Equivalence partitioning Also a measurement - Boundary value analysis technique? = Yes - State transition testing = No - Cause-effect graphing - Syntax testing - Random testing Also defines how to specify other techniques

Equivalence partitioning (EP)

- divide (partition) the inputs, outputs, etc. into areas which are the same (equivalent) - assumption: if one value works, all will work - one from each partition better than all from one
invalid 0 1 valid 100 101 invalid

Boundary value analysis (BVA)

- faults tend to lurk near boundaries - good place to look for faults - test values on both sides of boundaries
invalid 0 1 valid 100 101 invalid

Example: Loan application

Customer Name
Account number Loan amount requested
2-64 chars.

6 digits, 1st non-zero

500 to 9000 1 to 30 years Minimum 10

Term of loan
Monthly repayment Term:

Repayment:
Interest rate: Total paid back:

Customer name
Number of characters: 1 2 64 65

invalid

valid

invalid

Valid characters:

A-Z - a-z space

Invalid Partitions < 2 chars > 64 chars invalid chars

Any other
Valid Boundaries 2 chars 64 chars Invalid Boundaries 1 chars 65 chars 0 chars

Conditions Customer name

Valid Partitions 2 to 64 chars valid chars

Account number
first character:
valid: non-zero

invalid: zero
5 6 7

number of digits: invalid

valid

invalid

Conditions Account number

Valid Partitions 6 digits 1st non-zero

Invalid Partitions < 6 digits > 6 digits 1st digit = 0 non-digit

Valid Boundaries 100000 999999

Invalid Boundaries 5 digits 7 digits 0 digits

Loan amount

499
invalid
Valid Partitions 500 - 9000

500
valid
Invalid Partitions < 500 >9000 0 non-numeric null

9000

9001
invalid

Conditions Loan amount

Valid Boundaries 500 9000

Invalid Boundaries 499 9001

Condition template
Conditions Valid Partitions Tag V1 V2 V3 V4

Customer 2 - 64 chars name valid chars Account number 6 digits 1st non-zero

Loan amount

500 - 9000

V5

Valid Invalid Tag Tag Boundaries Boundaries < 2 chars X1 2 chars B1 1 char D1 > 64 chars X2 64 chars B2 65 chars D2 invalid char X3 0 chars D3 < 6 digits X4 100000 B3 5 digits D4 > 6 digits X5 999999 B4 7 digits D5 st 1 digit = 0 X6 0 digits D6 non-digit X7 < 500 X8 500 B5 499 D7 >9000 X9 9000 B6 9001 D8 0 X10 non-integer X11 null X12 Tag

Invalid Partitions

Design test cases

Test Case 1 Description Name: Acc no: Loan: Term: Name: Acc no: Loan: Term: John Smith 123456 2500 3 years AB 100000 500 1 year Expected Outcome Term: Repayment: Interest rate: Total paid: Term: Repayment: Interest rate: Total paid: 3 years 79.86 10% 2874.96 1 year 44.80 7.5% 537.60 New Tags Covered V1, V2, V3, V4, V5 ..... B1, B3, B5, .....

Why do both EP and BVA?

If you do boundaries only, you have covered all the partitions as well - technically correct and may be OK if everything works correctly! - if the test fails, is the whole partition wrong, or is a boundary in the wrong place - have to test midpartition anyway - testing only extremes may not give confidence for typical use scenarios (especially for users) - boundaries may be harder (more costly) to set up

Test objectives?
Condition Valid Tag Invalid Tag Valid Partition Partition Boundary Tag Invalid Boundary Tag

For a thorough approach: VP, IP, VB, IB Under time pressure, depends on your test objective - minimal user-confidence: VP only? - maximum fault finding: VB first (plus IB?)

Decision tables

explore combinations of inputs, situations or events, it is very easy to overlook specific combinations of input start by expressing the input conditions of interest so that they are either TRUE or FALSE
record found file exists code valid policy expired account in credit due date > current date

Example: student access

A university computer system allows students an allocation of disc space depending on their projects. If they have used all their allotted space, they are only allowed restricted access, i.e. to delete files, not to create them. This is assuming they have logged on with a valid username and password. What are the input and output conditions?

List the input and output conditions

list the input conditions in the first column of the table list the output conditions under the input conditions
Input Conditions Valid username Valid password Account in credit Output Conditions Login accepted Restricted access

Determine input combinations

add columns to the table for each unique combination of input conditions. each entry in the table may be either T for true, F for false.

Input Conditions Valid username T T T T F F F Valid password T T F F T T F Account in credit T F T F T F T

F F F

Rationalise input combinations

some combinations may be impossible or not of interest some combinations may be equivalent use a hyphen to denote dont care

Input Conditions Valid username F Valid password Account in credit -

T T T F T T - F T

Complete the table

determine the expected output conditions for each combination of input conditions

Input Conditions Valid username F Valid password Account in credit Output Conditions Login accepted F Restricted access -

T T T F T T - F T F T T T F

Determine test case groups

each column is at least one test case

Input Conditions Valid username F Valid password Account in credit Output Conditions Login accepted F Restricted access Tags A

T T T F T T - F T F T T - T F B C D

Design test cases

usually one test case for each column but can be none or several

Test Description
1 2 3 4 5 Username BrbU Username usernametoolong Username BobU Password abcd Valid user, no disc space Valid user with disc space

Expected Outcome
Invalid username Invalid username Invalid password Restricted access Unrestricted access

Tag
A A B C D

Rationalising outputs

if outputs or effects are mutually exclusive, I.e. T occurs in only one place in each column, we can combine them for example:
X Y Z T F F F T F F F T is equivalent to: Output X Y Z

Rationalising dangers

rationalising is based on assumptions assumptions may be wrong! assumptions should be stated assumptions may change over time be aware of the dangers - filling in the full table may find errors which will be missed if you rationalise - it is possible to rationalise too far

Extending decision tables

Entries can be more than just true or false - completing table needs to be done carefully - rationalising becomes more important E.g.

Code = 1, 2, or 3 1 1 1 1 2 2 2 2 3 3 3 3 Exp.date < now T T F F T T F F T T F F Class A product T F T F T F T F T F T F

Decision Tables in relation to EP and BVA

Input value Output value

FALSE TRUE

1 4

2 5

3 6

ISEB Foundation Certificate Course

Dynamic Testing Techniques

Contents
What is a testing technique? Black and White box testing Black box test techniques White box test techniques Error Guessing

White Box test design and measurement techniques

Techniques defined in BS 7925-2 Also a measurement - Statement testing technique? - Branch / Decision testing = Yes = No - Data flow testing - Branch condition testing - Branch condition combination testing - Modified condition decision testing - LCSAJ testing Also defines how to specify other techniques

Using structural coverage

Spec Software Enough tests? Results OK?

Tests What's covered?

More tests

Coverage OK?

Stronger structural techniques (different structural elements) Increasing coverage

The test coverage trap

Function exercised, insufficient structure
Functional testedness

better testing

Structure exercised, insufficient function

% Statement % Decision % Condition Combination

Structural testedness

100% coverage does not mean 100% tested!

Coverage is not Thoroughness

Statement coverage

Statement coverage is normally measured by a software tool.

percentage of executable statements exercised by a test suite number of statements exercised = total number of statements example: - program has 100 statements - tests exercise 87 statements - statement coverage = 87%
Typical ad hoc testing achieves 60 - 75%

Example of statement coverage

1 2 3 4 5

read(a)
IF a > 6 THEN b=a ENDIF print b

Test case 1

Input 7

Expected output 7

Statement numbers

As all 5 statements are covered by this test case, we have achieved 100% statement coverage

Decision coverage (Branch coverage)

Decision coverage is normally measured by a software tool.

percentage of decision outcomes exercised by a test suite number of decisions outcomes exercised False ? = total number of decision outcomes True example: - program has 120 decision outcomes - tests exercise 60 decision outcomes - decision coverage = 50%
Typical ad hoc testing achieves 40 - 60%

Paths through code

12 ?

1234

12
?

123
? ?

Paths through code with loops

1 2 3 4 5 6 7 8 . for as many times as it is possible to go round the loop (this can be unlimited, i.e. infinite)

Example 1
Wait for card to be inserted IF card is a valid card THEN display Enter PIN number IF PIN is valid THEN select transaction ELSE (otherwise) display PIN invalid ELSE (otherwise) reject card End

Wait
Yes

Valid card?
No

Display Enter..
Valid Yes Select PIN? trans...

Reject card

No

Display PIN in..

End

Example 2
Read A IF A > 0 THEN IF A = 21 THEN Print Key ENDIF ENDIF

Read
Yes Yes

A>0

A=21 No

No

Print

End

3 - Cyclomatic complexity: _____ - Minimum tests to achieve: 1 Statement coverage: ______ 3 Branch coverage: _____

Example 3
Read A Read B IF A > 0 THEN IF B = 0 THEN Print No values ELSE Print B IF A > 21 THEN Print A ENDIF ENDIF ENDIF

Read
Yes No

A>0 No

B=0

Print
Yes A>21 Print No

Yes

Print

End

4 - Cyclomatic complexity: _____ - Minimum tests to achieve: 2 Statement coverage: ______ 4 Branch coverage: _____

Example 4

Read

A<0 No

Yes

Print

Print Note: there Read A Read B are 4 paths IF A < 0 THEN Yes B<0 Print Print A negative No ELSE Print A positive Print ENDIF IF B < 0 THEN End Print B negative ELSE 3 - Cyclomatic complexity: _____ Print B positive - Minimum tests to achieve: ENDIF 2 Statement coverage: ______ 2 Branch coverage: _____

Example 5
Read A Read B IF A < 0 THEN Print A negative ENDIF IF B < 0 THEN Print B negative ENDIF

Read

A<0

Yes

Print

No
Yes

B<0 No

Print

End

3 - Cyclomatic complexity: _____ - Minimum tests to achieve: 1 Statement coverage: ______ 2 Branch coverage: _____

Example 6
Read A IF A < 0 THEN Print A negative ENDIF IF A > 0 THEN Print A positive ENDIF

Read

A<0

Yes

Print

No
Yes

A>0 No

Print

End

3 - Cyclomatic complexity: _____ - Minimum tests to achieve: 2 Statement coverage: ______ 2 Branch coverage: _____

1 4

2 5

3 6

ISEB Foundation Certificate Course

Dynamic Testing Techniques

Contents
What is a testing technique? Black and White box testing Black box test techniques White box test techniques Error Guessing

Non-systematic test techniques

Trial and error / Ad hoc Error guessing / Experience-driven User Testing Unscripted Testing

A testing approach that is only rigorous, thorough and systematic is incomplete

Error-Guessing

always worth including after systematic techniques have been used can find some faults that systematic techniques can miss a mopping up approach supplements systematic techniques

Not a good approach to start testing with

Error Guessing: deriving test cases

Consider: - past failures - intuition - experience - brain storming - What is the craziest thing we can do?

1 4

2 5

3 6

ISEB Foundation Certificate Course

Dynamic Testing Techniques

Summary: Key Points

Test techniques are best practice: help to find faults Black Box techniques are based on behaviour White Box techniques are based on structure Error Guessing supplements systematic techniques