Testing

Luca Vigan
Ingegneria del Software, AA 10/11, 04.05.11
The slides in this section are partly based on the courses Software Engineering by Prof. Peter Mller, ETH Zurich, Software Engineering I by Prof. Bernd Brgge, TU Mnchen and Software Engineering by Prof. Jan Vitek, Purdue University

The software testing process

Ian Sommerville 2006

!Software Engineering, 8th edition. Chapter 23

Slide 8!

7. Testing Basics Basics

Why Does Software Contain Bugs?

Our ability to predict the behavior of our creations is imperfect
- Software is extremely complex - No developer can understand the whole system

We make mistakes
Unclear requirements, miscommunication Wrong assumptions (e.g., behavior of operating system) Design errors (e.g., capacity of data structure too small) Coding errors (e.g., wrong loop condition)

Peter Mller Software Engineering, SS 11

7. Testing Basics

Goal of Testing
An error is a deviation of the observed behavior from the required (desired) behavior
- Functional requirements (e.g., user-acceptance testing) - Nonfunctional requirements (e.g., performance testing)

Testing is a process of executing a program with the intent of finding an error A successful test is a test that finds errors

Peter Mller Software Engineering, SS 11

Definition: testing
To test a software system is to try to make it fail

7. Testing Basics

Limitations of Testing
Testing can only show the presence of bugs, not their absence. [E. W. Dijkstra] It is impossible to completely test any nontrivial module or any system
- Theoretical limitations: termination - Practical limitations: prohibitive in time and cost

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

7. Testing Basics Basics

7.1 Test Stages 7.2 Test Strategies

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

Test Stages
Analysis System Test

System Design

Integration Test

Detailed Design

Unit Test

Implementation
Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

10

Unit Testing
Testing individual subsystems (collection of classes)
Detailed Design Model Subsystem Code

Unit Test

Goal: Confirm that subsystem is correctly coded and carries out the intended functionality

Peter Mller Software Engineering, SS 11

xUnit widely used testing frameworks

xUnit frameworks allow testing of different elements (units) of software, such as functions and classes. The main advantage of xUnit frameworks is that they provide an automated solution with no need to write the same tests many times, and no need to remember what should be the result of each test. Examples ! JUnit for Java ! NNnit for .NET ! CppUnit for C++

JUnit: Overview
Provides a framework for running test cases Test cases !! Written manually !! Normal classes, with annotated methods Input values and expected results defined by the tester Execution is the only automated step

7.1 Testing Basics Test Stages

17

Test Execution
Execute the test cases Re-execute test cases after every change
- Automate as much as possible - For instance, after each refactoring

Regression testing
- Testing that everything that used to work still works after changes are made to the system - Also important for system testing

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

18

Eight Rules of Testing

1. Make sure all tests are fully automatic and check their own results 2. A test suite is a powerful bug detector that reduces the time it takes to find bugs 3. Run your tests frequently every test at least once a day 4. When you get a bug report, start by writing a unit test that exposes the bug 5. Better to write and run incomplete tests than not run complete tests 6. Concentrate your tests on boundary conditions 7. Do not forget to test exceptions raised when things are expected to go wrong 8. Do not let the fear that testing cant catch all bugs stop you from writing tests that will catch most bugs [M. Fowler]

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

19

Integration Testing
Testing groups of subsystems and eventually the entire system
Subsystem Code Subsystem Code Subsystem Code

Software Architecture Integration Test

Goal: Test interfaces between subsystems

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

20

Steps in Integration-Testing
1. Select a component to be tested
- Unit test all the classes in the component

2. Put selected components together

- Make the integration test operational (drivers, stubs)

3. Do the testing
- Functional testing, structural testing, performance testing

4. Keep records of the test cases and testing activities 5. Repeat steps 1 to 4 until the full system is tested

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

21

Integration Testing Strategy

The order in which the subsystems are selected for testing and integration Typical strategies
- Big-bang integration (non-incremental) - Bottom-up integration - Top-down integration - Sandwich testing - Variations of the above

Call hierarchy
A

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

22

Big-Bang Strategy: Example

A B Dont try this!

C
D E F

Whole System
Integration Test

G
Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

23

Bottom-Up Strategy
Strategy
1. Start with subsystems in lowest layer of call hierarchy 2. Test subsystems that call the previously tested subsystems 3. Repeat until all subsystems are included Pros - Useful for integrating object-oriented systems and systems with strict performance requirements
Peter Mller Software Engineering, SS 11

Cons - Tests the most important subsystem (UI) last

7.1 Testing Basics Test Stages

24

Bottom-Up Strategy: Example

B

B
E B,E,F A C D E F G

Whole System

D,G
G
Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

25

Top-Down Strategy
Strategy
1. Start with subsystems in top layer of call hierarchy 2. Include subsystems that are called by the previously tested subsystems 3. Repeat until all subsystems are included

Pros - Supports test cases for the functionality of the system

Cons - Writing stubs is difficult: Stubs must allow all possible conditions to be tested - Possibly very large number of stubs required

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

26

Top-Down Strategy: Example

B A B A,B,C, D E F

C
D

E F G

Whole System

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

27

Sandwich Strategy
Combines top-down with bottom-up strategy

The system is viewed as having three layers

A target layer in the middle A layer above the target A layer below the target Testing converges at the target layer

How do you select the target layer if there are more than three layers?
- Try to minimize the number of stubs and drivers
Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

28

Sandwich Strategy: Example

F

Target layer
B B,E,F E Bottom / target D,G Whole System F G C D

E
B G D A C

A,B,C, D

Top / target

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

29

Sandwich Strategy: Discussion

Pros - Top and bottom layer can be tested in parallel - Fewer drivers and stubs needed (target layer instead of driver for bottom layer and stub for top layer) Cons - Does not test the individual subsystems thoroughly before integration

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

31

System Testing
Testing the entire system
Requirements Specification Entire System System Test

Goal: Determine if the system meets the requirements (functional and non-functional)

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

32

System Testing Stages

Entire System Functional requirements Non-functional requirements Clients understanding of requirements User Environment
Peter Mller Software Engineering, SS 11

Functional Test Performance Test Acceptance Test Installation Test

7.1 Testing Basics Test Stages

33

Functional Testing
Goal: . Test functionality of system
- System is treated as black box

Test cases are designed from requirements analysis document

- Based on use cases . - Alternative source: user manual

Test cases describe

- Input data - Flow of events - Results to check
Peter Mller Software Engineering, SS 11

Classification: by available information

White-box testing !! To define test cases, source code of IUT is available Alternative names: implementation-based, structural, glass box, clear box

Black-box testing !! Properties of IUT available only through specification Alternative names: responsibility-based, functional

A comparison

White-box
IUT internals Focus Origin of test cases Typical use Who?

Black-box

Knows internal structure No knowledge & implementation Ensure coverage of many Test conformance to execution possibilities specification Source code analysis Unit testing Developer Specification Integration & system testing Developers, testers, customers

7.1 Testing Basics Test Stages

34

Performance Testing
Stress Testing
- Stress limits of system (maximum number of users, peak demands)

Volume testing
- Large amounts of data

Configuration testing
- Various software and hardware configurations

Compatibility testing
- Backward compatibility with existing systems

Security testing
- Try to violate security requirements (red team)
Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

35

Performance Testing (contd)

Timing testing
- Response times and time to perform a function

Environmental testing
- Tolerances for heat, humidity, motion

Quality testing
- Reliability, maintainability, and availability

Recovery testing
- Systems response to presence of errors or loss of data

Usability testing
- Tests user interface with user
Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

36

Acceptance Testing
Goal: Demonstrate that the system meets customer requirements and is ready to use Choice of tests is made by client
- Many tests can be taken from integration testing

Performed by the client, not by the developer

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

37

Acceptance Testing (contd)

Majority of bugs is typically found by the client, not by the developers or testers Alpha test
- Client uses the software at the developers site - Software used in a controlled setting, with the developer always ready to fix bugs

Beta test
- Conducted at clients site (developer is not present) - Software gets a realistic workout in target environment - Potential client might get discouraged

Peter Mller Software Engineering, SS 11

Alpha and beta testing

Alpha testing The first test of newly developed hardware or software in a laboratory setting. When the first round of bugs has been fixed, the product goes into beta test with actual users. Beta testing A test of new or revised hardware or software that is performed by users at their facilities under normal operating conditions. An interesting example: proportional testing of Gmail.

7.1 Testing Basics Test Stages

38

Independent Testing
Programmers have a hard time believing they made a mistake
- Plus a vested interest in not finding mistakes - Often stick to the data that makes the program work

Designing and programming are constructive tasks

- Testers must seek to break the software

Testing is done best by independent testers

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

39

Independent Testing: Responsibilities

System Test

Performed by independent test team

- Exception: Acceptance test performed by client

Performed by independent test Integration Test team Performed by programmer

Unit Test - Requires detailed knowledge of the code - Immediate bug fixing

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

40

Independent Testing: Wrong Conclusions

The developer should not be testing at all
- Test before you code

Testers get only involved once software is done

Toss the software over the wall for testing

- Testers and developers collaborate in developing the test suite

Testing team is responsible for assuring quality

- Quality is assured by a good software process
Peter Mller Software Engineering, SS 11

7.2 Testing Basics Test Strategies

41

7. Testing Basics Basics

7.1 Test Stages 7.2 Test Strategies

Peter Mller Software Engineering, SS 11

7.2 Testing Basics Test Strategies

42

Testing Steps
Select what will be tested
What parts of the system? What aspects of the system?

Select test strategy

What integration strategy? How is the test data determined?

Define test cases

What are the test data? How is the test carried out?

Create test oracle

Peter Mller Software Engineering, SS 11

What are the expected results? Defined before executing tests

7.2 Testing Basics Test Strategies

43

Example: Solve Quadratic Equation

void roots( double a, double b, double c ) { double q = b*b 4*a*c; if( q > 0 && a != 0 ) { numRoots = 2; double r = Math.sqrt( q ); x1 = (-b + r) / (2 * a); Fails if a==0 and x2 = (-b - r) / (2 * a); b*b4*a*c == 0 } else if( q == 0 ) { numRoots = 1; x1 = -b / (2 * a); } else { b b 2 4ac x= numRoots = 0; 2a } }
Peter Mller Software Engineering, SS 11

7.2 Testing Basics Test Strategies

44

Strategy 1: Exhaustive Testing

Check UUT for all possible inputs
- Not feasible, even for trivial programs
void roots( double a, double b, double c ) { }

Assuming that double represents 64-bit values, we get (264)3 1058 possible values for a, b, c Programs with heap data structures have a much larger state space!

Peter Mller Software Engineering, SS 11

7.2 Testing Basics Test Strategies

45

Strategy 2: Random Testing

Select test data uniformly
void roots( double a, double b, double c ) { double q = b*b 4*a*c; } if( q > 0 && a != 0 ) { Fails if a==0 and b*b4*a*c == 0 } else if( q == 0 ) { numRoots = 1; x1 = -b / (2 * a); The likelihood of } else { } selecting a==0 and b==0 } randomly is 1/1038

Peter Mller Software Engineering, SS 11

7.2 Testing Basics Test Strategies

46

Random Testing: Observations

Random testing focuses on generating test cases fully automatically Advantages
- Avoids designer/tester bias - Tests robustness, especially handling of invalid input and unusual actions

Disadvantages
- Treats all inputs as equally valuable

Peter Mller Software Engineering, SS 11

7.2 Testing Basics Test Strategies

47

Strategy 3: Functional Testing

Use analysis knowledge to determine test cases that check requirements
Given three values, a, b, c, compute all solutions of the equation ax2 + bx + c = 0
Two solutions
a 0 and b2-4ac > 0

One solution
a = 0 and b 0 or a 0 and b2-4ac = 0

No solution
a = 0, b = 0, and c 0 or a 0 and b2-4ac < 0

Test each case of the specification

Peter Mller Software Engineering, SS 11

7.2 Testing Basics Test Strategies

48

Functional Testing: Observations

Functional testing focuses on input/output behavior
- Goal: Cover all the requirements

Attempts to find
- Incorrect or missing functions - Interface errors - Performance errors

Limitations
- Does not effectively detect design and coding errors (e.g., buffer overflow, memory management) - Does not reveal errors in the specification (e.g., missing cases)
Peter Mller Software Engineering, SS 11

7.2 Testing Basics Test Strategies

49

Strategy 4: Structural Testing

Use design knowledge about system structure, algorithms, data structures to determine test cases that exercise a large portion of the code
void roots( double a, double b, double c ) { double q = b*b 4*a*c; Test this if( q > 0 && a != 0 ) { case } else if( q == 0 ) { and this case } else { and this case } }
Peter Mller Software Engineering, SS 11

Error might still be missed, for instance, when case is tested with a==1, b==2, c==1

7.2 Testing Basics Test Strategies

50

Structural Testing: Observations

Structural testing focuses on thoroughness
- Goal: Cover all the code

Not well suited for system test

- Focuses on code rather than on requirements, for instance, does not detect missing logic - Requires design knowledge, which testers and clients do not have (and do not care about) - Thoroughness would lead to highlyredundant tests
Peter Mller Software Engineering, SS 11

7.2 Testing Basics Test Strategies

51

Testing Strategies: Summary

Functional testing Goal: Cover all the requirements Black-box test Suitable for all test stages Structural testing Goal: Cover all the code White-box test Suitable for unit testing

Random testing Goal: Cover corner cases Black-box test Suitable for all test stages
Peter Mller Software Engineering, SS 11

7. Testing Basics

52

Summary
Main objective
- Design tests that systematically uncover different classes of errors with a minimum amount of time and effort - A good test has a high probability of finding an error - A successful test uncovers an error

Secondary benefits
- Demonstrate that software appears to be working according to specification (functional and non-functional) - Data collected during testing provides indication of software reliability and software quality - Good testers clarify the specification (creative work)
Peter Mller Software Engineering, SS 11

GUI Testing

Why is GUI testing hard?

!! GUI "! Graphics: easy for humans, hard for machines "! Themable GUIs "! Simple change to interface, big impact !! Network & Databases "! Big effort to set up environment !! Computers !! Operating Systems !! Applications !! Data !! Network "! Reproducibility

Why is GUI testing hard?

!! In the old days things were easy "! Stdin / Stdout / Stderr !! Modern applications lack uniform interface "! GUI "! Network "! Database "!

Minimizing GUI code

!! GUI code is hard to test !! Try to keep it minimal !! How?

Model-View-Controller

Views

VIEW

Model

A = 50% B = 30% C = 20%

Model-View Controller

Model View Controller (2/2)

Model
Change Notification ! Encapsulates application state ! Exposes application functionality ! Notifies view of changes State change

View
! Renders the model ! Sends user gestures to controller ! Allows controller to select view Feature calls Events

View selection

Controller

! Defines application behavior ! Maps user actions to model User gestures updates ! Selects view for response ! One for each functionality

Test Data

8. Test Data Functional Testing

8. Test Data
8.1 Functional Testing 8.2 Structural Testing 8.3 Automatic Test Case Generation 8.4 Dynamic Test Generation 8.5 Mutation Testing

Peter Mller Software Engineering, SS 11

Applications of Functional Testing

Black-box test a unit against its requirements
Functional test Acceptance test Test interfaces between subsystems

System Test

Integration Test
During test-driven development, when code is not yet written

Unit Test
Peter Mller Software Engineering, SS 11

8. Test Data Functional Testing

8. Test Data
8.1 Functional Testing
8.1.1 Partition Testing 8.1.2 Selecting Representative Values 8.1.3 Combinatorial Testing

8.2 Structural Testing 8.3 Automatic Test Case Generation 8.4 Dynamic Test Generation 8.5 Mutation Testing
Peter Mller Software Engineering, SS 11

Finding Representative Inputs

Requirement not implemented

Divide inputs into equivalence classes

Requirement implemented incorrectly

- Each possible input belongs to at least one of the equivalence classes - Goal: some classes have higher density of failures

Failure No failure

Requirement implemented correctly

Choose test cases for each equivalence class

Peter Mller Software Engineering, SS 11

Equivalence Classes: Example

Given a month (an integer in [1;12]) and a year (an integer), compute the number of days of the given month in the given year (an integer in [28;31])
month
Month with 28 or 29 days month = 2 Leap years

year
(year mod 4 = 0 and year mod 100 0) or year mod 400 = 0

Months with 30 days

Months with 31 days

month {4, 6, 9, 11}

month {1, 3, 5, 7, 8, 10, 12}

Non-leap years

year mod 4 0 or (year mod 100 = 0 and year mod 400 0)

Invalid inputs missing

Peter Mller Software Engineering, SS 11

Equivalence Classes: Example (contd)

Given a month (an integer in [1;12]) and a year (an integer), compute the number of days of the given month in the given year (an integer in [28;31])
month
Month with 28 or 29 days month = 2 Leap years

year
(year mod 4 = 0 and year mod 100 0) or year mod 400 = 0

Months with 30 days

Months with 31 days Invalid

month {4, 6, 9, 11}

month {1, 3, 5, 7, 8, 10, 12} month < 1 or month > 12

Non-leap years

year mod 4 0 or (year mod 100 = 0 and year mod 400 0)

Partitioning seems too coarse

Peter Mller Software Engineering, SS 11

Equivalence Classes: Example (contd)

Given a month (an integer in [1;12]) and a year (an integer), compute the number of days of the given month in the given year (an integer in [28;31])
month
Month with 28 or 29 days month = 2 Standard leap years

year
year mod 4 = 0 and year mod 100 0

Months with 30 days

Months with 31 days Invalid

month {4, 6, 9, 11}

month {1, 3, 5, 7, 8, 10, 12} month < 1 or month > 12

Standard nonleap years

Special leap years Special nonleap years

year mod 4 0
year mod 400 = 0 year mod 100 = 0 and year mod 400 0

Peter Mller Software Engineering, SS 11

Sources of Information
Use analysis knowledge to determine test cases that check requirements
Analysis Model Functional Model Analysis Object Model Dynamic Model
Input / output behavior, valid and invalid input Data manipulated by the system

System states and protocols

Peter Mller Software Engineering, SS 11

10

Using the Functional Model

The functional model describes the input-output behavior of the whole system
- Valid and invalid inputs, entry conditions - Expected results, exit conditions

Basis for functional system testing

Peter Mller Software Engineering, SS 11

11

Using the Functional Model: Example

Actor steps
1. Authenticate (use case Authenticate) 3. Client selects Withdraw CHF
Inputs

System Steps
2. Bankomat displays options

4. Bankomat queries amount

5. Client enters amount
Expected outputs

6. Bankomat returns bank card 7. Bankomat outputs specified amount in CHF

Peter Mller Software Engineering, SS 11

12

Using the Analysis Object Model

The analysis object model contains the main concepts manipulated by the system, their properties and relationships
- Useful to determine equivalence classes - Useful to set up state of objects

Relevant information
- Classes and attributes - Subtypes - Aggregations and multiplicities

Peter Mller Software Engineering, SS 11

13

Using the Analysis Object Model: Example

Book
Test for super- and subclass Library Member borrow( Copy ) 0..1 Test both cases 0..1 0..* 0..* Borrowable

1..*
Copy Recall: library members may borrow at most 6 items (12 for staff members)

Staff Member
borrow( Journal )

Journal

Peter Mller Software Engineering, SS 11

14

Equivalence Classes
Consider the operation member.borrow( copy )
member copy

Library member
Staff member Invalid library member Invalid staff member

05 borrowed copies
0 11 borrowed items more than 5 borrowed copies more than 11 borrowed items

Copy on shelf
Copy borrowed by library member Copy borrowed by staff member Invalid

copy has zero library members

copy has one library member copy has one staff member null

Peter Mller Software Engineering, SS 11

15

Using the Dynamic Model

Sequence diagrams describe protocols for object interactions
- Benefit for testing is similar to use cases - Especially useful for integration testing

State diagrams describe state-dependent behavior

- Different states typically require different equivalence classes - State defines valid input and expected output - Expected output includes successor state - Useful for protocols, GUIs, and objects
Peter Mller Software Engineering, SS 11

16

Using the Dynamic Model: Example 1

:Sensor signal( floor ) opt <<Control>> :Controller

<<Entity>> :Engine

<<Entity>> :RequestPool

Test both cases

stop( ) served( floor )

[ floor = nextStop ]

nextStop := getNext( ) alt setIdle( )

start ( direction )

[ nextStop = ] [ else ]
Test both cases

Peter Mller Software Engineering, SS 11

17

Equivalence Classes
Consider the operation controller.signal( floor )
controller Serve only request Serve first request floor = nextStop, and there is no further request floor = nextStop, and there are further requests

Serve no request

floor nextStop

Peter Mller Software Engineering, SS 11

18

Using the Dynamic Model: Example 2

Test valid and invalid operations [ pool is empty ] request( floor )

Idle

Requested Stop

request( floor )

[ pool is not empty ] / choose nextStop

signal( floor ) [ floor = nextStop ]

Moving
Test all cases signal( floor ) [ floor nextStop ]

request( floor )
Peter Mller Software Engineering, SS 11

19

Equivalence Classes
Consider the operation controller.signal( floor )
controller Reach floor controller is in state Moving and floor = nextStop controller is in state Moving and floor nextStop

Keep moving
Invalid state

controller is in state Idle or RequestedStop

Peter Mller Software Engineering, SS 11

8. Test Data Functional Testing

20

8. Test Data
8.1 Functional Testing
8.1.1 Partition Testing 8.1.2 Selecting Representative Values 8.1.3 Combinatorial Testing

8.2 Structural Testing 8.3 Automatic Test Case Generation 8.4 Dynamic Test Generation 8.5 Mutation Testing
Peter Mller Software Engineering, SS 11

21

Selecting Representative Values

Once we have partitioned the input values, we need to select concrete values for the test cases for each equivalence class Input from a range of valid values
- Below, within, and above the range - Also applies to multiplicities on aggregations

Input from a discrete set of valid values

- Valid and invalid discrete value - Instances of each subclass
Peter Mller Software Engineering, SS 11

22

Boundary Testing
Given an integer x, determine the absolute value of x
x Valid all values

int abs( int x ) { if( 0 <= x ) return x; return x; } Negative result for x==Integer.MIN_VALUE

Large number of errors tend to occur at boundaries of the input domain

Overflows Comparisons (< instead of <=, etc.) Missing emptiness checks (e.g., collections) Wrong number of iterations

Peter Mller Software Engineering, SS 11

23

Boundary Testing: Example

Select elements at the edge of each equivalence class (in addition to values in the middle)
- Ranges: lower and upper limit - Sets and collections: empty set
month Month with 28 or 29 days Months with 30 days Months with 31 days Invalid month = 2 month {4, 6, 9, 11} month {1, 3, 5, 7, 8, 10, 12} month < 1 or month > 12

There is only one value Choose all values Choose 1 and 12 plus one more Choose MIN_VALUE, 0, 13, MAX_VALUE

Peter Mller Software Engineering, SS 11

24

Boundary Testing: Example (contd)

Choose for instance -200.004, -4, 4, 2012, 400.008 Choose for instance -200.003, -1, 1, 2011, 400.009 Choose for instance -200.000, 0, 2000, 400.000

year Standard leap years Standard nonleap years Special leap years Special nonleap years year mod 4 = 0 and year mod 100 0 year mod 4 0

year mod 400 = 0

year mod 100 = 0 and year mod 400 0

Choose for instance -200.100, 1900, 400.100

Peter Mller Software Engineering, SS 11

25

Parameterized Unit Test for Leap Years

[ Test ] public void TestDemo29( [ Values( -200004, -200000, -4, 0,4, 2000, 2012, 400000, 400008 ) ] int year ) Only one All selected values for { value leap years and special int d = Days( 2, year ); leap years Assert.IsTrue( d == 29 ); } Expected result

Analogous test cases for February in non-leap year, months with 30 days, and months with 31 days
Peter Mller Software Engineering, SS 11

26

Parameterized Unit Test for Invalid Inputs

Expected result: an exception

All selected [ Test ] invalid values [ ExpectedException( typeof(ArgumentException) ) ] for month public void TestDemoInvalid( [ Values( int.MinValue, 0, 13, int.MaxValue ) ] int month, [ Values( -200100, -200004, -200003, -200000, -4, -1, 0, 1, 4, 1900, 2000, 2011, 2012, 400000, 400008, 400009, 400100 ) ] int year ) { int d = Days( month, year ); All selected } values for year

Peter Mller Software Engineering, SS 11

27

Coverage of State Diagrams

We define coverage criteria to measure how thorough a state diagram is tested Path Coverage: execute each possible path
- Not feasible with many nested conditionals - Impossible for most loops

State Coverage: visit each state

- A minimum criterion

Transition Coverage: execute each edge

- Thorough testing

Peter Mller Software Engineering, SS 11

28

Coverage Example
request( floor )

Idle

[ pool is empty ]

Requested Stop

request( floor )

[ pool is not empty ] / choose nextStop

signal( floor ) [ floor = nextStop ] One test for state coverage

Moving
signal( floor ) [ floor nextStop ]

request( floor )
Peter Mller Software Engineering, SS 11

29

Coverage Example (contd)

Consider the operation controller.signal( floor )
One test case for state coverage
controller controller is in state Moving and floor = nextStop controller is in state Moving and floor nextStop controller is in state Idle or RequestedStop

Reach floor
Keep moving Invalid state

Two test cases for transition coverage: empty and non-empty pool Not required for state coverage

Peter Mller Software Engineering, SS 11

8. Test Data Functional Testing

30

8. Test Data
8.1 Functional Testing
8.1.1 Partition Testing 8.1.2 Selecting Representative Values 8.1.3 Combinatorial Testing

8.2 Structural Testing 8.3 Automatic Test Case Generation 8.4 Dynamic Test Generation 8.5 Mutation Testing
Peter Mller Software Engineering, SS 11

31

Combinatorial Testing
Combining equivalence classes and boundary testing leads to many values for each input
- Twelve values for month and 17 values for year in the Leap Year example

Testing all possible combinations leads to a combinatorial explosion (12 x 17 = 204 tests) Reduce test cases to make effort feasible
- Semantic constraints - Combinatorial selection - Random selection
Peter Mller Software Engineering, SS 11

32

Eliminating Combinations
Inspect test cases for unnecessary combinations
- Especially for invalid values - Use domain knowledge
month Month with 28 or 29 days month = 2 month {4, 6, 9, 11} month {1, 3, 5, 7, 8, 10, 12} month < 1 or month > 12

Test all combinations with year

Months with 30 days

Months with 31 days Invalid

Behavior is independent of year

Reduces test cases from 204 to 17 + 4 + 3 + 4 = 28

Peter Mller Software Engineering, SS 11

34

Selecting Object References

Objects are different from values because they have identity
a1 = new Account( 1000 ); a2 = new Account( 1000 ); a1.transfer( a2, 500 ); a1 = new Account( 1000 ); a1.transfer( a1, 500 ); Might behave differently (e.g., deadlock)

When selecting test data for objects, one has to consider object identities and aliasing Referenced objects lead to combination problem
Peter Mller Software Engineering, SS 11

35

Selecting Object References: Example

member Library member Staff member Invalid library member 05 borrowed copies 0 11 borrowed items more than 5 borrowed copies

Empty list, list of 1 and 5 (11) copies, list without duplicates, list with duplicates

List List of of 6 (12) 6 (12) copies copies and very large collection

Invalid staff member

more than 11 borrowed items

This is a case of combinatorial testing since it combines the Library Member and the collection
Peter Mller Software Engineering, SS 11

36

Semantic Constraints for Objects

Object invariants restrict the possible instances of a class diagram
- Expressed as comment or OCL constraint

In our example, let assume the following invariants

Library Member 0..1 0..* Copy

- A Library Member m contains a Copy c in its collection of borrowed items if and only if cs Library Member is m - For each Library Member, the collection of borrowed items contains no duplicates - For each Library Member, the collection of borrowed items contains at most 6 copies (12 items for staff)
Peter Mller Software Engineering, SS 11

37

Selecting Object References: Example (contd)

Empty list, listlist of 1 ofand 5 (11) 5 (11) copies, copies without list withou duplicates
member
Library member 05 borrowed copies

copy
Copy on shelf copy has zero library members

Staff member
Invalid library member Invalid staff member

0 11 borrowed items
more than 5 borrowed copies more than 11 borrowed items

Copy borrowed by library member

Copy borrowed by staff member Invalid copy is borrowed by member, member, copy is not borrowed by by member member

copy has one library member

copy has one staff member null

List of 6 (12) copies

Peter Mller Software Engineering, SS 11

38

Roots Example
Given three values, a, b, c, compute all solutions of the equation ax2 + bx + c = 0
a b any value c any value

Valid
Invalid

any value

infinity, NaN

infinity, NaN

infinity, NaN

Boundary testing: a, b, c { Double.MIN_VALUE, -5, 0, 5, Double.MAX_VALUE }

53 = 125 test cases for valid inputs

Peter Mller Software Engineering, SS 11

39

Roots Example (contd)

Given three values, a, b, c, compute all solutions of the equation ax2 + bx + c = 0

Look at dependencies between inputs

No solution

Two solutions a 0 and b2-4ac > 0

One solution

a = 0 and b 0 or a 0 and b2-4ac = 0

a = 0, b = 0, and c 0 or a 0 and b2-4ac < 0

Semantic constraints on combinations

Peter Mller Software Engineering, SS 11

Partitioning seems too coarse

40

Roots Example (contd)

Given three values, a, b, c, compute all solutions of the equation ax2 + bx + c = 0
Two solutions
Linear equation (Truly) quadratic equation

One solution
a = 0 and b 0

No solution
a = 0, b = 0, and c 0

a 0 and b2-4ac > 0 a 0 and b2-4ac = 0

a 0 and b2-4ac < 0

Not all inputs are covered: a=b=c=0

Peter Mller Software Engineering, SS 11

41

Roots Example (contd)

Given three values, a, b, c, compute all solutions of the equation ax2 + bx + c = 0; report an error if all three values are zero
Two solutions
Linear equation (Truly) quadratic equation Invalid input

One solution
a = 0 and b 0

No solution
a = 0, b = 0, and c 0

a 0 and b2-4ac > 0 a 0 and b2-4ac = 0

a 0 and b2-4ac < 0

a = 0, b = 0, c = 0

Peter Mller Software Engineering, SS 11

42

Roots Example: Summary

Classifying the combinations according to semantic constraints did not reveal any irrelevant test cases But we did identify an omission in the specification
- It is common that testers clarify the specification

One option is to manually choose a manageable number of test cases such that there is at least one test case for each semantic constraint
- Note that omitting test cases might leave errors such as arithmetic overflow undetected
Peter Mller Software Engineering, SS 11

43

Semantic Constraints : Discussion

Semantic constraints potentially reduce the number of (interesting) test cases
- They also help increasing the coverage

But too many combinations remain

- Especially when there are many input values, for instance, for the fields of objects

Peter Mller Software Engineering, SS 11

44

Influence of Variable Interactions

Empirical evidence suggests that most errors do not depend on the interaction of many variables
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1 2 3 4 5 6 Medical Devices Browser Server NASA GSFC

Vars 1 2 3 4 5 6

Medical NASA Network Browser Server Devices GSFC Security 66% 97% 99% 100% 29% 76% 95% 97% 99% 100% 42% 70% 89% 96% 96% 100% 68% 93% 98% 100% 20% 65% 90% 98% 100%

Network Security

Interactions of two or three variables trigger most errors

Peter Mller Software Engineering, SS 11

45

Pairwise-Combinations Testing
Instead of testing all possible combinations of all inputs, focus on all possible combinations of each pair of inputs
- Pairwise-combinations testing is identical to combinatorial testing for two or less inputs

Example: Consider a method with four boolean parameters

- Combinatorial testing requires 24 = 16 test cases - Pairwise-combinations testing requires 5 test cases: TTTT, TFFF, FTFF, FFTF, FFFT

Can be generalized to k-tuples (k-way testing)

Peter Mller Software Engineering, SS 11

46

Pairwise-Combinations Testing: Complexity

For n parameters with d values per parameter, the number of test cases grows logarithmically in n and quadratic in d
- Handles larger number of parameters, for instance, fields of objects - The number d can be influenced by the tester

Result holds for large n and d, and for all k in k-way testing

Peter Mller Software Engineering, SS 11

47

Pairwise-Combinations Testing: Example

Two solutions One solution a = 0 and b 0 a 0 and b2-4ac > 0 a 0 and b2-4ac = 0 a = 0, b = 0, c = 0 No solution a = 0, b = 0, and c 0 a 0 and b2-4ac < 0

Three parameters, five values each

- Double.MIN_VALUE, -5, 0, 5, Double.MAX_VALUE - 53 = 125 test cases for combinatorial testing - 25 test cases for pairwise-combinations testing

Bug is still detected (depends only on a and b) Some cases depend on three parameters, e.g., invalid input
Peter Mller Software Engineering, SS 11

48

Pairwise-Combinations Testing: Discussion

Pairwise-combinations testing (or k-way testing) reduces the number of test cases significantly while detecting most errors Pairwise-combinations testing is especially important when many system configurations need to be tested
- Hardware, operating system, database, application server, etc.

Should be combined with other approaches to detect errors that are triggered by more complex interactions among parameters
Peter Mller Software Engineering, SS 11

49

Functional Testing: Summary

Functional Requirements, Analysis Model

Independently Testable Feature

Equivalence classes, boundary testing, coverage

Representative Values

Exhaustive enumeration, semantic constraints, pairwise combinations

Test Case Specification

Test Cases

Peter Mller Software Engineering, SS 11