Software Testing

Zeeshan Ali Rana

Software Faults and Failures
Why Does Software Fail?

Wrong requirement: not what the customer wants

Missing requirement
Requirement impossible to implement
Faulty design
Faulty code
Improperly implemented design
Objective of Testing
Objective of testing: discover faults
A test is successful only when a fault is discovered
Fault identification is the process of determining what
fault caused the failure
Fault correction is the process of making changes to the
system so that the faults are removed
Elements of a Test Case
Purpose
Input
Expected Output
Actual Output
Sample Format:
Types of Faults
Algorithmic fault
Computation and precision fault
 a formula’s implementation is wrong
Documentation fault
 Documentation doesn’t match what program does
Capacity or boundary faults
 System’s performance not acceptable when certain limits are
reached
Timing or coordination faults
Performance faults
 System does not perform at the speed prescribed
Software Faults and Failures
Typical Algorithmic Faults
An algorithmic fault occurs when a component’s
algorithm or logic does not produce proper output
Branching too soon
Branching too late
Forgetting to initialize variable or set loop invariants
Comparing variables of inappropriate data types
Different Level of Failure
Severity
Catastrophic: causes death or system loss
Critical: causes severe injury or major system damage
Marginal: causes minor injury or minor system
damage
Minor: causes no injury or system damage
Testing Issues
Attitude Toward Testing
programs are viewed as components of a larger system,
not as the property of those who wrote them
Testing Issues
Who Performs the Test?
Independent test team
avoid conflict
 personal responsibility vs need to discover faults
improve objectivity
allow testing and coding concurrently
Testing Issues
Views of the Test Objects
Closed box or black box: functionality of the test
objects
Equivalence Class, Boundary Value Analysis, Scenario-
based, Decision Table based, State Machine based…
Clear box or white box: structure of the test objects
Control Flow
 Basis Path, Branch, Statement, Decision…
Data Flow
 Du Path, All-uses Path
Test Cases for Triangle Problem
Test cases based on input range
TC What to test A B C EO
Id
1 EC1 of A -10 15 15 Value of A not in valid range
2 EC1 of B 15 -10 15 Value of B not in valid range
3 EC1 of C 15 15 -10 Value of C not in valid range
4 EC3 of A -205 15 15 Value of A not in valid range
5 EC3 of B 15 -205 15 Value of B not in valid range
6 EC3 of C 15 15 -205 Value of C not in valid range
7 EC2 of A, B, C 15 15 15 Equilateral

Note that we have made only one test case instead of three separate test
cases to check ABC in valid range. This one test could be any test cases with
any valid combination of A, B, C; for example see any of the valid range based
test cases.
Test cases based on equivalence classes within valid range

TC Id What to test A B C EO
1 EC 1 15 15 15 Equilateral
2 EC 2 10 20 10 Isosceles
3 EC 3 5 6 7 Scalene
4 EC 4 1 2 10 Not a triangle
Whitebox Testing
Control Flow Graph

Cyclomatic Complexity = E – N + 2 = 9 – 9 + 2 = 2
Statement testing
Branch testing
Path testing
…
Example
MxN?
M?
N?
M=N=100,000?
Levels of Testing
Module testing, component testing, or unit testing
Integration testing
System Testing
Function testing
Performance testing
Acceptance testing
Installation testing
Levels of Testing
Testing Organization Illustrated
Unit Testing
Code Review
Code walkthrough
Code inspection
Unit Testing
Testing the unit for correct functionality
Testing the unit for correct execution
Unit Testing
Steps in Testing

Determining test objectives

Selecting test cases
Executing test cases
Unit Testing
public static double calculateBill (int usage)
{
double bill = 0;
Whitebox Test Thoroughness
if(usage > 0) 1*/ i = 0;
/*Line
{
/*Line 2*/ n=4;
 bill = 40;
Statement testing
}
 /*Line
> 100)3*/ while (i<n-1) do
Branch testing
if(usage
/*Line 4*/ j = i + 1;
{
 Path testing /*Line>=
if(usage <=5*/
200) while (j<n) do
{
 … bill = bill
/*Line 6*/ + (usageif -A[i]<A[j]
100) * 0.5;
200) then
1do{ } /*Line 7*/ swap(A[i], A[j]);
else = bill + 50 + (usage - 100) * 0.1;
bill
2 ptr = false{
/*Line 8*/ + 50end+ do;
3 if(x>k) /*Line 9*/ i=i+1;
} bill = bill (usage - 200) * 0.1;
return bill;
if(bill >= 100)
4 }ptr = true {
5 x=x+1 /*Line
bill10*/ end* do;
= bill 0.9;
}
6 result= } f(x,ptr)
7}while } (result≤0)
return bill;
8print
} result
Exercise
/*Line 1*/ boolean isOutsideGrid(int x,int y,int xl, int yl){
/*Line 2*/ boolean isOutside = false;
/*Line 3*/ if (x<xl && y<yl)
/*Line 4*/ isOutside = true;
/*Line 5*/ return isOutside;
/*Line 6*/ }
 Unit Testing
Comparing Techniques

• Effectiveness of fault-discovery techniques

Requirements Documentation
Faults Design Faults Code Faults Faults
Reviews Fair Excellent Excellent Good
Prototypes Good Fair Fair Not applicable
Testing Poor Poor Good Fair
Correctness Proofs Poor Poor Fair Fair
Integration Testing
Big-bang
Bottom-up
Top-down
Sandwich testing
Integration Testing
Terminology
Component Driver: a routine that calls a particular
component and passes a test case to it
Stub: a special-purpose program to simulate the
activity of the missing component
Integration

Testing
System viewed as a hierarchy of components
Integration Testing
Bing-Bang Integration Example
All components integrated at once
Locating faults?
Integration Testing
Top-Down Integration Example
Only A is tested by itself
Stubs of B, C and D are used at first level
N-1 stubs required (N=Number of nodes)
Locating faults?
Integration Testing
Top-Down Integration Example
Integration Testing
Bottom-Up Integration Example
Drivers are used to call the child functions
Drivers are relatively intelligent
N - leaves drivers
Locating faults?
Integration Testing
Bottom-Up Integration Example
Integration Testing
Sandwich Integration Example
Viewed system as three layers
Employ BU where
writing drivers is
not costly

Employ TD where
stubs are easier to
Write
Locating faults?
Integration Testing
Call-Graph Based Testing
• Adjacency Matrix
• NxN matrix that tells which components call the other
components
• Pairwise Integration
• Test each pair (i.e. each edge)
• E testing sessions
• Neighborhood based Integration
• Integrate each neighborhood
• The nodes at one edge distance from the node to be
integrated
• N-sink nodes sessions
 System Testing
System is tested as a whole
Different types of testing considered during System Testing:
- Functional Testing (GUI) - Scalability
- Performance Testing - Sanity
- Usability - Smoke
- Load - Regression
- Volume - Compatibility
- Stress - Installation
- Security - Ad hoc
Steps to do System Testing
The following steps are important to perform System Testing:
Step 1: Create a System Test Plan
Step 2: Create Test Cases
Step 3: Carefully Build Data used as Input for System Testing
Step 3: If applicable create scripts to
- Build environment and
- to automate Execution of test cases
Step 4: Execute the test cases
Step 5: Fix the bugs if any and re test the code
Step 6: Repeat the test cycle as necessary
Principles of System Testing
Regression Testing
Identifies new faults that may have been introduced as
current one are being corrected
Verifies a new version or release still performs the
same functions in the same manner as an older version
or release
Principles of System Testing
Regression Testing Steps
Inserting the new code
Testing functions known to be affected by the new
code
Testing essential function of m to verify that they still
work properly
Continuing function testing m + 1
Principles of System Testing
Configuration Management
Versions and releases
Production system vs. development system
Deltas, separate files and conditional compilation
Change control
Acceptance Tests
Purpose and Roles
Enable the customers and users to determine if the
built system meets their needs and expectations
Written, conducted and evaluated by the customers
Acceptance Tests
Types of Acceptance Tests
Pilot test: install on experimental basis
Alpha test: in-house test
Beta test: customer pilot
Parallel testing: new system operates in parallel with
old system
Result of Acceptance Tests
List of requirements that
are not satisfied
must be deleted
must be revised
must be added
Installation Testing
Before the testing
Configure the system
Attach proper number and kind of devices
Establish communication with other system
The testing
Regression tests: to verify that the system has been
installed properly and works
Software Testing Tools
Automated Testing Tools
Selenium, QTP, SilkTest, WinRunner, LoadRunner,
Jmeter
Testing Management Tools
TestManager, TestDirector
Bug Tracking/Configuration Management Tools
Bugzilla, Jitterbug, SilkRadar
When to Stop Testing
No time left
No money left
Statistical Criteria
Number of defects found per week becomes lower
than a set threshold
Test Planning
Establish test objectives
Design and Write test cases
Test test cases
Execute tests
Evaluate test results
Test Planning
Purpose of the Plan
Test plan explains
who does the testing
why the tests are performed
how tests are conducted
when the tests are scheduled
Test Planning
Contents of the Plan
What the test objectives are
How the test will be run
What criteria will be used to determine when the
testing is complete
Test Documentation
Test plan: describes system and plan for exercising all
functions and characteristics
Test specification and evaluation: details each test
and defines criteria for evaluating each feature
Test description: test data and procedures for each
test
Test analysis report: results of each test
Test Documentation
Documents Produced During Testing
Test Documentation
Test Plan
The plan begins by stating its objectives, which should
guide the management of testing
guide the technical effort required during testing
establish test planning and scheduling
explain the nature and extent of each test
explain how the test will completely evaluate system
function and performance
document test input, specific test procedures, and
expected outcomes
Test Documentation
Parts of a Test Plan
Testing Documentation
Test-Requirement Correspondence Chart
Requirement 2.4.1: Requirement 2.4.2: Requirement 2.4.3:
Generate and Selectively Retrieve Produced Specialized
Test Maintain Database Data Reports

1. Add new record X

2. Add field X

3. Change field X

4. Delete record X

5. Delete field X

6. Create index X
Retrieve record with a
requested
7. Cell number X

8. Water height X

9. Canopy height X

10. Ground cover X

11, Percolation rate X

12. Print full database X

Test Documentation
Test Description
Including
the means of control
the data
the procedures
Test Documentation
Test Description Example
INPUT DATA:
Input data are to be provided by the LIST program. The program generates randomly a list of N words of
alphanumeric characters; each word is of length M. The program is invoked by calling
RUN LIST(N,M)
in your test driver. The output is placed in global data area LISTBUF. The test datasets to be used for this
test are as follows:
Case 1: Use LIST with N=5, M=5
Case 2: Use LIST with N=10, M=5
Case 3: Use LIST with N=15, M=5
Case 4: Use LIST with N=50, M=10
Case 5: Use LIST with N=100, M=10
Case 6: Use LIST with N=150, M=10
INPUT COMMANDS:
The SORT routine is invoked by using the command
RUN SORT (INBUF,OUTBUF) or
RUN SORT (INBUF)
OUTPUT DATA:
If two parameters are used, the sorted list is placed in OUTBUF. Otherwise, it is placed in INBUF.
SYSTEM MESSAGES:
During the sorting process, the following message is displayed:
“Sorting ... please wait ...”
Upon completion, SORT displays the following message on the screen:
“Sorting completed”
To halt or terminate the test before the completion message is displayed, press CONTROL-C on the
keyboard.
Test Documentation
Test Script for Testing The “change field”
Step N: Press function key 4: Access data file.
Step N+1: Screen will ask for the name of the date file.
Type ‘sys:test.txt’
Step N+2: Menu will appear, reading
* delete file
* modify file
* rename file
Place cursor next to ‘modify file’ and press RETURN key.
Step N+3: Screen will ask for record number. Type ‘4017’.
Step N+4: Screen will fill with data fields for record 4017:
Record number: 4017 X: 0042 Y: 0036
Soil type: clay Percolation: 4 mtrs/hr
Vegetation: kudzu Canopy height: 25 mtrs
Water table: 12 mtrs Construct: outhouse
Maintenance code: 3T/4F/9R
Step N+5: Press function key 9: modify
Step N+6: Entries on screen will be highlighted. Move cursor to VEGETATION field. Type ‘grass’ over ‘kudzu’ and press RETURN key.
Step N+7: Entries on screen will no longer be highlighted.
VEGETATION field should now read ‘grass’.
Step N+8: Press function key 16: Return to previous screen.
Step N+9: Menu will appear, reading
* delete file
* modify file
* rename file
To verify that the modification has been recorded,place cursor next to ‘modify file’ and press RETURN key.
Step N+10: Screen will ask for record number. Type ‘4017’.
Step N+11: Screen will fill with data fields for record 4017:
Record number: 4017 X: 0042 Y: 0036
Soil type: clay Percolation: 4 mtrs/hr
Vegetation: grass Canopy height: 25 mtrs
Water table: 12 mtrs Construct: outhouse
Maintenance code: 3T/4F/9R
Test Documentation
Test Analysis Report
Documents the result of test
Provides information needed to duplicate the failure
and to locate and fix the source of the problem
Provides information necessary to determine if the
project is complete
Establish confidence in the system’s performance
Test Documentation
Problem Report Forms
Location: Where did the problem occur?
Timing: When did it occur?
Symptom: What was observed?
End result: What were the consequences?
Mechanism: How did it occur?
Cause: Why did it occur?
Severity: How much was the user or business affected?
Cost: How much did it cost?
Test Documentation
Example of Actual Problem Report Forms
Test Documentation
Example of Actual Discrepancy Report Forms
Real-Time Example
The Ariane-5 System
The Ariane-5’s flight control system was tested in four
ways
equipment testing
on-board computer software testing
staged integration
system validation tests
The Ariane-5 developers relied on insufficient reviews
and test coverage
8.11 What this Chapter Means for You
It is important to understand the difference between
faults and failures
The goal of testing is to find faults, not to prove
correctness
References
UCF Slides
Software Testing, A Craftsman’s Approach by
Jorgensen
Software Testing Tools by Prasad
Software Engineering, Business Continuity, and
Education. Kim, Adeli et al (Eds.)
https://www.guru99.com/cyclomatic-complexity.html