Software testing – Implementation

ICT 3312

Oshadhi Munasinghe
Software testing – implementation -
Overview

1 The testing process

2 Test case design
3 Automated testing
4 Alpha and beta site testing programs

2
The Testing Process

▪The main issues to be raised with respect to testing implementation:

➢Test effectiveness : reduction of the percentage of

undetected errors remaining in the software or system
tested

➢Test efficiency : performance of those tests with fewer

resources
3
Test Effectiveness

Test Effectiveness can be defined as how effectively testing is done or goal is achieved that
meets the customer requirement.

In SDLC (Software development Life Cycle), we have requirements gathering phase where SRS
(Software Requirements Specification) and FRD (Functional Requirements Document) are
prepared and based on that development team starts building the software application, at the
same time test cases are carved out of SRS and FRD documents by the testing team.

4
Test Effectiveness
Test effectiveness starts right at the beginning of the development and execution of test
cases and after development is completed to count the number of defects.

Defects can be valid or invalid. Valid defects are required to be fixed in the application or
product and invalid ones need to be closed or ignored. Thus, mathematically it is
calculated as a percentage of a number of valid defects fixed in software application
divided by the sum of a total number of defects injected and a total number of defects
escaped.

Test Effectiveness = ((Defects removed in a phase) / (Defect injected

+ Defect escaped)) * 100

5
Test Efficiency

Test Efficiency can be defined as the most efficient way in which the resources
associated with the software project are utilized to achieve an organization goal
◦ (for example, number of projects to be completed in that particular year).

Test efficiency is an internal process for an organization that evaluates the utilization of
resources which could be a timeline, hardware, manpower, expertise of test team, etc. to
develop a software product.

Mathematically test efficiency is calculated as a percentage of the number of alpha testing

(in-house or on-site) defects divided by sum of a number of alpha testing and a number of
beta testing (off-site) defects.
6
The Testing Process

▪ Test Planning and Designing activities

include:

➢ Determining the test methodology

➢ Planning unit and integration tests

➢ Planning the system test

➢ Designing the tests

➢ Implementation 7
Test Planning

▪Determining the test methodology : Deals mainly with decisions

about the required software quality standard and the software
strategy:
➢big bang or incremental testing (bottom-up or top- down)
➢ and the extent of automated testing.

8
Test Planning

▪Planning the tests : including…

➢ unit tests
➢ integration tests
➢System tests (refer to the entire software
package/system)

10
Test Planning- Unit Test
Unit Test
This type of Test Plan looks solely at the small component or modules which comprise the
system or software.

Evidence of Further Reading: MSDN defines unit test as,

“The primary goal of unit testing is to take the smallest piece of testable software in the
application, isolate it from the remainder of the code, and determine whether it behaves
exactly as you expect. Each unit is tested separately before integrating them into modules
to test the interfaces between modules. Unit testing has proven its value in that a large
percentage of defects are identified during its use.”
(http://msdn.microsoft.com/en-us/library/aa292197(VS.71).aspx)

11
0
Test Planning- Integration & System Test

Integration Test
Integration testing involves testing several modules or components together.

Evidence of Further Reading: MSDN defines integration test as,

“Integration testing is a logical extension of unit testing. In its simplest form, two units that have
already been tested are combined into a component and the interface between them is tested”
(http://msdn.microsoft.com/en-us/library/aa292128.aspx)

System Test
System test involves testing the whole system/software.

11
1
Test Planning : Main issues

▪What to test?
➢ which modules to be unit tested?
➢ which integrations to be tested?
➢for the system test : consider the already planned unit
and integration tests
➢ Prioritize based on the Damage severity level and
Software risk level

11
2
Test Planning : Main issues

▪ Which types of sources are to provide the test cases?

➢ samples of real-life and/or synthetic test cases

▪ Who performs the tests?

➢development team/ independent testing team(s)/ another
development team/ outsourcing…

▪ Where will the tests be performed?

➢ Developer’s site/ customer’s site/ third party site…

11
3
Test Planning : Main issues …

▪When to terminate the (system) tests?

➢ The completed implementation route : perfection approach

➢ The mathematical models application route :

o estimate the % of undetected errors, based on the rate of error detection

➢ The error seeding route :

o errors of various types are seeded (hidden) in the tested software prior to
testing
o assumes that the % of discovered seeded errors will correspond to the % of
real errors detected

11
4
Test Planning : Main issues …

➢ The dual independent testing teams route

oBy comparing the lists of detected errors provided by
two teams, the number of errors left undetected is
estimated
➢ Termination after resources have petered out : undesirable!!

11
5
Test Planning : Documentation

▪The planning stage of the software system tests is

commonly documented in a “Software Test Plan” (STP).

▪The main sections of the STP include:

1. Scope of the tests
2. Testing environment
3. Test details (for each test)
4. Test schedule (for each test or test group)
including time estimates

11
6
11
7
Test Designing

▪The test design is carried out on the basis of the Software Test Plan (STP).

▪The products of the test design stage are:

➢ Detailed design and procedure for each test → “software test procedure” document
➢ Test case database/file → “test case file” document

▪ Or the products of the test design stage may be documented in a single document
called the “Software Test description” (STD)

11
8
20
Test Implementation

▪Commonly, the testing implementation phase activities consist

of :
➢ a series of tests,
➢ corrections of detected errors
➢ and re-tests (regression tests).

12
0
Test Implementation

▪The tests are carried out by running the test cases according to the
test procedures given in the “Software Test description” (STD)

▪Re-testing (also termed “regression testing”) is conducted to verify

that the errors detected in the previous test runs have been
properly corrected, and that no new errors have entered as a
result of faulty corrections.
➢Testing is culminated when the re-test results
satisfy the developers

12
1
Automated Testing
▪Automated testing represents an additional step in the integration of computerized tools
into the process of software development.

▪These tools have joined computer aided software engineering (CASE) tools in performing
a growing share of software analysis and design tasks.

▪ Motivating Factors :
➢anticipated cost savings,
➢shortened test duration,
➢heightened thoroughness of the tests performed, improvement of test
accuracy,
➢improvement of result reporting,
➢statistical processing and subsequent reporting.
12
2
Automated testing…
▪ Typically, automated software testing requires :
➢ test planning, test design,
➢ test case preparation,
➢ test performance, test log and report preparation,
➢ re-testing after correction of detected errors (regression tests),
➢ final test log and report preparation including comparison reports.

▪ The last two activities may be repeated several times.

▪ The planning, design and test case preparation of automated testing require
substantial investment of professional manpower.

▪ It is the computerized test performance and the reporting that yield the main economic,
quality and timetable advantages of the process.

12
3
Types of automated testing

▪ Code auditing
▪ Coverage monitoring
▪ Functional tests
▪ Load tests
▪ Test management

12
4
Code Auditing

▪ Code auditors :
➢Checks the compliance of code to specified standards and procedures of coding.
➢ An automated type of qualification testing.
➢Auditor’s report includes a list of the deviations from the standards and a
statistical summary of the findings.

12
5
Coverage Monitoring
▪ Coverage monitors :
➢ Produce reports about the line coverage achieved by implementing a given test
case file.
➢ Code Coverage is a measurement of how many lines, statements, or blocks of your
code are tested using your suite of automated tests. It’s an essential metric to
understand the quality of your QA efforts. Code coverage shows you how much of
your application is not covered by automated tests and is therefore vulnerable to
defects. Code coverage is typically measured in percentage values – the closer to
100%, the better.

12
6
Functional Tests
▪ Functional tests :
➢ Automated functional tests replace manual black-box correctness tests.
➢ The first test runs as well as the regression test runs, applied with the same test
files and the same test cases, are performed by a computer program that
replaces the “classic” tester.

12
7
 Load Tests
▪ Load tests :
➢The load tests are based on simulated scenarios of maximal load
situations the software system will confront.
➢An automated testing system enables measurement of the
expected performance of the software system under various load
levels.

40
Test Management
▪ Test management :
The main objectives of these automated tools are…
➢to provide comprehensive follow-up and
➢reporting of the testing and correction of detected errors.

12
9
13
0
13
1
Alpha and Beta site testing programs
▪ Alpha site and beta site tests are employed to obtain comments about quality from
the package’s potential users.

▪ They are additional commonly used tools to identify software design and code errors
in software packages in commercial over-the-counter sale (COTS).

▪ In a way, alpha and beta site tests replace the customer’s acceptance test, a test that
is impractical under the conditions of commercial software package development.

▪ However, an analysis of the characteristics of these tests leads one to conclude that in
no case should they replace the formal software tests performed by the developer.

13
2
Alpha site testing
▪ Alpha site testing is a method by which customers try out the new software
package at the developer’s site.

▪ The customer, by applying the new software to the specific requirements of his
organization, tends to examine the package from angles not expected by the testing
team.

▪ The errors identified by alpha site tests are expected to include the errors that only
use by a real user can reveal, and thus should be reported to the developer.

13
3
Alpha site testing
Advantages of Alpha Testing:
• Clear view of the reliability of the product is provided at an early stage.
• It helps in simulating the real-time user behaviour and environment.
• It helps in detecting serious threats or bugs and allows immediate action towards
rectification.
Disadvantage of Alpha Testing:
• In depth functionality of the software cannot be tested as it is still under development
stage.

13
4
Beta site testing
▪ Beta site testing is a method by which a selected group of users or customers
receive an advanced version of the software to be installed in their sites, and
report the errors they find in the process of their experiments with the program
and the program’s regular use.

▪ Beta site tests are much more commonly applied than are alpha site tests.

▪ Participants in beta site testing are often users of previously released packages,
sophisticated software professionals and the like.

▪ Because beta site tests are considered to be a valuable tool, some developers involve
hundreds or even thousands of participants in the process.

13
5
Beta site testing …

Advantages :
➢ Identification of unexpected errors
➢ Wide scope of coverage in search for errors
➢ Low costs

Disadvantages :
➢ Lack of systematic testing
➢ Error reports of low quality
➢Much human effort expended to examine participants reports.

13
6
Test Implementation : Documentation

▪The results of the individual tests and re-tests

are documented in a “Software Test Report”
(STR).

▪The summary of the set of planned tests is

documented in the “Test Summary Report”
(TSR).

13
7
Test Implementation : Documentation

▪The main sections of the STR include:

1. Test identification, site, schedule and participation
2. Test environment
3. Test results
4. Summary tables for total number of errors, their distribution
and types
5. Special events and testers’ proposals

13
8
13
9