Chapter 3

Error Fault and Failure:

In general, software engineers distinguish software faults from software failures. In case of a
failure, the software does not do what the user expects. A fault is a programming bug that
may or may not actually manifest as a failure. A fault can also be described as an error in the
correctness of the semantic of a computer program. A fault will become a failure if the exact
computation conditions are met, one of them being that the faulty portion of computer
software executes on the CPU. A fault can also turn into a failure when the software is ported
to a different hardware platform or a different compiler, or when the software gets extended.
The term error is used to refer to the discrepancy between a computed, observed or measured
value and the true, specified or theoretically correct value. Basically it refers to the difference
between the actual output of a program and the correct output.
Fault is a condition that causes a system to fail in performing its required functions.
Failure is the inability of a system to perform a required function according to its
specification. In case of a failure the observed behaviour of the system is different
from the specified behaviour. Whenever there is a failure, there is a fault in the system but
vice-versa may not be true. That is, sometimes there is a fault in the software but failure is not
observed. Fault is just like an infection in the body. Whenever there is fever there is an
infection, but sometimes body has infection but fever is not observed,

Software Testing Fundamentals:

Software testing may be viewed as a sub-field of software quality assurance but typically
exists independently (and there may be no SQA areas in some companies). In SQA, software
process specialists and auditors take a broader view on software and its development. They
examine and change the software engineering process itself to reduce the amount of faults
that end up in the code or deliver faster.
Regardless of the methods used or level of formality involved the desired result of testing is a
level of confidence in the software so that the developers are confident that the software has
an acceptable defect rate. What constitutes an acceptable defect rate depends on the nature of
the software.
A problem with software testing is that the number of defects in a software product can be
very large, and the number of configurations of the product larger still. Bugs that occur
infrequently are difficult to find in testing. A rule of thumb is that a system that is expected to
function without faults for a certain length of time must have already been tested for at least
that length of time. This has severe consequences for projects to write long-lived reliable
software.
A common practice of software testing is that it is performed by an independent group of
testers after the functionality is developed but before it is shipped to the customer. This
practice often results in the testing phase being used as project buffer to compensate for
project delays. Another practice is to start software testing at the same moment the project
starts and it is a continuous process until the project finishes.
Another common practice is for test suites to be developed during technical support
escalation procedures. Such tests are then maintained in regression testing suites to ensure
that future updates to the software don't repeat any of the known mistakes.
It is commonly believed that the earlier a defect is found the cheaper it is to fix it.
Unit tests are maintained along with the rest of the software source code and generally
integrated into the build process (with inherently interactive tests being relegated to a
partially manual build acceptance process). The software, tools, samples of data input and
output, and configurations are all referred to collectively as a test harness.
Testing is the process of finding the differences between the expected behaviour specified by
system models and the observed behaviour of the system. Software testing consists of the
dynamic verification of the behaviour of a program on a finite set of test cases, suitably
selected from the usually infinite executions domain, against the specified expected
behaviour.

Testing Objectives:

There are a number of rules that can serves as testing objectives:

􀂾 Testing is a process of executing a program with the intent of finding an error.
􀂾 A good test case is one that has the high probability of finding an as-yet undiscovered
error.
􀂾 A successful test is one that uncovers an as-yet undiscovered error.

Testing principles:

There are the following testing principles:

􀂾 All tests should be traceable to customer requirements.
􀂾 Tests should be planned long before testing begins.
􀂾 The Pareto principle applies to software testing. According to this principle 80 percent of
all errors uncovered during testing will likely to be traceable to 20 percent of all program
modules. The problem is to isolate these 20 percent modules and test them thoroughly.
􀂾 Testing should begin “in the small” and progress toward testing “in the large”.
􀂾 Exhaustive testing is not possible.
􀂾 To be most effective, testing should be conducted by an independent third party.

Psychology of Testing:

“Testing cannot show the absence of defects, it can only show that software errors are
present”. So devising a set of test cases that will guarantee that all errors will be detected is
not feasible. Moreover, there are no formal or precise methods for selecting test cases. Even
though, there are a number of heuristics and rules of thumb for deciding the test cases,
selecting test cases is still a creative activity that relies on the ingenuity of the tester. Due to
this reason, the psychology of the person performing the testing becomes important.
The aim of testing is often to demonstrate that a program works by showing that it has no
errors. This is the opposite of what testing should be viewed as. The basic purpose of the
testing phase is to detect the errors that may be present in the program. Hence, one should not
start testing with the intent of showing that a program works; but the intent should be to show
that a program does not work. With this in mind, we define testing as follows: testing is the
process of executing a program with the intent of finding errors.
This emphasis on proper intent of testing is a trivial matter because test cases are designed by
human beings, and human beings have a tendency to perform actions to achieve the goal they
have in mind. So, if the goal is to demonstrate that a program works, we may consciously or
subconsciously select test cases that will try to demonstrate that goal and that will beat the
basic purpose of testing. On the other hand, if the intent is to show that the program does not
work, we will challenge our intellect to find test cases towards that end, and we are likely to
detect more errors. Testing is the one step in the software engineering process that could be
viewed as destructive rather than constructive. In it the engineer creates a set of test cases that
are intended to demolish the software. With this in mind, a test case is "good" if it detects an
as-yet undetected error in the program, and our goal during designing test cases should be to
design such "good" test cases.
Due to these reasons, it is said that the creator of a program (i.e. programmer) should not be
its tester because psychologically you cannot be destructive to your own creation. Many
organizations require a product to be tested by people not involved with developing the
program before finally delivering it to the customer. Another reason for independent testing is
that sometimes errors occur because the programmer did not understand the specifications
clearly. Testing of a program by its programmer will not detect such errors, whereas
independent testing may succeed in finding them.

Test Levels:

􀂾 Unit testing: It tests the minimal software item that can be tested. Each component is tested
independently.
􀂾 Module testing: A module is a collection of dependent components. So it is component
integration testing and it exposes defects in the interfaces and interaction between integrated
components.
􀂾 Sub-system testing: It involves testing collection of modules which have been integrated
into sub-systems. The sub-system test should concentrate on the detection of interface errors.
􀂾 System testing: System testing tests an integrated system to verify that it meets its
requirements. It is concerned with validating that the system meets its functional and non-
functional requirements.
􀂾 Acceptance testing: Acceptance testing allows the end-user or customer to decide whether
or not to accept the product.

System Testing:

System testing involves two kinds of activities: integration testing and acceptance testing.
Strategies for integrating software components into a functioning product include the bottom-
up strategy, the top-down strategy, and the sandwich strategy. Careful planning and
scheduling are required to ensure that modules will be available for integration into the
evolving software product when needed. The integration strategy dictates the order in which
modules must be available, and thus exerts a strong influence on the order in which modules
are written, debugged, and unit tested.
Acceptance testing involves planning and execution of functional tests, performance tests,
and stress tests to verify that the implemented system satisfies its requirements. Acceptance
tests are typically performed by the quality assurance and/or customer organizations.
Depending on local circumstances, the development group may or may not be involved in
acceptance testing. Integration testing and acceptance testing are discussed in the following
sections.

Integration Testing:
Three are two important variants of integration testing, (a) Bottom-up integration and (b)top-
down integration, which are discussed in the following sections:
 (a)Bottom-up integration
Bottom-up integration is the traditional strategy used to integrate the components of a
software system into a functioning whole. Bottom-up integration consists of unit testing,
followed by subsystem testing, followed by testing of the entire system. Unit testing has the
goal of discovering errors in the individual modules of the system. Modules are tested in
isolation from one another in an artificial environment known as a "test harness," which
consists of the driver programs and data necessary to exercise the modules. Unit testing
should be as exhaustive as possible to ensure that each representative case handled by each
module has been tested. Unit testing is eased by a system structure that is composed of small,
loosely coupled modules. A subsystem consists of several modules that communicate with
each other through well-defined interfaces.
Normally, a subsystem implements a major segment of the total system. The primary purpose
of subsystem testing is to verify the operation of the interfaces between modules in the
subsystem. Both control and data interfaces must be tested. Large software may require
several levels of subsystem testing; lower level subsystems are successively combined to
form higher-level subsystems. In most software systems, exhaustive testing of subsystem
capabilities is not feasible due to the combinational complexity of the module interfaces;
therefore, test cases must be carefully chosen to exercise the interfaces in the desired manner.
System testing is concerned with subtleties in the interfaces, decision logic, control flow,
recovery procedures, throughput; capacity, and timing characteristics of the entire system.
Careful test planning is required to determine the extent and nature of system testing to be
performed and to establish criteria by which the results will be evaluated.
Disadvantages of bottom-up testing include the necessity to write and debug test harnesses
for the modules and subsystems, and the level of complexity that result from combining
modules and subsystems into larger and larger units. The extreme case of complexity results
when each module is unit tested in isolation and all modules are then linked and executed in
one single integration run. This is the "big bang" approach to integration testing. The main
problem with big-bang integration is the difficulty of isolating the sources of errors.
Test harnesses provide data environments and calling sequences for the routines and
subsystems that are being tested in isolation. Test harness preparation can amount to 50
percent or more of the coding and debugging effort for a software product.

(b)Top-down integration:
Top-down integration starts with the main routine and one or two immediately subordinate
routines in the system structure. After this top-level "skeleton" has been thoroughly tested, it
becomes the test harness for its immediately subordinate routines. Top-down integration
requires the use of program stubs to simulate the effect of lower-level routines that are called
by those being tested.
Top-down integration offers several advantages:
1. System integration is distributed throughout the implementation phase. Modules are
integrated as they are developed.
2. Top-level interfaces are tested first and most often.
3. The top-level routines provide a natural test harness for lower-Level routines.
4. Errors are localized to the new modules and interfaces that are being added.
While it may appear that top-down integration is always preferable, there are many situations
in which it is not possible to adhere to a strict top-down coding and integration strategy. For
example, it may be difficult to find top-Level input data that will exercise a lower level
module in a particular desired manner. Also, the evolving system may be very expensive to
run as a test harness for new routines; it may not be cost effective to relink and re-execute a
system of 50 or 100 routines each time a new routine is added. Significant amounts of
machine time can often be saved by testing subsystems in isolation before inserting them into
the evolving top-down structure. In some cases, it may not be possible to use program stubs
to simulate modules below the current level (e.g. device drivers, interrupt handlers). It may be
necessary to test certain critical low-level modules first.
The sandwich testing strategy may be preferred in these situations. Sandwich integration is
predominately top-down, but bottom-up techniques are used on some modules and
subsystems. This mix alleviates many of the problems encountered in pure top-down testing
and retains the advantages of top-down integration at the subsystem and system level.

Regression testing:

After modifying software, either for a change in functionality or to fix defects, a regression
test re-runs previously passing tests on the modified software to ensure that the modifications
haven't unintentionally caused a regression of previous functionality. Regression testing can
be performed at any or all of the above test levels. These regression tests are often automated.
In integration testing also, each time a module is added, the software changes. New data flow
paths are established, new I/O may occur, and new control logic is invoked. Hence, there is
the need of regression testing.
Regression testing is any type of software testing which seeks to uncover regression bugs.
Regression bugs occur whenever software functionality that previously worked as desired
stops working or no longer works in the same way that was previously planned. Typically
regression bugs occur as an unintended consequence of program changes Common methods
of regression testing include re-running previously run tests and checking whether previously
fixed faults have reemerged.
Experience has shown that as software is developed, this kind of reemergence of faults is
quite common. Sometimes it occurs because a fix gets lost through poor revision control
practices (or simple human error in revision control), but just as often a fix for a problem will
be "fragile" - i.e. if some other change is made to the program, the fix no longer works.
Finally, it has often been the case that when some feature is redesigned, the same mistakes
will be made in the redesign that were made in the original implementation of the feature.
Therefore, in most software development situations it is considered good practice that when a
bug is located and fixed, a test that exposes the bug is recorded and regularly retested after
subsequent changes to the program. Although this may be done through manual testing
procedures using programming techniques, it is often done using automated testing tools.
Such a 'test suite' contains software tools that allows the testing environment to execute all
the regression test cases automatically; some projects even set up automated systems to
automatically rerun all regression tests at specified intervals and report any regressions.
Common strategies are to run such a system after every successful compile (for small
projects), every night, or once a week.
Regression testing is an integral part of the extreme programming software development
methodology. In this methodology, design documents are replaced by extensive, repeatable,
and automated testing of the entire software package at every stage in the software
development cycle.

Uses of regression testing

Regression testing can be used not only for testing the correctness of a program, but it is also
often used to track the quality of its output. For instance in the design of a compiler,
regression testing should track the code size, simulation time, and compilation time of the test
suites.
System testing is a series of different tests and each test has a different purpose but all work
to verify that all system elements have been properly integrated and perform allocated
functions.

Recovery testing:

Many systems must recover from faults and resume processing within a specified time.
Recovery testing is a system test that forces the software to fail in a variety of ways and
verifies that recovery is properly performed.
Stress testing:

Stress tests are designed to confront programs with abnormal situations. Stress testing
executes a program in a manner that demands resources in abnormal quantity, frequency, or
volume. For example, a test case that may cause thrashing in a virtual operating system.

Performance Testing:

For real time and embedded systems, performance testing is essential. In these systems, the
compromise on performance is unacceptable. Performance testing is designed to test run-time
performance of software within the context of an integrated system.

Acceptance testing:

Acceptance testing involves planning and execution of functional tests, performance tests,
and stress tests in order to demonstrate that the implemented system satisfies its requirements.
Stress tests are performed to test the limitations of the systems. For example, a compiler may
be tested to determine the effect of symbol table overflow.
Acceptance test will incorporate test cases developed during unit testing and integration
testing. Additional test cases are added to achieve the desired level of functional,
performance and stress testing of the entire system.

(A)Alpha testing
Alpha testing is simulated or actual operational testing by potential users/customers or an
independent test team at the developers’ site. Alpha testing is often employed for off-the-
shelf software as a form of internal acceptance testing, before the software goes to beta
testing.
(B)Beta testing
Beta testing comes after alpha testing. Versions of the software, known as beta versions, are
released to a limited audience outside of the company. The software is released to groups of
people so that further testing can ensure the product has few faults or bugs. Sometimes, beta
versions are made available to the open public to increase the feedback field to a maximal
number of future users.