Fundamentals of Testing – Part 1

This introductory chapter will explain basic facts of software testing, covering what you will need
to know to understand the following chapters. Important concepts and essential vocabulary will be
explained. The fundamental test process with the different testing activities will be illustrated.
Psychological problems with testing will be discussed.

Case study, “VirtualShowRoom” – VSR

Throughout this course, we’ll use one example application to illustrate the software test methods
and techniques. The fundamental scenario is as follows.

A car manufacturer develops a new electronic sales support system called VirtualShowRoom
(VSR). The final version of this software system will be installed at every car dealer worldwide.
Customers who are interested in purchasing a new car will be able to configure their favorite model
(model, type, color, extras, etc.), with or without the guidance of a salesperson.

The system shows possible models and combinations of extra equipment and instantly calculates
the price of the car the customer configures. A subsystem called DreamCar will provide this
functionality.

When the customer has made up her mind, she will be able to calculate the most suitable financing
(EasyFinance) as well as place the order online (JustInTime). She will even get the option to
sign up for the appropriate insurance (NoRisk). Personal information and contract data about the
customer is managed by the ContractBase subsystem.
Figure shows the general architecture of this software system:
2.1 Terms and Motivation
Requirements: During the construction of an industry product, the parts and the final product are
usually examined to make sure they fulfill the given ➞requirements, that is, whether the product
solves the required task.

Software is immaterial: What generally counts for the production of industry products is also
appropriate for the production or development of software. However, testing (or evaluation) of
partial products and the final product is more difficult, because a software product is not a tangible
physical product. Direct examination is not possible. The only way to examine the product is by
reading (reviewing) the development documents and code.

The dynamic behavior of the software, however, cannot be checked this way. It must be done
through ➞testing, by executing the software on a computer. Its behavior must be compared to the
requirements. Thus, testing of software is an important and difficult task in software development.
It contributes to reducing the ➞risk of using the software because ➞defects can be found in
testing.

2.1.1 Error, Defect, and Bug Terminology

Failure: When does a system behave incorrectly, not conforming to requirements?

A situation can be classified as incorrect only after we know what the correct situation is supposed
to look like. Thus, a ➞failure means that a given requirement is not fulfilled; it is a discrepancy
between the ➞actual result or behavior and the ➞expected result or behavior.

To describe the event when a user experiences a problem, uses the term failure. However, other
terms, like problem, issue, and incident, are often used. During testing or use of the software, the
failure becomes visible to the ➞tester or user; for example, an output is wrong or the program
crashes.
Fault/Defect/Bug: We have to distinguish between the occurrence of a failure and its Fault. A
failure is caused by a fault in the software. This fault is also called a defect or internal error.
Programmer slang for a fault is bug. For example, faults can be incorrect or forgotten ➞statements
in the program.

Defect masking: It is possible that a fault is hidden by one or more other faults in other parts of
the program (➞defect masking). In that case, a failure occurs only after the masking defects have
been corrected. This demonstrates that corrections can have side effects. One problem is that a
fault can cause none, one, or many failures for any number of users.

Error or mistake: The cause of a fault or defect is an ➞error or ➞mistake by a person— for
example, defective programming by the developer. However, faults may even be caused by
environmental conditions, like radiation and magnetism that introduce hardware problems.
People err, especially under time pressure. Defects may occur, for example, by bad programming
or incorrect use of program statements. Forgetting to implement a requirement leads to defective
software. Another cause is changing a program part because it is complex and the programmer
does not understand all consequences of the change.

2.1.2 Testing Terms

Testing is not debugging: To be able to correct a defect or bug, it must be localized in the
software. Initially, we know the effect of a defect but not the precise location in the software.
Localization and correction of defects are tasks for a software developer and are often called
➞debugging.

Repairing a defect generally increases the ➞quality of the product because the ➞change in most
cases does not introduce new defects. However, in practice, correcting defects often introduces
one or more new defects. The new defects may then introduce failures for new, totally different
inputs. Such unwanted side effects make testing more difficult. The result is that not only must we
repeat the ➞test cases that have detected the defect, we must also conduct even more test cases to
detect possible side effects.
Debugging is often equated with testing, but they are entirely different activities. Debugging is the
task of localizing and correcting faults. The goal of testing is the (more or less systematic) detection
of failures (that indicate the presence of defects).
Testing software has different purposes:
■ Executing a program to find failures
■ Executing a program to measure quality
■ Executing a program to provide confidence5
■ Analyzing a program or its documentation to prevent failures

Testing terms: Besides execution of the test object with ➞test data, planning, design,
implementation, and analysis of the test (➞test management) also belong to the ➞test process. A
➞test run or ➞test suite includes execution of one or more ➞test cases. A test case contains
defined test conditions. In most cases, these are the preconditions for execution, the inputs, and
the expected outputs or the expected behavior of the test object. A test case should have a high
probability of revealing previously unknown faults.

Several test cases can often be combined to create ➞test scenarios, whereby the result of one test
case is used as the starting point for the next test case. For example, a test scenario for a database
application can contain one test case writing a date into the database, another test case changing
that date, and a third test case reading the changed date from the database and deleting it. (By
deleting the date, the database should be in the same state as before executing this scenario.) Then
all three test cases will be executed, one after another, all in a row.

No large software system is bug free: At present, there is no known bug-free software system,
and there will probably not be any in the near future (if a system has nontrivial complexity). Often
the reason for a fault is that certain exceptional cases were not considered during development and
testing of the software.

Testing cannot produce absence of defects: Even if all the executed test cases do not show any
further failures, we cannot safely conclude (except for very small programs) that there are no
further faults or that no further test cases could find them.

Naming tests:
The following terms describe the different ways tests are named:

➞Test objective or test type: A test is named according to its purpose (for example, ➞load test).

➞Test technique: A test is named according to the technique used for specifying or executing the
test (for example, ➞business-process-based test).

Test object: The name of a test reflects the kind of the test object to be tested (for example, a GUI
test or DB test [database test]).

Test level: A test is named after the level of the underlying life cycle model (for example, ➞system
test).

Test person: A test is named after the personnel group executing the tests (for example, developer
test, ➞user acceptance test).

Test extent: A test is named after the level of extent (for example, partial ➞regression test, full
test).

2.1.3 Software Quality

Software testing contributes to improvement of ➞software quality. This is done by identifying
defects and subsequently correcting them. If the test cases are a reasonable sample of software use,
quality experienced by the user should not be too different from quality experienced during testing.

But software quality is more than just the elimination of failures found during testing. According
to the ISO/IEC Standard 9126-1 [ISO 9126], software quality comprises the following factors:
➞functionality, ➞reliability, usability, ➞efficiency, ➞maintainability, and portability.

Testing must consider all these factors, also called ➞quality characteristics and ➞quality
attributes, in order to judge the overall quality of a software product. Which quality level the test
object is supposed to show for each characteristic should be defined in advance. Appropriate tests
must then check to make sure these requirements are fulfilled.

Functionality: When we talk about functionality, we are referring to all of the required capabilities
of a system. The capabilities are usually described by a specific input/output behavior and/or an
appropriate reaction to an input. The goal of the test is to prove that every single required capability
in the system was implemented as described in the specifications.

Reliability: Reliability describes the ability of a system to keep functioning under specific use
over a specific period. In the standard, the reliability characteristic is split into maturity, ➞fault
tolerance, and recoverability.
Maturity means how often a failure of the software occurs as a result of defects in the software.

Fault tolerance is the capability of the software product to maintain a specified level of
performance or to recover from faults such as software faults, environment failures, wrong use of
interface, or incorrect input.

Recoverability is the capability of the software product to reestablish a specified level of

performance (fast and easily) and recover the data directly affected in case of failure.
Recoverability describes the length of time it takes to recover, the ease of recovery, and the amount
of work required to recover. All this should be part of the test.

Usability: Usability is very important for acceptance of interactive software systems. Users won’t
accept a system that is hard to use. What is the effort required for the usage of the software for
different user groups? Understandability, ease of learning, operability, and attractiveness as well
as compliance to standards, conventions, style guides, and user interface regulations are aspects of
usability. These quality characteristics are checked in ➞nonfunctional tests.

Efficiency: Efficiency tests may give measurable results. An efficiency test measures the required
time and consumption of resources for the execution of tasks. Resources may include other
software products, the software and hardware ➞configuration of the system, and materials (for
example, print paper, network, and storage)

Maintainability and portability: Software systems are often used over a long period on various
platforms (operating system and hardware). Therefore, the last two quality criteria are very
important: maintainability and portability.

A software system cannot fulfill every quality characteristic equally well. Sometimes it is possible
that meeting one characteristic results in a conflict with another one. For example, a highly
efficient software system can become hard to port because the developers usually use special
characteristics (or features) of the chosen platform to improve efficiency. This in turn negatively
affects portability.
2.1.4 Test Effort
Complete testing is impossible: Testing cannot prove the absence of faults. In order to do this, a
test would need to execute a program in every possible situation with every possible input value
and with all possible conditions. In practice, a ➞complete or exhaustive test is not feasible. Due
to combinational effects, the outcome of this is an almost infinite number of tests. Such a “testing”
for all combinations is not possible.

Test effort between 25% and 50%: Thus, in practice it is not possible to test even a small
program exhaustively. It is only possible to consider a part of all imaginable test cases. But even
so, testing still accounts for a large portion of the development effort. However, a generalization
of the extent of the ➞test effort is difficult because it depends very much on the character of the
project.

Defects can cause high costs: Test effort is often shown as the proportion between the number of
testers and the number of developers. The proportion varies from 1 tester per 10 developers to up
to 3 testers per developer.
Faults that were not found during testing can cause high costs when the software is used. The
German newspaper Frankfurter Allgemeine Zeitung from January 17, 2002, had an article titled
“IT system breakdowns cost many millions.” A one-hour system breakdown in the stock exchange
is estimated to cost $7.8 million. When safety-critical systems fail, the lives and health of people
may be in danger. Since a full test is not possible, the testing effort must have an appropriate
relation to the attainable result. “Testing should continue as long as costs of finding and correcting
a defect are lower than the costs of failure”. Thus, the test effort is always dependent on an
estimation of the application risk.

Systems with high risks must be tested more thoroughly than systems that do not generate big
losses if they fail.

Define test intensity and test extent depending on risk: Thus, for every software program it
must be decided how intensively and thoroughly it shall be tested. This decision must be made
based upon the expected risk of failure of the program. Since a complete test is not possible, it is
important how the limited test resources are used. To get a satisfying result, the tests must be
designed and executed in a structured and systematic way. Only then is it possible to find many
failures with appropriate effort and avoid ➞unnecessary tests that would not give more
information about system quality.

Select adequate test techniques: There exist many different methods and techniques for testing
software. Every technique especially focuses on and checks particular aspects of the test object.
Thus, the focus of examination for the control-flow-based test techniques is the program flow. In
case of the ➞data flow test techniques, the examination focuses on the use and flow of data. Every
test technique has its strengths and weaknesses in finding different kinds of faults. There is no test
technique that is equally well suited for all aspects. Therefore, a combination of different test
techniques is always necessary to detect failures with different causes.

Test of extra functionality: During the test execution phase, the test object is checked to
determine if it works as required by the ➞specifications. It is also important— and thus naturally
examined while testing—that the test object does not execute functions that go beyond the
requirements. The product should provide only the required functionality.
Test case explosion: The testing effort can grow very large. Test managers face the dilemma of
possible test cases and test case variants quickly becoming hundreds or thousands of tests. This
problem is also called combinatorial explosion, or ➞test case explosion.

Limited resources: Participants in every software development project will sooner or later
experience a fight about resources. The complexity of the development task is underestimated, the
development team is delayed, the customer pushes for an earlier release, or the project leader wants
to deliver “something” as soon as possible. The test manager usually has the worst position in this
“game.” Often there is only a small time window just before delivery for executing the test cases
and very few testers are available to run the test.