Software Testing and Quality Assurance

Theory and Practice

Chapter 1
Basic Concepts and Preliminaries

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 1
 Outline of the Chapter
• Software Quality
• Role of Testing
• Verification and Validation
• Failure, Error, Fault and Defect
• The Notion of Software Reliability
• The Objectives of Testing
• What is a Test Case?
• Expected Outcome
• The Concept of Complete Testing
• The Central Issue in Testing
• Testing Activities
• Testing Level
• Source of Information for Test Selection
• White-box, Black-box Testing and Gray-Box Testing
• Test Planning and Design

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 2
 What is software testing???
Software Testing is a process where the software is
evaluated to determine that it meets the user needs
(validation) and that the process to build the software
was followed correctly (verification).

Put another way, it ensures that "you built the right

thing" (validation) and ensures that "you built it right"
(verification).

Software Testing Theory and Practice (Chapter 1: Basic Concepts and 3

© Naik & Tripathy
Preliminaries)
 Why testing is important???
• Testing is critical to make sure that the software has
appropriate quality and works correctly.
• Insufficient testing results into errors and failures of the
product, can have significant impacts on the business.

For example: when the product is unstable and breaks

easily or frequently, new potential buyers may decide to
wait for the next release of the product or look for a
different product all together.

In either case the business will suffer significant

financial losses as well as a loss in their company’s
reputation, which in turn may impact the sale of other
products.
Software Testing Theory and Practice (Chapter 1: Basic Concepts and 4
© Naik & Tripathy
Preliminaries)
 Software failure cases
Case 1: Potentially a deadly software failure in
Medicine infusion
Some software failures are non-negotiable and deadly, too, just like this Medicine infusion pump.

These medicines pump equipment, aka Alaris Pump, were manufactured by CareFusion.

due to some failure in the software, the equipment had the potential to either overdose or not deliver
the drugs to patients at all.

Case 2: A bug made the robbery easy

Once upon a time in Bangladesh (exactly in 2016, precisely in March), a group of wicked, witty hackers
stole 81 million and got away with it via a bug

the bank found this heist after they found a bug in the printer that automatically prints the transactions
and that too after several days.

Case 3: A glitch caused target detection issues in F-

35 fighter planes
We have another pivotal software failure that caused a target detection problem in F-35 fighter planes.
As you can see, these failed software systems that we are addressing are sensitive and typically need
close attention.

Software Testing Theory and Practice (Chapter 1: Basic Concepts and © Naik & Tripathy
Preliminaries)
 Case 4: Israel’s Attempt to land on Moon fails seconds before touchdown

On April 11, 2019, Israel’s first lunar lander Beresheet crash-arrived on the Moon.

Case 5 : Boeing 737 Max Crashes

The US aeroplane producer Boeing dispatched one of its most up-to-date models, 737 Max, and
its business flight smashed off the shoreline of Indonesia in October 2018.

Case 6: Self-driving Uber car kills pedestrian in Arizona

A person on foot was killed by a Uber vehicle in Arizona on March 21.

Software Testing Theory and Practice (Chapter 1: Basic Concepts and © Naik & Tripathy
Preliminaries)
 Why testing is important…?
When the product causes an injury or loss of life of a user, the
impacts are even greater because the company then may be sued
for negligence, resulting in more financial and reputation loss.
Therac 25 was a cancer treatment machine that due to a software
defect mistakenly gave some patients deadly amounts of radiation.
Mars climate orbiter crashed on the red planet also due to a
software issue.

Software Testing Theory and Practice (Chapter 1: Basic Concepts and 7

© Naik & Tripathy
Preliminaries)
 Cost of late testing..
Defects are more expensive to fix in the later stages of development and once the product
is already in the field.

IBM mainframe z/OS system defect that was found by a customer in the field was
estimated to cost at minimum 10K per defect.

Stecklein, J. et al. (2004) conducted a study for NASA type of products (e.g. spacecraft or
satellite) and found that if finding an error in requirements phase costs 1 unit then it
increases so significantly that the same defect if undetected until operations phase can cost
to fix anywhere from 29 units to more than 1,500 units.
Example of costly defects :
Ariane 5 rocket – the rocket was supposed to deliver four satellite systems to conduct
research into the Earth’s magnetosphere on June 4, 1996 but it failed to achieve orbit and
self-destructed, resulting in a financial loss of more than $370 million.
The Mars climate orbiter- cost $193 million.
Radiotherapy system Therac-25: 3 people were killed.

Software Testing Theory and Practice (Chapter 1: Basic Concepts and 8

© Naik & Tripathy
Preliminaries)
 software developement life cycle(SDLC)

Software Testing Theory and Practice (Chapter 1: Basic Concepts and 9

© Naik & Tripathy
Preliminaries)
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 • Benefits of Waterfall model

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 Agile and waterfall model

Software Testing Theory and Practice (Chapter 1: Basic Concepts and 16

© Naik & Tripathy
Preliminaries)
 Agile Development Models

Scrum is one of the most popular software testing methodologies, but it’s
actually a framework for managing work with an emphasis on software
development.

Key quality of Scrum is the introduction of the roles

• Scrum Master and Product Owner who organise and manage the rest of the
Scrum team including software developers, designers, QA, testing and
operations.
• Sprints, which are ‘time-boxed’ periods of often around two weeks during
which a pre-determined amount of development work is completed, are
integral to the Scrum framework.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 Scrum tool

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 Types of Design

High-Level Design (HLD)

High-Level Design (HLD) is a general system design. It includes the description
of the following parts:
• System architecture
• Database design
• Brief mention of all the platforms, systems, services, and processes the
product would depend on
• Brief description of relationships between the modules and system features

Low-Level Design (HLD)

Low-Level Design (LLD) is a component-level design process that follows a
step-by-step refinement process. It provides the details and definitions for the
actual logic for every system component. It is based on HLD but digs deeper,
going into the separate modules and features for every program in order to
document their specifications.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 Software Quality

What is Software quality?

• Quality is a complex concept—it means different things to different
people, and it is highly context dependent.
• Defined as a field of study and practice that describes the desirable
attributes of software products.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 Software Quality
• Five Views of Software Quality:
1. Transcendental view-

– quality as something that can be recognized but is difficult to define.

– The transcendental view is not specific to software quality alone but has been
applied in other complex areas.

Example:
I wasn’t quite certain what color I wanted (although I knew what I didn’t
want – no shades of white, cream).
But I was confident that I’d know it when I see it.

2. User’s view- It perceives quality as fitness for purpose.

Example: “Does the product satisfy user needs and expectations?”

The vehicle had to be fit for use…my use. If not, will I truly be satisfied?

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 22
 Software Quality
3. Manufacturing view-
• Here quality is understood as conformance to the specification.
• The quality level of a product is determined by the extent to which the product
meets its specifications.

4. Product view-

– quality is viewed as tied to the inherent characteristics of the product.

– It assumes that measuring and controlling internal product properties

(internal quality indicators) will result in improved external product behavior
(quality in use).

5. Value-based view-Quality, in this perspective, depends on the amount a

customer is willing to pay for it.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 23
Scenario: I leased a vehicle I found myself entrenched in the “five perspectives” as I navigated
through my decision-making process.

First, transcendent based:

I wasn’t quite certain what color I wanted (although I knew what I didn’t want – no shades of
white, cream).
But I was confident that I’d know it when I see it. THAT is “transcendent based”.

Then we have product based:

I knew, there are specific features the vehicle had to have enough space in the trunk

Third on our list – user based:

The vehicle had to be fit for use…my use. If not, will I truly be satisfied? Whether the seats are
comfortable or cheap seats? Can I drive it safely?

The fourth – manufacturer based:

Obviously, the car manufacturer has certain specifications and design requirements of their own.
And they may nail it. They may get it 100% right. Put a check in “meets all requirements” But to
me, that still may not be a “quality” vehicle.

And last on the list – value based:

I had a budget. I had to make sure the vehicle I chose met my definition of quality within certain
guidelines of what I could afford.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 Software Quality
The concept of software quality is first studied in terms of quality
factors and quality criteria.
A quality factor represents a behavioural characteristic of a
system.
Some examples of high-level quality factors are
• correctness,
• reliability ,
• efficiency,
• testability ,
• maintainability, and
• reusability.
A quality criterion is an attribute of a quality factor that is related
to software development.
For example:
Modularity is an attribute of the architecture of a software system.
A highly modular software allows designers to put cohesive
components in one module, thereby improving the maintainability of
the system.
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 Software Quality

Software quality models have been proposed to define quality and its
related attributes.
• Quality Models
– ISO(International Organization for Standardization) 9126 & CMM
• The document ISO 9126 defines six broad, independent categories of
quality characteristics:
1. functionality,
2. reliability,
3. usability,
4. efficiency,
5. maintainability, and
6. portability.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 Software Quality

The CMM framework, a development process is evaluated on a scale

of level 1 through level 5.
For example,
Level 1 is called the initial level, whereas
level 5—optimized—is the highest level of process maturity.

There are two well-known process models, namely,

• the test process improvement (TPI) model and
• the test maturity Model (TMM)

These two models allow an organization to assess the current state of

their software testing processes,
• identify the next logical area for improvement, and
• recommend an action plan for test process improvement.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 Role of Testing
• Testing plays an important role in achieving and assessing the
quality of a software product.
• we improve the quality of the products as we repeat a test–find
defects–fix cycle during development.
• we assess how good our system is, when we perform system-level
tests before releasing a product.
• Software testing is a verification process for software quality
assessment and improvement.
• The activities for software quality assessment can be divided into
two broad categories, namely,
– static analysis and
– dynamic analysis.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 28
 Role of Testing
– Static analysis
• Term “static” suggests , it is based on the examination of a number of
documents, namely
– requirements documents,
– software models,
– design documents, and
– source code.
• Traditional static analysis includes
– code review,
– inspection,
– walk-through,
– algorithm analysis, and
– proof of correctness.
• It does not involve actual execution of the code under development.
• Instead, it examines code over, all possible behaviours that might arise
during run time.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 29
What are the Advantages of Static Analysis

• It can find weaknesses in the code at the exact location.

• It can be conducted by trained software assurance developers who fully

understand the code.

• Source code can be easily understood by other or future developers.

• It allows a quicker turn around for fixes.

• Weaknesses are found earlier in the development life cycle, reducing the cost
to fix. Less defects in later tests

• Unique defects are detected that cannot or hardly be detected using dynamic
tests

Unreachable code
Variable use (undeclared, unused)
Uncalled functions
Boundary value violations
 Role of Testing
– Dynamic analysis

• Actual program execution to expose possible program failure.

• The behavioural and performance properties of the program are also observed.

• Programs are executed with both typical and carefully chosen input values

• If the input set of a program can be impractically large - a finite subset of the input
set can be selected.

• One observe some representative program behavior and reach conclusion

about the quality of the system.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 31
 DYNAMIC ANALYSIS

Dynamic code analysis advantages

• It identifies vulnerabilities in a runtime environment.
• It allows for analysis of applications in which you do not have access to the actual code.
• It identifies vulnerabilities that might have been false negatives in the static code analysis.
• It permits you to validate static code analysis findings.
• It can be conducted against any application.

Dynamic code analysis limitations

• Automated tools provide a false sense of security that everything is being addressed.
• Cannot guarantee the full test coverage of the source code
• Automated tools produce false positives and false negatives.
• It is more difficult to trace the vulnerability back to the exact location in the code, taking
longer to fix the problem

• Static and Dynamic analyses helps to identify as many faults as possible so that those faults are fixed
at an early stage of the software development.
• Static and Dynamic Analysis are complementary in nature, both must be performed repeatedly and
alternated.
 Verification and Validation
• Verification
– Evaluation of software system that help in determining whether the product of
a given development phase satisfy the requirements established before the start
of that phase
• Building the product correctly

• Validation
– Evaluation of software system that help in determining whether the product
meets its intended use
• Building the correct product

VERIFICATION : Are we building the product right?

VALIDATION : Are we building the right product.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 33
Main differences between verification and validation
 Failure, Error, Fault and Defect

• A mistake in coding is called Error,

• Error found by tester is called Defect,
• Defect accepted by development team then it is called
Bug,
• Build does not meet the requirements then it Is Failure.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 37
What is Error?

The Problem in code leads to errors, which means that a mistake can occur
• due to the developer's coding error as the developer misunderstood the requirement or
the requirement was not defined correctly.
• Developers not following coding standards – mistakes in syntax, mistake in some
connectivity i.e. database connectivity, mistake in invocation of a variable.
• The developers use the term error.
What is Fault?

Any incorrect step, data definition in a computer program which causes the
program to behave in an unintended manner is considered as fault.
It is result of error.

The fault may occur in software because it has not added the code for fault
tolerance, making an application act up .A fault may happen in a program
because of the following reasons:
i) Lack of resources
ii) An invalid step
iii) Inappropriate data definition

Bug is a fault in program which causes the program to behave in an

unintended or unanticipated manner in early stages.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
What is a failure?

Once the product is deployed and customers find any issues then they call
the product as a failure product. After release, if an end user finds an issue
then that particular issue is called as failure

What is a Defect?

The variation between the actual results and expected results is known as defect.
If a developer finds an issue and corrects it by himself in the development phase then
it’s called a defect.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 Failure, Error, Fault and Defect
• Failure
– A failure is said to occur whenever the external behavior of a system does not
conform to that prescribed in the system specification
• Error
– An error is a state of the system.
– An error state could lead to a failure in the absence of any corrective action by
the system
• Fault
– A fault is the adjudged cause of an error
• Defect
– It is synonymous of fault
– It a.k.a. bug

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 42
• This Defect report or Bug report consists of the following information:
• Defect ID – Every bug or defect has it’s unique identification number
• Defect Description – This includes the abstract of the issue.
• Product Version – This includes the product version of the application in
which the defect is found.
• Detail Steps – This includes the detailed steps of the issue with the
screenshots attached so that developers can recreate it.
• Date Raised – This includes the Date when the bug is reported
• Reported By – This includes the details of the tester who reported the bug
like Name and ID
• Status – This field includes the Status of the defect like New, Assigned,
Open, Retest, Verification, Closed, Failed, Deferred, etc.
• Fixed by – This field includes the details of the developer who fixed it like
Name and ID
• Date Closed – This includes the Date when the bug is closed
• Severity – Based on the severity (Critical, Major or Minor) it tells us about
impact of the defect or bug in the software application
• Priority – Based on the Priority set (High/Medium/Low) the order of fixing the
defect can be made. (Know more about Severity and Priority)
 NOTION OF SOFTWARE RELIABILITY

• It is defined as the probability of failure-free operation of a

software system for a specified time in a specified
environment.
• It can be estimated via random testing suggested by Hamlet.
• Its a quantitative measure that is useful in assessing the
quality of a software is its reliability.
• Test data must be drawn from the input distribution to
closely resemble the future usage of the system.
• Future usage pattern of a system is described in a form called
operational profile.
• The concept of operational profile of a system was pioneered
by John D. Musa at AT&T Bell Laboratories between the
1970s and the 1990s.
• A software product having a large number of defects is
unreliable.
https://www.youtube.com/watch?v=ZVHDRj42MZI
 The Objectives of Testing
• A stakeholder is a person or an organization who
influences a system’s behaviours or who is impacted by
that system.
• Different stakeholders view a test process from different
perspectives as explained below:
1. It does work:
– While implementing a program unit, the programmer may want
to test whether or not the unit works in normal circumstances.
– The programmer gets much confidence if the unit works to his or
her satisfaction.
– The same idea applies to an entire system as well—once a
system has been integrated, the developers may want to test
whether or not the system performs the basic functions.
– Here, the objective of testing is to show that the system works.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 49
 The Objectives of Testing
2. It does not work :
– Once the programmer (or the development team) is satisfied that a
unit (or the system) works to a certain degree, more tests are
conducted with the objective of finding faults in the unit (or the
system).
– Here, the idea is to try to make the unit (or the system) fail.

3. Reduce the risk of failures:

– Most of the complex software systems contain faults, which cause the
system to fail from time to time.
– This concept of “failing from time to time” gives rise to the notion of
failure rate.
– As faults are discovered and fixed while performing more and more
tests, the failure rate of a system generally decreases.
– Thus, a higher level objective of performing tests is to bring down the
risk of failing to an acceptable level.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 50
 The Objectives of Testing
4. Reduce the cost of testing:
– The different kinds of costs associated with a test process includes:
• the cost of designing, maintaining and executing test cases,
• the cost of analyzing the result of executed each test case,
• the cost of documenting the test cases, and
• the cost of actually executing the system and documenting it.
– Therefore, the less the number of test cases designed, the less will
be the associated cost of testing.
– However, producing a small number of arbitrary test cases is not a
good way of saving cost.
– The highest level of objective of performing tests is to produce low-
risk software with fewer number of test cases.
– This idea leads us to the concept of effectiveness of test cases.
– Test engineers must therefore judiciously select fewer, effective test
cases.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 51
 Test Scenario

• Test Scenario gives the idea of what we have to test.

• Test Scenario is like a high-level test case where end to end functionality of
application under test is included.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 What is a Test Case?

• A Test Case is a set of actions executed to verify a

particular feature or functionality of your software
application.
• A Test Case contains,
– test steps,
– test data,
– precondition,
– postcondition developed for specific test scenario to verify any
requirement.
The test case includes specific variables or conditions, using
which a testing engineer can compare expected and actual results
to determine whether a software product is functioning as per the
requirements of the customer.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 What is a Test Case?
• Test Case is a simple pair of
<input, expected outcome>
• State-less systems: A compiler is a stateless system
– Test cases are very simple
• Outcome depends solely on the current input
• State-oriented: ATM is a state oriented system
– Test cases are not that simple. A test case may consist of a sequences of
<input, expected outcome>
• The outcome depends both on the current state of the system and the
current input
• ATM example:
⮚ < check balance, $500.00 >,
⮚ < withdraw, “amount?” >,
⮚ < $200.00, “$200.00” >,
⮚ < check balance, $300.00 >

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 54
 How To Write Test Cases in Manual Testing

Step #1 – Test Case ID: Each test case should be represented by a unique ID.
It’s good practice to follow some naming convention for better understanding and discrimination
purposes.

Step #2 – Test Case Description:

Pick test cases properly from the test scenarios

Example:
Test scenario: Verify the login of Gmail
Test case: Enter a valid username and valid password

Step #3 – Pre-Conditions:
Conditions that need to meet before executing the test case. Mention if any preconditions are
available.
Example: Need a valid Gmail account to do login

Step #4 – Test Steps:

To execute test cases, you need to perform some actions. So write proper
test steps. Mention all the test steps in detail and in the order how it could
be executed from the end-user’s perspective.
Example:
Enter Username
Enter Password
Click Login button
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 How To Write Test Cases in Manual Testing
Step #5 – Test Data:
You need proper test data to execute the test steps. So gather appropriate test data.
The data which could be used an input for the test cases.
Username: [email protected]
Password: STM
Step #6 – Expected Result:
The result which we expect once the test cases were executed. It might be
anything such as Home Page, Relevant screen, Error message, etc.,
Example: Successful login

Step #7 – Post Condition:

Conditions that need to achieve when the test case was successfully executed.
Example: Gmail inbox is shown

Step #8 – Actual Result:

The result which system shows once the test case was executed. Capture the result after the execution.
Based on this result and the expected result, we set the status of the test case.
Example: Redirected to Gmail inbox

Step #9 – Status:
Finally set the status as Pass or Fail based on the expected result against the actual result. If the
actual and expected results are the same, mention it as Passed. Else make it as Failed. If a test
fails, it has to go through the bug life cycle to be fixed.
Example:
Result: Pass

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
WRITE THE TEST CASES FOR THE BELOW
FUNCTIONALITY OF SPOTIFY APP
1.Login 2. Forgot password 3.Search songs
4. Download song 5. Adding songs to playlist
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
Functional Requirements
These are the requirements that the end user specifically demands as basic
facilities that the system should offer.
Non-Functional Requirements
These are the quality constraints that the system must satisfy according to the
project contract.
Like : Portability,Security ,Maintainability, Reliability, Scalability,
Performance,Reusability,Flexibility
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy
 Expected Outcome
• An outcome of program execution may include
– Value produced by the program
– State Change:
– State change of the program
– State change of the database (due to add, delete, and update
operations)
– A sequence of values which must be interpreted together for the outcome
to be valid
• An important concept in test design is the concept of an oracle,
that tells us the expected outcome of a particular test or set of
tests.
• Ideally, the expected outcome of a test should be computed
while designing the test case. The test outcome is computed
before the program is executed with the selected test input.
• The idea here is that one should be able to compute the
expected outcome from an understanding of the program’s
requirements.
67
 Expected Outcome
• A test oracle is a mechanism that verifies the correctness of
program outputs
– Generate expected results for the test inputs
– Compare the expected results with the actual results of execution

• In exceptional cases, where it is extremely difficult, impossible,

or even undesirable to compute a single expected outcome, one
should identify expected outcomes by examining the actual test
outcomes, as explained in the following:
1. Execute the program with the selected input.
2. Observe the actual outcome of program execution.
3. Verify that the actual outcome is the expected outcome.
4. Use the verified actual outcome as the expected outcome in subsequent
runs of the test case.

68
 The Concept of Complete Testing
• Complete or exhaustive testing means
“There are no undisclosed faults at the end of test phase”

• Complete testing is near impossible for most of the system

– The domain of possible inputs of a program is too large
• Valid inputs
• Invalid inputs
– There may be timing constraints on the inputs, that is, an input may be
valid at a certain time and invalid at other times.
– The design issues may be too complex to completely test
• For example, a programmer may use a global variable or a static variable to
control program execution.
– It may not be possible to create all possible execution environments of the
system. This becomes more significant when the behaviour of the software
system depends on the real, outside world, such as weather, temperature,
altitude, pressure, and so on.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 69
 The Central Issue in Testing

Figure 1.5: A subset of the input domain exercising a subset of the program behavior

• Divide the input domain D into D1 and D2

• Select a subset D1 of D to test program P
• It is possible that D1 exercise only a part P1 of P

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 70
 The Central Issue in Testing
• The next is to select a subset of the input domain to test a
program.
• Let D be the input domain of a program P.
– We select a subset D1 of D, that is, D1 ⊂D, to test program P.
– It is possible that D1 exercises only a part P1, that is, P1 ⊂P,
of the execution behaviour of P, in which case faults with the
other part, P2, will go undetected.
– By selecting a subset of the input domain D1, the test
engineer attempts to deduce properties of an entire program P
by observing the behaviour of a part P1 of the entire
behaviour of P on selected inputs D1.
– Therefore, selection of the subset of the input domain must be
done in a systematic and careful manner so that the deduction
is as accurate and complete as possible.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 71
 Testing Activities

Figure 1.6: Different activities in process testing

1) Identify the objective to be tested

2) Select inputs
3) Compute the expected outcome
4) Set up the execution environment of the program
5) Execute the program
6) Analyze the test result

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 72
 Testing Activities

1) Identify the objective to be tested -The objective defines the intention, or

purpose, of designing one or more test cases to ensure that the program
supports the objective.
2) Select inputs -Selection of test inputs can be based on the requirements
specification, the source code, or our expectations. Test inputs are selected by
keeping the test objective in mind.
3) Compute the expected outcome: This can be done from an overall, high-
level understanding of the test objective and the specification of the program
under test.
4) Set up the execution environment of the program: all the assumptions
external to the program must be satisfied. A few examples of assumptions
external to a program are as follows:
• Initialize the local system, external to the program. This may include
making a network connection available, making the right database system
available, and so on.
• Initialize any remote, external system (e.g., remote partner process in a
distributed application.) For example, to test the client code, we may need
to start the server at a remote site.
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 73
 Testing Activities
5) Execute the program: The test engineer executes the program with the
selected inputs and observes the actual outcome of the program.
6) Analyze the test results: compare the actual outcome of program
execution with the expected outcome.
• At the end of the analysis step, a test verdict is assigned to the program.
• There are three major kinds of test verdicts, namely, pass, fail, and
inconclusive:
• Pass: If the program produces the expected outcome and the purpose of
the test case is satisfied, then a pass verdict is assigned.
• Fail: If the program does not produce the expected outcome, then a fail
verdict is assigned.
• inconclusive : In some cases it may not be possible to assign a clear pass
or fail verdict.
• For example, if a timeout occurs while executing a test case on a
distributed application, we may not be in a position to assign a clear pass
or fail verdict. In those cases, an inconclusive test verdict is assigned. An
inconclusive test verdict means that further tests are needed to be done to
refine the inconclusive verdict into a clear pass or fail verdict.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 74
 Testing Activities
• A test report must be written after analyzing the test result.
• The motivation for writing a test report is to get the fault fixed if the
test revealed a fault.
• A test report contains the following items to be informative:
– Explain how to reproduce the failure.
– Analyze the failure to be able to describe it.
– A pointer to the actual outcome and the test case, complete with the input, the
expected outcome, and the execution environment.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 75
 Testing Level
• Testing is performed at different
levels involving the complete
system.
• A software system goes through
four stages of testing before it is
actually deployed. These four
stages are known as unit,
integration, system, and
acceptance level testing.
• The first three levels of testing are
performed by a number of different
stakeholders in the development
organization, where as Figure 1.7: Development and testing phases
acceptance testing is performed in the V model
by the customers.
• The four stages of testing have
been illustrated in the form of what
is called the classical V model in
Figure 1.7.
Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 76
 Testing Level
• Unit testing
– Individual program units, such as procedure, methods in isolation
• Integration testing
– Modules are assembled to construct larger subsystem and tested
• System testing
– Includes wide spectrum of testing such as functionality, testing, security testing,
robustness testing and load testing, stability testing, stress testing, performance
testing, and reliability testing.
– System testing comprises a number of distinct activities: creating a test plan,
designing a test suite, preparing test environments, executing the tests by
following a clear strategy, and monitoring the process of test execution.
• Acceptance testing
– The objective of acceptance testing is to measure the quality of the product.
– A key notion: is Customer’s expectations from the system
– Two types of acceptance testing
• UAT
• BAT
– UAT(user acceptance testing): Its conducted by customer and verifying the user's
requirements have been met.
– BAT(business acceptance testing): assess whether the product meets the business
goals set out in the design.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 77
 Testing Level
• Regression testing is another level
of testing that is performed
throughout the life cycle of a
system.
• It is performed whenever a
component of the system is
modified.
• The key idea is to ascertain that the
modification has not introduced
any new faults.
• Regression testing is considered as Figure 1.8: Regression testing at
a sub-phase of unit, integration, different software testing levels
and system-level testing.

• New test cases are not designed

• Test are selected, prioritized and executed
• To ensure that nothing is broken in the new version of the
software

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 78
 Source of Information for Test Selection
In order to generate effective tests at a lower cost, test designers
analyze the following sources of information:

1. Requirement and Functional Specifications

2. Source Code
3. Input and output Domain
4. Operational Profile
5. Fault Model
– Error Guessing
– Fault Seeding
– Mutation Analysis

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 79
 Source of Information for Test Selection
• Requirement and Functional Specifications:
– The process begins by capturing user needs. The nature and amount
of user needs identified at the beginning of system development will
vary depending on the specific life-cycle model to be followed.
– Waterfall model - a requirements engineer tries to capture most of
the requirements.
– Agile software development model- such as XP or the Scrum only a
few requirements are identified in the beginning.
A test engineer considers all the requirements the program is expected
to meet whichever life-cycle model is chosen to test a program
• Source Code : describes the actual behaviour of the system
• Input and output Domain: Some values in the input domain of
a program have special meanings, and hence must be treated
separately. For eg: the factorial of a nonnegative integer.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 80
 Source of Information for Test Selection
• Operational Profile: is a quantitative characterization of how a system
will be used. It was created to guide test engineers in selecting test cases
(inputs) using samples of system usage.
• Fault Model:
• Previously encountered faults are an excellent source of information in
designing new test cases.
• The known faults are classified into different classes, such as initialization
faults, logic faults, and interface faults, and stored in a repository.
• There are three types of fault-based testing:
1. Error Guessing: A test engineer applies his experience to
(i) assess the situation and guess where and what kinds of faults
might exist, and
(ii) design tests to specifically expose those kinds of faults.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 81
 Source of Information for Test Selection
2. Fault Seeding:
– known faults are injected into a program, and
– the test suite is executed to assess the effectiveness of the test suite.
– fault seeding is based on the idea of fault injection – a fault is inserted
into a program, and an oracle is available to assert that the inserted
fault indeed made the program incorrect.

3. Mutation Analysis:
– mutate the program statements in order to determine the fault
detection.
– If the test cases are not capable of revealing such faults, the test
engineer may specify additional test-cases to reveal the faults.
– It is based on the idea of fault simulation – one may modify an
incorrect program and turn it into a correct program

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 82
Software Testing Theory and Practice (Chapter 1: Basic Concepts and © Naik & Tripathy
Preliminaries)
 We execute test cases on various Mutated versions of source code. Mutated versions are
nothing but fault-injected versions of the original Source code. We can call those fault-
injected versions as mutants.

Once we run all the tests we can analyze our test results which is what is basically called
as Mutation analysis or Mutation testing. A mutant for which our tests have not failed
as “Survived Mutant”. A mutant for which all tests have failed is called as “Killed
Mutant”.

Software Testing Theory and Practice (Chapter 1: Basic Concepts and © Naik & Tripathy
Preliminaries)
White-box testing :
• The white box testing is done by Developer,
where they check every line of a code before
giving it to the Test Engineer.
• Since the code is visible for the Developer
during the testing, that's why it is also known
as White box testing.

Black box testing :

• The black box testing is done by the Test
Engineer, where they can check the
functionality of an application or the software
according to the customer /client's needs.
• In this, the code is not visible while performing
the testing; that's why it is known as black-box
testing.
Gray Box testing
• Gray box testing is a combination of white box
and Black box testing. It can be performed by
a person who knew both coding and testing.
• And if the single person performs white box, as
well as black-box testing for the application, is
known as Gray box testing.
 White-box and Black-box Testing
• White-box testing a.k.a. structural • Black-box testing a.k.a. functional
testing testing
• Examines source code with focus on: • Examines the program that is
– Control flow accessible from outside
– Data flow • Applies the input to a program and
• Control flow refers to flow of control observe the externally visible outcome
from one instruction to another • It is applied to both an entire program
• Data flow refers to propagation of as well as to individual program units
values from one variable or constant • It is performed at the external
to another variable interface level of a system
• It is applied to individual units of a • It is conducted by a separate software
program quality assurance group
• Software developers perform
structural testing on the individual
program units they write.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 87
Software Testing Theory and Practice (Chapter 1: Basic Concepts and © Naik & Tripathy
Preliminaries)
 Test Planning and Design
• The purpose is to get ready and organized for test execution
• A test plan provides a:
– Framework
• A set of ideas, facts or circumstances within which the tests will be
conducted
– Scope
• Outline the domain or extent of the test activities
– Details of resource needed
– Effort required
– Schedule of activities
– Budget

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 89
Software Testing Theory and Practice (Chapter 1: Basic Concepts and © Naik & Tripathy
Preliminaries)
 Test Planning and Design
• Test design is a critical phase of software testing.
• In this phase:
– the system requirements are critically studied,
– system features to be tested are thoroughly identified, and
– the objectives of test cases and
– the detailed behaviour of test cases are defined.
• Test objectives are identified from different sources namely, the
requirement specification and the functional specification.
• Each test case is designed as a combination of modular test
components called test steps.
• Test steps are combined together to create more complex tests.

Software Testing and QA Theory and Practice (Chapter 1: Basic Concepts and Preliminaries) © Naik & Tripathy 91
Software Testing Theory and Practice (Chapter 1: Basic Concepts and © Naik & Tripathy
Preliminaries)
 Continued,…
New test centric approach to system development is gradually
emerging (called as Test driven Development-TDD)
Here programmers design, develop & implement test case before
production code is written.
This approach is key practice in modern agile s/w development
process.
Main characteristics of agile s/w development process are:
i) Incremental development
ii) Coding of unit & acceptance testing along with customers
iii) Frequent regression testing
iv) Writing test code 1 test case at a time, before production code.

Software Testing Theory and Practice (Chapter 1: Basic Concepts and 93

© Naik & Tripathy
Preliminaries)
Software Testing Theory and Practice (Chapter 1: Basic Concepts and © Naik & Tripathy
Preliminaries)
• Identify the different test scenarios for Swiggy
app & write the functional test scenarios and
test cases for placing an order through Swiggy
app using standard test case template.
https://www.qable.io/blog/testing-online-food-delivery-app