Chapter Three

Testing and V&V

What is validation and verification(V & V)?

• Verification refers to the set of tasks that ensure that software

correctly implements a specific function.

• Validation refers to a different set of tasks that ensure that the

software that has been built is traceable to customer requirements.

Verification: “Are we building the product right?”

Validation: “Are we building the right product?”
• In verification one may note that a product can be an intermediate product, such as
• requirement specification,
• design specification,
• code,
• user manual,
• or even the final product.

• Activities that check the correctness of a development phase are called verification
activities
• Validation activities focus on the final product, which is
extensively tested from the customer point of view.

• Validation establishes whether the product meets overall

expectations of the users
• The verification process establishes the correspondence of an

implementation phase of the software development process with its

specification, whereas validation establishes the correspondence

between a system and users’ expectations.

 Summery of V&V
Validation Verification
Aim at confirming that one is building the correct Aims at confirming that one is building the
product product correctly

validation is performed toward the end of system Verification activities review interim work
development to determine if the entire system products, such as requirements specification,
meets the customer’s needs and expectations. design, code, and user manual, during a project
life cycle to ensure their quality. The quality
attributes sought by verification activities
are consistency, completeness, and correctness at
each major stage of system development.
Validation is performed on the entire system by Verification activities are performed on interim
actually running the system in its real products by applying mostly static analysis
environment and using a variety of tests. techniques, such as inspection, walkthrough, and
reviews, and using standards and checklists.
Verification can also include dynamic analysis,
such as actual program execution.
Verification activities
• Walkthrough
• Walkthrough is a method of conducting informal group/individual review. In a
walkthrough, author describes and explain work product in a informal meeting to his
peers or supervisor to get feedback.
• Here, validity of the proposed solution for work product is checked.

• Inspection
• An inspection is defined as formal, rigorous, in depth group review designed to
identify problems as close to their point of origin as possible.
• Inspections improve reliability, availability, and maintainability of software product.
• Anything readable that is produced during the software development can be
inspected.
• Inspection activity follows a specified process and participants play
well-defined roles.

• An inspection team consists of three to eight members who plays

roles of moderator, author, reader, recorder and inspector.
• Author: The person who created the work product being inspected.

• Moderator: This is the leader of the inspection. The moderator plans

the inspection and coordinates it.

• Reader: The person reading through the documents, one item at a

time. The other inspectors then point out defects.

• Recorder/Scribe: The person that documents the defects that are

found during the inspection.

• Inspector: The person that examines the work product to identify

possible defects.
• Stages in the inspections process :
• Planning : Inspection is planned by moderator.
• Overview meeting : Author describes background of work product.
• Preparation : Each inspector examines work product to identify possible
defects.
• Inspection meeting : During this meeting, reader reads through work
product, part by part and inspectors points out the defects for every part.
• Rework : Author makes changes to work product according to action plans
from the inspection meeting.
• Follow-up : Changes made by author are checked to make sure that
everything is correct.
Preparation phase
Meeting phase
Inspection Checks
• Requirements Inspection Checklist
• Do requirements exhibit a clear distinction between functions and data?
• Do requirements define all the information to be displayed to users?
• Do requirements address system and user response to error conditions?
• Is each requirement stated clearly, concisely, and unambiguously?
• Is each requirement testable?
• Are there ambiguous or implied requirements?
• Are there conflicting requirements?
• Are there areas not addressed in the Software Requirements Specification (SRS) that need to
be?
• Are performance requirements (such as response time, data storage requirements) stated?
• Sample Design Inspection Checklist Questions (Error Handling and
Recovery)
• Is there adequate error condition testing?
• Are error conditions tested where the probability of an error is high or results of an
error would be fatal to the system?
• Are return codes documented?
• Are return messages understandable?
• Does the program allow for successful error recovery...
• across module or process failures?
• across operating system failure?
• across interrupts?
• across hardware failures?
• Many different metrics can be calculated during an inspection process

including the following:

• The number of major and minor defects found

• The number of major defects found to total found.

• (If the proportion of minor defects to major defects is too large a moderator may request

that the reviewer repeat the review, focusing on major defects, prior to commencing the

logging meeting)
• The size of the artifact (pages, LOC, ...)

• The rate of review - the size of the reviewed artifact divided by time
(normally expressed in hours)
• E.g., 15 pages /hour

• The defect detection rate - the number of major defects found per review
hour.
• Total Number of Defects Found
• Total number of defects found is the sum of the total
number of defect found by each reviewer, minus the
number of common defects found.
• For instance, with 2 reviewers, the metric is computed by
• Total Defects Found = A + B – C
• Where A and B are the number found by reviewer A and B
respectively and C is the number found by both A and B.
• Defect density
• It is the ratio of the number of defects found to the size of
the artifact. It is given by
• Defect Density = Total Defects Found / Size
• Where the size of the artifact is measured in number of pages, loc, or
other size measure.
• Estimated Total Number of Defects
• The estimated total number of defects is the sum of the total
number of defects found and the estimated total number of
defects remaining.
• Using the population sampling approach to estimate the number
of defects remaining leads to the following steps:
• Let the number of defects found by one reviewer be the tagged population (A)

• Assume an even likelihood of finding all defects (even distribution,...)

• Count the number of defects found by the second reviewer (B).

• Count the number of defects found by the second reviewer that were also found by the
first (C the common defects).

• Calculate the proportion of common defects in the second reviewers defects (T=C/B).

• We assume that T is representative of the proportion of common defects in the total

number of defects (EstTot), so

• T∝EstTot=A, or for our purposes

• EstTot = A/T = (A∝B)/C.

How would you classify the following activity?

• Testing handling of a new car in snow condition in Alaska.

• Frontal crash test in the lab
• Testing a new toy in a kindergarten
• Vehicle emissions testing on a dynamo.
Review

• In a review (also called a technical review), a work product is

examined for defects by individuals other than the person who
produced it.
• A work product is any important deliverable created during the requirements,
design, coding, or testing phase of software development
Goal of V&V

• The ultimate goal of the verification and validation process is to

establish confidence that the software system is ‘fit for purpose’.

• All of the following factors must be considered when deciding how

much effort should be spent on the V & V process.
• Software function
• The level of confidence depends on how critical the software is to an organization.

• User expectations
• Users may have low expectations of certain kinds of software.

• Marketing environment
• Getting a product to market early may be more important than finding defects in the
program.
Testing and debugging

• Defect testing and debugging are distinct processes.

• Verification and validation is concerned with establishing the

existence of defects in a program.

• Debugging is concerned with locating and repairing these errors.

• Debugging involves formulating a hypothesis about program behavior

then testing these hypotheses to find the system error.
Planning verification and validation

• Verification and validation is an expensive process.

• Careful planning is needed to get the most out of inspections and

testing and to control the costs of the verification and validation
process.
• You should start planning system validation and verification early in
the development process.
• The V model helps achieve this goal
The structure of a software test plan

• The testing process

• A description of the major phases of the testing process (V&V)

• Requirements traceability
• Users are most interested in the system meeting its requirements and testing should
be planned so that all requirements are individually tested.

• Tested items
• The products of the software process that are to be tested should be specified.
• Testing schedule
• An overall testing schedule and resource allocation for this schedule is, obviously,
linked to the more general project development schedule.

• Test recording procedures

• It is not enough simply to run tests; the results of the tests must be systematically
recorded.
• It must be possible to audit the testing process to check that it has been carried out
correctly.
• Hardware and software requirements
• This section should set out the software tools required and estimated
hardware utilization.

• Constraints
• Constraints affecting the testing process such as staff shortages should be
anticipated in this section