MAKE OR BUY

the software engineering organization can

(1) build system X from scratch
(2) reuse existing “partial-experience” components to construct the
System
(3) buy an available software product and modify it to meet local needs, or
(4) contract the software development to an outside vendor.

1
MAKE OR BUY expected cost = (path probability)i x (estimated path cost)I
where i is the decision
tree path.
expected costbuild = 0.30 ($380K) + 0.70 ($450K) = $429K

2
Software Quality Assurance

3
 Project Planning
- Quality Concepts
- Quality
- Quality Control
- Quality Assurance
- Cost of Quality
- Software Quality Assurance
- Software Reviews
- Formal Technical Reviews
- The Review Meeting
- Review Reporting and Record Keeping
- Review Guidelines
- Formal Approaches to SQA
- Statistical Quality Assurance
- Software Reliability
- The SQA Plan

4
 Quality Concepts

Software quality assurance is an umbrella activity that is

applied throughout the software process.

SQA encompasses:
i. a quality management approach
ii. effective software engineering technology
iii. formal technical reviews
iv. a multi-tiered testing strategy
v. document change control
vi. software development standard and its control procedure
vii. measurement and reporting mechanism

5
 Quality Concepts
Quality
- refers to measurable characteristics of a software.
These items can be compared based on a given standard

Two types of quality control:

- Quality design
-> the characteristics that designers specify for an item.
-> includes: requirements, specifications, and the design of the
system.
- Quality of conformance
-> the degree to which the design specification are followed.
-> It focuses on implementation based on the design.

6
 Quality Control
What is quality control
-- the series of inspections, reviews, and test used throughout the
develop cycle of a software product

Quality control includes a feedback loop to the process.

Objective
---> minimize the produced defects, increase the product quality

Implementation approaches:
- Fully automated
- Entirely manual
- Combination of automated tools and human interactions

7
 Quality Control

Key concept of quality control:

--> compare the work products with the specified and
measurable standards

Quality assurance consists of:

- the auditing and reporting function of management

Goal
--> provide management with the necessary data about
product quality.
--> gain the insight and confidence of product quality

8
 Quality Concepts
How do I ensure that
I’ve done the right thing or have I done it right?
Variation control is the heart of quality control.

Minimize the difference between the predicted resources

needed to complete a project and the actual resources used,
including staff, equipment, and calendar time.

Minimize the number of defects that are released to the

field, we’d like to ensure that the variance in the number of
bugs is al so minimized from one release to another.

User satisfaction = compliant product + good quality + delivery within

budget and schedule

9
 Cost of Quality
Cost of quality --> includes all costs incurred in the pursuit of
quality or perform quality related work

Quality cost includes:

- Prevention cost: - Appraisal cost:
- quality planning
- formal technical reviews - in-process and inter-process inspection
- testing equipment - equipment calibration and maintenance
- training - testing

- Failure cost:
internal failure cost:
- rework, repair, and failure mode analysis
external failure cost:
- complaint resolution
- product return and replacement
- help line support
- warranty work
10
 Software Quality Assurance
Goal: to achieve high-quality software product

Quality definition:
“Conformance to explicitly stated functional and performance requirements,
explicitly documented development standards, and implicit characteristics that
expected of al professional developed software.”

Three import points for quality measurement:

- Use requirements as the foundation
- Use specified standards as the criteria
- Considering implicit requirements

11
 Software Quality Assurance
About Quality Assurance:

- The first formal quality assurance and control function was introduced at

Bell Labs in 1916 in the manufacturing world.

- During the 1950s and 1960s, the programmers controlled their product
quality.

- During the 1970s, quality assurance standards were introduced first in

military contract software development.

12
 SQA Group
Who involves quality assurance activities?
Software engineers, project managers, customers, sale people, SQA group

Engineers involved in the quality assurance work:

- apply technical methods and measures
- conduct formal technical review
- perform well-planned software testing

The SQA group’s role

- serves as the customer’s in-house representative assist the software
engineering team in achieving high-quality

The SQA group’s responsibility:

- quality assurance planning oversight, record keeping, analysis and
reporting

13
 SQA Group
The SQA group’s tasks:

- Prepare a SQA plan for a project

- Participate in the development of the project’s software process
description
- Review engineering activities to verify compliance with the defined
process
- Audits designated software work products to verify compliance in the
defined process
- Ensure the deviations in software work and products according to a
documented procedure
- Records any noncompliance and reports to senior management

14
 Software Reviews
What is software reviews?
- a “filter” for the software engineering process.

Purpose:
- serves to uncover errors in analysis, design, coding, and testing.

Why software reviews?

- To err is human
- Easy to catch the errors in engineers’ work

A review --> a way to

- identify the needed improvements of the parts in a product
- confirm the improvement parts of a product.
- achieve technical work of more uniform, predicable, and
manageable.

15
 Reviews

... there is no particular reason

why your friend and colleague
cannot also be your sternest critic.
- Jerry Weinberg

16
 What Are Reviews?

a meeting conducted by technical people for technical people

a technical assessment of a work product created during the

software engineering process

a software quality assurance mechanism

a training ground

17
 What Reviews Are Not

A project summary or progress assessment

A meeting intended solely to impart information

A mechanism for political or personal reprisal!

18
 What Do We Look For?

Errors and Defects

Error
- a quality problem found before the software is released
to end users

Defect
- a quality problem found only after the software has
been released to end-users

Errors and defects have very different economic,

business, psychological, and human impact!!

19
 Defect Amplification
A Defect Amplification Model can be used to illustrate the
generation and detection of errors during the design and code
generation actions of a software process.

Defects Detection
Errors from Errors passed through
Previous step Percent Errors passed
Amplified errors 1:x Efficiency To next step

Newly generated errors

Development step

20
Defect Amplification

21
Formal Technical Reviews (FTR)
Objectives of FTR:

- to uncover errors in function, logic, or implementation

- to verify the software under review meets its requirements

- to ensure that the software has been represented according to

predefined standards

- to develop software in a uniform manner

- to make projects more manageable

The FTR is actually a class of reviews that includes walkthroughs

and inspections.

22
Formal Technical Reviews (FTR)

Purposes of FTR:
- serves as a training ground for junior engineers
- promote backup and continuity

Review meeting’s constraints:

- 3-5 people involved in a review
- advanced preparation (no more than 2 hours for each person)
- the duration of the review meeting should be less than 2 hours
- focus on a specific part of a software product

People involved in a review meeting:

- producer, review leader, 2 or 3 reviewers (one of them is recorder)

23
Formal Technical Reviews (FTR)
The preparation of a review meeting:
- a meeting agenda and schedule (by review leader)
- review material and distribution (by the producer)
- review in advance (by reviewers)

Review meeting results:

- a review issues list
- a simple review summary report (called meeting minutes)
- meeting decisions:
- accept the work product w/o further modification
- reject the work product due to errors
- accept work under conditions (such as change/review)
- sign-off sheet

24
Formal Technical Reviews (FTR)

Review summary report (a project historical record)

answers the following questions:
- what was reviewed?
- who reviewed it?
- what were the findings and conclusions

Review issues list serves two purposes:

- to identify problem areas in the project
- to serve as an action item checklist (a follow-up procedure is
needed)

25
Review Guidelines (for FTR)
A minimum set of guidelines for FTR:

- Review the product, not the producer

- Set an agenda and maintain it
- Limit debate and rebuttal
- Enunciate problem areas, but don’t attempt to solve every
problem noted
- Take written notes
- Limit the number of participants and insist upon advance
preparation
- Develop a checklist for each work product that is likely to be
reviewed
- Allocate resources and time schedule for FTRs
- Conduct meaningful training for all reviewers
- Review your early reviews

26
 The Players
review
leader standards bearer (SQA)

producer

maintenance
oracle

recorder reviewer
user rep

27
 Statistical Quality
Assurance
Statistical quality assurance reflects a growing trend
throughout industry to become more quantitative about
quality.

Statistical quality assurance implies the following steps:

- Information about software defects is collected and categorized
- An attempt is made to trace each defect to its underlying cause
- Using the Pareto principle (80 percent of the defects can be traced to
20 percent of all possible causes, and isolate the 20 percent)
- Once the vital few causes have been identified, correct the defects.

28
 Statistical Quality
Assurance
Causes of Errors:

- incomplete or erroneous specification (IES)

- misinterpretation of customer communication(MCC)
- intentional deviation from specification (IDS)
- violation of programming standards (VPS)
- error in data representation (EDR)
- inconsistent module interface (IMI)
- error in design logic (EDL)
- incomplete or erroneous testing (IET)
- inaccurate or incomplete documentation (IID)
- error in programming language translation of design (PLT)
- ambiguous or inconsistent human-computer interface (HCI)
- miscellaneous (MIS)

29
 Statistical Quality
Assurance
In conjunction with the collection of defect information,
software developers can calculate an error index (EI) for
each major step in the software engineering process.

After analysis, design, coding, testing, and release, the

following data are collected:

Ei = the total no. of errors uncovered during the ith step in the process.
Si = the no. of serious errors
Mi = the no. of moderate errors
Ti = the no. of minor errors
PS = the size of the product at the ith step.

30
 Statistical Quality
Assurance

At each step in the software engineering process, a phase

index ( PIi ) is computed:
PIi = ws (Si/Ei) + wm (Mi/Ei) + wt (Ti/Ei)
ws, wm, wt = weighting factors for serious,
moderate, and trivial errors, where
recommended values are ws = 10, wm = 3, wt = 1.
Error index (EI) can be computed as follows:
EI = ∑ ( i x Pii ) / PS
or, EI = ( PI1 + 2 PI2 + 3 PI3 + i PIi ) / PS

31
Statistical Quality
Assurance

32
ISO

A consortium of 63 countries to formulate and

foster standardization
A reference for contract between independent
parties
Specifies guidelines for maintaining a quality
system

33
ISO 9000

A series of 3 standards ISO 9001,9002,9003

If proper process is followed for production then
good quality products are bound to follow
automatically!!

34
ISO 9000
A quality assurance system may be defined
as the organizational structure,
responsibilities, procedures, processes,
Quality assurance systems are created to
and resources for implementing quality
help organizations ensure their products
management
and services satisfy customer
expectations by meeting their
specifications.

35
ISO 9000
These systems cover a wide variety of
activities encompassing a product’s
entire life cycle including planning,
controlling, measuring, testing and
reporting, and improving
After adopting quality
the standards, levels
a country
throughout the development
typically permits and
only ISO registered
manufacturing process.
companies to supply goods and services to
government agencies and public utilities.

36
ISObecome
To 9000 registered to one of the
quality assurance system models
contained in ISO 9000, a company’s
quality system and operations are
Upon successful registration, a company
scrutinized by third party auditors for
is issued a certificate from a registration
compliance to the standard and for
body represented by the auditors.
effective operation.
Semi-annual surveillance audits ensure
continued compliance to the standard.

37
ISO 9000

For a quality system to be ISO compliant,

these processes must address the areas
identified in the standard and must be
documented and practiced as described.

38
terms.
ISO 9000
These elements include the
organizational structure, procedures,
processes, and resources needed to
implement quality planning, quality
control, quality assurance, and quality
improvement.

However, ISO 9000 does not describe

how an organization should implement
these quality system elements.

39
ISO 9001

Applies to organizations engaged in design,

development, production and servicing of goods

Applicable to most software development organizations

40
ISO 9001

The standard contains 20 requirements

that must be present for an effective
Because the ISO 9001 standard
quality assurance system. is
applicable to all engineering disciplines, a
special set of ISO guidelines (ISO 9000-3)
have been developed to help interpret the
standard for use in the software process.

41
ISO 9001

The standard contains 20 requirements

that must be present for an effective
In orderassurance
quality for a software organization to
system.
become registered to ISO 9001, it must
establish policies and procedures to
address each of the requirements just
noted (and others) and then be able to
demonstrate that these policies and
procedures are being followed.
42
ISO 9002

Applies to organizations which do not design products

but are only involved in production

e.g. steel and car manufacturing industries

(they buy plant and product design from others)

43
ISO 9003

Applies to organizations involved only in installation

and testing of products

44
Why get ISO 9000 Certification?

Increase of confidence of customers in an

organization
It requires well documented software
production process (repeatable and higher quality )

Points out weak points of an organizations

45
How to get ISO 9000 Certification

Application Pre-assessment

Document Review

Compliance Audit

Registration

Surveillance
46
Shortcomings of ISO Certification
Requires software production process to
adhere to high quality. (No guidelines for defining an
appropriate process)

of awarding certificates
Variations in the norms
among different accreditation agencies

In manufacturing industry there exists a link

between process quality and product quality.
Good process quality product but software is not manufactured!!
47
 SEI Capability Maturity Model

Proposed by Software Engineering Institute of Carnegie

Mellon University,
Originally developed to assist DOD in software
acquisition
Provide a way to assess the software process capability of
an organization

48
SEI Capability Maturity Model (CMM)

Used to improve the process capability of an

organization
This assessment is purely for internal use

49
CMM Level1 - Initial

Competent People

Success depends on individual efforts

50
CMM Level2 - Repeatable

activities e.g. tracking cost and

Project management
schedule are established FP, COCOMO

Repetition of earlier success with similar applications

51
CMM Level3 - Defined

Process for management and development is defined

But Process and Product quality not measured

52
CMM Level4 - Managed
Product and Process metrics collected

Product Metrics: Measures characteristics of product

e.g, size, reliability, time complexity etc.

Process Metrics: Reflects effectiveness of process

being followed
Results used to evaluate project
performance rather than improve process
53
CMM Level5 - Optimizing

Process and product measurement data analyzed

for continuous process improvement

Process/technology change management

54
ISO and CMM

ISO:

Awarded by international standards body

Not software industry specific

Confined to Quality Assurance

55
ISO and CMM

CMM:

Assessment for internal use only!!

Developed specifically for software industry use

Quality Assurance, provides a

Goes beyond
way to achieve Gradual Quality
Achievement 56
Key Process Areas (KPA)
• A list of areas to focus to improve CMM
rating of an organization

57
CMM Focus Key Process Areas

Initial Competent People --------------------------

Repeatable Project Management SW Project Planning

SW Config. Mgmnt.

Defined Definition of Processes Process Definition

Training Program
Peer Reviews

Managed Product and Process Quantitative Process Metrics

Quality SW Quality Management

Optimizing Continuous Process Defect Prevention

Improvement Process/Technology Change
Management
58
 The SQA Plan

The SQA plan provides a road map for instituting software

quality assurance.

Basic items:
- purpose of plan and its scope
- management
- organization structure, SQA tasks, their placement in the process
- roles and responsibilities related to product quality
- documentation
- project documents, models, technical documents, user documents.

59
 The SQA Plan

- standards, practices, and conventions

- reviews and audits
- test - test plan and procedure
- problem reporting, and correction actions
- tools
- code control
- media control
- supplier control
- records collection, maintenance, and retention
- training
- risk management

60