Advance Software Quality

Assurance

Anas Bilal
 Quality
• Synonyms
– Excellence, superiority, class, eminence, value,
worth
• Antonym
– Inferiority
• Everyone claims to manufacture / develop /
sell / market “good” quality products /
services
 Software Quality
• Quality as it relates to software
• Difficult to define
• Good software quality characteristics can be
identified
• Bad or undesirable characteristics can also be
identified
• Relates to all aspects of software (requirements /
design / code / tests / documents / training)
 Software Quality Definitions - 1
• Low levels of defects when deployed,
ideally approaching zero
• High reliability, or the capability of running
without crashes or strange results
• A majority of clients with high user-
satisfaction when surveyed
 Software Quality Definitions - 2
• A structure that can minimize “bad fixes” or
insertion of new defects during repairs
• Effective customer support when problems
do occur
• Rapid repairs for defects, especially for
high-severity defects
 Why Software Quality? - 1
• Reduces time to market for new products
• Enhances market share compared to direct
competitors
• Minimizes “scrap and rework” expenses
• Attracts and keeps “top-gun” personnel
• Minimizes the risk of serious litigation
 Why Software Quality? - 2
• Minimizes the risk of serious operating
failures and delays
• Minimizes the risk of bankruptcy or
business failures, which may be attributed
directly to poor quality or poor software
quality
 Achieving Software Quality
• “For a software application to achieve high
quality levels, it is necessary to begin
upstream and ensure that intermediate
deliverables and work products are also of
high quality levels. This means that the
entire process of software development
must itself be focused on quality”
– Capers Jones
 Six Software Defect Origins
• Requirements
• Design
• Source code
• User manuals/training material
• “Bad fixes” or mistakes made during repairs
• Flawed test cases used by the application
 Six Root Causes of Poor
Software Quality
• Inadequate training of managers and staff
• Inadequate defect and cost measurement
• Excessive schedule pressure
• Insufficient defect removal
• High complexity levels
• Ambiguous and creeping requirements and
design (feature race & gimmicks)
Six Software Defect Elimination
Strategies
• Effective defect prevention
• High levels of defect removal efficiency
• Accurate defect prediction before the
project begins
• Accurate defect tracking during
development
• Useful quality measurements
• Ensuring high levels of user-satisfaction
 Quality Attributes
• Quality attributes set is a way to represent
customer quality requirements
• Ask your current and prospective customers
about their definition of quality
• Develop a quality assurance program based
on the requirements of your customers
 Product-Specific Attributes
• Ease of use
• Documentation
• Defect tolerance
• Defect frequency
• Defect impact
• Packaging
• Price versus reliability
• Performance
Organization-Specific Attributes
• Service and support
• Internal processes
 Achieving High Levels of
Software Quality - 1
• Enterprise-wide quality programs
• Quality awareness and training methods
• Quality standards and guidelines
• Quality analysis methods
• Quality measurement methods
• Defect prevention methods
• Non-test defect removal methods
 Achieving High Levels of
Software Quality - 2
• Testing methods
• User-satisfaction methods
• Post-release quality control
 Best in Class Quality Results - 1
• Quality measurements
• Defect prevention
• Defect and quality estimation automation
• Defect tracking automation
• Complexity analysis tools
• Test coverage analysis tools
• Formal inspections
Best in Class Quality Results - 2
• Formal testing by test specialists
• Formal quality assurance group
• Executive and managerial understanding of
quality
 Two Components of
Software Quality Improvement
• Reductions in total defect potentials using
methods of defect prevention
• Improvements in cumulative removal
efficiency levels
 Categories of Software Defects
• Errors of commission: something wrong is
done
• Errors of omission: something left out by
accident
• Errors of clarity and ambiguity: different
interpretations
• Errors of speed and capacity
 Software Defect Prevention
Req. Design Code Document Perf.
Defects Defects Defects Defects Defects

JAD Excellent Good N/A Fair Poor

Prototypes Excellent Excellent Fair N/A Excellent

Structured Fair Good Excellent Fair Fair

Methods

CASE Tools Fair Good Fair Fair Fair

Blueprints / Excellent Excellent Excellent Excellent Good

Reusable Code

QFD Good Excellent Fair Poor Good

 Joint Application Development
• Users are active participants in the
requirements sessions
• Both client and MIS sides agree on
uninterrupted time commitments
• JAD-based requirements are more
“complete” versus the traditional
requirements
 Quality Function Deployment
• QFD is a very formal, structured group
activity involving clients and development
personnel
• During QFD sessions, user’s quality criteria
are exhaustively enumerated and defined.
This is followed by the product’s quality
response to these requirements
 Quality Laggards - 1
• No software quality measurement program
of any kind
• No usage of formal design and code
inspections
• No knowledge of the concepts of defect
potentials and defect removal efficiency
• Either no quality assurance group or a group
that is severely understaffed
 Quality Laggards - 2
• No trained testing specialists available
• Few or no automated quality assurance
tools
• No quality and reliability estimation
capability
• Minimal or no software project
management tools available
 Quality Laggards - 3
• No automated risk assessment or
avoidance capability
• From a low of one to a high of perhaps four
distinct testing stages
• No test library or test-case management
tools available
• No complexity analysis tools utilized
 Quality Laggards - 4
• Defect potentials averaging more than 6
defects per function point
• Defect removal efficiency averaging less
than 80%
• Executive and managerial indifference (and
ignorance) of quality matters
Project Management Approaches
and High Software Quality - 1
• Use of automated project estimation
methods
• Use of automated project planning methods
• Use of early and automated estimates of
software defect potentials
• Use of early and automated estimates of
software defect removal efficiency
• Formal risk-analysis
Project Management Approaches
and High Software Quality - 2
• Provision of adequate time for pre-test
inspections
• Historical quality data from similar projects
available
• Milestone tracking automated and thorough
• Defect tracking automated and thorough
• Management focus concentrated on
achieving excellent results
Project Management Approaches
and Poor Software Quality
• Exact opposite of the project management
approaches correlating with high software
quality
 SQA Group’s Activities - 1
• Preparation of an SQA plan for a project
– Evaluations to be performed
– Audits and reviews to be performed
– Standards that are applicable to the project
– Procedures for error reporting and tracking
– Documents to be produced by the SQA group
– Amount of feedback provided to the software
project team
 SQA Group’s Activities - 2
• Participation in the development of the
project’s software process description
• Review of software engineering activities to
verify compliance with the defined software
process
• Audit of designed software work products
to verify compliance with those defined as
part of the software process
 SQA Group’s Activities - 3
• Ensure that deviations in software work and
work products are documented and handled
according to a documented procedure
• Record any noncompliance and reports to
senior management
 Costs of Software Quality - 1
• Defects prevention costs
• User satisfaction optimization costs
• Data quality defect prevention costs
• Data quality defect removal costs
• Quality awareness/training costs
• Non-test defect removal costs
• Testing defect removal costs
 Costs of Software Quality - 2
• Post-release customer support costs
• Litigation and damage award costs
• Quality savings from reduced scrap/rework
• Quality savings from reduced user
downtime
• Quality value from reduced time-to-market
intervals
 Costs of Software Quality - 3
• Quality value from enhanced
competitiveness
• Quality value from enhanced employee
morale
• Quality return on investment
 Cost Per Defect Hazards
• Test cases must be created whether there are
many bugs, only a few, or none at all
• Test cases must be run whether there are
any bugs or not, although tests will be run
more often for buggy software
• During testing, programmers are waiting
(and getting paid) for bugs to be found
 Data Quality - 1
• Extremely important to understand issues of
data quality
• Data results in (useful | useless) information
• Usually, governments are holders of largest
data banks (are they consistent?)
• Companies are increasingly using data to
their advantage over competitors
 Data Quality - 2
• Data warehouses present a unique challenge
to keep data consistent
• Another problem is the interpretation of
data
 Defect Discovery
• Defects are discovered by developers &
testers (usually) before release
• Defects are discovered by customers and
users (usually) after release
• Defects discovered after release can be
embarrassing for the development team
 Defect Discovery by Customers
• Rule 1: Defect discovery is directly related
to the number of users
• Rule 2: Defect discovery is inversely related
to the number of defects
 Quality Measurement Questions
• What should be measured for quality?
• How often should quality measurement be
taken and reported?
Quality Measurement Categories
• Measurement of defects or bugs in software
– 100% of software projects
• Measurement of user-satisfaction levels
– Only for software projects where clients can be
queried and actually use the software
consciously
 Software Defect Quality
Measurements - 1
• Defect volumes (by product, by time period,
by geographic region)
• Defect severity levels
• Special categories (invalid defects,
duplicates, un-duplicatable problems)
• Defect origins (i.e., requirements, design,
code, documents, or bad fixes)
 Software Defect Quality
Measurements - 2
• Defect discovery points (i.e., inspections,
tests, customer reports, etc.)
• Defect removal efficiency levels
• Normalized data (i.e., defects per function
point or per KLOC)
• Causative factors (i.e., complexity, creeping
requirements, etc.)
 Software Defect Quality
Measurements - 3
• Defect repair speeds or intervals from the
first report to the release of the fix
 Software User-Satisfaction
Quality Measurements - 1
• User perception of quality and reliability
• User perception of features in the software
product
• User perception of ease of learning
• User perception of ease of use
• User perception of customer support
• User perception of speed of defect repairs
 Software User-Satisfaction
Quality Measurements - 2
• User perception of speed of adding new
features
• User perception of virtues of competitive
products
• User perception of the value versus the cost
of the package
 References
• Software Quality: Analysis and Guidelines
for Success by Capers Jones
• Customer-Oriented Software Quality
Assurance by Frank Ginac
• A Practitioner’s Approach to Software
Engineering by Roger Pressman