Software Quality Engineering
Software Quality Engineering
A reliability growth model is a model of how the system reliability changes over time during the
testing process.As system failures are discovered, the underlying faults causing these failures
are repaired so that the reliability of the system should improve during system testing and
debugging. To predict reliability, the conceptual reliability growth model must then be translated
into a mathematical model.
Reliability growth modeling involves comparing measured reliability at a number of points of time
with known functions that show possible changes in reliability
# What is quality?
It is the standard of something as measured against other things of a similar kind; the degree of
excellence of something.
Examples:
Let’s look at an example to understand how important software quality is.
● On June 4, 1996, Ariane 5 crashed on its maiden journey just 40 seconds after the take off.
At an altitude of about 3700 m, the launcher veered off and broke and exploded. After
detailed investigation, it was found that the failure in software caused the crash that resulted
in the loss of half a billion dollars.
● Similarly, on September 23rd, in the year 1999, Mars Climate Orbiter disappeared after it
began to orbit Mars. The cost of this mission was about 125 US million dollars. On
investigating, it was found that the error was due to the transfer of information between the
team in Colorado and the team in California. One team used English units whereas the
other one used metric unit for key space craft operation.
If the quality of the software was taken care of, these incidents could have been avoided easily.
There are 2 aspects of software quality – functional aspect and structural aspect.
The structural aspect includes all the non functional aspects like reliability, consistency,
maintenance and so on.
FAILURE: A failure is the inability of a software system or component to perform its required
functions within specified performance requirements. When a defect reaches the end customer
it is called a Failure. During development Failures are usually observed by testers.
FAULT: An incorrect step, process or data definition in a computer program which causes the
program to perform in an unintended or unanticipated manner. A fault is introduced into the
software as the result of an error. It is an anomaly in the software that may cause it to behave
incorrectly, and not according to its specification. It is the result of the error.
# Defect prevention:
Making sure the defects do not happen in the first place
Defect Prevention Methods and Techniques:
Some traditional and common methods that have been in use since a long time for defect
prevention are listed below;
1) Review and Inspection: This method includes the review by an individual team member
(self-checking), peer reviews and inspection of all work products.
Use of formal methods like formal specification and formal verification.Formal specification is
concerned with producing consistent requirements specification, constraints and designs so that
it reduces the chances of accidental fault injections. With formal verifications, correctness of
software system is proved.
2) Walkthrough: This is more or less like a review but it’s mostly related to comparing the
system to the prototype which will give a better idea regarding the correctness and/or the
look-and-feel of the system.
3) Defect Logging and Documentation: This method provides some key information,
arguments/parameters that can be used to support analyzing defects.
4) Root Cause Analysis: Root cause analysis includes two major approaches:
I) Pareto Analysis
2) Fishbone Analysis:
# LOC (Lines of code): A line of code is any line of program text that is not a comment or blank
line, regardless of the number of statements or fragments of statements on the line.This
specifically includes all lines containing program headers, declarations, and executable and
non-executable statements. Thus their method is to count physical lines including prologues and
data definitions (declarations) but not comments
# Function point:
A function can be defined as a collection of executable statements that performs a certain task,
together with declarations of the formal parameters and local variables manipulated by those
statements (Conte et al., 1986).
A function point is a unit of measurement to express the amount of business functionality an
information system (as a product) provides to a user.
The functional user requirements of the software are identified and each one is categorized into
one of five types:
inputs, outputs, inquiries, internal and external interfaces
2)Functionally sizing the requirements for the application quantifies the different types of
functionality delivered by an application. The function point count assigns function points to
each of the function types: External Inputs, Outputs, Enquiries, and Internal and External Files.
3)Knowing the software size facilitates the creation of more accurate estimates of project
resources and delivery dates
6) Once the scope of the project is agreed, the estimates for effort, staff resources, costs and
schedules need to be developed
Process metrics can be used to improve development and maintenance activities of the
software. Examples include the effectiveness of defect removal during development, the pattern
of testing defect arrival, and the response time of the fix process.
Project metrics describe the project characteristics and execution. Examples include the number
of software developers, the staffing pattern over the life cycle of the software, cost, schedule,
and productivity.
Some metrics belong to multiple categories. For example, the in- process quality metrics of a
project are both process metrics and project metrics.
Software quality metrics are a subset of software metrics that focus on the quality aspects of the
product, process, and project. In general, software quality metrics are more closely associated
with process and product metrics than with project metrics.
Defect Density
It measures the defects relative to the software size expressed as lines of code or function
point, etc. i.e., it measures code quality per unit. This metric is used in many commercial
software systems.
number of defects over the opportunities for error (OFE) during a specific time frame.
Defect Density is the number of defects confirmed in software/module during a specific period of
operation or development divided by the size of the software/module.
Defect density is counted per thousand lines of code also known as KLOC.
Formula to measure Defect Density:
● Defect Density = Defect count/ size of the release
Size of release can be measured in terms of line of code (LoC).
Customer Problems
It measures the problems that customers encounter when using the product. It contains the
customer’s perspective towards the problem space of the software, which includes the
non-defect oriented problems together with the defect problems.
rom the customers' standpoint, all problems they encounter while using the software product,
not just the valid defects, are problems with the software. Problems that are not valid defects
may be usability problems, unclear documentation or information, duplicates of valid defects
(defects that were reported by other customers and fixes were available but the current
customers did not know of them), or even user errors. These so-called non-defect-oriented
problems, together with the defect problems, constitute the total problem space of the software
from the customers' perspective.
Customer Satisfaction
Customer satisfaction is often measured by customer survey data through the five-point scale −
● Very satisfied
● Satisfied
● Neutral
● Dissatisfied
● Very dissatisfied
Satisfaction with the overall quality of the product and its specific dimensions is usually obtained
through various methods of customer surveys. Based on the five-point-scale data, several
metrics with slight variations can be constructed and used, depending on the purpose of
analysis. For example −
● Percent of completely satisfied customers
● Percent of satisfied customers
● Percent of dis-satisfied customers
● Percent of non-satisfied customers
Usually, this percent satisfaction is used.
- one measures the time between failures, the other measures the defects relative to the
software size (lines of code, function points, etc.).
- although it is difficult to separate defects and failures in actual measurements and data
tracking, failures and defects (or faults) have different meanings.
- For special-purpose software systems such as the air traffic control systems or the
space shuttle control systems, the operations profile and scenarios are better defined
and, therefore, the time to failure metric is appropriate. For general-purpose computer
systems or commercial-use software, for which there is no typical user profile of the
software, the MTTF metric is more difficult to implement and may not be representative
of all customers.
- gathering data about time between failures is very expensive. It requires recording the
occurrence time of each software failure. It is sometimes quite difficult to record the time
for all the failures observed during testing or operation. To be useful, time between
failures data also requires a high degree of accuracy. This is perhaps the reason the
MTTF metric is not widely used by commercial developers.
- Finally, the defect rate metric (or the volume of defects) has another appeal to
commercial software development organizations. The defect rate of a productor the
expected number of defects over a certain time period is important for cost and resource
estimates of the maintenance phase of the software life cycle.
The fix response time metric is usually calculated as the mean time of all problems from open to
close. Short fix response time leads to customer satisfaction.
The important elements of fix responsiveness are customer expectations, the agreed-to fix time,
and the ability to meet one's commitment to the customer.
For example, John takes his car to the dealer for servicing in the early morning and needs it
back by noon. If the dealer promises noon but does not get the car ready until 2 o'clock, John
will not be a satisfied customer. On the other hand, Julia does not need her mini van back until
she gets off from work, around 6 P.M. As long as the dealer finishes servicing her van by then,
Julia is a satisfied customer. If the dealer leaves a timely phone message on her answering
machine at work saying that her van is ready to pick up, Julia will be even more satisfied. This
type of fix responsiveness process is indeed being practiced by automobile dealers who focus
on customer satisfaction.
Percent delinquent fixes
For each fix, if the turnaround time greatly exceeds the required response time, then it is
classified as delinquent:
Fix Quality
Fix quality or the number of defective fixes is another important quality metric for the
maintenance phase. A fix is defective if it did not fix the reported problem, or if it fixed the
original problem but injected a new defect. For mission-critical software, defective fixes are
detrimental to customer satisfaction. The metric of percent defective fixes is the percentage of
all fixes in a time interval that is defective.
A defective fix can be recorded in two ways: Record it in the month it was discovered or record it
in the month the fix was delivered. The first is a customer measure; the second is a process
measure. The difference between the two dates is the latent period of the defective fix.
Usually the longer the latency, the more will be the customers that get affected. If the number of
defects is large, then the small value of the percentage metric will show an optimistic picture.
The quality goal for the maintenance process, of course, is zero defective fixes without
delinquency.
# SQA:
Software Quality Assurance (SQA) consists of a means of monitoring the software engineering
processes and methods used to ensure quality. It does this by means of audits of the quality
management system under which the software system is created.It is distinct from software
quality control which includes reviewing requirements documents, and software testing. SQA
encompasses the entire software development process, which includes processes such as
software design, coding, source code control, code reviews, change management, configuration
management, and release management. Whereas software quality control is a control of
products, software quality assurance is a control of processes.
SQA is used to test the software, rather than checking the quality after completion. SQA
processes, test for quality in each phase of development until the software is complete. With the
SQA the software development moves to the next phase if the current phase of software is
compiled with required quality standards.
SQA encompasses:
1. analysis, design, coding and testing methods and tools
2. formal technical reviews that are applied during each software engineering step 3. a multi
tiered testing strategy
4. control of software documentation and the changes made to it
5. a procedure to assure compliance with software development standards
6. measurement and reporting mechanism
Below are the Quality assurance criteria against which the software would be evaluated against:
● correctness
● efficiency
● flexibility
● integrity
● interoperability
● maintainability
● portability
● reliability
● reusability
● testability
● usability
Different activities/role of SQA are as follows
● Maintaining the quality of the system as per the specification and business requirements.
● Defect prevention and formal methods for other defect prevention technique.
● Defect Reduction
● Direct fault detection and removal without executing the project scenario.
● Testing the project for Failure observation and bug removal.
● Risk identification
● Defect tracking techniques and methods. Software fault tolerance. Concluding remarks
and maintaining reports.
SQA ACTIVITIES
http://www.academia.edu/9760547/LECTURE_NOTES_2_Software_Quality_Assurance
Prepares an SQA plan for a project. The plan is developed during project planning and is
reviewed by all interested parties. Quality assurance activities performed by the software
engineering team and the SQA group are governed by the plan.
The plan identifies • evaluations to be performed • audits and reviews to be performed •
standards that are applicable to the project • procedures for error reporting and tracking •
documents to be produced by the SQA group • amount of feedback provided to the software
project team
Participates in the development of the project’s software process description. The software team
selects a process for the work to be performed. The SQA group reviews the process description
for compliance with organizational policy, internal software standards, externally imposed
standards (e.g., ISO-9001), and other parts of the software project plan.
Reviews software engineering activities to verify compliance with the defined software process.
The SQA group identifies, documents, and tracks deviations from the process and verifies that
corrections have been made.
Audits designated software work products to verify compliance with those defined as part of the
software process. The SQA group reviews selected work products; identifies, documents, and
tracks deviations; verifies that corrections have been made; and periodically reports the results
of its work to the project manager.
Ensures that deviations in software work and work products are documented and handled
according to a documented procedure. Deviations may be encountered in the project plan,
process description, applicable standards, or technical work products.
Records any noncompliance and reports to senior management. Noncompliance items are
tracked until they are resolved
# Statistical Quality Assurance (SQA). is used to identify the potential variations in the
manufacturing process and predict potential defects on a parts-per-million (PPM) basis. It
provides a statistical description of the final product and addresses quality and safety issues
that arise during manufacturing.
This relatively simple concept represents an important step towards the creation of an adaptive
software engineering process in which changes are made to improve those elements of the
process that introduce error. To illustrate this, assume that a software engineering organization
collects information on defects for a period of one year. Some of the defects are uncovered as
software is being developed. Others are encountered after the software has been released to its
end-users.
Once the vital few causes are determined, the software engineering organization can begin
corrective action. For example, to correct MCC, the software developer might implement
facilitated application specification techniques (Chapter 11) to improve the quality of customer
communication and specifications. To improve EDR, the developer might acquire CASE tools
for data modeling and perform more stringent data design reviews. It is important to note that
corrective action focuses primarily on the vital few. As the vital few causes are corrected, new
candidates pop to the top of the stack.
# standard of quality which is a criterion or statement that describes the acceptable level of
something
Quality assurance is the process of defining how software quality can be achieved and how the
development organization knows that the software has the required level of quality. The main
activity of the quality assurance process is the selection and definition of standards that are
applied to the software development process or software product.
2) The process standards define the processes that should be followed during software
development.Process quality standards protect the business owner from unnecessary
costs in product repair or manufacturing rejects. These standards ensure that employees
building products or providing services follow a specific procedure so that the results
always meet the design quality standards. These process guidelines come in the form of
instructions that describe each step and explain how to differentiate a well-performed
action from an error.
3) The software standards are based on best practices and they provide a framework for
implementing the quality assurance process.
ISO 9000 is an international set of standards that can be used in the development of a quality
management system in all industries. ISO 9000 standards can be applied to a range of
organizations from manufacturing to service industries. ISO 9001 is the most general of these
standards. It can be applied to organizations that design, develop and maintain products and
develop their own quality processes.
. The ISO 9001 standard describes various aspects of the quality process and defines the
organizational standards and procedures that a company should define and follow during
product development. These standards and procedures are documented in an organizational
quality manual.
Documentation standards in a software project are important because documents can represent
the software and the software process. Standardized documents have a consistent appearance,
structure and quality, and should therefore be easier to read and understand. There are three
types of documentation standards:
1. Documentation process standards. These standards define the process that should be
followed for document production.
2. Document standards. These standards describe the structure and presentation of
documents.
3. Documents interchange standards. These standards ensure that all electronic copies of
documents are compatible.
#http://blog.pilgrimquality.com/iso90012015-quality-mgmt-principles/
The International Standard for Quality management (ISO 9001:2015) adopts a number of
management principles, that can be used by top management to guide their organizations
towards improved performance.
1. Customer focus[edit]
The primary focus of quality management is to meet customer requirements and to strive to
exceed customer expectations.
Rationale
Sustained success is achieved when an organization attracts and retains the confidence of
customers and other interested parties on whom it depends. Every aspect of customer
interaction provides an opportunity to create more value for the customer. Understanding
current and future needs of customers and other interested parties contributes to sustained
success of an organization [5]
2. Leadership[edit]
Leaders at all levels establish unity of purpose and direction and create conditions in which
people are engaged in achieving the organization’s quality objectives.
Rationale
Creation of unity of purpose and direction and engagement of people enable an organization to
align its strategies, policies, processes and resources to achieve its objectives [6]
3. Engagement of people[edit]
Competent, empowered and engaged people at all levels throughout the organization are
essential to enhance its capability to create and deliver value.
Rationale
To manage an organization effectively and efficiently, it is important to involve all people at all
levels and to respect them as individuals. Recognition, empowerment and enhancement of
competence facilitate the engagement of people in achieving the organization’s quality
objectives.[7]
4. Process approach[edit]
Consistent and predictable results are achieved more effectively and efficiently when activities
are understood and managed as interrelated processes that function as a coherent system.
Rationale
The quality management system consists of interrelated processes. Understanding how results
are produced by this system enables an organization to optimize the system and its
performance.[8]
5. Improvement[edit]
Successful organizations have an ongoing focus on improvement.
Rationale
Improvement is essential for an organization to maintain current levels of performance, to react
to changes in its internal and external conditions and to create new opportunities.[9]
7. Relationship management[edit]
Further information: Relationship management
For sustained success, an organization manages its relationships with interested parties, such
as suppliers, retailers.
Rationale
Interested parties influence the performance of an organization. Sustained success is more
likely to be achieved when the organization manages relationships with all of its interested
parties to optimize their impact on its performance. Relationship management with its supplier
and partner networks is of particular importance
Custi focus, ppl engagement, leadership, improvement, evidence based decision making,
relationship mgmt
Celier
Quality costs are the costs associated with preventing, finding, and correcting defective work.
These costs are huge, running at 20% - 40% of sales. Many of these costs can be significantly
reduced or completely avoided. One of the key functions of a Quality Engineer is the reduction
of the total cost of quality associated with a product.
Cost of Quality (COQ) is a measure that quantifies the cost of control( cost of conformance) and
the cost of failure of control (Cost of Non-Conformance).
In other words, it sums up the costs related to prevention and detection of defects and the costs
due to occurrences of defects.
● Definition by ISTQB: cost of quality: The total costs incurred on quality activities and
issues and often split into prevention costs, appraisal costs, internal failure costs and
external failure costs.
# Benchmarking
Benchmarking is a technique in which a company measures its performance against that of best
in class companies, determines how those companies achieved their performance levels and
uses the information to improve its own performance. Subjects that can be benchmarked
include strategies, operations and processes.
Benchmarking is the process of measuring products, services, and processes against those of
organizations known to be leaders in one or more aspects of their operations. Benchmarking
provides necessary insights to help you understand how your organization compares with
similar organizations, even if they are in a different business or have a different group of
customers.
Additionally, benchmarking can help you identify areas, systems, or processes for
improvements—either incremental (continuous) improvements or dramatic (business process
reengineering) improvements.
Competitive benchmarking — Compares how well (or poorly) an organization is doing with
respect to the leading competition, especially with respect to critically important attributes,
functions, or values associated with the organization’s products or services. For example, on a
scale of one to four, four being best, how do customers rank your organization’s products or
services compared to those of the leading competition? If you cannot obtain hard data,
marketing efforts may be misdirected and design efforts misguided.
http://www.qasigma.com/2008/11/benchmarking.html
PERFORMANCE.
Key Points:
● Benchmarking is an increasingly popular improvement tool.
● Benchmarking concerns processes and practices.
● Benchmarking is a respected means of identifying processes that require major change.
● Benchmarking is done between consenting companies that may or may not be
competitors.
● Benchmarking compares your process or practice with the target company’s
best-in-class process or practice.
● The goal of benchmarking is to find “secrets of success” and then adapt and improve for
your own application.