CST 309 MANAGEMENT OF SOFTWARE SYSTEMS

Module 5
Software Quality, Software Quality Dilemma, Achieving Software Quality Elements of Software
Quality Assurance, SQA Tasks , Software measurement and metrics. Software Process
Improvement(SPI), SPI Process CMMI process improvement framework, ISO 9001:2000 for
Software. Cloud based Software - Virtualisation and containers, Everything as a service(IaaS, PaaS),
Software as a service. Microservices Architecture - Microservices, Microservices architecture,
Microservice deployment.

SOFTWARE QUALITY
 Today, software quality remains an issue.
 Customers blame developers, arguing that sloppy practices lead to low quality software.
 Developers blame customers (and other stakeholders), arguing that irrational delivery dates
and a continuing stream of changes force them to deliver software before it has been fully
validated.
 American Heritage Dictionary defines quality as “a characteristic or attribute of
something.”
 For software, two kinds of quality may be encountered:
 Quality of design encompasses requirements, specifications, and the design of the system.
 Quality of conformance is an issue focused primarily on implementation.
 User satisfaction = compliant product + good quality + delivery within budget and
schedule
 Software quality can be defined as:
o An effective software process applied in a manner that creates a useful product
that provides measurable value for those who produce it and those who use it.

Effective Software Process

• An effective software process establishes the infrastructure that supports any effort at building a
high quality software product.
•The management aspects of process create the checks and balances that help avoid project chaos
—a key contributor to poor quality.
•Software engineering practices allow the developer to analyze the problem and design a
solid solution—both critical to building high quality software.

Useful Product
• A useful product delivers the content, functions, and features that the end-user desires.
• It delivers these assets in a reliable, error free way.
•A useful product always satisfies those requirements that have been explicitly stated
by stakeholders.
•It satisfies a set of implicit requirements (e.g., ease of use) that are expected of all high
quality software.

Adding Value
• By adding value for both the producer and user of a software product, high quality software
provides
benefits for the software organization and the end-user community.
• The software organization gains added value because high quality software requires less maintenance
effort, fewer bug fixes, and reduced customer support.
• The user community gains added value because the application provides a useful capability in a way
that expedites some business process. 1
• The end result is:
 (1) greater software product revenue
(2) better profitability when an application supports a businessprocess
(3) improved availability of information that is crucial for the business.

Garvin’s Quality Dimensions

1. Performance Quality - Does the software deliver all content, functions, and features that are
specified
as part of the requirements model?
2. Feature quality - Does the software provide features that surprise and delight first-time end users?
3.Reliability - Does the software deliver all features and capability without failure? Is it available
when it is needed? Does it deliver functionality that is error free?
4.Conformance - Does the software conform to local and external software standards that are
relevant to the application?
5.Durability - Can the software be maintained (changed) or corrected (debugged) without
the inadvertent generation of unintended side effects? Will changes cause the error rate or
reliability to degrade with time?
6.Serviceability - Can the software be maintained (changed) or corrected (debugged) in an
acceptably short time period? Can support staff acquire all information they need to make
changes or correct defects?
7. Aesthetics - Overall visual effect of the design.
8.Perception – Describes how customers feel about the system (opinion about the system it can
be positive or negative).

Quality Factors

1. Correctness - The extent to which a program satisfies its specification and fulfills the customer’s
mission objectives.
2. Reliability - The extent to which a program can be expected to perform its
intended function with required precision.
3. Efficiency - The amount of computing resources and code required by a program to
perform its function.
4. Integrity - Extent to which access to software or data by unauthorized persons
can be controlled.
5. Usability - Effort required to learn, operate, prepare input for, and interpret output of a
program.
6. Maintainability - Effort required to locate and fix an error in a program.
7. Flexibility - Effort required to modify an operational program.
8. Testability - Effort required to test a program to ensure that it performs its intended
function.

2
 THE SOFTWARE QUALITY DILEMMA
•If you produce a software system that has terrible quality, you lose because no one will want to
buy it.
•If on the other hand you spend infinite time, extremely large effort, and huge sums of money to
build the absolutely perfect piece of software, then it's going to take so long to complete and it
will be so expensive to produce that you'll be out of business anyway.
• Either you missed the market window, or you simply exhausted all your resources.
•So people in industry try to get to that magical middle ground where the product is good enough
not to be rejected right away, such as during evaluation, but also not the object of so much
perfectionism and so much work that it would take too long or cost too much to complete.
“Good Enough” Software
•Good enough software delivers high quality functions and features that end users desire, but at
the same time it delivers other more obscure or specialized functions and features that contain known
bugs.
• Arguments against “good enough.”
o It is true that “good enough” may work in some application domains and for a few
major software companies.
o If you deliver a “good enough” (buggy) product, you risk permanent damage to your
company’s reputation.

Cost of Quality
Prevention costs include
 quality planning
 formal technical reviews
 test equipment
 Training
Internal failure costs include
 Rework
 Repair
 failure mode analysis
External failure costs
 complaint resolution
 product return and replacement
 help line support
 warranty work

The relative costs to find and repair an error or defect increase dramatically as we go from prevention
to detection to internal failure to external failure costs.

Negligence and Liability

• A governmental or corporate entity hires a major software developer or consulting company to
analyze requirements and then design and construct a software-based “system” to support
some major activity.
3
 • The system might support a major corporate function (e.g., pension management) or
some governmental function (e.g., healthcare administration or homeland security).
• Work begins with the best of intentions on both sides, but by the time the system is delivered,
things have gone bad.
• The system is late, fails to deliver desired features and functions, is error-prone, and does
not meet with customer approval.

Quality and Security

• “Software security relates entirely and completely to quality.
• And there are two kinds of software problems.
• One is bugs, which are implementation problems.
• The other is software flaws—architectural problems in the design.
• People pay too much attention to bugs and not enough on flaws.”

ACHIEVING SOFTWARE QUALITY

• Management and practice are applied within the context of four broad activities that help
a software team achieve high software quality:
o software engineering methods
o project management techniques
o quality control actions
o software quality assurance.

Software Engineering Methods

• In order to build high-quality software, understand the problem to be solved.
• Also be capable of creating a design that conforms to the problem while at the same time
exhibiting characteristics that lead to software that exhibits the quality dimensions and factors.
Project Management Techniques
(1) a project manager uses estimation to verify that delivery dates are achievable
(2) schedule dependencies are understood and the team resists the temptation to use shortcuts,
(3)risk planning is conducted so problems do not breed
chaos. Quality Control
• Quality control encompasses a set of software engineering actions that help to ensure that
each work product meets its quality goals.
• Models are reviewed to ensure that they are complete and consistent.
• Code may be inspected in order to uncover and correct errors before testing commences.
• A series of testing steps is applied to uncover errors in processing logic, data manipulation,
and interface communication.
• A combination of measurement and feedback allows a software team to tune the process when
any of these work products fail to meet quality goals.
Quality Assurance
• Quality assurance establishes the infrastructure that supports solid software engineering
methods, rational project management, and quality control actions.
• Quality assurance consists of a set of auditing and reporting functions that assess the
effectiveness and completeness of quality control actions.
• The goal of quality assurance is to provide management and technical staff with the data
necessary
to be informed about product quality, thereby gaining insight and confidence that actions to
achieve product quality are working.

4
 SOFTWARE QUALITY ASSURANCE
Software quality assurance (SQA) encompasses:
(1) an SQA process
(2)specific quality assurance and quality control tasks (including technical reviews and a
multi- tiered testing strategy)
(3) effective software engineering practice (methods and tools)
(4) control of all software work products and the changes made to them
(5) a procedure to ensure compliance with software development standards
(6) measurement and reporting mechanisms.

ELEMENTS OF SOFTWARE QUALITY ASSURANCE

Ten Elements of SQA are
1. Standards
2. Reviews and audits
3. Testing
4. Error/defect collection and analysis
5. Change management
6. Education
7. Vendor management
8. Security management
9. Safety
10. Risk management

11. Standards - The job of SQA is to ensure that standards that have been adopted are followed and
that all work products conform to them.
12. Reviews and audits –
• Technical reviews are a quality control activity.
• Their intent is to uncover errors.
• Audits are a type of review performed by SQA personnel with the intent of ensuring
that quality guidelines are being followed for software engineering work.
13. Testing - The job of SQA is to ensure that testing is properly planned and efficiently conducted
so that it has the highest likelihood of achieving its primary goal.
14. Error/defect collection and analysis - SQA collects and analyzes error and defect data to
better
understand how errors are introduced and what software engineering activities are best suited
to eliminating them.
5. Change management - SQA ensures that adequate change management practices have been
instituted.
6. Education - The SQA organization takes the lead in software process improvement and is a key
proponent and sponsor of educational programs.
7. Vendor management - The job of the SQA organization is to ensure that high-quality software
results by suggesting specific quality practices that the vendor should follow (when possible),
and incorporating quality mandates as part of any contract with an external vendor.
8. Security management - SQA ensures that appropriate process and technology are used to
achieve software security.
9. Safety - SQA may be responsible for assessing the impact of software failure and for initiating
those steps required to reduce risk.
10. Risk management - Although the analysis and mitigation of risk is the concern of software
engineers, the SQA organization ensures that risk management activities are properly conducted 5
 and that risk- related contingency plans have been established.

SQA NEED

• Quality Assurance testing is employed to reduce possible defects in every stage of the
software development life cycle.
• The focus of SQA is to monitor continuously throughout the Software Development Life Cycle
to ensure the quality of the delivered products.
• In process assurance, SQA provides management with objective feedback regarding compliance
to approved plans, procedures, standards, and analyses.

SQA TASKS

1. Prepares an SQA plan for a project:

• The plan identifies evaluations to be performed, audits and reviews to be conducted, standards
that are applicable to the project, procedures for error reporting and tracking, work products that
are produced by the SQA group.
2. Participates in the development of the project’s software process description.
• The SQA group reviews the process description for compliance with organizational policy,
internal software standards, externally imposed standards (e.g., ISO-9001), and other parts of
the software project plan.
3. Reviews software engineering activities to verify compliance with the defined software process
• The SQA group identifies, documents, and tracks deviations from the process and verifies that
corrections have been made.
4.Audits designated software work products to verify compliance with those defined as part of
the software process:
• The SQA group reviews selected work products; identifies, documents, and tracks deviations;
verifies that corrections have been made; and periodically reports the results of its work to
the project manager.
5.Ensures that deviations in software work and work products are documented and
handled according to a documented procedure:
• Deviations may be encountered in the project plan, process description, applicable standards,
or software engineering work products.
6. Records any non compliance and reports to senior management
• Non compliance items are tracked until they are resolved.

SQA Goals

Following are SQA Goals:

• Requirements quality
• Design quality
• Code quality
• Quality control effectiveness.

6
 SOFTWARE MEASUREMENT AND METRICS

SOFTWARE PROCESS IMPROVEMENT

SPI implies that
 elements of an effective software process can be defined in an effective manner
 an existing organizational approach to software development can be assessed against
those elements
 a meaningful strategy for improvement can be defined.
The SPI strategy transforms the existing approach to software development into something that is
more focused, more repeatable, and more reliable.

SPI Framework
 a set of characteristics that must be present if an effective software process is to be achieved
 a method for assessing whether those characteristics are present
 a mechanism for summarizing the results of any assessment
 a strategy for assisting a software organization in implementing those process characteristics
that have been found to be weak or missing.
7
  An SPI framework assesses the “maturity” of an organization’s software process and provides a
qualitative indication of a maturity level.

Elements of a SPI Framework

Constituencies
Quality certifiers
Quality(
Process)
-->
Quality(
Product)
Formalists -
This group
wants to
understand
process
workflow.
Tool advocates
- This group
insists on a tool-
assisted
approach to SPI
Practitioners -
This
constituency
uses a pragmatic
approach
Reformers - The goal of this group is organizational change that might lead to a better software process
Ideologists - This group focuses on the suitability of a particular process model for a specific
application domain or organizational structure.

Maturity Models
• A maturity model is applied within the context of an SPI framework.
• The intent of the maturity model is to provide an overall indication of the “process maturity”.
• An indication of the quality of the software process, the degree to which practitioner’s
understand and apply the process,

Is SPI for Everyone? 8

• Can a small company initiate SPI activities and do it successfully?
• Answer: a qualified “yes”
• Small organizations are more informal, apply fewer standard practices, and tend to be
 SPI PROCESS

The SPI Process—I - Assessment and Gap Analysis

Assessment examines a wide range of actions and tasks that will lead to a high quality process.
Consistency - Are important activities, actions and tasks applied consistently across all software
projects and by all software teams?
Sophistication - Are management and technical actions performed with a level of sophistication
that implies a thorough understanding of best practice?
Acceptance - Is the software process and software engineering practice widely accepted by
management
and technical staff?
Commitment - Has management committed the resources required to achieve consistency,
sophistication and acceptance?
Gap analysis—The difference between local application and best practice represents a “gap” that offers
opportunities for improvement.

The SPI Process—II Education and Training

Three types of education and training should be conducted:

Generic concepts and methods - Directed toward both managers and practitioners, this category
stresses both process and practice. The intent is to provide professionals with the intellectual tools
they need to apply the software process effectively and to make rational decisions about improvements
to the process. Specific technology and tools - Directed primarily toward practitioners, this
category stresses technologies and tools that have been adopted for local use. For example, if UML
has been chosen for analysis and design modeling, a training curriculum for software engineering
using UML would be established.
Business communication and quality-related topics - Directed toward all stakeholders, this category
focuses on “soft” topics that help enable better communication among stakeholders and foster a
greater
quality focus.

The SPI Process—III Selection and Justification

• Choose the process model that best fits your organization, its stakeholders, and the software
that you build
• Decide on the set of framework activities that will be applied, the major work products that
will be produced and the quality assurance checkpoints that will enable your team to assess
progress
• Develop a work breakdown for each framework activity (e.g., modeling), defining the task
set that would be applied for a typical project
• Once a choice is made, time and money must be expended to install it within an
organization and these resource expenditures should be justified.

The SPI Process—IV Installation/Migration

• Software process redesign (SPR) activities.
• SPR is concerned with identification, application, and refinement of new ways to
dramatically improve and transform software processes.
• ”Three different process models are considered:
o the existing (“as-is”) process
o a transitional (“here-to-there”) process 9
o the target (“to be”) process.

The SPI Process—V Evaluation

 • prior to the initiation of the SPI roadmap,
• during the execution of SPI activities (assessment, education, selection, installation),
•during the evaluation activity that follows the instantiation of some process
characteristic. In general, the following categories can be identified for SPI risk factors:
• budget and cost
• content and deliverables culture
• maintenance of SPI deliverables
• mission and goals
• organizational management and organizational stability
• process stakeholders
• schedule for SPI development
• SPI development environment and process
• SPI project management and SPI staff

SPI Return on Investment(ROI)

benefits include the cost savings associated with higher product quality (fewer defects), less rework,
reduced effort associated with changes, and the income.
costs include both direct SPI costs (e.g., training, measurement) and indirect costs associated with
greater
emphasis on quality control and change management activities and more rigorous application of
software engineering methods (e.g., the creation of a design model).

CMMI PROCESS IMPROVEMENT FRAMEWORK

• Capability Maturity Model Integration (CMMI)

• A comprehensive process meta-model that is predicated on a set of system and
software engineering capabilities that should be present as organizations reach different levels
of process capability and maturity
• A process meta-model in two different ways:
(1) as a “continuous” model
(2) as a “staged” model
• Defines each process area in terms of “specific goals” and the “specific practices” required to
achieve these goals.
• Specific goals establish the characteristics that must exist if the activities implied by a
process
area are to be effective.
• Specific practices refine a goal into a set of process-related activities.

The People CMM

• “a roadmap for implementing workforce practices that continuously improve the capability of
an organization’s workforce
• Set of five organizational maturity levels

10
 Process areas for the People CMM

Other SPI Frameworks

SPICE— a international initiative to support the International Standard ISO/IEC for Process
Assessment Bootstrap—a SPI framework for small and medium sized organizations that conforms to
SPICE
PSP and TSP—individual and team specific SPI frameworks that focus on process in-the-small, a
more rigorous approach to software development coupled with measurement
TickIT—an auditing method that assesses an organization compliance to ISO Standard 9001:2000

ISO 9001:2000 for Software

• ISO 9001:2000 is the quality assurance standard that applies to software engineering.
• The standard contains 20 requirements that must be present for an effective quality assurance
system.
• The requirements delineated by ISO 9001:2000 address topics such as
o management responsibility
o quality system
o contract review
o design control
o document and data control
o product identification and traceability
o process control
o inspection and testing
o corrective and preventive action
o control of quality records
11
 o internal quality audits
o training
o servicing
o statistical techniques.
CLOUD-BASED SOFTWARE

• The cloud is made up of very large number of remote servers that are offered for rent by
companies that own these servers.
• Cloud-based servers are ‘virtual servers’, which means that they are implemented in
software rather than hardware.
• You can rent as many servers as you need, run your software on these servers and make them
available to your customers.
• Your customers can access these servers from their own computers or other networked devices
such as a tablet or a TV.
• Cloud servers can be started up and shut down as demand changes.
• You may rent a server and install your own software, or you may pay for access to software
products that are available on the cloud.
Scalability, elasticity, and resilience
Scalability reflects the ability of your software to cope with increasing numbers of users. As the load
on
your software increases, your software automatically adapts so that the system performance and
response time is maintained.
Elasticity is related to scalability but also allows for scaling-down as well as scaling-up. That is, you
can monitor the demand on your application and add or remove servers dynamically as the number of
users
change.
Resilience means that you can design your software architecture to tolerate server failures. You can
make several copies of your software concurrently available. If one of these fails, the others continue to
provide a service.

Benefits of using the cloud for software development

12
 VIRTUALISATION AND CONTAINERS
• A virtual server runs on an underlying physical computer and is made up of an operating system
plus a set of software packages that provide the server functionality required.
• A virtual server is a stand-alone system that can run on any hardware in the cloud.
• This ‘run anywhere’ characteristic is possible because the virtual server has no
external dependencies.
• Virtual machines (VMs), running on physical server hardware, can be used to implement virtual
servers.
• A hypervisor provides hardware emulation that simulates the operation of the underlying
hardware.
• If you use a virtual machine to implement virtual servers, you have exactly the same
hardware platform as a physical server

Container-based virtualization
• If you are running a cloud-based system with many instances of applications or services, these
all use the same operating system, you can use a simpler virtualization technology called
‘containers’.
• Using containers accelerates the process of deploying virtual servers on the cloud.
• Containers are usually megabytes in size whereas VMs are gigabytes.
• Containers can be started and shut down in a few seconds rather than the few minutes required
for a VM.
• Containers are an operating system virtualization technology that allows independent servers
to share a single operating system.
• They are particularly useful for providing isolated application services where each user sees
their own version of an application.

13
Docker

• Containers were developed by Google around 2007 but containers became a mainstream
technology around 2015.
• An open-source project called Docker provided a standard means of container management that
is fast and easy to use.
• Docker is a container management system that allows users to define the software to be
included in a container as a Docker image.
• It also includes a run-time system that can create and manage containers using these
Docker images.

Benefits of containers
• They solve the problem of software dependencies.
•Instead of shipping your product as stand-alone software, you can ship a container that includes all
of the support software that your product needs.
• They provide a mechanism for software portability across different clouds.
•Docker containers can run on any system or cloud provider where the Docker daemon is available.
• They provide an efficient mechanism for implementing software services and so support the
development of service-oriented architectures.
• They simplify the adoption of DevOps.
•This is an approach to software support where the same team are responsible for both developing
and supporting operational software.
EVERYTHING AS A SERVICE
• The idea of a service that is rented rather than owned is fundamental to cloud computing.
Infrastructure as a service (IaaS)
•Cloud providers offer different kinds of infrastructure service such as a compute service, a
network service and a storage service that you can use to implement virtual servers.
Platform as a service (PaaS)
•This is an intermediate level where you use libraries and frameworks provided by the cloud provider
to implement your software.
• These provide access to a range of functions, including SQL and NoSQL databases.
Software as a service (SaaS)
• Your software product runs on the cloud and is accessed by users through a web browser or mobile
app.

14
 SOFTWARE AS A SERVICE
•Increasingly, software products are being delivered as a service, rather than installed on
the buyer’s computers.
• If you deliver your software product as a service, you run the software on your servers, which
you may rent from a cloud provider.
•Customers don’t have to install software and they access the remote system through a
web browser or dedicated mobile app.
• The payment model for software as a service is usually a subscription model.
• Users pay a monthly fee to use the software rather than buy it outright.

15
 Advantages and disadvantages of SaaS for customers

Data storage and management issues for SaaS

Design issues for SaaS

1. Local/remote processing
•A software product may be designed so that some features are executed locally in the user’s browser
or mobile app and some on a remote server.
• Local execution reduces network traffic and so increases user response speed.
• This is useful when users have a slow network connection.

16
• Local processing increases the electrical power needed to run the system.
2. Authentication
•If you set up your own authentication system, users have to remember another set of
authentication credentials.
• Many systems allow authentication using the user’s Google, Facebook or LinkedIn credentials.
•For business products, you may need to set up a federated authentication system, which
delegates authentication to the business where the user works.
3. Information leakage
•If you have multiple users from multiple organizations, a security risk is that information leaks
from one organization to another.
•There are a number of different ways that this can happen, so you need to be very careful in
designing your security system to avoid this.
4. Multi-tenant and multi-instance systems
•In a multi-tenant system, all customers are served by a single instance of the system and a
multitenant database.
• In a multi-instance system, a separate copy of the system and database is made available for each
user.

MULTI-TENANT SYSTEM
•A multi-tenant database is partitioned so that customer companies have their own space and can
store and access their own data.
•There is a single database schema, defined by the SaaS provider, that is shared by all of the
system’s users.
•Items in the database are tagged with a tenant identifier, representing a company that has stored data
in the system.
•The database access software uses this tenant identifier to provide ‘logical isolation’, which means
that users seem to be working with their own database.

MULTI-INSTANCE SYSTEMS
•Multi-instance systems are SaaS systems where each customer has its own system that is adapted to
its needs, including its own database and security controls.
•Multi-instance, cloud-based systems are conceptually simpler than multi-tenant systems and
avoid security concerns such as data leakage from one organization to another.
• There are two types of multi-instance system:
•VM-based multi-instance systems are multi-instance systems where the software instance
and database for each customer runs in its own virtual machine.
• All users from the same customer may access the shared system database.
• Container-based multi-instance systems
•These are multi-instance systems where each user has an isolated version of the software and
database running in a set of containers.
•This approach is suited to products in which users mostly work independently, with relatively
little data sharing.
•Therefore, it is best used for software that serves individuals rather than business customers or
for business products that are not data-intensive.

MICROSERVICES
• Microservices are small-scale, stateless, services that have a single responsibility.
• They are combined to create applications.
• They are completely independent with their own database and UI management code.
• Software products that use microservices have a microservices architecture. 17
 •If you need to create cloud-based software products that are adaptable, scaleable and resilient then
design them around a microservices architecture.

A microservice example
System authentication
•User registration, where users provide information about their identity, security information,
mobile (cell) phone number and email address.
• Authentication using UID/password.
• Two-factor authentication using code sent to mobile phone.
• User information management e.g. change password or mobile phone number.
• Reset forgotten password.
•Each of these features could be implemented as a separate service that uses a central shared database
to hold authentication information.
• However, these features are too large to be microservices.
•To identify the microservices that might be used in the authentication system, you need to break
down the coarse-grain features into more detailed functions.

Characteristics of microservices

18
 MICROSERVICES ARCHITECTURE
•A microservices architecture is an architectural style – a tried and tested way of implementing a
logical software architecture.
• This architectural style addresses two problems with monolithic applications
•The whole system has to be rebuilt, re-tested and re-deployed when any change is made. This can be
a slow process as changes to one part of the system can adversely affect other components.
•As the demand on the system increases, the whole system has to be scaled, even if the demand is
localized to a small number of system components that implement the most popular system
functions. A microservices architecture for a photo-printing system

Example:

Microservice architecture key design questions

MICROSERVICE DEPLOYMENT
•After a system has been developed and delivered, it has to be deployed on servers, monitored
for problems and updated as new versions become available.
• When a system is composed of tens or even hundreds of microservices, deployment of the
system is
19
more complex than for monolithic systems.
•The service development teams decide which programming language, database, libraries and
other support software should be used to implement their service. Consequently, there is no
‘standard’
deployment configuration for all services.
•It is now normal practice for microservice development teams to be responsible for deployment
and service management as well as software development and to use continuous deployment.
•Continuous deployment means that as soon as a change to a service has been made and validated,
the modified service is redeployed.
Deployment automation
•Continuous deployment depends on automation so that as soon as a change is committed, a series
of automated activities is triggered to test the software.
•If the software ‘passes’ these tests, it then enters another automation pipeline that packages
and deploys the software.
•The deployment of a new service version starts with the programmer committing the code changes to
a code management system such as Git.
•This triggers a set of automated tests that run using the modified service. If all service tests
run successfully, a new version of the system that incorporates the changed service is created.
• Another set of automated system tests are then executed. If these run successfully, the service
is ready
for deployment.

20