Lecturer: Assoc. Prof. PhD R.

Deliyski
 INTRODUCTION TO THE SOFTWARE TESTING
 DEFINITIONS AND SOFTWARE LIFE CYCLE ACCORDING
TO ISO 29119-1
 TEST PROCESSES ACCORDING TO ISO 29119-2
 REFERENCES

Slide 2/52
INTRODUCTION TO THE SOFTWARE TESTING

ISO/IEC/IEEE 29119 – Software and System Engineering – Software

Testing

The aim of the standard is to define internationally recognized rules

and principles for software testing, that can be used by any
organization working in this problem.

The standard is applicable for all types of software products and

systems.

ISO/IEC/IEEE 29119 consists of 5 parts:

ISO/IEC/IEEE 29119-1: Concepts & Definitions
ISO/IEC/IEEE 29119-2: Test Processes
ISO/IEC/IEEE 29119-3: Test Documentation
ISO/IEC/IEEE 29119-4: Test Techniques
ISO/IEC/IEEE 29119-5: Keyword-Driven Testing

Slide 3/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Testing – set of activities conducted to facilitate discovery and/or
evaluation of properties of one or more test items.

The primary goal of the testing is: to provide information about the
quality of the test item and any residual risk in relation to how much
the test item has been tested; to find defects in the test item prior to
its release for use; and to migrate the risk to the stockholders of poor
product quality.

Slide 4/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Software testing is necessary because:

• Information on the quality characteristics of the test item(s) is

required by decision makers;

• The test item(s) being tested does not always do what is expected
to do;

• The test item(s) being tested needs to be verified;

• The test item(s) being tested needs to be validated;

• Evaluation of the test item(s) needs to be conducted throughout the

software and system development life cycle.

Slide 5/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Test item (object) – work product that is an object of testing.

Test result – indication of whether or not a specific test case has

passed or failed, i.e. if the actual result observed as test item output
corresponds to the expected result or if deviations were observed.

Actual results – set of behavior or conditions of a test item, or set of

conditions of associated data or test environment, observed as a
result of test execution.

Expected results – observable predicted behavior of a test item

under specified conditions based on its specification or another
source.

Slide 6/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Decision – types of statements in which a choice between 2 or more
possible outcomes controls which set of actions will result.

Pass/fail criteria – decision rule used to determine whether a test

item, or feature of a test item, has passed or failed after testing.

Test execution - process of running a test on the test item,

producing actual result(s).

Re-testing (confirmation testing) - re-execution of test cases that

previously returned “fail” result, to evaluate the effectiveness of
intervening corrective actions.

Slide 7/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Test case – set of test preconditions, inputs (including actions,
where applicable), and expected results, developed to drive the
execution of a test item to meet test objectives, including correct
implementation, error identification, checking quality, and other
valued information.

Test data – data created or selected to satisfy the input

requirements for executing one or more test cases, which may be
defined in a test plan, test case or test procedure.

Test basis – body of knowledge used as the basis for design of test
cases

Slide 8/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Dynamic testing – testing that requires the execution of test items.

Static testing – testing in which a test item is examined against a

set of quality or other criteria without code being executed.

Risk-based testing – testing in which the management, selection,

prioritization, and use of testing activities and resources are
consciously based on corresponding types and levels of analyzed
risk.

Regression testing – testing following modifications to a test item or

to its operational environment, to identify whether regression failures
occurs.

Slide 9/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Specification-based testing (black-box testing) – testing in which
the principal test bases is the external input and outputs of the test
item, commonly based on a specification, rather than its
implementation in source code or executable software.

Structure-based testing (white-box testing) – dynamic testing in

which the tests are derived from an examination of the structure of
the test item.

Slide 10/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Test plan – detailed description of test objectives to be achieved and
the means and schedule for achieving them, organized to coordinate
testing activities for some test item or set of test items.

The project test plan describes the overall strategy for testing and
the test processes to be used. It established the context of testing for
the project by determining the objectives, practices, resources, and
schedule.

The test plan also describes the appropriate test design techniques
(static or dynamic) to use in order to complete the testing required by
the sub-process plan.

Slide 11/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Test procedure – sequence of test cases in execution order, and
any associated actions that may be required to set up the initial
preconditions and any wrap up activities post execution.

Test strategy – part of the test plan that describes the approach to
testing for a specific test project or test sub-process or sub-
processes

Slide 12/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Test process – used to provide information on the quality of a
software product, often comprised of a number of activities, grouped
into one or more test sub-processes.

Testing as a process could be planned, monitor and control.

Test Sub-process:

• Test Objectives
• Test Item
• Testing of Quality Characteristics
• Test Basis
• Retesting and Regression Testing
• Test Design Techniques (static and dynamic)

Slide 13/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Development of sub-processes:

Requirements Architectural Detailed

Coding Integration Acceptance
engineering design design

Requirements Architectural Unit 1 Coding Sub-system 1 Accepted

design Design Unit 1 system

Unit 2 Coding Sub-system 2

Design Unit 2

Unit n Coding Sub-system n

Design Unit n

Completed
system

Slide 14/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Test management – planning, scheduling, estimating, monitoring,
reporting, control and completion of test activities.

Test techniques – activities, concepts, processes, and patterns

used to construct a test model that is used to identify test conditions
for a test item, derive corresponding test coverage items, and
subsequently derive or select test cases.

Slide 15/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS

Slide 16/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Software Life cycle models:

• Agile - It is based on iterative and incremental development, where

requirements and solutions evolve through collaboration between
cross-functional teams.

• Evolutionary - prototypes that evolve into the final system through

an iterative incorporation of user feedback.

• Sequential (waterfall model) - a linear sequential flow. In which

progress is seen as flowing steadily downwards (like a waterfall)
through the phases of software implementation. This means that any
phase in the development process begins only if the previous phase
is complete. The waterfall approach does not define the process to
go back to the previous phase to handle changes in requirement.

Slide 17/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Test Practices
Requirements-Based Testing
• to ensure that the requirements of the test item have been addressed
(covered) during testing, to determine whether the test item meets end-user
requirements.
• It is also used to collect and provide other valuable information to
stakeholders during the life cycle, such us defects identified in the test item.
• Using requirement-based testing is highly dependent on the context.

Model-Based Testing
• Model of the expected behavior
• With model-based testing the model has to be formal enough and/or
detailed enough so that useful test information can be derived from the
model.

Slide 18/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Test Practices
Mathematical-Based Testing
Mathematical-based test practices can be used to plan, design, select data
and set-up input conditions when the test item input or output space can be
described in sufficient detail.

• Combinatorial testing;
• Random test case selection;
Statistical modeling as:
• Fuzzy testing
• Stratified sampling;
• Cluster sampling;
• Expert judgment sapling.

Mathematical-based test practices can also be used to objectively sample

the test case space using mathematically-based tools and techniques.

Slide 19/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Test Practices
Experience-Based Testing (trial and error)

Previous testing experience

• Knowledge of particular software and systems;

• Domain knowledge;
• Metrics from previous projects.

Scripted and Unscripted Testing (props and negs)

Slide 20/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Verification – conformation, through the provision of objectives
evidence, that specified requirements have been fulfilled in a given
work item.

Validation – demonstrates that the work item can be used by the

users for their specific tasks.

ISO 29119 addresses to some verification and validation activities.

Specifically it addresses software testing, which is major activity of V&V.
Other standards (ISO/IEC 12207 and IEEE 1012) address other V&V
activities

Slide 21/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS

Slide 22/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS
Automation in Testing

Next processes can be automated:

• Test case management

• Test monitoring and control
• Test data generator
• Static analysis
• Test case generator
• Test case execution
• Test environmental implementation and execution

Annex D 29119-1 примери подходящи за упражнения

Slide 23/52
ISO/IEC/IEEE 29119-1: CONCEPTS AND
DEFINITIONS

Slide 24/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

The aim of ISO/IEC/IEEE 29119-2 is to define a generic process

model for software testing that can be used within any software
development life cycle. The model specifies test processes that can
be used to govern, manage and implement software testing in any
organization, project or testing activity. The testing process is based
on a three-layer process model that covers:

• Organizational Test Specifications

• Test Management
• Dynamic Testing

A risk-based approach to testing is used throughout the standard.

Risk-based testing is a best-practice approach to strategizing and
managing testing, as it allows testing to be prioritized and focused on
the most important features and quality attributes of each system
under test.

Slide 25/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Multy-layer Test Process Model

Slide 26/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Organizational test processes – to define a process for the

creation and maintenance of organizational test specifications, such
as organizational test policies, strategies, processes, procedures
and other assets.

Test management processes – to define processes that cover the

management of testing for a whole test project or any test phase
(system testing) or test type (performance testing) within a test
project (project test management, system test management,
performance test management).

Dynamic test processes – to define processes for performing

dynamic testing. Dynamic testing may be performed at a particular
phase of testing (unit, integration, system, and acceptance) or for a
particular type of testing (performance testing, security testing, and
functional testing) within a test project.

Slide 27/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Organizational Test Processes

Slide 28/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Organizational Test Processes

Organizational test policy – an executive-level document that

describes the purpose, goals, and overall scope of testing within the
organization. It is also established organizational testing practices
and provide a framework for establishing, reviewing and continually
improving the organization's Test policy, Test strategy and approach
to project test management.

Organizational test Strategy – detailed technical document that

defines how testing is carried out within organization. It is a generic
document that provides guidelines for a number of projects in the
organization and is non project-specific.

Slide 29/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Organizational Test Processes

Slide 30/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Organizational Test Processes

Outcomes:
• The requirements for organizational test specifications are
identified;
• The organizational test specifications are developed;
• The organizational test specifications are agreed to by the
stakeholders;
• The organizational test specifications are made accessible;
• Conformance to the organizational test specifications is monitored;
• Updates to organizational test specifications are agreed to by
stakeholders;
• Updates to the organizational test specifications are made.

Slide 31/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Test Management Processes

Slide 32/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Test Management Processes

Test Planning Process

Used to develop, agree, record and communicate to relevant
stakeholders the scope and approach that will be taken to testing,
enabling, early identification of resources, environments and other
requirements of testing.

Outcomes:
• The scope of work of the test project is analyzed and understood;
• The stakeholders who will participate in the test planning are
identified and informed;
• Risks that can be treated by testing are identified, analyzed and
classified with an agreed level of risk exposure;
• Test strategy, test environment, test tools and test data are
identified.

Slide 33/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Slide 34/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Test Management Processes

Test Monitoring And Control Process

Used to determine whether testing processes are in accordance with
the Test Plan and with organizational test specifications
(Organizational test policy and Organizational test strategy). It is also
initiates control actions as necessary and identifies necessary
updates to the Test Plan (revise completion criteria or identify new
actions to compensate for deviations from the Test Plan).

It is also used to determine whether testing progresses are in

accordance with higher level test plans, such as the Project Test
Plan, and to manage the testing performed at specific test phases
(system testing) or for specific test types (performance testing).

Slide 35/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Slide 36/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Test Management Processes

Test Monitoring And Control Process

Outcomes:
• The means of collecting suitable measures to monitor test process
and changing risk are set up;
• Progress against the test plan is monitored;
• New and changed test-related risks are identified, analyzed and
necessary action(s) invoked;
• Necessary control actions are identified;
• Necessary control actions are communicated to the relevant
stakeholders;
• The decision to stop testing is approved;
• Test progress and changes to the risks are reported to the
stakeholders.

Slide 37/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Test Management Processes

Test Completion Process

Used to make available useful test assets for later use, leave the test
environment in a satisfactory condition, and record and communicate
the results of the testing to relevant stakeholders. Test assets
include:

• Test Plans,
• Test Case specifications,
• Test scripts,
• Test tools,
• Test data,
• Test environment infrastructure.

Slide 38/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Test Management Processes

Slide 39/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Test Management Processes

Test Completion Process

Outcomes:
• Test assets are either archived or passed directly to the relevant
stakeholders;
• The test environment is in its agreed state (so that it is available for
the next project)
• All test requirements are satisfied and verified;
• The test Completion Report is recorded;
• The test Completion Report is approved;
• The test Completion Report is communicated to relevant
stakeholders.

Slide 40/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Dynamic Test Process

Slide 41/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Dynamic Test Process

Test Design And Implementation Process

Used to derive test procedures that will be executed during the Test
Execution Process. As a part of this process the test basis is
analyzed, features are combined into feature sets, test conditions,
test coverage items, test cases, test procedures are derived and test
sets are assembled (used to derive test cases and test procedures).

Outcomes:
• The Test bases for each test item is analyzed;
• The features to be tested are combined into Feature Sets;
• The Test Conditions are derived;
• The Test Coverage Items are derived;
• Test cases are derived;
• Test sets are assembled;
• Test Procedures are derived.
Slide 42/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Slide 43/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Dynamic Test Process

Test Environment Set-up & Maintenance Process

Used to establish and maintain the required test environment and to
communicate its status to all relevant stakeholders.

Outcomes:
• The Test environment is set-up in a state ready for testing;
• The status of test environment is communicated to all relevant
stakeholders;
• The test environment is maintained.

Slide 44/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Dynamic Test Process

Slide 45/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Dynamic Test Process

Test Execution Process

Used to execute the test procedures created in the Test Design and
Implementation Process in the prepared test environment and record
the results.

Outcomes:
• The Test Procedure/s is/are executed
• The actual results are recorded
• The actual and expected results are compared
• The test results are determined.

Slide 46/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Dynamic Test Process

Test Execution
Process
Test
Test Test Execution
Execution Test Outcome Compare Results Record Test Log
Procedure(s) Test Results Execution

Inputs activities in this process may include:

- Test plan(s);
- Test procedures(s);
- Test item;
- Test bases;
- Test environment readiness report (where available);
- Test environment updates (where available)

Slide 47/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Dynamic Test Process

Test Incident Reporting Process

Used to report to the relevant stakeholders incidents requiring further

action identified as a result of test execution. In the case of a new
test this will require an incident report to be created. In the case of a
retest, this will require the status of a previously-raised incident
report to be updated, but may also require a new incident report to
be raised where further incidents are identified.

Slide 48/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Dynamic Test Process

Test Incident
Reporting Process
Incident
Report Incident
Updates Report
Analyze Test Create/Update
Results Incident Report

Inputs
- Test results;
- Test procedures(s);
- Test cases;
- Test item;
- Test bases;
- Test execution log (where available)

Slide 49/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

Dynamic Test Process

Test Incident Reporting Process

Outcomes:
• Test results are analyzed;
• New incidents are confirmed;
• New incident report details are created;
• The status and details of previously-raised incidents are
determined;
• Previously-raised incident report details are updated as appropriate
• New and/or updated incident reports are communicated to the
relevant stakeholders.

Slide 50/52
ISO/IEC/IEEE 29119-2: TEST PROCESSES

ISO/IEC/IEEE 29119-2 Annex A. Partial Example Test Design

Process

За упражнения

Slide 51/52
REFERENCES

ISO/IEC/IEEE 29119 – Software and System Engineering – Software

Testing

https://www.slideshare.net/fdgiraldo/iso-29119-and-sw-testing

https://www.ida.liu.se/~TDDB62/www-pub-01/lectures/testing.pdf

https://melsatar.wordpress.com/2012/03/15/software-development-life-
cycle-models-and-methodologies/

Slide 52/52