The History of

Software Testing
Software Testing Introduction
20 th Century: 50 th -60 th

Mathematical Exhaustive
Debugging
Approach Testing

This is not the “Testing”, this is just some sort of activity during
the Development

50th-60th 70th 80th 90th 2000th Now

2
20 th Century: 70 th

Prove that software works Prove that software fails

(Positive Testing) (Negative Testing)

Both first attempts failed, but the basis for modern approaches
has been built

50th-60th 70th 80th 90th 2000th Now

3
20 th Century: 80 th

Lifecycle Error Prevention First Automation

Integration Approach Attempts

Software Testing is recognized globally. A lot of

methodologies, approaches, standards, etc. appears.

50th-60th 70th 80th 90th 2000th Now

4
20 th Century: 90 th

Quality Integration with Numerous Tools

Assurance Development and Systems

For the first time Software Testing looks a lot like today

50th-60th 70th 80th 90th 2000th Now

5
21 st Century Beginning

TDD and other Automation

Agile Testing
approaches Everywhere

Software Testing becomes inseparable part of software

development process

50th-60th 70th 80th 90th 2000th Now

6
Modern Situation

Testing as a part Testing as a basis

Agile Testing
of… for…

Software Testing becomes a part of Development, DevOps and

other activities. It becomes a basis for CI/CD/etc. approaches.

50th-60th 70th 80th 90th 2000th Now

7
More About Our Days

8
The History of
Software Testing
Software Testing Introduction
Why Software
Testing is Important
and What to Test
Software Testing Introduction
For Individual Users (and B2C Businesses)

Years ago Now

You want a Tool? Please,

You want a Tool? Go write it
select from the following
yourself!
100500 options.

2
For Enterprise (and B2B Businesses)

Individual Users Enterprise Customers

They count. And count. And

Do like quality, but often fall
count again. And again. The
victims of aggressive
quality affects this
marketing.
calculations essentially.

3
For Everyone

Trains, airplanes, Nuclear power Medical

ships, etc… plants… equipment…

Today software is literally EVERYWHERE. Smartphone failure is

bad, but airplane crash is incomparably worse.

4
What to Test: Main Testing Areas

We can test…

Literally EVERYTHING

5
What to Test: Main Testing Areas

We can test…

Software Code Prototype Documentation

6
Why Software
Testing is Important
and What to Test
Software Testing Introduction
Soft Skills and Hard Skills
of Software Tester

Software Testing Introduction

Soft Skills

Enhanced responsibility
Good communication skills
Presentation skills
Performance and goal oriented approach
Observation and attentiveness to details
Flexible thinking and ability to learn fast
Good abstract and analytical thinking
Tendency to experimenting
2
Hard Skills

Good English (B2+)

Programming (including web and mobile development) basics
Databases (and SQL) basics
OS administration basics
Network administration basics
Virtualization and cloud computing basics
Automated testing basics
General IT knowledge
3
Hard Skills

Good English
General IT knowledge

This is the ultimate minimum to start with!

4
Soft Skills and Hard Skills
of Software Tester

Software Testing Introduction

Basic Software
Testing Terminology
Software Testing Introduction
Testing and Quality

Quality

Quality
Management Testing

Quality Quality
Assurance Control

2
Testing and Quality by Example
Quality
Healthy Patient

Quality
Management
Treatment
Quality Quality
Assurance Control
Medication and Ongoing
Procedures Diagnostics

3
Testing and Quality Read and remember!

Testing – the process concerned with planning, preparation

and evaluation of software products and related work products
to determine that they satisfy specified requirements, to
demonstrate that they are fit for purpose and to detect defects.

Quality – the degree to which a component, system or process

meets specified requirements and/or user/customer needs and
expectations.

4
Testing and Quality Read and remember!

Quality management – coordinated activities to direct and

control an organization with regard to quality.

Quality assurance – part of quality management focused on

providing confidence that quality requirements will be fulfilled.

Quality control – the operational techniques and activities,

part of quality management, that are focused on fulfilling
quality requirements.

5
Defect, Expected Result, Actual Result

Expected result

Defect

Actual result

6
Defect, Expected Result, Actual Result Read and remember!

Defect – an imperfection or deficiency in a work product where

it does not meet its requirements or specifications.

Expected result – the predicted observable behavior of a

component or system executing under specified conditions,
based on its specification or another source.

Actual result – the behavior produced/observed when a

component or system is tested.

7
Defect Report: Sample

8
Checklist, Test Case, Test Suite

Checklist

Test Cases

Test Suites

9
Checklist, Test Case, Test Suite Read and remember!

Checklist – a set of ideas.

Test case – a set of preconditions, inputs, actions (where
applicable), expected results and postconditions, developed
based on test conditions.

Test suite – a set of test cases or test procedures to be

executed in a specific test cycle.

10
Checklist: Sample

11
Test Case: Sample

12
Test Suite: Sample

13
Please, visit this URL

https://glossary.istqb.org

14
Basic Software
Testing Terminology
Software Testing Introduction
Software
Development Models
and Testing
Software Testing Introduction
Disclaimer

This is NOT a detailed review or a

comparison of Software Development
Models. This is just a quick answer to
“Where is the Testing in the … Model?”

2
Waterfall

General Planning

User Requirements

System Requirements

Technical Architecture

Detailed Design

Coding and Debugging

Integration- and Unit-Testing

Testing (as we now understand it)
comes to life barely in the middle Installation Testing
of the project lifecycle. Testing
System Testing
reaches its maximum at the end
of the project. Usually – too late. Acceptance Testing

Final Reporting

3
V-Model

Testing starts from the very

General Planning Final Reporting
beginning but it is
concentrated mostly on the
User Requirements transition between stages. Acceptance Testing

System Requirements System Testing

Technical Architecture Installation Testing

Detailed Design Integration- and Unit-Testing

Coding and Debugging

4
Iterative Incremental Model

Final Reporting General Planning

Detailed Planning and

Reporting
Req-s Analysis

Architecture and With each and every cycle-based

Result Evaluation
Design
model Testing becomes “a part
of every action” (although there
System Testing Coding and Debugging are dedicated stages).

Integration- and Unit-

Build Setup
Testing

5
Agile Model

With each and every cycle-based

model Testing becomes “a part
Iteration of every action” (although there
(24 h) are dedicated stages).
Project Backlog Sprint Backlog

Sprint
(2-4 weeks)

Deliverable

6
And what about…

With non-iterative models Testing

RAD Model lies within dedicated stages and
between stages.

Spiral Model
With iterative models Testing not
only has dedicated stages, but fills
Prototype Model any other activities like air or
water.

Any Other Model With such approach Testing helps

us to find defects ASAP or even to
prevent them.

7
Software
Development Models
and Testing
Software Testing Introduction
Software Testing
Lifecycle
Software Testing Introduction
General Overview General planning
and requirements
analysis

1 Acceptance criteria
establishment
Test results
reporting 2

Test results 8
analysis
Test strategy
3
establishment
7

6
4
Defect reporting
5 Test cases creation

Test cases
execution

2
General planning and requirements analysis
General planning
and requirements
analysis

1. General planning and requirements analysis

Here we have to find out: what to test; how much work is ahead;
what difficulties we may face; do we have all necessary resources; are
requirements good enough.
3
Acceptance criteria establishment
Acceptance criteria
establishment

2. Acceptance criteria establishment

Here we have to establish or adjust metrics and criteria for test

process to start, pause, resume, complete or abort. We also have to
know key quality criteria and goals for the current test cycle.
4
Test strategy establishment

Test strategy
establishment

3. Test strategy establishment

Once again we return to planning to find out HOW shall we achieve

all those goals and criteria from the previous step. Here we speak
about approaches, tools, schedule, roles, responsibility and so on.
5
Test cases creation

Test cases creation

4. Test cases creation

Here we create, review, adjust, rework (and so on) checklists, test

cases, test scenarios and other similar artifacts.

6
Test cases execution and defect reporting

Defect reporting

Test cases execution

5. Test cases execution

6. Defect reporting

These two stages are inseparable as we report defects during test

cases execution and defects detection.
7
Test results analysis and reporting
Test results reporting

Test results analysis

7. Test results analysis

8. Test results reporting

Here we analyze if we managed to achieve the goals set during stages

1-3. The result of such analysis is presented in Test Result Report and
used as the basis for the planning of the next test cycle.
8
Software Testing
Lifecycle
Software Testing Introduction
Software Testing
Classification
Software Testing Introduction
Two Fundamental Approaches: Functional and Non-Functional Testing

Testing

Functional Testing Non-Functional Testing

Functional Non-Functional
Requirements Requirements

What to do (e.g. How to do (e.g. how

encrypt, compress, fast, how reliable, how
send, archive, etc) secure, etc)
2
Functional and Non-Functional Testing Read and remember!

Functional testing – testing conducted to evaluate the

compliance of a component or system with functional
requirements. Functional requirement – a requirement that specifies a
function that a component or system must be able to
perform.

Non-functional testing – testing conducted to evaluate the

compliance of a component or system with non-functional
requirements. Non-functional requirement – a requirement that describes
how the component or system will do what it is intended to
do.
3
General Classification Approach

Classification by…
Static testing Manual
Code execution Automation level
Dynamic testing Automated

White-box testing Extended testing

Access to application By functions under test
code and architecture importance
Black-box testing Critical path testing

Gray-box testing Smoke testing

System testing Positive testing

By specification level By ways of dealing with
(by testing level) application
Integration testing Negative testing

Unit testing

4
General Classification Approach: There Is Much More!

Please, visit:
http://svyatoslav.biz/urls/stc_en/

5
Classification by code execution

Classification by…
Static testing Manual
Code execution Automation level
Dynamic testing Automated

White-box testing Extended testing

Access to application By functions under test
code and architecture importance
Black-box testing Critical path testing

Gray-box testing Smoke testing

System testing Positive testing

By specification level By ways of dealing with
(by testing level) application
Integration testing Negative testing

Unit testing

6
Classification by code execution Read and remember!
Static testing
Code execution
Dynamic testing

Static testing – testing a work product without code being

executed.

Dynamic testing – testing that involves the execution of the

software of a component or system.

7
… by access to application code … Read and remember!
White-box testing
Access to application
code and architecture
Black-box testing

Gray-box testing

White-box testing – testing based on an analysis of the internal

structure of the component or system.
Black-box testing – testing, either functional or non-functional, without
reference to the internal structure of the component or system.
Gray-box testing – testing, which is a combination of Black-box and
White-box (i.e. the internal structure is partially known).
8
Classification by specification level Read and remember!
System testing
By specification level
(by testing level)
Integration testing

Unit testing

Unit testing – the testing of individual hardware or software

components.
Integration testing – testing performed to expose defects in the
interfaces and interactions between integrated components or systems.
System testing – testing an integrated system to verify that it meets
specified requirements.
9
Classification by automation level Read and remember!
Manual
Automation level
Automated

Manual testing – testing performed by the tester who carries

out all the actions on the tested application manually.

Automated testing – the use of software to control the

execution of tests, the comparison of actual outcomes to
predicted outcomes, the setting up of test preconditions, and
other test control and test reporting functions.

10
… by functions under test importance Read and remember!
Extended testing
By functions under test
importance
Critical path testing

Smoke testing

Smoke test – a subset of all defined test cases that cover the main
functionality of a component or system, the most crucial functions.
Critical path test – test cases that cover the functionality used most of
the time by the majority of users.
Extended test – test cases that cover the “nice-to-have” functionality
(not used most of the time by the majority of users).
11
… by ways of dealing with application Read and remember!
Positive testing
By ways of dealing with
application
Negative testing

Positive testing – testing process where the system validated

against the valid input data and valid set of actions.

Negative testing – tests aimed at showing that a component or

system does not work.

12
Let’s look at the big picture one more time…

Classification by…
Static testing Manual
Code execution Automation level
Dynamic testing Automated

White-box testing Extended testing

Access to application By functions under test
code and architecture importance
Black-box testing Critical path testing

Gray-box testing Smoke testing

System testing Positive testing

By specification level By ways of dealing with
(by testing level) application
Integration testing Negative testing

Unit testing

13
Software Testing
Classification
Software Testing Introduction
Test Planning and
Its Tasks and Goals
Software Testing Introduction
Main Definitions Read and remember!

Test planning – the activity of establishing or updating a test

plan.

Test plan – documentation describing the test objectives to be

achieved and the means and the schedule for achieving them,
organized to coordinate testing activities.

2
Planning Tasks and Goals (part 1)

Planning Tasks and Goals

Designation of
Estimation of work Definition of schedule
resources and ways to
scope and complexity and milestones
acquire them

Risk mitigation and Coordination of team-

Distribution of duties
countermeasures effort between
and responsibilities
definition teams/projects/groups

3
Why is Test Plan so important? General planning
and requirements
analysis
Acceptance
1
Test Plan is the criteria
establishment
basis for all further Test results
reporting 2
activities, it is a
8
map to the Test results
analysis
defined 3 Test strategy
establishment
7
destination, a
starting point to 6
4
the reporting and Defect reporting
so on and so 5 Test cases creation

forth… Test cases

execution

4
Planning Tasks and Goals (part 2)

Test Plan helps us to…

Get better results with Optimize resources See the progress at any
less effort usage, avoid waste given moment

Get better
Understand what to do
understanding between Cooperate better
in any situation
people

5
Test Planning and
Its Tasks and Goals
Software Testing Introduction
Test Plan
Sections
Software Testing Introduction
Main Definition Read and remember!

Test plan – documentation describing the test objectives to be

achieved and the means and the schedule for achieving them,
organized to coordinate testing activities.

2
General Test Plan Sections Overview

Project scope and main goals Schedule

Requirements to be tested Roles and responsibilities

Requirements NOT to be tested Risk evaluation

Test strategy and approach Documentation

Criteria Metrics

Resources …

3
Project scope and main goals Read and remember!

Project scope and main goals – a very brief description of the

purpose of the application development.
This section is reminiscent of business requirements, but here the information
is presented in an even more condensed form with an emphasis on the most
important tasks.

E.g.: Correct automated conversion of text documents in different source

encodings to one destination encoding with performance significantly
higher than human performance during the same actions.

4
Requirements to be tested Read and remember!

Requirements to be tested – the list of functional and / or non-

functional requirements to be tested.
Not each and every requirement should be tested. Here we list those ones that
should be covered with test-cases.

E.g.: • UR-1.*: smoke test.

• UR-2.*: smoke test, critical path test.
• UR-3.*: critical path test.
• BR-1.*: smoke test, critical path test.
• QA-2.*: smoke test, critical path test
5
Requirements NOT to be tested Read and remember!

Requirements NOT to be tested – the list of functional and / or

non-functional requirements NOT to be tested.
As not each and every requirement should be tested, here we list those ones
that should NOT be covered with test-cases. For each requirement we write
the reason why shouldn't we test it.

E.g.: • SC-1: the application is a console one by design.

• SC-2, L-1: the application is developed with proper PHP version.
• QA-1.1: this performance characteristic is at the bottom border of typical
operations performance for such applications.

6
Test strategy and approach Read and remember!

Test strategy and approach – the description of the testing

process in terms of the methods used, approaches, types of
testing, technologies, tools, etc.
This description allows us to use the most effective and efficient way to achieve
project goals in terms of quality.
E.g.: Due to the team cross-functionality, a significant contribution to quality
improvement can be expected from the code review combined with
manual testing using the white box method. Unit-testing will not be
applied due to extreme time limitations.

7
Criteria Read and remember!

Criteria – the list of miscellaneous testing criteria, such as

acceptance criteria, entry criteria, suspension criteria,
resumption criteria, exit criteria, etc.
Usually, each criterion has a reference to proper metric. Otherwise the decision
on criterion fulfillment may be too subjective.
E.g.: Testing resumption criteria: more than 50% of bugs found during the
previous iteration are fixed (see “Ongoing defects fixed percentage”
metric).

8
Resources Read and remember!

Resources – the list of miscellaneous project resources, such as

software, hardware, staff, time, finance, etc.
Usually, finance resources are too confidential to be included in test-plan, so
only a reference to the budget is mentioned here.

E.g.: • Software: four virtual machines (two with Windows 10 Ent x64, two with
Linux Ubuntu 18 LTS x64), two PHP Storm licenses (latest version
available).
• Hardware: two standard workstations (8GB RAM, i7 3GHz).

9
Schedule Read and remember!

Schedule – in fact, this is a kind of calendar with milestones

and periods marked on it.
When the project is long enough (months and years long) there is no need to
have a detailed schedule for the far future: the schedule may be adjusted and
filled with details once we reach one milestone or another.

E.g.: • 25.05 – requirements testing and finalizing.

• 26.05 – test-cases and scripts for automated testing creation.
• 27.05-28.05 – main testing stage.
• 29.05 – testing finalization, reporting.

10
Roles and responsibilities Read and remember!

Roles and responsibilities – the list of all key project roles

(related to the testing process) with their key areas of
responsibility.
Though this list is rather unified for most projects, there may be some special
cases, so this section usually is not omitted.
E.g.: • Senior developer: participation in requirements testing and code review.
• Tester: documentation creation, test-cases execution, participation in
code-review.

11
Risk evaluation Read and remember!

Risk evaluation – the list of risks that are likely to arise during
the project. For each risk there is a probability evaluation and
some options for resolving the situation.
There are a lot of typical risks (actual for every project) so no need to list such
risks here. This section is for risks specific (or even unique) to the project.
E.g.: • Personnel (low probability): if any team member is inaccessible, we can
contact the representatives of the “Cataloger” project to get a temporary
replacement (the commitment from the “Cataloger” PM John Smith was
received).

12
Documentation Read and remember!

Documentation – the list of test documentation with details on

who should prepare it, when, how, etc.
This section is extremely useful when onboarding a new project member. As
sometimes documentation list may consist of dozens of items, it’s good to have
them listed in one place.

E.g.: • Requirements. Responsible person – tester, deadline – 25.05.

• Test cases and defect reports. Responsible – tester, creation period –
26.05-28.05.
• Test result report. Responsible person – tester, deadline – 29.05.

13
Metrics Read and remember!

Metrics – the list of numerical characteristics of quality

indicators, methods for their evaluation, formulas, etc.
This section is actively referenced by “Criteria” section. Metrics have their dark
side too, but in general it is better to have some objective way to tell the
current project situation.

E.g.: Test cases success percentage:

𝑇 𝑆𝑢𝑐𝑐𝑒𝑠𝑠
𝑇 𝑆𝑃 = 𝑇 𝑇𝑜𝑡𝑎𝑙
∙ 100%, where

𝑇 𝑆𝑃 – percentage of successfully passed test cases,

𝑇 𝑆𝑢𝑐𝑐𝑒𝑠𝑠 – quantity of successfully passed test cases,
𝑇 𝑇𝑜𝑡𝑎𝑙 – total quantity of executed test cases.

14
Test Plan
Sections
Software Testing Introduction
 «File Converter» Project

Test Plan
SAMPLE
Project Documentation

Background Estimations, schedule, strategy, and metrics are needed to

organize the testing process efficiently.
Purpose To organize the testing process effective and efficient during
the whole project period.
Scope Testing process description, metrics, schedule, resources.
Audience Management staff, QA team, project team.
File 02 03 - Test Plan Sample.docx

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:12:23
 «File Converter» Project

Contents

1. Project scope and main goals ...................................................................... 3

2. Requirements to be tested ........................................................................... 3
3. Requirements NOT to be tested .................................................................. 3
4. Test strategy and approach ......................................................................... 3
4.1. General approach ................................................................................... 3
4.2. Functional testing levels ......................................................................... 3
5. Criteria .......................................................................................................... 4
6. Resources .................................................................................................... 4
7. Schedule ...................................................................................................... 4
8. Roles and responsibilities ............................................................................ 4
9. Risk evaluation ............................................................................................. 4
10. Documentation .......................................................................................... 5
11. Metrics ....................................................................................................... 5

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:12:23
 «File Converter» Project

1. Project scope and main goals

Correct automated conversion of text documents in different source encodings to one
destination encoding with performance significantly higher than human performance during
the same actions.

2. Requirements to be tested
See referenced sections in “File Converter Requirements.docx”:
 UR-1.*: smoke test.
 UR-2.*: smoke test, critical path test.
 UR-3.*: critical path test.
 BR-1.*: smoke test, critical path test.
 QA-2.*: smoke test, critical path test.
 L-4: smoke test.
 L-5: smoke test.
 DS-*: smoke test, critical path test.

3. Requirements NOT to be tested

See referenced sections in “File Converter Requirements.docx”:
 SC-1: the application is a console one by design.
 SC-2, L-1, L-2: the application is developed with proper PHP version.
 QA-1.1: this performance characteristic is at the bottom border of typical operations
performance for such applications.
 L-3: no implementation required.
 L-6: no implementation required.

4. Test strategy and approach

4.1. General approach
The application is to be configured once by an experienced specialist and later used
by end users, for whom only one operation is available – placing the file into the input
directory. Therefore, issues of usability, security, etc. not explored during testing.

4.2. Functional testing levels

 Smoke test: automated with batch files under Windows and Linux.
 Critical path test: executed manually.
 Extended test: not executed as the probability of defects detection on this level is
negligibly small.

Due to the team cross-functionality, a significant contribution to quality improvement

can be expected from the code review combined with manual testing using the white box
method. Unit-testing will not be applied due to extreme time limitations.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:12:23
 «File Converter» Project

5. Criteria
 Acceptance criteria: 100% success of test cases on smoke test level and 90%
success of test cases on critical path test level (see “Test cases success
percentage” metric) if 100% of critical and major bugs are fixed (see “Overall defects
fixed percentage” metric). Final requirements coverage by tests (see “Requirements
coverage by tests” metric) should be at least 80%.
 Testing start criteria: new build.
 Testing pause criteria: critical path test must begin only after 100% success of test-
cases on the smoke test (see “Test cases success percentage”); test process may
be paused is with at least 25% test-cases executed there is at least 50% failure rate
(see “Stop-factor” metric).
 Testing resumption criteria: more than 50% of bugs found during the previous
iteration are fixed (see “Ongoing defects fixed percentage” metric).
 Testing finish criteria: more than 80% planned for the current iteration test cases are
executed (see “Test-cases execution percentage”).

6. Resources
 Software: four virtual machines (two with Windows 10 Ent x64, two with Linux
Ubuntu 18 LTS x64), two PHP Storm licenses (latest version available).
 Hardware: two standard workstations (8GB RAM, i7 3GHz).
 Personnel:
o One senior developer with testing experience (100% workload during all
project time). Roles: team lead, senior developer.
o One tester with PHP knowledge (100% workload during all project time).
Role: tester.
 Time: one workweek (40 work hours).
 Finances: according to the approved budget.

7. Schedule
 25.05 – requirements testing and finalizing.
 26.05 – test-cases and scripts for automated testing creation.
 27.05-28.05 – main testing stage (test-cases execution, defect reports creation).
 29.05 – testing finalization, reporting.

8. Roles and responsibilities

 Senior developer: participation in requirements testing and code review.
 Tester: documentation creation, test-cases execution, participation in code-review.

9. Risk evaluation
 Personnel (low probability): if any team member is inaccessible, we can contact the
representatives of the “Cataloger” project to get a temporary replacement (the
commitment from the “Cataloger” PM John Smith was received).

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:12:23
 «File Converter» Project

 Time (high probability): the customer has indicated a deadline of 01.06, therefore
time is a critical resource. It is recommended to do our best to complete the project
by 28.05 so that one day (29.05) remains available for any unexpected issues.
 Other risks: no other specific risks have been identified.

10. Documentation
 Requirements. Responsible person – tester, deadline – 25.05.
 Test cases and defect reports. Responsible – tester, creation period – 26.05-28.05.
 Test result report. Responsible person – tester, deadline – 29.05.

11. Metrics
 Test cases success percentage:
𝑇 𝑆𝑢𝑐𝑐𝑒𝑠𝑠
𝑇 𝑆𝑃 = 𝑇 𝑇𝑜𝑡𝑎𝑙
∙ 100%, where

𝑇 𝑆𝑃 – percentage of successfully passed test cases,

𝑇 𝑆𝑢𝑐𝑐𝑒𝑠𝑠 – quantity of successfully passed test cases,
𝑇 𝑇𝑜𝑡𝑎𝑙 – total quantity of executed test cases.
Minimally acceptable borders:
o Beginning project phase: 10%.
o Main project phase: 40%.
o Final project phase: 80%.

 Overall defects fixed percentage:

𝐹𝑇𝑃 𝐷𝐶𝑙𝑜𝑠𝑒𝑑
𝐷𝐿𝑒𝑣𝑒𝑙 = 𝐷𝐿𝑒𝑣𝑒𝑙
𝐹𝑜𝑢𝑛𝑑 ∙ 100%, where
𝐿𝑒𝑣𝑒𝑙

𝐹𝑇𝑃
𝐷𝐿𝑒𝑣𝑒𝑙 – overall defects fixation percentage by 𝐿𝑒𝑣𝑒𝑙 during all project lifetime,
𝐶𝑙𝑜𝑠𝑒𝑑
𝐷𝐿𝑒𝑣𝑒𝑙 – quantity of defects of 𝐿𝑒𝑣𝑒𝑙 fixed during all project lifetime,
𝐹𝑜𝑢𝑛𝑑
𝐷𝐿𝑒𝑣𝑒𝑙 – quantity of defects of 𝐿𝑒𝑣𝑒𝑙 found during all project lifetime.

Minimally acceptable borders:

Defect severity
Minor Medium Major Critical
Beginning 10% 40% 50% 80%
Project
Main 15% 50% 75% 90%
phase
Final 20% 60% 100% 100%

 Ongoing defects fixed percentage:

𝐹𝐶𝑃 𝐷𝐶𝑙𝑜𝑠𝑒𝑑
𝐷𝐿𝑒𝑣𝑒𝑙 = 𝐷𝐿𝑒𝑣𝑒𝑙
𝐹𝑜𝑢𝑛𝑑 ∙ 100%, where
𝐿𝑒𝑣𝑒𝑙

𝐹𝐶𝑃
𝐷𝐿𝑒𝑣𝑒𝑙 – defects fixation percentage by 𝐿𝑒𝑣𝑒𝑙 (defects found in the previous build and fixed in the current build),
𝐶𝑙𝑜𝑠𝑒𝑑
𝐷𝐿𝑒𝑣𝑒𝑙 – quantity of defects of 𝐿𝑒𝑣𝑒𝑙 fixed in the current build,
𝐹𝑜𝑢𝑛𝑑
𝐷𝐿𝑒𝑣𝑒𝑙 – quantity of defects of 𝐿𝑒𝑣𝑒𝑙 found in the previous build.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:12:23
 «File Converter» Project

Minimally acceptable borders:

Defect severity
Minor Medium Major Critical
Beginning 60% 60% 60% 60%
Project
Main 65% 70% 85% 90%
phase
Final 70% 80% 95% 100%

 Stop-factor:
𝑌𝑒𝑠, 𝑇 𝐸 ≥ 25% && 𝑇 𝑆𝑃 < 50%
𝑆={ , where
𝑁𝑜, 𝑇 𝐸 < 25% || 𝑇 𝑆𝑃 ≥ 50%
𝑆 – decision to pause the testing process,
𝑇 𝐸 – current 𝑇 𝐸 value,
𝑇 𝑆𝑃 – current 𝑇 𝑆𝑃 value.

 Test-cases execution percentage:

𝑇 𝐸𝑥𝑒𝑐𝑢𝑡𝑒𝑑
𝑇𝐸 = ∙ 100%, where
𝑇 𝑃𝑙𝑎𝑛𝑛𝑒𝑑

𝑇 𝐸 – test-cases execution percentage,

𝑇 𝐸𝑥𝑒𝑐𝑢𝑡𝑒𝑑 – quantity of executed test-cases,
𝑇 𝑃𝑙𝑎𝑛𝑛𝑒𝑑 – quantity of planned (to execution) test-cases.
Levels (borders):
o Minimal: 80%.
o Desired: 95%-100%.

 Requirements coverage by tests:

𝑅 𝐶𝑜𝑣𝑒𝑟𝑒𝑑
𝑅𝐶 = ∙ 100%, where
𝑅 𝑇𝑜𝑡𝑎𝑙

𝑅𝐶 – requirements coverage by tests (percentage),

𝑅𝐶𝑜𝑣𝑒𝑟𝑒𝑑 – quantity of requirements covered with test-cases,
𝑅𝑇𝑜𝑡𝑎𝑙 – overall quantity of requirements.
Minimally acceptable borders:
o Beginning project phase: 40%.
o Main project phase: 60%.
o Final project phase: 80% (90%+ recommended).

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:12:23
What is Requirement and
Why Requirements are
Important
Software Testing Introduction
Main Definitions Read and remember!

Requirement – a condition or capability needed by a user to

solve a problem or achieve an objective that must be met or
possessed by a system or system component to satisfy a
contract, standard, specification, or other formally imposed
document.

2
General Documentation Overview

Product documentation Project documentation

Project management plan User and accompanying

Test plan documentation

Requirements Market requirements

Architecture and design
Test cases, test suites
Technical specifications

3
Why Documentation Is Important

One of the
greatest
reasons for
the Failure is
here

4
Why Documentation Is Important

5
Why Documentation Is Important

Other
15%

Code
¾ of defects are 10%
Requirements
originated from 50%
documentation

Architecture
and Design
25%

6
What is Requirement and
Why Requirements are
Important
Software Testing Introduction
Ways of Requirements
Gathering – Part 1
Software Testing Introduction
General Overview

Meetings and
Interview Questioning
Brainstorming

Observation Prototyping Modeling

Documentation Work with Focus

Self-(re)writing
Analysis Groups

2
Interview Read and remember!
Interview – the most common way of identifying requirements. Usually,
two main roles involved: interviewer (project specialist, BA) and
interviewee (customer representative or expert, user, etc.)

Fast, easy, no special sophisticated techniques required.

Everybody understands the concept, so everybody can participate.
Variations possible: in-person, via (video) call, e-mail, messenger, etc.

Often some visual aids are required.

It takes time to rework the results as a useable set of data.
Interviewer bears the most responsibility for the results.

3
Meetings and Brainstorming Read and remember!
Meetings and Brainstorming – unlike with Interview, several people are
playing active role here to share information, discuss issues and be on
the same page immediately.

Several key people share information and receive feedback at once.

Meetings are faster than e-mail/messenger communication.
“Collective thinking” may create new good ideas.

Meetings take time. A lot of time. No, really, A LOT OF TIME.

Sometimes it’s difficult to gather all necessary people together.
With intense rhythm meetings become boring and inefficient.

4
Questioning Read and remember!
Questioning (and questionnaires) – is a fast and easy way of gathering
and aggregation of information from thousands of respondents. When
organized properly, questioning brings a lot of useful results.

Great audience coverage – thousands and millions of respondents.

Nice data processing and visualizing capabilities.
Possibility to easily compare several results (over time).

If held improperly (bad questions, wrong target audience, demotivated

respondents and so on), the questioning can give us no data, mess data
or (in worst case) wrong data that looks like a good one.

5
Observation Read and remember!
Observation – is a method that shows things you never hear about.
People are apt to lie (not only to others but to themselves), they forget
things, do not know things… But observation shows the truth.

Experienced observer may pick some mission critical details.

This method is as objective, as possible.
Observation results may lead to the whole problem re-evaluation.

It’s hard to organize observation well most of the time.

Experienced observers are rare.
When improperly organized, observation gives a lot of “bad data”.

6
Ways of Requirements
Gathering – Part 1
Software Testing Introduction
Ways of Requirements
Gathering – Part 2
Software Testing Introduction
General Overview

Meetings and
Interview Questioning
Brainstorming

Observation Prototyping Modeling

Documentation Work with Focus

Self-(re)writing
Analysis Groups

2
Prototyping Read and remember!
Prototyping allows to use “something real” (not imaginary) both as (and
always these two ways): a) the source of the new information; b) real
material thing to discuss.

Most people prefer to see the object in discussion.

No matter how detailed the description is, real object provides more
data.

Prototyping takes time. Some work may be discarded after discussion.

It is crucial to do enough and stop, otherwise a lot of work may be
wasted.

3
Modeling Read and remember!
Modeling is somehow similar to prototyping, but it (usually) does not
require “physical” development. A model may be mathematical one,
computational one and so on.

Model makes a lot of difficult things easy and understandable.

Model allows to see hidden properties of some object / process.
Model allows to see if something is even possible / good.

Modeling requires a lot of experience, knowledge, tools.

Improper model may give us bad data.
Models are harder to understand than prototypes.

4
Documentation Analysis Read and remember!
Documentation analysis allows us to find issues with existing
project/product documentation and to get extra information (when
reading standards, manuals, books, etc.)

This method requires no special actions, tools, and so on: you may just
sit and read a document. Anytime you like. Anywhere you like.
Reading documentation makes you a better professional.

Sometimes with new data new questions rise (you have to write them
down for further discussion). Working alone increases the probability to
miss a defect or to create one.

5
Work with Focus Groups Read and remember!
Work with focus groups looks a lot like questioning except it involves in-
person communication, discussions, end-user involvement in beta-
testing, deep end-user participation in product development.

We can gather a lot of information from real end-users.

Unlike questioning this activity creates deep involvement.
We may not only gather information, but influence people opinion.

Sometimes it’s difficult to gather a proper focus group.

Experienced professional is required to organize the process.
Some NDA issues may arise.

6
Self-(re)writing Read and remember!
Self-(re)writing is mostly a way not for gathering requirements, but for
formalizing them (because it’s extremely difficult to think for the
customer and somehow telepathically know his ideas ☺).

Each and every just mentioned way of requirements gathering provides

us with information that has to be re-worked, re-formulated, and
included into Requirements document. Self-(re)writing gives us that.

Sometimes it’s difficult to avoid adding your own ideas into the
Requirements. You can propose ideas but absolutely have to get the
approval from the Customer side.

7
Ways of Requirements
Gathering – Part 2
Software Testing Introduction
Requirements Levels
and Types
Software Testing Introduction
General Overview

2
Where to read MORE

3
Simplified Model

Business Requirements

Project Vision

User Requirements

Use-cases / user
stories
Non-Functional
Functional Requirements
Requirements

Software Requirements Specification (SRS)

4
Business Requirements Read and remember!

Business requirements express the

purpose for which the product is
developed (why is it needed at all, what
benefits are expected from it).

E.g.: • We need a tool to display the most profitable currency exchange rate in real
time.
• It is necessary to increase by 2-3 times the number of tickets processed by one
operator per shift.
• It is necessary to automate the process of issuing invoices based on contracts.

5
User Requirements Read and remember!

User requirements describe the tasks that

a user can perform with the system
(system response to user actions, user
scenarios).

E.g.: • The license agreement should be displayed on the user first login.
• The administrator should be able to view a list of all users currently
working in the system.
• When you first save a new article, the system should produce a request
to either save it as a draft or to publish it immediately.
6
Requirements Types Read and remember!

Functional requirement – a requirement that specifies a

function that a component or system must be able to perform.

Non-functional requirement – a requirement that describes

how the component or system will do what it is intended to do.

E.g.: “A bird should fly high”.

“A singer should sing nicely”.

7
Functional Requirements Read and remember!

Functional requirements describe the

behavior of the system, i.e. its actions
(calculations, transformations, checks,
processing, etc.)

E.g.: • The application should check the remaining free space on the target
media during installation.
• The system should automatically backup data daily at the specified time.
• The user's email address must be verified against RFC822.

8
Non-Functional Requirements Read and remember!

Non-functional requirements describe the

properties of the system (usability, security,
reliability, extensibility, etc.) that it must
possess when implementing its behavior.

E.g.: • The minimum “time to failure” must be >= 100 hours with simultaneous
continuous work of 1000 users.
• Under no circumstances the total memory used by the application should
exceed 2 GB.
• The font size for any label on the screen must support a range of 5-15 points.
9
Requirements Levels
and Types
Software Testing Introduction
Good Requirements
Properties – Part 1
Software Testing Introduction
General Overview

Obligation Up-to-date

Consistency Atomicity

Correctness and
Unambiguousness Completeness
verifiability
Traceability Feasibility

Ranked for…
Modifiability

Importance Stability Priority

2
Atomicity Read and remember!

A requirement is atomic if it cannot be broken down into

separate requirements without loss of completeness, and it
describes one and only one situation.

Typical problems
• One requirement actually contains several independent requirements.
• The requirement allows discrepancies due to the grammatical features of
the language.
• Description of several independent situations combined into one
requirement.

3
Atomicity Read and remember!

How to detect typical problems

Think, discuss with colleagues and use common sense. If we
consider that a certain section of requirements is overloaded
and requires decomposition, most likely it is so.

How to fix typical problems

Process, re-write and structure the requirements: split them
into sections, subsections, paragraphs, etc.

4
Completeness Read and remember!

The requirement is complete if it provides all the necessary

information, if nothing is missed or left out for reasons such as
“this is already clear to everyone”.

Typical problems
• There is no non-functional part of the requirement or link to the corresponding non-
functional requirement (e.g.: “passwords must be stored in encrypted form” - what is the
encryption algorithm?).
• Only a part of some enumeration is mentioned (e.g.: “export is carried out in PDF, PNG, etc.”
formats - what should we understand by “etc.”?).
• The references cited are ambiguous (e.g.: “see above” instead of “see section 123.45.b”).

5
Completeness Read and remember!

How to detect typical problems

Most requirements testing techniques are applicable, but the
best are: asking questions and using graphical
representations of the system being developed.

How to fix typical problems

Once we see that something is missing, it is necessary to
obtain the missing information and add it to the
requirements. A little re-writing may be needed.

6
Consistency Read and remember!

A consistent requirement should not contain internal

contradictions and/or contradictions with other requirements
and documents.

Typical problems
• Contradictions within one requirement.
• Contradictions between two or more requirements, between a table and text, a
picture and text, a requirement and a prototype, etc.
• Incorrect terminology or different terms to refer to the same phenomenon.

7
Consistency Read and remember!

How to detect typical problems

Good memory helps best , but tool for graphical
representation of the system being developed are also
extremely useful.

How to fix typical problems

Once we detect an inconsistency, it is necessary to clarify the
situation with the Customer and make the necessary changes
to requirements.

8
Unambiguousness Read and remember!

A requirement is unambiguous (clear) if it is described without

the use of jargon, non-obvious abbreviations and vague
wording, and allows only one objective understanding.

Typical problems
• Use of terms or phrases that allow subjective interpretation.
• Use of non-obvious or ambiguous abbreviations without decoding.
• Writing the requirement assuming that some parts are obvious to everybody by
default and therefore needs no explanation.

9
Unambiguousness Read and remember!

Indicators of problems with unambiguousness

Adequate, be able to, easy, provide for, as a minimum, be capable of, effectively,
timely, as applicable, if possible, to be determined (TBD), as appropriate, if
practical, but not limited to, capability of, capability to, normal, minimize,
maximize, optimize, rapid, user-friendly, simple, often, usual, large, flexible, robust,
state-of-the-art, improved, efficient.

E.g. (real quotation)

“In case of necessity to optimize large files transfer, the app should effectively use
the minimum of RAM (if possible).”

10
Unambiguousness Read and remember!

How to detect typical problems

Look for specific words and phrases. Writing check-lists is
also effective: it is hard to come up with a good check-list for
a requirement that is ambiguous.

How to fix typical problems

Use numbers and formulas instead of a verbal description. If
this is not possible, at least use the most accurate technical
terms, references to standards, etc.

11
Good Requirements
Properties – Part 1
Software Testing Introduction
Good Requirements
Properties – Part 2
Software Testing Introduction
General Overview

Obligation Up-to-date

Consistency Atomicity

Correctness and
Unambiguousness Completeness
verifiability
Traceability Feasibility

Ranked for…
Modifiability

Importance Stability Priority

2
Obligation and up-to-date state Read and remember!

Any requirement should be obligatory and up-to-date,

otherwise (if the requirement is not mandatory for
implementation or is outdated) it should be removed.

Typical problems
• The requirement was added for “why not” reason although there is no real need
for it.
• The requirement has wrong values for priority/importance properties.
• The requirement is outdated, but it has not been removed.

3
Obligation and up-to-date state Read and remember!

How to detect typical problems

Periodic review of requirements (preferably with the participation
of the Customer) allows you to notice fragments that have lost
their obligation or have become outdated ones.

How to fix typical problems

Re-write requirements to eliminate fragments that have lost
obligation or have become outdated ones.

4
Feasibility Read and remember!

A feasible requirement is the one that technologically

achievable and may be implemented within the project budget
and schedule.

Typical problems
• So called “Gold plating” – requirements that are extremely expensive to
implement and at the same time almost useless for end users.
• Requirements that are technically unrealistic at the present level of technology.
• Unrealizable requirements (usually – requirements not to the product, but to the
user, environment, laws of physics and so on).

5
Feasibility Read and remember!

How to detect typical problems

Use every bit of your subject matter experience to
understand that a certain requirement “costs” too much or is
infeasible at all within current Projects limitations.

How to fix typical problems

There is nothing else to do, but to discuss the situation with the
Customer and either change the requirement, or reconsider the
terms of the Project (making this requirement feasible).

6
Traceability Read and remember!

Vertical traceability shows the connection between levels of

requirements, horizontal traceability shows the connection between a
requirement and a test plan paragraph, test cases, architectural
solutions, etc.

Typical problems
• Requirements have no ids, not structured, don’t have a table of contents, don’t
have cross-references.
• No tools and/or techniques of requirements management were used during the
development of requirements.
• The set of requirements is incomplete, it is fragmented and has obvious “blind
spots".
7
Traceability Read and remember!

How to detect typical problems

Traceability problems mean no answers to questions like “where
did this requirement came from?”, “where are the related
requirements?”, “what does it affect and what is it affected by?”

How to fix typical problems

Re-work requirements: change the structure, arrange cross-
references, add tags, bookmarks, etc.

8
Modifiability Read and remember!

Modifiability characterizes the ease of making changes to

individual requirements or to a set of requirements. With good
modifiability changes go well without unexpected side effects.

Typical problems
• Requirements are non-atomic and/or untraceable, therefore changes lead to
inconsistency.
• Requirements are initially contradictory, so changes (not related to eliminating
inconsistencies) only worsen the situation.
• Requirements are in an inconvenient form (e.g., requirements management tools
are not used, so the team has to work with dozens of huge text documents).
9
Modifiability Read and remember!

How to detect typical problems

No traceability – no modifiability. Also, the modifiability
decreases with the violation of almost any “good
requirement” property.

How to fix typical problems

Re-write requirements with a primary goal is to increase the
traceability. While doing this, you can fix a lot of other
problems with requirements.

10
Good Requirements
Properties – Part 2
Software Testing Introduction
Good Requirements
Properties – Part 3
Software Testing Introduction
General Overview

Obligation Up-to-date

Consistency Atomicity

Correctness and
Unambiguousness Completeness
verifiability
Traceability Feasibility

Ranked for…
Modifiability

Importance Stability Priority

2
Rank for importance, stability, priority Read and remember!

Importance shows how the success of the Project depends on the

success of the implementation of the requirement.
Stability shows the likelihood that in the foreseeable future the
requirement will not be changed.
Priority shows what requirements should be implemented earlier or
later.

Typical problems
No rank (or incorrect rank, or outdated rank) for importance, stability, priority

3
Rank for importance, stability, priority Read and remember!

How to detect typical problems

Review requirements (with the participation of the
Customer) to detect incorrect values of importance, stability,
and priority. Repeat such review periodically.

How to fix typical problems

Make corrections during the review process.

4
Correctness and verifiability Read and remember!

Correctness and verifiability state that it is possible to create an

objective test case (test cases) that clearly shows that the requirement
is implemented correctly and the behavior of the application exactly
corresponds to the requirement.

Typical problems
As these properties follow from compliance with all of the above mentioned ones,
if any other “good property” is violated the requirement most likely becomes
incorrect and unverifiable.

5
Correctness and verifiability Read and remember!

How to detect typical problems

Since we are dealing with an “integral problem”, we can use
methods described earlier. There are no unique techniques
here.

How to fix typical problems

Make necessary changes to the requirements. Starting from
simple typo correction, and up to global re-writing of the
entire requirement set.

6
General Overview

Obligation Up-to-date

Modifiability Atomicity

Correctness and
Traceability Completeness
verifiability
Unambiguousness Feasibility

Ranked for…
Consistency

Importance Stability Priority

7
The main idea!

If we can not verify the requirement, we have a problem.

Testers are responsible for ensuring that the requirement is verified,

so we have to have a way to verify it, so the requirement has to be a
good one with no “good properties” violated.

8
Good Requirements
Properties – Part 3
Software Testing Introduction
Requirements
Testing Techniques
Software Testing Introduction
General Overview

Peer review
Asking questions
Writing check-lists

Visualization

Modeling and prototyping

2
Peer review Read and remember!

Peer review – a form of review of work products performed by

others qualified to do the same work.

3
Peer review types

Walkthrough

Quick feedback from colleagues

Technical review

Review by a group of specialists (preferable with different skillsets)

Formal inspection

Formal and systematical approach with a lot of specialists following special procedures

4
Asking questions

If something bothers you, if you don’t understand something, if

any kind of clarification is needed – ASK A QUESTION!

5
Asking questions

Bad requirement Bad questions Good questions

“The app should start fast”. • “How fast?” (What do you expect • “What is the maximum time to
to hear? “Extremely fast”? “As fast as start and what hardware / OS are we
possible”? “Fast enough”?) talking about? How busy is the
system with other tasks?”
• “What if it wouldn't start fast?”
(Do you really want to upset the • “Why is the start time so
Customer or even make him angry?) important? What is depending on
it?”
• “Always?” (“Yes, always!” Did you
expect some other answer?) • “May we start/load some
components of the app in
background?”

• “What state/conditions should we

see as the mark that the app is
started?”

6
Writing check-lists

Ask yourself: “How shall I test this requirement? Are there tests
that show objectively that this requirement is implemented
correctly?”

If you have a lot of ideas in mind – that’s good, but not enough:
this requirement may be outdated, inconsistent and so on.

If you have no ideas – that’s a reason to investigate this

situation more thorough, gather more info, ask questions.

7
Visualization

Either to see the overall big

picture or to drill into details
you can use drawings,
diagrams, schemas, mind maps,
concept maps, etc.

8
Modeling and prototyping
Point «abstract» Ellypsoid
Shape
Triangle - x
- radiusA: var
- y Sphere

Use mental simulation of the

# id - radiusB: var
- sideA: var
- z
- sideB: var - radiusLength: var
+ getId() : var + getId() : var
- sideC: var
+ getId() : var + setId(var) : void + getSquare() : var + getId() : var
+ setId(var) : void + getVolume() : var
+ getId() : var + getSquare() : var
+ getSquare() : var + rotate2Dimensional(var) : void + getVolume() : var
+ rotate2Dimensional(var) : void + rotate3Dimensional(var, var, var) : void + rotate2Dimensional(var) : void
+ setId(var) : void + setId(var) : void + rotate3Dimensional(var, var, var) : void

user experience, think about

+ setId(var) : void

«abstract»
Shape2D Parallelepiped
Rectangle
«abstract»
+ getId() : var - sideA: var
- sideA: var Shape3D
+ getSquare() : var - sideB: var

ambiguous or completely
- sideB: var + rotate2Dimensional(var) : void - sideC: var
+ getId() : var
+ setId(var) : void
+ getId() : var + getSquare() : var
+ getId() : var
+ getSquare() : var + getVolume() : var
+ getSquare() : var
+ rotate2Dimensional(var) : void + rotate2Dimensional(var) : void
+ getVolume() : var
+ setId(var) : void + rotate3Dimensional(var, var, var) : void
+ rotate2Dimensional(var) : void
+ setId(var) : void
+ rotate3Dimensional(var, var, var) : void

undescribed options for the

+ setId(var) : void

Square
«interface»
- sideLength: var Rotation2D
Cube
+ rotate2Dimensional(var) : void
+ getId() : var - sideLength: var

behavior of the system.

+ getSquare() : var
+ rotate2Dimensional(var) : void + getId() : var
+ rotate2Dimensional(var) : void + getSquare() : var
«interface»
+ setId(var) : void + getVolume() : var
Rotation3D
+ rotate2Dimensional(var) : void
+ rotate3Dimensional(var, var, var) : void + rotate3Dimensional(var, var, var) : void
+ setId(var) : void

Prototyping and modeling may

require special tools and
knowledge, but it can save a lot
of time and effort.
9
Requirements
Testing Techniques
Software Testing Introduction
 «File Converter» Project

Requirements
SAMPLE
Project Documentation

Background Full set of requirements specification.

Purpose To organize both development and testing process.
Scope Business requirements, user requirements, detailed
specification, limitations.
Audience Management staff, project team.
File 03 06 - Requirements Sample.docx

EPAM Training Center: Software Testing Introduction Most Recent Update: 15.03.2022 12:05:26
 «File Converter» Project

Contents

1. Project scope ................................................................................................................. 3

2. Main goals ..................................................................................................................... 3
3. Criteria for main goals achievement .............................................................................. 3
4. Risks .............................................................................................................................. 3
5. System characteristics ................................................................................................... 3
6. User requirements ......................................................................................................... 3
7. Business rules ............................................................................................................... 4
8. Quality attributes ............................................................................................................ 4
9. Limitations...................................................................................................................... 5
10. Detailed specifications ................................................................................................ 5

EPAM Training Center: Software Testing Introduction Most Recent Update: 15.03.2022 12:05:26
 «File Converter» Project

1. Project scope
Development of a tool to eliminate encoding multiplicity in text documents stored
locally.

2. Main goals
• Eliminate the necessity for manual detection and conversion of encoding in text
documents.
• Decrease document-processing time by the amount needed for manual encoding
detection and conversion.

3. Criteria for main goals achievement

• Full automation of encoding detection and conversion.
• Document-processing time reduction by 1-2 minutes (average) due to elimination the
necessity for manual encoding detection and conversion.

4. Risks
• High complexity of accurate detection of text document initial encoding.

5. System characteristics
• SC-1: The application should be a console one.
• SC-2: The application should be developed using PHP (see L-1 for the explanation;
PHP-related details are described in DS-1).
• SC-3: The application should be a multi-platform one (taking into account L-4).

6. User requirements
• See also the use cases diagram below for details.
• UR-1: Start and stop of the application.
o UR-1.1: The application start should be performed by the following console
command: “php converter.phar SOURCE_DIR DESTINATION_DIR
[LOG_FILE_NAME]” (see DS-2.1 for parameters description, see DS-2.2, DS-
2.3, and DS-2.4 for error messages on any misconfiguration situation).
o UR-1.2: The application stop (shutdown) should be performed by applying
Ctrl+C to the console window, which holds the running application.
• UR-2: Configuration of the application.
o UR-2.1: The only configuration available is through command line parameters
(see DS-2).
o UR-2.2: Target encoding for text file conversion is UTF8 (see also L-5).
• UR-3: Application log.
o UR-3.1: The application should output its log both to the console and to a log-
file (see DS-4). Log file name should comply with the rules described in DS-
2.1.

EPAM Training Center: Software Testing Introduction Most Recent Update: 15.03.2022 12:05:26
 «File Converter» Project

o UR-3.2: Log contents and format are described in DS-4.1, the application
reaction to log file presence/absence is described in DS-4.2 and DS-4.3
accordingly.

uc Primary Use Cases

Application
Other functions are provided by
external tools and are NOT part of
the application

Configure the
application

Administrator

Start the application

Stop the application

View the application

log

7. Business rules
• BR-1: The source directory and the destination directory
o BR-1.1: The source directory and the destination directory may NOT be the
same directory (see also DS-2.1 and DS-3.2).
o BR-1.2: The destination directory may NOT be inside the source directory or
any of its subdirectories (see also DS-2.1 and DS-3.2).

8. Quality attributes
• QA-1: Performance
o QA-1.1: The application should provide the processing speed of at least 5
MB/sec with the following (or equivalent) hardware: CPU i7, RAM 4 GB,
average disc read/write speed 30 MB/sec. See also L-6.
• QA-2: Resilience to input data
o QA-2.1: See DS-5.1 for the requirements to input file formats.
o QA-2.2: See DS-5.2 for the requirements to input file size.
o QA-2.3: See DS-5.3 for the details on application reaction on incorrect input

EPAM Training Center: Software Testing Introduction Most Recent Update: 15.03.2022 12:05:26
 «File Converter» Project

file format.

9. Limitations
• L-1: The application should be developed using PHP as the Customer is going to
support the application with their own IT-department.
• L-2: See DS-1 for PHP version and configuration details.
• L-3: PHP setup and configuration process are out of this project scope and therefore
are NOT described in any product/project documentation.
• L-4: Multi-platform capabilities of the application are the next: it should work with
Windows and Linux assuming that proper PHP version (see DS-1.1) works there.
• L-5: The target encoding (UTF8) is fixed. There is no option to change it.
• L-6: The QA-1.1 may be violated in case of objective reasons (e.g., system overload,
low-performing hardware and so on).

10. Detailed specifications

DS-1: PHP
DS-1.1: Minimal version – 5.5.
DS-1.2: The mbstring extension should be installed and enabled.
DS-2: Command line parameters
DS-2.1: The application receives three command line parameters during the start
process:
• SOURCE_DIR – mandatory parameter, points to the directory with files to be
processed;
• DESTINATION_DIR – mandatory parameter, points to the directory to store
converted files (see also BR-1.1 and BR-1.2);
• LOG_FILE_NAME – optional parameter, points to the log file (if omitted,
“converter.log” file should be created in the same directory where “converter.phar”
is located);
DS-2.2: If some mandatory command line parameter is omitted, the application should
shut down displaying standard usage-message (see DS-3.1).
DS-2.3: If more than three command line parameters are passed to the application, it
should ignore any parameter except listed in DS-2.1.
DS-2.4: If the value of any command line parameter is incorrect, the application should
shut down displaying standard usage-message (see DS-3.1) and incorrect parameter name,
value, and proper error message (see DS-3.2).
DS-3: Messages
DS-3.1: Usage message: “USAGE converter.phar SOURCE_DIR
DESTINATION_DIR LOG_FILE_NAME”.
DS-3.2: Error messages:
• Directory not exists or inaccessible.
• Destination dir may not reside within source dir tree.
• Wrong file name or inaccessible path.
DS-4: log

EPAM Training Center: Software Testing Introduction Most Recent Update: 15.03.2022 12:05:26
 «File Converter» Project

DS-4.1: The log format is the same for the console and the log file: YYYY-MM-DD
HH:II:SS operation_name operation_parameters operation_result.
DS-4.2: If the log-file is missing, a new empty one should be created
DS-4.3: If the log file exists, the application should append new records to its tail.
DS-5: File format and size
DS-5.1: The application should process input files in Russian and English languages
with the following encodings: WIN1251, CP866, and KOI8R.
Supported file formats (defined by extension) are:
• Plain Text (TXT).
• Hyper Text Markup Language Document (HTML).
• Mark Down Document (MD).
DS-5.2: The application should process files up to 50 MB (inclusive), the application
should ignore any file with the size larger than 50 MB.
DS-5.3: If a file with supported format (see DS-5.1) contains format-incompatible data,
such data may be damaged during file processing, and this situation should be treated as
correct application work.

EPAM Training Center: Software Testing Introduction Most Recent Update: 15.03.2022 12:05:26
Checklists

Software Testing Introduction

Checklist Read and remember!

Checklist – a set of ideas.

In testing checklists come first: this is a kind of draft for further
activities. Writing and re-writing checklists is (relatively) fast
and simple. When checklist is ready (and reviewed by
colleagues) each item may become a source for one or several
test cases.

2
Example (famous “Triangle Task” by Glenford J. Myers)

A console C++ program (32-bit, Win32) reads three integers

from the command line. These integers represent the three
lengths of the sides of a triangle. Then the program displays a
message which states whether the triangle is scalene (i.e., no
two sides are equal), isosceles (two sides equal) or equilateral
(all sides equal).

Take a pause and write down your own ideas on how to test
such program.

3
Checklist example

1. Valid scalene triangle.

2. Valid isosceles triangle: (a = b) or (b = c) or (a = c).
3. Valid equilateral triangle.
4. The length of one side = 0: (side a) or (side b) or (side c).
5. The length of one side <0: (side a) or (side b) or (side c).
6. The sum of two sides is equal to the third: (a + b = c) or (a + c = b) or (b + c = a).
7. The sum of two sides is less than the third: (a + b < c) or (a + c < b) or (b + c < a).
9. Non-integers (i.e. real values).
10. Not-numbers (i.e. letters, special symbols, etc.).
11. Invalid number of entered values.

4
General ideas

This prevents you from forgetting

Write down your checklists! any ideas and allows you to easily
share and re-use a lot of checklists.

It is recommended to start with

ideas of simple tests to check basic
functionality and/or user scenarios.
Start with simple ideas! Don’t try to check rare extreme
cases until you’re done with simple
and obvious ones.

5
Checklists writing approaches

By application parts, sub-parts, sub-sub-parts

By user scenarios and actions

By subject matter scenarios and cases

By testing objectives and quality criteria

By requirements

By interface
6
Checklists

Software Testing Introduction

Basic Testing
Techniques
Software Testing Introduction
General overview

It is much more simple to produce good checklists when you

follow special techniques. We’ll overview some.

For more
details see
this
outstanding
book:

2
Basic definitions Read and remember!

An equivalence class consists of a set of data that is treated the

same by a module or that should produce the same result.

The boundaries — the “edges” of each equivalence class.

So we have to find out which input data is treated the same

way and use boundary values technique to select values to test.

3
Equivalence classes and boundary values: example

The function determines if a given string value represents

correct (return true) or incorrect (return false) user name.
Username requirements:
• Length: 3-20 symbols.
• Allowed symbols: numbers, underscores, English letters (in
upper and lower case).

Take a pause and write down your own ideas on how to test
such function.

4
Equivalence classes and boundary values: example

For those who is too optimistic: don’t be !

Username requirements:
• Length: 3-20 symbols.
• Allowed symbols: numbers, underscores, English letters (in
upper and lower case).

Valid names: 18-digits 63-demical number, i.e.

2.4441614509104E+32. And the infinity of invalid names.

5
Equivalence classes and boundary values: example

Equivalence classes by length:

• 3-20 (valid).
• 0-2 (invalid).
Boundary values
• 21+ (invalid).

Invalid Valid Invalid

3 20

0 2 21

6
 Equivalence classes and boundary values: example

Equivalence classes by symbols:

• Letters (A-Z, a-z), numbers (0-9), underscore (_) – (valid).
• Other symbols (invalid).

Bad decision. Software Good Valid

Invalid
doesn’t work this way. decision.
Letters A-Z
Numbers Letters A-Z _ Letters a-z Letters a-z
48 57 65 90 95 97 122 Other symbols
Numbers
Underscore _
0 47 58 64 91 94 96 123 255

7
Equivalence classes and boundary values: example

Finally we have the following values to test:

• Valid length, valid symbols: ABC, ABCDEFGHIJKLMNOPQRST,
abc_12_def.
• Invalid length, valid symbols: AA, {empty string}.
• Invalid length, valid symbols: AAAAAAAAAAAAAAAAAAAAA
(21 characters).
• Valid length, invalid symbols: Abcd#23456%@#&#%^#.

Seven tests instead of «2.4441614509104E+32 of positive» + «infinity of

negative».
8
And some more ideas…

Are there more ways to pick values in

such situations? Yes. See this book:

9
Functional approach Read and remember!

Other powerful approach to generating test-ideas is thinking

about software functions: what should it do (or shouldn’t do) in
some situation.

10
Functional approach

Example 1: e-mail field.

1. What is it used for?
2. What are special requirements (if any)?
3. What is the reaction to valid/invalid input?
4. What is the reaction to empty field (is it valid or not)?
5. What is the reaction to already existing e-mail?
6. Is the data filtering secure enough? Can we use SQL
injection or code injection?
7. And so on and so forth…
11
Functional approach

Example 2: two date fields (vacation start and finish).

1. What are special requirements (if any)?
2. What is the reaction to valid/invalid input?
3. Can both values be in the past? In the future?
4. What is the reaction to empty field (is it valid or not)?
5. Can start date and end date be the same?
6. What is the minimum/maximum vacation length?
7. Can start/end date be Saturday and/or Sunday? Or any holiday?
8. And so on and so forth…

12
Please read these books!

A lot (and lot, and lot!) of nice ideas on testing techniques you
can find in these books:

13
Basic Testing
Techniques
Software Testing Introduction
Test cases

Software Testing Introduction

Test case Read and remember!

Test case – a set of preconditions, inputs, actions (where

applicable), expected results and postconditions, developed
based on test conditions. Test case always follows some
purpose.

These four parts are

crucial for a good
test case.

2
Key test case fields

Disclaimer: each and every test case management tool has

much more fields to fill. There are different approaches in
details, but now we are going to review only KEY FIELDS that
remains more or less the same for past 3-4 decades.

3
Key test case fields: general overview

UG_U1.12 A R97 Gallery Uploader File upload, name with 1. Upload window appears.
special symbols 2. Standard file selection
Preparations: create a dialog window appears.
non-empty file named 3. Chosen file name
#$%^&.jpg. appears in “File” field.
1. Click “Upload photo”. 4. File selection dialog
2. Click “Choose”. window closes, “File” field
3. Select the prepared contains full path to the
file from the list. selected file.
4. Click “OK”. 5. Uploaded file appears in
5. Click “Add to the the gallery contents list.
gallery”.

4
Key test case fields: general overview

Priority Related requirement Test case title Expected results

(for each step)
UG_U1.12 A R97 Gallery Uploader File upload, name with 1. Upload window appears.
special symbols 2. Standard file selection
Preparations: create a dialog window appears.
ID non-empty file named 3. Chosen file name
#$%^&.jpg. appears in “File” field.
Necessary
1. Click “Upload photo”. 4. File selection dialog
preparations (if any)
2. Click “Choose”. window closes, “File” field
Module and 3. Select the prepared contains full path to the
submodule file from the list. selected file.
4. Click “OK”. 5. Uploaded file appears in
Steps
5. Click “Add to the the gallery contents list.
gallery”.

5
Key test case fields: ID

UG_U1.12 A R97 Gallery Uploader File upload, name with 1. Upload window appears.
special symbols 2. Standard file selection
• Unique. Preparations: create a
non-empty file named
dialog window appears.
3. Chosen file name
ID
• Meaningful (if test #$%^&.jpg.
management toolappears allows).in “File” field.
1. Click “Upload photo”. 4. File selection dialog
2. Click “Choose”. window closes, “File” field
3. Select the prepared contains full path to the
file from the list. selected file.
4. Click “OK”. 5. Uploaded file appears in
5. Click “Add to the the gallery contents list.
gallery”.

6
Key test case fields: priority

Priority
• Shows how important this test case is.
UG_U1.12 A • Represented
R97 Gallery UploaderbyFile
letters
upload, (A,
nameB, C, D,1.E),
with numbers
Upload window appears.
special symbols 2. Standard file selection
(1, 2, 3, 4, 5), words («extremely
Preparations: create a high»,
dialog «high»,
window appears.
«medium», «low», «extremely
non-empty file named low») orfile
3. Chosen anyname
#$%^&.jpg. appears in “File” field.
other convenient way.
1. Click “Upload photo”. 4. File selection dialog
• Correlated with:2.3. Click “Choose”.
Select the prepared
window closes, “File” field
contains full path to the
• the requirementfile fromimportance;
the list. selected file.
• potential severity
4. Click “OK”.
of a defect
5. Click “Add to the
5. Uploaded file appears in
that this test
the gallery contents list.
case may detect.
gallery”.

7
Key test case fields: related requirement

Related requirement

UG_U1.12 A R97 Gallery Uploader

File upload, name with 1. Upload window appears.
special symbols 2. Standard file selection
Preparations: create a dialog window appears.
non-empty file named 3. Chosen file name
• Contains the ID of the requirement this
#$%^&.jpg. testin “File”
appears casefield.
is
1. Click “Upload photo”. 4. File selection dialog
intended to test. 2. Click “Choose”. window closes, “File” field
• 3. Select the prepared
Facilitates traceability between
file from the list.
contains full path to the
requirements
selected file.
and
test cases. 4. Click “OK”. 5. Uploaded file appears in
5. Click “Add to the the gallery contents list.
gallery”.

8
Key test case fields: module and submodule

Module and submodule

UG_U1.12 A R97 Gallery Uploader

File upload, name with 1. Upload window appears.
special symbols 2. Standard file selection
• Preparations: create a
Shows the application (sub)parts covered
non-empty file named
by dialog window appears.
the test case.
3. Chosen file name
• Simplifies understanding of test case purpose.appears in “File” field.
#$%^&.jpg.
• Module and submodule are 1. Click “Upload
parts of thephoto”. 4. File selection
application, NOT dialog
2. Click “Choose”. window closes, “File” field
actions! See the difference3. (using
Select thehuman
prepared as example):
contains full path to the
«respiratory system, lungs»file–from
module
the list. and submodule,
selected file. but
4. Click “OK”. 5. Uploaded file appears in
«breathing», «snuffling», «sneezing»
5. Click “Add to the– are NOT; «head,
the gallery brain»
contents list.
– module and submodule,gallery”.
but «nodding», «thinking» – are
NOT.
9
Key test case fields: title

Test case title

UG_U1.12 A R97 Gallery Uploader

File upload, name with 1. Upload window appears.
special symbols 2. Standard file selection
Preparations: create a dialog window appears.
• Shows the main idea (purpose) of
non-empty file named
the test case.
3. Chosen file name
• Populates the test cases list.
#$%^&.jpg. appears in “File” field.
• Is one of the main data1. Click “Upload
sources photo”.searching
when
2. Click “Choose”.
4. File selection
for adialog
window closes, “File” field
particular test case. 3. Select the prepared contains full path to the
• file fromhave
A test case should ALWAYS the list.a title! selected file.
4. Click “OK”. 5. Uploaded file appears in
5. Click “Add to the the gallery contents list.
gallery”.

10
Key test case fields: preparations

Necessary preparations (if any)

UG_U1.12 A R97 Gallery Uploader

File upload, name with 1. Upload window appears.
special symbols 2. Standard file selection
Preparations: create a dialog window appears.
non-empty file named 3. Chosen file name
#$%^&.jpg. appears in “File” field.
1. Click “Upload photo”. 4. File selection dialog
2. Click “Choose”. window closes, “File” field
• Is optional. 3. Select the prepared contains full path to the
• file from
Describes actions and conditions tothebelist.done or selected file.
met before the
4. Click “OK”. 5. Uploaded file appears in
test case itself begins. 5. Click “Add to the the gallery contents list.
• Often does not interfere withgallery”.
the application under test (unlike
“Steps” data).
11
Key test case fields: steps and expected results

Steps Expected results (for each step)

UG_U1.12 A R97 Gallery Uploader

File upload, name with 1. Upload window appears.
special symbols 2. Standard file selection
Preparations: create a dialog window appears.
non-empty file named 3. Chosen file name
• Steps describe action sequence to perform
#$%^&.jpg.
during the test case execution.
appears in “File” field.
• Expected results describe the application reaction
1. Click “Upload photo”.to the specific
4. File input
selection dialogor
action. 2. Click “Choose”. window closes, “File” field
• Both steps and expected results 3. Select
have thesame
the prepared
numeric contains
ids. full path to the
file from the list. selected file.
4. Click “OK”. 5. Uploaded file appears in
5. Click “Add to the the gallery contents list.
gallery”.

12
Key test case fields: general overview

Priority Related requirement Test case title Expected results

(for each step)
UG_U1.12 A R97 Gallery Uploader File upload, name with 1. Upload window appears.
special symbols 2. Standard file selection
Preparations: create a dialog window appears.
ID non-empty file named 3. Chosen file name
#$%^&.jpg. appears in “File” field.
Necessary
1. Click “Upload photo”. 4. File selection dialog
preparations (if any)
2. Click “Choose”. window closes, “File” field
Module and 3. Select the prepared contains full path to the
submodule file from the list. selected file.
4. Click “OK”. 5. Uploaded file appears in
Steps
5. Click “Add to the the gallery contents list.
gallery”.

13
Useful ideas

Use active voice and simple phrases in steps section

Use objective description in expected results section
Write simple, this is not a fiction novel

Use exact names for interface elements

Don’t explain basics

14
Useful ideas

Use active voice and simple phrases in steps section

Bad:
Use Good: results section
objective description in expected
“When a user seems to “Click ‘Open’ button.”
Write simple,
try to this is not a fiction novel
click ‘Open’
button…”
Use exact names for interface elements

Don’t explain basics

15
Useful ideas

Use active voice and simple phrases in steps section

Use objective description in expected results section
Bad:simple, this is not a fictionGood:
Write novel
“The application works as “The ‘New document’
Use exact
it has to”,names for interface elements
“Operation label appears”,
successful”, “OK”… “Calculation result equals
Don’t explain basics
to A*(B+C)”, “Error
message ‘Bad email
appears’”…
16
Useful ideas

Use active voice and simple phrases in steps section

Use objective description in expected results section
Write simple, this is not a fiction novel

Use exact names for interface elements

Don’t explain basics

17
Test cases

Software Testing Introduction

Good Test Cases
Properties – Part 1
Software Testing Introduction
A little step aside: let’s consider good requirements properties

Obligation Up-to-date

Modifiability Atomicity

Correctness and
Traceability Completeness
verifiability
Unambiguousness Feasibility

Ranked for…
Consistency

Importance Stability Priority

2
A little step aside: let’s consider good requirements properties

Obligation Up-to-date

Modifiability Atomicity

Correctness and
Traceability Completeness
verifiability
Unambiguousness Feasibility

Ranked for…
Consistency

Importance Stability Priority

3
And there are more specific properties

… too specific
Somewhere between…

… too general

… too simple
Somewhere between…

… too complex

… independent …
Either… or…

… reasonably linked …

4
Somewhere between being too specific or too general

Which test case is good and which test case is bad?

1. Enter 10 to the 3. The “C” field 1. Check that the 1. The app adds
“A” field. contains 25. app adds two two numbers
2. Enter 15 to the numbers properly. properly.
“B” filed.
3. Click the “Add”
button.

Both test cases are BAD!

5
Somewhere between being too specific or too general

1. Enter 10 to the 3. The “C” field 1. Check that the 4. The app adds
“A” field. contains 25. app adds two two numbers
2. Enter 15 to the numbers properly. properly.
“B” filed.
3. Click the “Add”
button.

Too specific: Too general:

• the probability of detecting an error is • it’s difficult for beginners (or new
reduced due to re-executing exactly the people on the project) to understand
same actions; such test cases;
• the time of creation and support of the • even huge effort to understand the
test case increases. initial idea may leave some gaps.

6
Somewhere between being too specific or too general

Summation of two numbers 1. The value appears in the field.

1. Enter a valid integer into the “A” field. 2. The value appears in the field.
2. Enter a valid integer into the “B” field. 3. The “C” field contains the sum of values
3. Click the “Add” button. from “A” and “B” fields.
4. Repeat steps 1-3 for the following
values: 0, max valid values, min valid
values.

Balanced:
• we are not tied to specific values (so each time we use different ones);
• we know exactly how to check the result (no guessing needed);
• we reduce the time to write and support the test case by referring its own steps;
• we have the list of values we particularly interested in.

7
Between being too simple or too complex Read and remember!

A simple test case operates with single object (or the main
object is obvious), it contains a small number of trivial actions.

A complex test case operates with several equal objects and/or

contains many nontrivial actions.

8
Somewhere between being too simple or too complex

The simplicity or the complexity themselves are OK. Problems

begin with extreme simplicity or extreme complexity.
Repeated conversion 1. The app starts and is ready for file
Filling the “A” field 1. The value appears in Preparations: conversion.
* Create three separate folders for input files, 2. Files are moved from the input folder to the
1. Enter the value into the “A” field. output files, and log file inside the root of any output folder, messages about successful file
drive. conversion appear in the console and in the
the “A” filed. * Prepare a set of several files of the log file.
maximum supported size and supported 3. Files are moved from the input folder to the
formats with supported encodings, as well as output folder, messages about successful file
several files of the supported size, but in conversion appear in the console and in the
invalid format. log file.
1. Start the application specifying the 5. Files are moved from the input folder to the
corresponding paths from the preparation for output folder, messages about successful
the test case in the parameters (the name of conversion of files of the valid format and
the log file is mandatory). messages about ignoring files of the invalid
2. Copy several files with the valid format into format appear in the console and in the log
the input folder. file.
3. Move the converted files from the output 6. Files are moved from the input folder to the
folder into the input folder. output folder, messages about successful
4. Move the converted files from the output conversion of files of the valid format and
folder into the folder containing initial set of messages about ignoring files of the invalid
files prepared for the test case. format appear in the console and in the log
5. Move all the files from the initial folder with file.
the set of files prepared for the test into the
input folder.
6. Move all the converted files from the output
folder into the input folder.

9
Somewhere between being too simple or too complex

Simplicity: pros and cons

Easy to understand
Too simple test cases are nothing
Easy to execute more than a step (or several steps) of
more complex test cases. That’s why
Easy to see the defects such extremely simple test cases are
useless.
Found defects are obvious

10
Somewhere between being too simple or too complex

Complexity: pros and cons

More chances to break something Too complex test cases take a lot of
time to write and maintain. And they
More similar to real user actions
often need huge maintenance with
Developers rarely test such things each application change.

It’s recommended to move from simple test cases to complex

ones during the testing process.
11
Good Test Cases
Properties – Part 1
Software Testing Introduction
Good Test Cases
Properties – Part 2
Software Testing Introduction
And there are more specific properties

… too specific
Somewhere between…

… too general

… too simple
Somewhere between…

… too complex

… independent …
Either… or…

… reasonably linked …

2
Either independent or reasonably linked Read and remember!

An independent test case does not reference to any other test

case and is not referenced by any other test case.

A linked test case either references to some other test

case(es), or has a fixed place in a chain of test cases.

Independent test cases are industrial standard

3
Either independent or reasonably linked

Independency: pros and cons

Easy to execute in any order or set Usually need more preparations

Easy to combine into sets/scenarios Usually need more steps

Work if some other test cases fail May produce some redundancy

4
Either independent or reasonably linked

Being linked: pros and cons

Usually need less preparations More difficult to maintain

Usually need less steps The structure is fixed and rigid

Start work from the point where If previous test case fails, all next-in-
previous test case ends chain test cases fail too

Once again: independent test cases are industrial standard

5
Some more useful ideas about good test cases…

A good test case exposes some defect with high probability

This one is better…

1/1 = 1 -5/0 = err!

6
Some more useful ideas about good test cases…

A good test case follows its goals without retreat

Really? What for?!

File saving
1. Open “File -> Print”.
2. …

7
Some more useful ideas about good test cases…

A good test case does not contain unnecessary steps

What for?!
…
3. Open “File -> Print” dialog.
4. Click several times in random places of the
windows.
…
8
Some more useful ideas about good test cases…

A good test case is not redundant with other test cases

If we have this… … we don’t need this (at
17+98
the same iteration)

1+1 3+4 1+98

1+2 7+5 20+21
2+1 8+1 30+12
9
Some more useful ideas about good test cases…

A good test case makes found defects obvious

237465689645896589236892 100 + 50 = 212

365 *
2375641647647647816478 =
5.64133389910162755434020 Yes, this is a defect!
79018219e+47

Is there any defect?!

10
Some more useful ideas about good test cases…

A good test case performs some non-trivial actions

…
3. Upload the file with the following name
“%^##76 / // \ ^^ [ ] : .jpg” as user avatar.
…

11
Good Test Cases
Properties – Part 2
Software Testing Introduction
Test Suites

Software Testing Introduction

Test Suite Read and remember!

Test Suite – a set of test cases or test procedures to be

executed in a specific test cycle.

Usually test suites are sets of test cases selected with some
common goal or on some common basis.

The main idea here is to manage testing process, to organize

huge number of test cases into small convenient sets.

2
Test Suites creation logic

When composing test suites, we may use any common sense.

I.e. we may compose a test suite with test cases that…

Cover the same requirements Have the same priority,

section module/submodule/etc…

Work with the same Cover some user scenario or

functionality (e.g. FS, DBMS) typical set of actions

3
Test Suites: free sets and linked sets Read and remember!

Free set of test cases doesn’t imply any specific execution

order, it only composes several test cases together.

Linked set of test cases implies specific execution order.

Free Linked

4
Test Suites: free sets and linked sets

Free Linked

Easy to compose Need less preparations and steps

Easy to change execution order Continues work from the point

where previous test case ends
Still work if some test cases fail
Easy to emulate user scenarios
May produce some redundancy
If previous test case fails, all next-in-
Hard to emulate user scenarios chain test cases fail too

5
Test Suites

Software Testing Introduction

 «File Converter» Project

Checklists, Test Cases, Test

Suites sample
SAMPLE

This is NOT a part of project documentation! This is just a way to

demonstrate the whole process of checklists, test cases, and test
suites creation.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:48:27
 «File Converter» Project

Contents
1. Stage 1: read and improve the requirements ................................................................. 3
2. Stage 2: checklist creation logic and checklist sample................................................... 3
2.1. Basic functions ........................................................................................................ 3
2.2. Functions used by most users in their daily work .................................................... 3
2.3. Remaining functions ................................................................................................ 5
3. Stage 3: test cases sample ............................................................................................ 6
4. Stage 4: test suites sample ............................................................................................ 9

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:48:27
 «File Converter» Project

1. Stage 1: read and improve the requirements

First, let us take a pause and once again read this document attentively:

Requirements
Sample.docx

2. Stage 2: checklist creation logic and checklist sample

As we cannot “test the entire application at once” (this is a too huge task to solve), we
have to follow some logic to build checklists – yes, there will be several of them (as a result,
they can be combined into one, but this is optional action).
Typical approaches to create separate checklists are:
 by levels of functional testing;
 by individual parts (modules and submodules) of the application;
 by individual requirements, groups of requirements, levels and types of requirements;
 by typical user scenarios;
 by functions of the application that are mostly at risk.

Let’s use the logic of splitting application functions according to their degree of
importance:
 Basic functions (i.e., the most important ones; failure with these functions make the
application unusable).
 Functions used by most users in their daily work.
 Remaining functions (a variety of "little things"; failure with these function do not really
affect the value of the application for the end user).

2.1. Basic functions

 Configuration and start.

 File processing:
Input file format
TXT HTML MD
WIN1251 + + +
Input file
CP866 + + +
encoding
KOI8R + + +
 Stop.
Yes, that’s all. These are all basic functions of this small application.

2.2. Functions used by most users in their daily work

A very common question here is: can we copy here some ideas from the previous
level? The answer is “yes and no at the same time”. “No” because it doesn’t make sense to
repeat the same checks. “Yes” because any idea can be expanded and provided with
additional details.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:48:27
 «File Converter» Project

 Configuration and start:

o With correct parameters:
 Values for SOURCE_DIR, DESTINATION_DIR, LOG_FILE_NAME are
passed to the application. These values (all of them) should contain
spaces and Cyrillic symbols. Repeat this test for Windows and *nix file
systems (pay attention to directory separators (“/” и “\”) and logical
drives).
 LOG_FILE_NAME value is omitted.
o Without any parameter.
o With some mandatory parameters omitted.
o With incorrect parameters:
 Wrong path for SOURCE_DIR.
 Wrong path for DESTINATION_DIR.
 Wrong file name for LOG_FILE_NAME.
 DESTINATION_DIR is inside SOURCE_DIR.
 DESTINATION_DIR and SOURCE_DIR are the same.
 File processing:
o Miscellaneous formats, encodings and sizes:
Input file format
TXT HTML MD
WIN1251 100 KB 50 MB 10 MB
CP866 10 MB 100 KB 50 MB
KOI8R 50 MB 10 MB 100 KB
Input file
Any 0 bytes
encoding
Any 50 MB + 1 B 50 MB + 1 B 50 MB + 1 B
- Any non-supported format
Any Damaged file of any supported format
o Inaccessible input files:
 No access rights.
 File is opened and locked.
 File with “read-only” attribute.
 Stop:
o Close the console window (do not use Ctrl-C).
 Log:
o Creation (if no log was created yet).
o Appending (for the existing log) with application restart.
 Performance:
o Execute some elementary rough test.
Please note that the checklist may contain not only “just ideas”, but also comments,
details, and examples, if necessary.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:48:27
 «File Converter» Project

2.3. Remaining functions

The time has come to pay attention to various potential problems that may hardly
trouble the user, but formally such problems are still application defects.
 Configuration and start:
o Values for SOURCE_DIR, DESTINATION_DIR, LOG_FILE_NAME:
 In different styles (Windows-like + *nix-like): one value in one style and
the other value in another style at the same time.
 UNC-names.
 LOG_FILE_NAME inside SOURCE_DIR.
 LOG_FILE_NAME inside DESTINATION_DIR.
o Log file size at the application start:
 2–4 GB.
 4+ GB.
o Start of two (and more) copies of the application:
 With the same SOURCE_DIR, DESTINATION_DIR,
LOG_FILE_NAME.
 With the same SOURCE_DIR and LOG_FILE_NAME, but different
DESTINATION_DIR.
 With the same DESTINATION_DIR and LOG_FILE_NAME, but
different SOURCE_DIR.
 File processing:
o File of valid supported format containing text in two or more encodings.
o Input file size:
 2–4 GB.
 4+ GB.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:48:27
 3. Stage 3: test cases sample
Here we have only some test-cases samples (in real life there will be hundreds of test cases).

Test case sample 1:

Id Priority Requirement Module Submodule Steps Expected Results
ST.7 A DS-5.1 File processor Encoding Conversion from all 1. The app starts and writes the
converter supported encodings startup message to the
Preparations: console and to the log file.
 Create four separate 2. Files from INPUT directory
directories in the root folder move to OUTPUT directory.
of any logical drive (or in Messages about each file
the user home directory for conversion (with information
*nix systems): one for about source encoding) are
INPUT files, one for displayed both in the console
OUTPUT files, one for LOG and in the log file.
file, one for TEMPORARY 3. The app writes shutdown
test files storage. message into the log file and
 Unpack files from the stops.
attached archive into
TEMPORARY directory.
1. Start the app with
corresponding paths from
preparations as parameters
(use any log file name).
2. Copy all files from
TEMPORARY directory into
INPUT directory.
3. Stop the app.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:48:27
 «File Converter» Project

Test case sample 2:

Id Priority Requirement Module Submodule Steps Expected Results
CPT.17 B - Scanner Traverser Multiple copies of the app 3. All three copies of the app are
working simultaneously, file started, the log file contains three
operations conflict sequential startup records.
Preparations:
 Create three separate 5. Files gradually move from INPUT to
directories in the root folder of OUTPUT directory. The console and
any logical drive (or in the user the log file contains messages about
home directory for *nix successful file conversion for each
systems): one for INPUT files, file. The console and the log file may
one for OUTPUT files, one for also contain the following error
LOG file. messages:
a. “source file inaccessible, retrying”.
 Create several files of the
b. “destination file inaccessible,
maximum supported size,
retrying”.
supported formats, and
c. “log file inaccessible, retrying”.
supported encodings.
1. Start the first copy of the app The key success indicator for this test
with corresponding paths from is the successful conversion of all
preparations as parameters files while no copy of the app should
(use any log file name). crash. The log file should contain at
2. Start the second copy of the least one record (thee – maximum)
app with the same parameters for each converted file.
(see step 1).
3. Start the third copy of the app Mentioned above error messages are
with the same parameters (see OK too. They may appear, or may
step 1). not – it depends on too many
4. Change the process priority for conditions and too hard to predict.
the second copy to “high” and
for the third copy to “low”.
5. Copy the prepared set of files
(see preparations section) into
the INPUT directory.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:48:27
 «File Converter» Project

Test case sample 3:

Id Priority Requirement Module Submodule Steps Expected Results
ST.5 A DS-2.4 App Error handler Incorrect start parameters, 1. The console contains the following
loader nonexisting paths messages, the app stops.
1. Start the app with all three Messages:
parameters (SOURCE_DIR, a. Usage message.
DESTINATION_DIR, b. SOURCE_DIR: directory not
LOG_FILE_NAME) pointing to exists or inaccessible.
nonexisting in current file c. DESTINATION_DIR: directory
system paths (e.g.: z:\src\, not exists or inaccessible.
z:\dst\, z:\log.txt in case there is d. LOG_FILE_NAME: wrong file
no z: drive in the system). name or inaccessible path.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:48:27
 4. Stage 4: test suites sample
Imagine we have the set of the following tests:

Id Priority Requirement Module Submodule Title

CPT.1 B DS-2.1 App loader Parameters Default log file name
analyzer
CPT.10 B - Scanner Traverser No access rights to the output
directory
CPT.11 C - Scanner Traverser UNC names in parameters
CPT.12 C - App loader Parameters Windows-like and *nix-like
analyzer paths in parameters
CPT.13 C - Logger - Log file size is 4+ GB
CPT.14 C DS-5.3 File processor Encoding converter Text file contains binary data
CPT.15 C - Scanner Traverser Locked file conversion
CPT.16 C - Scanner Traverser “Read-only” file conversion
CPT.17 B - Scanner Traverser Multiple copies of the app
working simultaneously, file
operations conflict
CPT.2 B BR-1.1 App loader Error handler Destination and input
directory are the same
CPT.3 B BR-1.2 App loader Error handler Destination directory is inside
input directory
CPT.4 B DS-5.2 File processor Encoding converter Conversion of file with
maximum allowed size
CPT.5 B DS-5.2 Scanner Traverser Too big files conversion
CPT.6 B - Scanner Traverser Empty files conversion
CPT.7 C - Scanner Traverser Directory tree inside the input
directory
CPT.8 C - File processor Encoding converter Multiple encodings inside one
file
CPT.9 B DS-2.4 Scanner Traverser No access rights to the input
directory
ST.1 A DS-2.2 App loader Error handler Start with no parameters
ST.2 A DS-2.2 App loader Error handler Start with some mandatory
parameter omitted
ST.3 B DS-2.4 App loader Error handler Invalid path to the log file
ST.4 A DS-2.4 App loader Error handler Invalid paths to the
input/output directory
ST.5 A DS-2.4 App loader Error handler Incorrect start parameters,
nonexisting paths
ST.6 B - App loader Parameters Valid parameters with spaces
analyzer and Cyrillic symbols
ST.7 A DS-5.1 File processor Encoding converter Conversion from all supported
encodings

Now we can compose the following (or any other) sets:

 If we want tests for “Error handler” submodule, we get the following test cases: CPT.2,
CPT.3, ST.1, ST.2, ST.3, ST.4, ST.5.
 If we want tests with the priority “A”, we get the following test cases: ST.1, ST.2, ST.4,
ST.5, ST.7.
 If we want tests with the priority “B” for the “App loader” module, we get the following
test cases: CPT.1, CPT.2, CPT.3, ST.3, ST.6.
 If we want tests for the “BR” section of Requirements, we get the following test cases:
CPT.2, CPT.3.
 If we want tests with the priority “C” for the “File processor” module and the “DS”
section of Requirements, we get the following test case: CPT.14.
 And so on: we choose the set of criteria and get the resulting set of test cases.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:48:27
Focusing on
Important Things
Software Testing Introduction
Closer look to some familiar things…

Defect – an imperfection or deficiency in a work product where

it does not meet its requirements or specifications.

Expected result – the predicted observable behavior of a

component or system executing under specified conditions,
based on its specification or another source.

Actual result – the behavior produced/observed when a

component or system is tested.

2
Closer look to some familiar things…

Defect – an imperfection or deficiency in a work product where

it does not meet its requirements or specifications.

Expected result – the predicted observable behavior of a

component or system executing under specified conditions,
based on its specification or another source.

Actual result – the behavior produced/observed when a

component or system is tested.

3
So, the Quality is…

Quality – is some value for the

customer

This is not necessarily (not only)

functionality

4
How to deliver the value

Imagine a colleague asks you to develop an application (see the

requirements below). He may wait for only 30 minutes and
agrees to pay you a chocolate bar for this work .
The console application searches for all text files in the specified directory and
subdirectories.
Found files are displayed in two ways:
• [mandatory] As an HTML table (columns: file name, file size, full path to file).
• [optional] As a text file: each found file is listed in a separate line: file name,
file size, full path to file.

5
What to do and what not to do? Applying common sense.

1. Search for text files in a given directory:

• Without subdirectories.
• With subdirectories.
• Directory without txt-files.
2. File Names:
• Latin letters only.
• Russian letters, spaces, special characters.
• Unicode.
• Unicode special characters (including “unprintable”).
• Names of different (including invalid) lengths.
• Text files with the extension written in different ways (including upper case
only).

6
What to do and what not to do? Applying common sense.

3. Output (in txt-file and HTML-file):

• Output data correctness:
+ Encodings.
+ The names of different files in different encodings.
• Output data validity:
+ HTML compliance to the standards.

7
What to do and what not to do? Applying common sense.

4. Work with critical parameters:

• The directory nesting depth.
• The path length.
• File size:
+ Zero.
+ 1 byte - 2 GB.
+ 2-4 GB.
+ 4+ GB.
• Number of files. (??? See in the documentation how many items in the
catalog are supported by different FS.)

8
What to do and what not to do? Applying common sense.

5. “Looped” symbolic links.

6. Access rights missing to the analyzed FS object.
7. Output file unavailable for writing:
• Opened by other application.
• Read-only.
• No more disk space.

9
What to do and what not to do? Applying common sense.

8. File system:
• FAT16, FAT32, NTFS4, NTFS5, ext3fs, ext4fs;
• raiserfs (?), FS for MacOS (???);
• Check each of the FSs under different OSs.
9. Location:
• On a local drive.
• On a network:
+ Network folder.
+ Mounted network drive.

10
What to do and what not to do? Applying common sense.

O! M! G!
According to the most pessimistic estimations this testing process may take
several months. But we only have 30 minutes!

11
What to do and what not to do? Applying common sense.

1. Search for text files in a given directory:

Lets create a directory tree with 5-7 nesting
• Without subdirectories. levels. There we’ll have empty, non-empty, etc.
• With subdirectories. directories with and without txt-files and other
• Directory without txt-files. files. (2-3 minutes using FAR manager.)
2. File Names:
• Latin letters only.
• Russian letters, spaces, special characters.
• Unicode.
• Unicode special characters (including “unprintable”).
• Names of different (including invalid) lengths.
• Text files with the extension written in different ways (including upper case
only).

12
What to do and what not to do? Applying common sense.

3. Output (in txt-file and HTML-file): For such a “huge” project it is enough to
• Output data correctness: just look at the output results.
+ Encodings.
+ The names of different files in different encodings.
• Output data validity:
+ HTML compliance to the standards.

13
What to do and what not to do? Applying common sense.

4. Work with critical parameters:

• The directory nesting depth. Another 1-2 minutes using
• The path length. FAR manager.
• File size:
+ Zero.
+ 1 byte - 2 GB.
+ 2-4 GB.
+ 4+ GB.
• Number of files. (??? See in the documentation how many items in the
catalog are supported by different FS.)

14
What to do and what not to do? Applying common sense.

5. “Looped” symbolic links.

6. Access rights missing to the analyzed FS object.
7. Output file unavailable for writing:
• Opened by other application.
• Read-only.
• No more disk space.

15
What to do and what not to do? Applying common sense.

8. File system:
• FAT16, FAT32, NTFS4, NTFS5, ext3fs, ext4fs;
• raiserfs (?), FS for MacOS (???);
• Check each of the FSs under different OSs.
9. Location:
• On a local drive.
• On a network:
+ Network folder. Run the application from a virtual
+ Mounted network drive. machine or from another PC in the LAN.

16
What to do and what not to do? Applying common sense.

So, 5-10 minutes instead of several months.

Yes, IF we have much-much-much more

resources, we may add… some percent to
the quality.

17
So, the Quality is…

Quality – is some value for the

customer

This is what the customer pays for. So, provide

affordable results, not “a tea cup with a complexity
and a price of a space ship”.
18
Focusing on
Important Things
Software Testing Introduction
Good Test Cases with
Four Questions
Software Testing Introduction
First, some obvious ideas…

Test Cases help us

to find defects

But it’s impossible

to find ALL defects

So, we have to find as

many IMPORTANT
defects as we may with
the time available
2
BAD! Don’t do such things!

1. Enter «abc».
2. Enter «%^#^#^%$&^#^^#&^^%$%#^#&^».
3. Enter «; delete from `users`; -- »
4. Leave the field empty.

3
BAD! Don’t do such things!

WHAT?! IS?!
THIS?!

Positive tests?
1. Enter «abc».
No…
2. Enter «%^#^#^%$&^#^^#&^^%$%#^#&^».
3. Enter «; delete from `users`; -- »
4. Leave the field empty. Button click?
What is the purpose No…
of this form?!
4
BAD! Don’t do such things!

Test 1
open http://site.com/subscribe/
Test 2
verifyElementNotPresent //form
Test 3
type //input $%@%^#$%$%^^
5
BAD! Don’t do such things!

This is
Test 1
insanity… 
open http://site.com/subscribe/
Test 2
verifyElementNotPresent //form
Test 3
type //input $%@%^#$%$%^^
6
Three mortal sins in testing

“False-positive tests” Tests that pass successfully and does

not find the existing defect.

“Blind tests” Tests that follow some mysterious logic,

do complex things and all this… for
nothing.

“Garbage tests” Tests that analyze insignificant behavior

aspects (usually covered by other tests),
e.g. tests like “fill this field, then this
field, then this field… and that’s all”.

7
Good! Still may be improved, but really good!

1. Subscribe to the mailing list.

2. Check for the confirmation email.
3. Check for the mailing list emails.
4. Subscribe again with the same email.
5. Unsubscribe (how?).
6. What may go wrong during the subscription
process? How can we simulate this situation?
8
So, the final idea:

What is this?

Who needs it and

what for?

What is the usage

process?

How can something

go wrong?

9
What to do and what not to do? Applying common sense.

Yes, it seems that simple. All the questions

are simple for real, the difficulty is in the
answers. The real answers. The answers that
sometimes took days to find.

10
Good Test Cases with
Four Questions
Software Testing Introduction
Defects and
Defect Reports
Software Testing Introduction
Definitions

Expected result

Defect

Actual result

2
Definitions Read and remember!

Defect – an imperfection or deficiency in a work product where

it does not meet its requirements or specifications.

Expected result – the predicted observable behavior of a

component or system executing under specified conditions,
based on its specification or another source.

Actual result – the behavior produced/observed when a

component or system is tested.

3
More scientific technical approach Read and remember!

Defect, Problem, Bug,

Error, Mistake Failure, Interruption
Fault

Anomaly, Incident, Deviation

Anomaly, incident, deviation – any deviation of the observed

(actual) result, state, behavior, value, property from the observer's
expectations, formed on the basis of requirements, specifications,
other documentation or experience and common sense.

4
Defect Report Read and remember!

Defect report – documentation of the occurrence, nature, and

status of a defect.

5
It’s important!

A defect report should…

Define the severity and
Describe the issue in details
priority of the issue

Provide all necessary data to Help developer to fix the

reproduce and fix the issue issue

6
It’s important!

Bad defect report Good defect report

Lacks significant details Provides significant details

Takes time to understand Easy and obvious

Forces the developer to re- Facilitates quick and easy

do testers work solutions

7
It’s important!

The main goal of a defect

report – is to get the defect
fixed

8
Some simple logic

Before you write down a defect report, make sure you have
answers to the following questions

What have I done? Steps to reproduce

What have I got? Actual result

What did I expect to get? Expected result

9
Defect Report Lifecycle

Submitted Assigned Fixed Verified

Disclaimer: these are the most common and universalClosed

Reopened lifecycle
stages. Yes, each and every bug tracking system has some
specific alterations to this chart. And that’s absolutely OK.
To be
declined

Deferred Declined

10
Defect Report Lifecycle

Submitted Assigned Fixed Verified

Reopened Closed

To be
declined

Deferred Declined

11
Defects and
Defect Reports
Software Testing Introduction
Defect Report
Fields – Part 1
Software Testing Introduction
Key defect report fields

Disclaimer: each and every bug tracking system has much more
fields to fill. There are different approaches in details, but now
we are going to review only KEY FIELDS that remains more or
less the same for past 3-4 decades.

2
Key defect report fields: general overview

ID Summary Description Steps to reproduce

Infinite loop on input file If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19 move the processed file into the destination directory: so the the input directory.
with read-only attribute app processes the file again and again and thus falls into the 2. Set the “read-only” attribute on
infinite loop. this file.
3. Start the app.
Exp: the processed file is moved from the input directory to
the destination directory. Bug: the processing result
appears inside the destination dir
Act: the processed data (new file) appears inside the
(and the file is repeatedly updated
destination directory, but the original file is not deleted from
according to the last write time),
the input directory.
but the original file stays inside the
Req: DS-2.1. input directory.

Reproducibility
Severity Priority Symptom Workaround Comments Attachments
If the customer has no special plans for
Always Medium Normal Incorrect No using “read-only” attribute on files in the -
operation input directory, the Easiest solution is to
remove the attribute once it is detected.

3
Key defect report fields: ID

ID

Infinite loop on input file If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19 move the processed file into the destination directory: so the the input directory.
with read-only attribute app processes the file again and again and thus falls into the 2. Set the “read-only” attribute on
infinite loop. this file.
• Unique. Exp: the processed file is moved from the input directory to
3. Start the app.
the destination directory. Bug: the processing result
• Meaningful (if bugAct:tracking
the processed datasystem allows).
(new file) appears inside the
appears inside the destination dir
(and the file is repeatedly updated
destination directory, but the original file is not deleted from
according to the last write time),
the input directory.
but the original file stays inside the
Req: DS-2.1. input directory.

If the customer has no special plans for

Always Medium Normal Incorrect No using “read-only” attribute on files in the -
operation input directory, the Easiest solution is to
remove the attribute once it is detected.

4
Key defect report fields: summary

Summary

Infinite loop on input file If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19 move the processed file into the destination directory: so the the input directory.
with read-only attribute app processes the file again and again and thus falls into the 2. Set the “read-only” attribute on
infinite loop. this file.
3. Start the app.
Exp: the processed file is moved from the input directory to
the destination directory. Bug: the processing result
• Answers questions:
Act: thewhat did
processed data (newhappen,
file) appears insidewhere
the did it happen,appears inside the destination dir
(and the file is repeatedly updated
destination directory, but the original file is not deleted from
and in what conditions did it happen.
the input directory.
according to the last write time),
but the original file stays inside the
Req: DS-2.1. input directory.
• Should at the same time:
• Provide as much information as possible.
• Be as short as possible.
Always Medium Normal Incorrect No Ifusing the customer has no special plans for
-
• Be easily distinguishable
“read-only” attribute on files in the
operation from other summaries. input directory, the Easiest solution is to
remove the attribute once it is detected.

5
Key defect report fields: description

Description

Infinite loop on input file If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19 move the processed file into the destination directory: so the the input directory.
with read-only attribute app processes the file again and again and thus falls into the 2. Set the “read-only” attribute on
infinite loop. this file.
3. Start the app.
Exp: the processed file is moved from the input directory to
the destination directory. Bug: the processing result
appears inside the destination dir
Act: the processed data (new file) appears inside the
(and the file is repeatedly updated
destination directory, but the original file is not deleted from
according to the last write time),
the input directory.
but the original file stays inside the
Req: DS-2.1. input directory.

• Contains detailed defect description.

• Unlike Summary, Description may be long enough.
• In many
Always MediumBTSNormal
testers use DescriptionNoto write
Incorrect down attributethe actual
If the customer has no special plans for
using “read-only” on files in the -
operation input directory, the Easiest solution is to
result, the expected result and the reference to the
remove the attribute once it is detected.

corresponding requirement. 6
Key defect report fields: steps to reproduce

Steps to reproduce

Infinite loop on input file If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19 move the processed file into the destination directory: so the the input directory.
with read-only attribute app processes the file again and again and thus falls into the 2. Set the “read-only” attribute on
infinite loop. this file.
3. Start the app.
Exp: the processed file is moved from the input directory to
the destination directory. Bug: the processing result
appears inside the destination dir
Act: the processed data (new file) appears inside the
(and the file is repeatedly updated
destination directory, but the original file is not deleted from
according to the last write time),
the input directory.
but the original file stays inside the
Req: DS-2.1. input directory.

• Contains detailed description of actions to be done to

reproduce the defect.
• May contain a short description of the defect
If the customer has no special plans for
Always Medium Normal Incorrect No orattribute
using “read-only” theonfinalfiles in the -
operation input directory, the Easiest solution is to

erroneous state of the application. remove the attribute once it is detected.

7
Key defect report fields: general overview

ID Summary Description Steps to reproduce

Infinite loop on input file If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19 move the processed file into the destination directory: so the the input directory.
with read-only attribute app processes the file again and again and thus falls into the 2. Set the “read-only” attribute on
infinite loop. this file.
3. Start the app.
Exp: the processed file is moved from the input directory to
the destination directory. Bug: the processing result
appears inside the destination dir
Act: the processed data (new file) appears inside the
(and the file is repeatedly updated
destination directory, but the original file is not deleted from
according to the last write time),
the input directory.
but the original file stays inside the
Req: DS-2.1. input directory.

Reproducibility
Severity Priority Symptom Workaround Comments Attachments
If the customer has no special plans for
Always Medium Normal Incorrect No using “read-only” attribute on files in the -
operation input directory, the Easiest solution is to
remove the attribute once it is detected.

8
Defect Report
Fields – Part 1
Software Testing Introduction
Defect Report
Fields – Part 2
Software Testing Introduction
Key defect report fields: general overview

ID Summary Description Steps to reproduce

Infinite loop on input file If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19 move the processed file into the destination directory: so the the input directory.
with read-only attribute app processes the file again and again and thus falls into the 2. Set the “read-only” attribute on
infinite loop. this file.
3. Start the app.
Exp: the processed file is moved from the input directory to
the destination directory. Bug: the processing result
appears inside the destination dir
Act: the processed data (new file) appears inside the
(and the file is repeatedly updated
destination directory, but the original file is not deleted from
according to the last write time),
the input directory.
but the original file stays inside the
Req: DS-2.1. input directory.

Reproducibility
Severity Priority Symptom Workaround Comments Attachments
If the customer has no special plans for
Always Medium Normal Incorrect No using “read-only” attribute on files in the -
operation input directory, the Easiest solution is to
remove the attribute once it is detected.

2
Key defect report fields: reproducibility

• Shows if the defect If an input

appear
file has the
each
“read-only”
time
attribute, the app
we
can not
follow
1. Place a
steps
valid file
totype) into
(size,
19 Infinite loop on input file
withreproduce (“always”), or if the defect sometimes appears
move the processed file into the destination directory: so the the input directory.
read-only attribute app processes the file again and again and thus falls into the 2. Set the “read-only” attribute on

and sometimes infinite doesn’tloop.

(“sometimes”).
Exp: the processed file is moved from the input directory to
this file.
3. Start the app.

• Defects with thethe destination directory.

reproducibility “always”
Act: the processed data (new file) appears inside the
areBug: the processing result
much
appears inside themore
destination dir
(and the file is repeatedly updated
destination directory, but the original file is not deleted from
easy to fix. the input directory.
according to the last write time),
but the original file stays inside the
Req: DS-2.1. input directory.

Reproducibility

If the customer has no special plans for

Always Medium Normal Incorrect No using “read-only” attribute on files in the -
operation input directory, the Easiest solution is to
remove the attribute once it is detected.

3
Key defect report fields: severity

• Shows If an input
the
file has
damage
the “read-only”
thethedefect
attribute, app can not
causes.
1. Place a valid file (size, type) into
19 Infinite loop on input file
• Typical
with read-only attribute values are:
move the processed file into the destination directory: so the
app processes the file again and again and thus falls into the
the input directory.
2. Set the “read-only” attribute on

• Critical
infinite loop.
Exp: the processed file is moved from the input directory to
this file.
3. Start the app.

• Major
the destination directory.
Act: the processed data (new file) appears inside the
Bug: the processing result
appears inside the destination dir
(and the file is repeatedly updated
• Medium
destination directory, but the original file is not deleted from
the input directory.
according to the last write time),
but the original file stays inside the

• Minor
Req: DS-2.1. input directory.

Severity
If the customer has no special plans for
Always Medium Normal Incorrect No using “read-only” attribute on files in the -
operation input directory, the Easiest solution is to
remove the attribute once it is detected.

4
Key defect report fields: priority

• Shows the urgency to fix the defect.

• Typical values
Infinite loop on input file If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19
with read-only attribute
move the are:
processed file into the destination directory: so
app processes the file again and again and thus falls into the
the the input directory.
2. Set the “read-only” attribute on
• ASAPExp: the processed file is moved from the input directory to this
infinite loop. file.
3. Start the app.

• High the destination directory.

Act: the processed data (new file) appears inside the
Bug: the processing result
appears inside the destination dir

• Normal
(and the file is repeatedly updated
destination directory, but the original file is not deleted from
according to the last write time),
the input directory.
but the original file stays inside the
• Low Req: DS-2.1. input directory.

Priority
If the customer has no special plans for
Always Medium Normal Incorrect No using “read-only” attribute on files in the -
operation input directory, the Easiest solution is to
remove the attribute once it is detected.

5
Key defect report fields: severity vs priority

Severity Priority
Shows, how dangerous the Shows, how quickly the
defect is defect should be fixed

There may be any combination of “severity & priority”

6
Key defect report fields: symptom

Allows defects classification by typical indication:

• Cosmetic flaw.on input file
Infinite loop • Scalability
If an input file has issue.
the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19 •
• Data
withcorruption/loss.
read-only attribute Low performance.
move the processed file into the destination directory: so the
app processes the file again and again and thus falls into the
the input directory.
2. Set the “read-only” attribute on
• Documentation issue. • System
infinite loop. crash. this file.

• Incorrect operation. • Unexpected

Exp: behavior.
the processed file is moved from the input directory to
3. Start the app.
the destination directory. Bug: the processing result
• Installation problem. • Unfriendly behavior. appears inside the destination dir
Act: the processed data (new file) appears inside the
• Localization issue. • Variance
destination frombutspecs.
directory, the original file is not deleted from
(and the file is repeatedly updated
according to the last write time),
• Missing feature. • theEnhancement.
input directory.
but the original file stays inside the
Req: DS-2.1. input directory.

Symptom
If the customer has no special plans for
Always Medium Normal Incorrect No using “read-only” attribute on files in the -
operation input directory, the Easiest solution is to
remove the attribute once it is detected.

7
Key defect report fields: workaround

Infinite loop on input file If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19 move the processed file into the destination directory: so the the input directory.
with read-only attribute app processes the file again and again and thus falls into the 2. Set the “read-only” attribute on
infinite loop. this file.
3. Start the app.
• Shows, Exp:
if destination
the there is a way to achieve the
the processed file is moved from the input directory to
directory. desired
Bug: the processing result
appears inside the destination dir
result without being interrupted
file is not deleted fromby the defect.
Act: the processed data (new file) appears inside the
destination directory, but the original
(and the file is repeatedly updated
according to the last write time),
the input directory.
• Typical values
Req: DS-2.1. are: “yes”, “no”.
but the original file stays inside the
input directory.

Workaround
If the customer has no special plans for
Always Medium Normal Incorrect No using “read-only” attribute on files in the -
operation input directory, the Easiest solution is to
remove the attribute once it is detected.

8
Key defect report fields: comments

Infinite loop on input file If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19 move the processed file into the destination directory: so the the input directory.
with read-only attribute app processes the file again and again and thus falls into the 2. Set the “read-only” attribute on
infinite loop. this file.
3. Start the app.
Exp: the processed file is moved from the input directory to

• Optional field.
the destination directory.
Act: the processed data (new file) appears inside the
Bug: the processing result
appears inside the destination dir
(and the file is repeatedly updated
• May contain any data useful to the process
destination directory, but the original file is not deleted from
the input directory. ofwrite time),
according to the last
but the original file stays inside the

bug fixing.
Req: DS-2.1. input directory.

Comments
If the customer has no special plans for
Always Medium Normal Incorrect No using “read-only” attribute on files in the -
operation input directory, the Easiest solution is to
remove the attribute once it is detected.

9
Key defect report fields: attachments

Infinite loop on input file If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19 move the processed file into the destination directory: so the the input directory.
with read-only attribute app processes the file again and again and thus falls into the 2. Set the “read-only” attribute on
infinite loop. this file.
3. Start the app.
Exp: the processed file is moved from the input directory to
• Optional field.the destination directory.
Act: the processed data (new file) appears inside the
Bug: the processing result
appears inside the destination dir

• May contain any files, screenshots,

destination directory, but the original file is not deleted from
the input directory.
(and the file is repeatedly updated
etc.
accordinguseful totime),
to the last write
but the original file stays inside the
the process of bug fixing.
Req: DS-2.1. input directory.

Attachments
If the customer has no special plans for
Always Medium Normal Incorrect No using “read-only” attribute on files in the -
operation input directory, the Easiest solution is to
remove the attribute once it is detected.

10
Useful ideas

Use active voice and simple phrases in steps description

Use objective description in actual/expected results
Write simple, this is not a fiction novel

Use exact names for interface elements

Don’t explain basics

11
Key defect report fields: general overview

ID Summary Description Steps to reproduce

Infinite loop on input file If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into
19 move the processed file into the destination directory: so the the input directory.
with read-only attribute app processes the file again and again and thus falls into the 2. Set the “read-only” attribute on
infinite loop. this file.
3. Start the app.
Exp: the processed file is moved from the input directory to
the destination directory. Bug: the processing result
appears inside the destination dir
Act: the processed data (new file) appears inside the
(and the file is repeatedly updated
destination directory, but the original file is not deleted from
according to the last write time),
the input directory.
but the original file stays inside the
Req: DS-2.1. input directory.

Reproducibility
Severity Priority Symptom Workaround Comments Attachments
If the customer has no special plans for
Always Medium Normal Incorrect No using “read-only” attribute on files in the -
operation input directory, the Easiest solution is to
remove the attribute once it is detected.

12
Defect Report
Fields – Part 2
Software Testing Introduction
Typical Defect
Reporting Mistakes
Software Testing Introduction
The defect report is a bad one if…

Not enough information to understand and reproduce the

defect.
Description
The search returns wrong results if some administrative
settings are changed.

Really!? Such a
shame…

2
The defect report is a bad one if…

The “defect” is found in a functionality not yet marked as

“ready for testing”.

Always read build release notes

and look to the project
management system to be sure the
functionality is ready for testing.

3
The defect report is a bad one if…

Any field contains wrong information (for any reason – copy-

paste issues, misunderstanding, inattentiveness, etc…)
Summary
Search page is not available

The “Search” page is

OK. It’s the “Catalog”
page that’s missing.

4
The defect report is a bad one if…

A defect report contains slang or strong words.

Description
Some s[censored] has happened again to this f[censored]
back-redirectors!

5
The defect report is a bad one if…

A defect report contains criticism to someone's work.

Description
….

P.S. What a stupid fool may fail with such simple feature
implementation?!

6
The defect report is a bad one if…

Some critical details (e.g. the environment specifics) are

missing in the report.

W7, Chr, EN
W10, FF, RU

7
The defect report is a bad one if…

A defect report has inappropriate severity/priority values.

Summary Severity Priority

Each third restart leads to BSOD Minor Low
and loss of ALL user data.

Bad joke 

8
The defect report is a bad one if…

A defect report is written messy, illiterately, confusingly.

Summary Description Steps to reproduce

No save DB. 1. Save DB.
2. Change DB.
3. Create DB.
4. Save no.

9
The defect report is a bad one if…

Necessary logs, screenshots, etc. are missing from the report.

Summary Description STR
The second click on “New” brings With a document loaded, the 1. Open a
up a window with unreadable second click on “New” menu document.
signs. item brings up a window with 2. Click “New”.
unreadable signs (nothing like 3. Click “New”
letters). again.

But the developer sees this

window. So, where is the bug?!

10
The defect report is a bad one if…

The tester didn’t manage to convince the team about the

defect consequences.

Is it trifle?

They don’t think so…

11
Conclusion

Most mistakes come from the inattentiveness. So be careful,

re-read your defect report.

Try to see the technical background, the real cause of the

defect. Don’t stop at the interface behavior only.

The more “tricky” defect you found, the more time it may take
to deal with. But it saves much more time in the future.

12
Typical Defect
Reporting Mistakes
Software Testing Introduction
Defect Reporting
Recommendations
Software Testing Introduction
General idea…

All the next points are just variations of one synonymous series

Attentiveness Particularity

Thoroughness Nicety

Accuracy Scrupulousness

Exactness …

2
Follow this advice!

Describe the STR with even the smallest details.

STR
1. Clear cookies or create a fresh new user profile.
2. Open http://pathtoapplication/admin_login/
3. Enter “dba_admin” into the “Login” field.
4. Enter “globaladmin12” into the “Password” field.
5. Click the logo at the left top corner of the page.

Bug: there is a redirection to the DBMS main administration page without additional
authentication.

3
Follow this advice!

Add details not only to STR, but to all other fields

Description
Wrong calculation of the total price of the order in the basket. It seems like
the last item in the list is processed twice.
Act: the “Total” field contains the prices sum + the price of the last item.
Exp: the “Total” field contains the exact sum of all prices of all items in the list.
Req: R892.34b/73.

4
Follow this advice!

Write down every detail, don’t rely on the “esoteric” knowledge.

STR
1. Clear cookies or create a fresh new user profile.
2. Open http://pathtoapplication/admin_login/
3. Change the current language to any different from default.
4. Authenticate with your domain credentials.
5. Perform a search for any dismissed employee (search has to return 2+ results).

Bug: the first match doesn’t contain a photo.

5
Follow this advice!

Reference to the requirement the defect violates.

6
Follow this advice!

Provide key details explicitly!

Description
After re-authentication caused by logout after 10 minutes timeout, the each-minute
auto saving of current document stops working.
Act: the app either doesn’t auto save the document, or the auto saving process fails
with no error messages.
Exp: the app automatically saves the current document each minute, corresponding
hint appears for two seconds at the right top corner of the main app window.
Req: R1752.2a/4.

7
Follow this advice!

Even if your BTS doesn’t provide such feature, name the

environment the defect found within.

Summary
[XP, IE6, ru] JQuery fails to initialize.

8
Follow this advice!

Don’t be emotional. Even if you consider the defect to be

someone’s epic fail, just provide details, not your opinion.

Summary
The license agreement main page contains 72 typos.

9
Follow this advice!

Write separate defect report for each found defect. Don’t

combine several defects into one defect report.
Summary
The catalog map export to JPG causes BSOD.
The search doesn’t take into account goods description.

But! If you are absolutely sure you found one defect with
several indications, you can write one defect report and
provide the list of indications in the Description field.

10
Follow this advice!

Make the root cause analysis. Try to find the real cause of the
defect. Don’t stop with only system behavior description.

Comments
According to the symptoms, the performance falls to almost zero once the DB
size reaches 70-80% of the RAM. It seems that the app tries to load all the DB
into the RAM, while it really needs only 3-5% of records.

11
Follow this advice!

If you have enough technical experience, provide a

recommendation on how to fix the defect.
Comments
…
We may try optimizing search queries, add caching or at least removing “old datasets”
from RAM. This may help reduce the memory usage and should really help those users
who perform a lot of “heavy” queries.

Even better: we can try to adjust our DAO-layer not to buffer “ahead data”. See the
description of the similar problem here: http://habr.com/articles/art97234/

12
Follow this advice!

Write down the defect report immediately once you found it

and performed all the necessary actions (like root cause
analysis, checking the BTS for duplicates and so on).

The less time has passed, the more details you remember.

The sooner the defect report appears, the sooner someone

starts working on the defect.

13
Follow this advice!

Analyze the most critical consequences of the defect. This

information will not only help with Severity and Priority fields,
but may lead you to some useful ideas on additional tests.

E.g., this is not just “page not

found” issue, this is also
security issue and an indication
of web app engine
misconfiguration.

14
Follow this advice!

It’s likely that you yourself will verify that the defect is fixed. So
writing a good defect report now saves the future you a lot of
time and effort.

15
Follow this advice!

Use spellchecking…

And remember about…

Attentiveness Particularity

Thoroughness Nicety

Accuracy Scrupulousness

Exactness …
16
Defect Reporting
Recommendations
Software Testing Introduction
 «File Converter» Project

Defect Reports Sample

SAMPLE

This is NOT a part of project documentation! This is just a way to

demonstrate some defect reports.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:50:48
Defect report sample 1:

ID Summary Descriptions Steps to reproduce

19 Infinite loop on input If an input file has the “read-only” attribute, the app can not 1. Place a valid file (size, type) into the input directory.
file with read-only move the processed file into the destination directory: so the 2. Set the “read-only” attribute on this file..
attribute app processes the file again and again and thus falls into the 3. Start the app.
infinite loop.
Bug: the processing result appears inside the destination dir
Exp: the processed file is moved from the input directory to (and the file is repeatedly updated according to the last write
the destination directory. time), but the original file stays inside the input directory.
Act: the processed data (new file) appears inside the
destination directory, but the original file is not deleted from
the input directory.
Req: DS-2.1.

Reproducibility Severity Priority Symptom Workaround Comments Attachments

Always Medium Normal Incorrect No If the customer has no special plans for using “read-only” -
operation attribute on files in the input directory, the Easiest solution is to
remove the attribute once it is detected.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:50:48
 «File Converter» Project

Defect report sample 2:

ID Summary Descriptions Steps to reproduce
27 Non-empty The app doesn’t produce any warning and deletes any non- 1. Create a directory with any valid name (e.g. TEST).
subdirectory in empty (empty as well) subdirectory in SOURCE_DIR Create a subdirectory (e.g. BUG) inside it and place
SOURCE_DIR is directory. some files inside this subdirectory.
deleted w/o any Exp: if the app detects a non-empty subdirectory inside 2. Start the app with the SOURCE_DIR parameter
warning SOURCE_DIR, it produces the error message and stop. The pointing to the TEST directory (see the step 1), e.g.:
message should be: “Non-empty subfolder [name] in “php file_converter.phar c:\TEST c:\OUT”.
SOURCE_DIR folder detected. Remove it manually or
restart application with --force_file_operations key to remove Bug: the subdirectory BUG is deleted with all its
automatically.” contents.
Act: any non-empty subdirectory (along with all its contents)
in SOURCE_DIR is deleted without any warning.
Req: DS-8.5.12.

Reproducibility Severity Priority Symptom Workaround Comments Attachments

Always Major Normal Data loss No -

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:50:48
 «File Converter» Project

Defect report sample 3:

ID Summary Descriptions Steps to reproduce
41 File processing stops After something about 30 minutes of work the app stops 1. Start the app and let it work for at least 30 minutes.
after 0.5 hour of work reacting to the new valid files inside the SOURCE_DIR. 2. Put a valid file inside the SOURCE_DIR.
Exp: the app continues to process the new valid files in the
SOURCE_DIR until it is stopped by an appropriate Bug: the app does not process the file.
command.
Act: the app still utilizes some system resources (CPU,
RAM, drives) but does not react to any new valid file inside
the SOURCE_DIR.
Req: UR-1.

Reproducibility Severity Priority Symptom Workaround Comments Attachments

Always Critical Normal System crash No It looks like some descriptors of file -
system objects are closed
automatically due to some timeout.

This behavior may have the same

cause as in bug 38 (once started,
the app does not see that
SOURCE_DIR and/or
DESTINATION_DIR are deleted).

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:50:48
 «File Converter» Project

Defect report sample 4:

ID Summary Descriptions Steps to reproduce
2352 Multiple instances With 2+ instances of the application running with the same 1. Start several (3-5-7+) instances of the app with the
crash on competition input directory configured, they compete for files in this same valid parameters.
for the SOURCE_DIR directory that leads to multiple file operation failures and 2. Start putting valid files into the SOURCE_DIR.
(finally) to crash of some instances.
Exp: all instances work (not crashes); if a file operation fails, Bug: one by one most instances crash.
the affected instance makes a log record and retries the
operation.
Act: the affected instance crashes.
Req: UR-12.

Reproducibility Severity Priority Symptom Workaround Comments Attachments

Always Critical Normal System crash No It looks like some semaphore or -
synchronization mechanism is
either broken or do not take the
SOURCE_DIR into account. Or
some file operation failure handler
does not work as it has to.

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:50:48
Test Result
Reports
Software Testing Introduction
Definitions Read and remember!

Test Progress Report – a test report produced at regular

intervals about the progress of test activities against a baseline,
risks, and alternatives requiring a decision.

Test Summary Report – a test report that provides an

evaluation of the corresponding test items against exit criteria.

Test Report, Test Result Report – documentation summarizing

test activities and results.

2
Why is Test Result Report so important? General planning
and requirements
analysis
Acceptance
1
TRR is the basis for criteria
establishment
the planning of the Test results
reporting 2
next iteration. We
8
have to know Test results
analysis
“where are we” in 3 Test strategy
establishment
7
order to decide
“where to go” and 6
4
“how to get Defect reporting
there”. 5 Test cases creation

Test cases
execution

3
Test Result Report Purpose

Test Result Report

Reliable source of
Basis for the planning of
information for
the next iteration
stakeholders

4
Principles of Test Result Reporting

The test result report is published on known schedule

Test lead is responsible for the test result report creation

If needed, the test result report is discussed on a meeting

The test result report is created based on some template

5
Who uses the Test Result Report

Who What for

Stakeholders and Customer Get information on progress

Project manager Make managerial decisions

Development team leader Get another point of view

Testing team leader Summarize knowledge

6
Test Result
Reports
Software Testing Introduction
Test Result Report
Sections
Software Testing Introduction
General Test Result Report Sections Overview

Summary Overall defects statistics

Test team Recommendations

Testing process description Attachments / appendixes

Timetable

New defects statistics

New defects list

2
Summary Read and remember!

Summary – a very brief and meaningful description of the main

results, achievements, problems, etc. – all that really worth
mentioning.
With best case scenario most stakeholders may only read Summary and get all
the necessary information they want.
E.g.: During May 26-28 four builds were released. The latest build has
successfully passed 100% of the smoke test, and 76% of the critical path
test. 98% of the requirements of high importance are implemented
correctly. All key quality metrics are in the green zone … [and so on]

3
Test team Read and remember!

Test team – a section containing the list of testers involved in

the project during the reporting time.

E.g.:
Name Role Responsibility
Joe Black Tester Documentation creation, test-
cases execution, participation in
code-review
Jim White Senior Participation in requirements
developer testing and code review

4
Testing process description Read and remember!

Testing process description – a section containing not

final facts and conclusions, but literally the description
of what and how the test team has been doing.
E.g.: Each of the four builds (3-6) released during the reporting period was
tested under Windows 10 Ent x64 and Linux Ubuntu 16 LTS x64 in the PHP
7.3.0 runtime environment. The smoke test (see
http://projects/FC/Testing/SmokeTest) was performed using automation
based on batch files (see \\PROJECTS\FC\Testing\Aut\Scripts). The critical
path test (see http://projects/FC/Testing/CriticalPathTest) was performed
… [and so on]
5
Timetable Read and remember!

Timetable – is a section containing… the timetable ☺, i.e.

automatic data export from time reporting system. It helps to
see the workload, time distribution, time consumption and so
on.
E.g.: Name Date Activity Duration, hours
Joe Black 27.05.2015 Test cases creation 2
Joe Black 27.05.2015 Pair testing 2
Joe Black 27.05.2015 Smoke test automation 1
Joe Black 27.05.2015 Defect reporting 2
Jim White 27.05.2015 Code review 1
Jim White 27.05.2015 Pair testing 2

6
New defects statistics Read and remember!

New defects statistics – is a section, containing the number of

defects found/fixed/etc during the reporting time. This is the
automatic export from bug-tracking system.
E.g.: Severity
Status Quantity Low Medium Major Critical
Submitted 23 2 12 7 2
Fixed 17 0 9 6 2
Verified 13 0 5 6 2
Reopened 1 0 0 1 0
Declines 3 0 2 1 0

7
New defects list Read and remember!

New defects list – is a section, containing the list of defects

found during the reporting time. This is the automatic export
from bug-tracking system.
E.g.:
ID Severity Summary

BR 21 Major The app does not distinguish files and symbolic

links to files
BR 22 Critical The app ignores .md input files
[And so on for all the submitted defects]

8
Overall defects statistics Read and remember!

Overall defects statistics – is a section, containing the number

of all defects found/fixed/etc from the beginning of the project.
This is the automatic export from bug-tracking system.
E.g.: Severity
Status Quantity Low Medium Major Critical
Submitted 34 5 18 8 3
Fixed 25 3 12 7 3
Verified 17 0 7 7 3
Reopened 1 0 0 1 0
Declines 4 0 3 1 0

9
Recommendations Read and remember!

Recommendations – is a tricky section, that usually post-

factum contains previously discussed and accepted decisions
on changes in team, schedule, test plan, technology scope and
so on.

E.g.: We recommend rescheduling production servers S12 and S931 update to

Tuesday (May 18) morning as there is high risk of unexpected
incompatibility issues.

10
Attachments / appendixes Read and remember!

Attachments / appendixes – is a section containing any useful

data (documents, logs, charts, drawings, schemas, etc.)
referenced from the report text or self-meaningful.

E.g.:

11
Test Result Report
Sections
Software Testing Introduction
 «File Converter» Project

Test Result Report

SAMPLE
Project Documentation

Background Information needed to assess the effectiveness of the testing

process and the current state of the project.
Purpose To provide the key information about the testing process and
testing results.
Scope Testing process description, test results, metrics, timetable,
recommendations.
Audience Management staff, QA team, project team.
File 06 03 - Test Result Report Sample.docx

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:51:43
 «File Converter» Project

Contents

1. Summary ...................................................................................................... 3
2. Test team ..................................................................................................... 3
3. Testing process description ......................................................................... 3
4. Timetable ...................................................................................................... 3
5. New defects statistics ................................................................................... 4
6. New defects list ............................................................................................ 4
7. Overall defects statistics .............................................................................. 4
8. Recommendations ....................................................................................... 5
9. Attachments ................................................................................................. 5

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:51:43
 «File Converter» Project

1. Summary
During May 26-28 four builds were released. The latest build has successfully passed
100% of the smoke test, and 76% of the critical path test. 98% of the requirements of high
importance are implemented correctly. All key quality metrics are in the green zone, so there
is every reason to expect the project completion on time (at the moment, real progress
exactly corresponds to the plan). At the next iteration (starting May 29) the remaining low-
priority test cases are scheduled for execution.

2. Test team

Name Role Responsibility

Joe Black Tester Documentation creation, test-
cases execution, participation in
code-review
Jim White Senior developer Participation in requirements
testing and code review

3. Testing process description

Each of the four builds (3-6) released during the reporting period was tested under
Windows 10 Ent x64 and Linux Ubuntu 16 LTS x64 in the PHP 7.3.0 runtime environment.
The smoke test (see http://projects/FC/Testing/SmokeTest) was performed using
automation based on batch files (see \\PROJECTS\FC\Testing\Aut\Scripts). The critical path
test (see http://projects/FC/Testing/CriticalPathTest) was performed manually. Regression
testing shows high stability of functionality (only one defect was found with the importance
of “medium”). Re-testing shows a noticeable quality increase (83% of previously detected
defects were fixed).

4. Timetable

Name Date Activity Duration, hours

Joe Black 27.05.2015 Test cases creation 2
Joe Black 27.05.2015 Pair testing 2
Joe Black 27.05.2015 Smoke test automation 1
Joe Black 27.05.2015 Defect reporting 2
Jim White 27.05.2015 Code review 1
Jim White 27.05.2015 Pair testing 2
Joe Black 28.05.2015 Test cases creation 3
Joe Black 28.05.2015 Pair testing 1
Joe Black 28.05.2015 Defect reporting 2
Joe Black 28.05.2015 Test result reporting 1
Jim White 28.05.2015 Code review 1
Jim White 28.05.2015 Pair testing 1

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:51:43
 «File Converter» Project

5. New defects statistics

Severity
Status Quantity Low Medium Major Critical
Submitted 23 2 12 7 2
Fixed 17 0 9 6 2
Verified 13 0 5 6 2
Reopened 1 0 0 1 0
Declined 3 0 2 1 0

6. New defects list

ID Severity Summary
BR 21 Major The app does not distinguish files and symbolic
links to files
BR 22 Critical The app ignores .md input files
And so on for all the 23 submitted defects

7. Overall defects statistics

Severity
Status Quantity Low Medium Major Critical
Submitted 34 5 18 8 3
Fixed 25 3 12 7 3
Verified 17 0 7 7 3
Reopened 1 0 0 1 0
Declined 4 0 3 1 0

Overall defects statistics

40
35
30
25
20
15
10
5
0
26.05.2015 26.05.2015 27.05.2015 27.05.2015 28.05.2015 28.05.2015
1 2 3 4 5 6

Submitted Fixed Total submitted Total fixed

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:51:43
 «File Converter» Project

8. Recommendations
No significant changes are needed at the current moment.

9. Attachments
Metrics through time changes:

Metrics
200
180
160
140
120
100
80
60
40
20
0
26.05.2015 26.05.2015 27.05.2015 27.05.2015 28.05.2015 28.05.2015
1 2 3 4 5 6

Tsp Dftp Dfcp S Te Rc

EPAM Training Center: Software Testing Introduction Most Recent Update: 14.06.2019 10:51:43
What is Test
Automation
Software Testing Introduction
Definitions Read and remember!

Test automation – the use of software to perform or support

test activities, e.g., test management, test design, test
execution and results checking.

Test automation framework – a tool that provides an

environment for test automation. It usually includes a test
harness and test libraries.

Test harness – a test environment comprised of stubs and

drivers needed to execute a test.
2
Test automation pros and cons: time consumption

Manual testing

Design Execution Execution Execution … Execution

Development, debugging and support Execution

Automated testing

3
Test automation pros and cons: some factors to consider

Time (manual vs automated)

Times to repeat

Support effort

Staff

4
Test automation pros and cons: pros

Speed

Reliability

Power

Statistics

Low-level actions

5
Test automation pros and cons: cons

Staff

Strategy

Tools

Changes

Hopes 

6
Test automation: local conclusion

Useful approach if managed wisely

Become more and more required

Manual testing still stays on the stage

Automation not really replaces manual testing

7
And one more thing…

Automation is good not only for testing, but in “everyday life”...

Batch files to automate Scripts to process office

system operations documents

Reporting (data aggregation, Backups for your workstation

cross-checking and so on) and mobile devices

E-mail rules, auto-replies, Virtual machines for quick

processing scripts environment switch
8
What is Test
Automation
Software Testing Introduction
Areas of Test
Automation High
Efficiency – Part 1
Software Testing Introduction
General overview

Regression testing Performance testing

Multi-environment testing Smoke test for large systems
Configuration testing Applications without human interface
Combinatory techniques Technical and low-level operations
Unit testing Long routine operations
Some cases of integration testing Standard functionality
Basic security testing Classic well-automated areas

2
Regression testing

3
Multi-environment testing

OS LNG DBMS Java

W7 RU My5 J8
W7 EN My5 J9
W7 RU My8 J8
W7 EN My8 J9
W8 RU My5 J8
W8 EN My8 J9
W8 EN My5 J8
U12 RU My8 J9
U14 EN My5 J8
U16 EN My8 J9
W10 RU My8 J8
W10 EN My5 J9
W10 RU My8 J8
W10 EN My5 J9

4
Configuration testing

5
 Combinatory techniques
Valid

Free Invalid

Valid

Local Reserved Invalid

Valid

Free Invalid

Valid

Windows Network Reserved Invalid

SOURCE_DIR

Linux Network Reserved Invalid

Valid

Free Invalid

Valid

Local Reserved Invalid

Valid

Free Invalid

Valid

6
Unit testing

7
Some cases of integration testing

START
Начало
Предположим, что страница отправки файла уже
загружена в браузер
Веб-сервер в это время User enters an URL into
some web-client

Проверка
обрабатывает другие запросы
Кодирование данных
Выбор файла в диалоге корректности или находится в режиме
выбора файла и заполнения всех
Все поля Да формы (метод POST,
MySQL запущен и работает
отправка формы полей формы (с
заполнены верно? способ кодирования ожидания PHP может быть подключён как модуль веб-сервера, работать как No Client displays
multipart/form-data) совершенно независимо от веб- Client sends request to Firewall checks if this
an error
помощью JavaScript)
независимый «сервис» или запускаться веб-сервером по факту DNS-server connection is allowed
message
Нет сервера и интерпретатора PHP,
получения запроса – в любом случае PHP «узнаёт» о запросе
он просто ожидает поступления Yes
Извлечение данных только после того, как веб-сервер выполнит свою часть анализа END
Блокировка отправки Финальная подготовка и запроса, определение запросов (или входящих No
запроса и передаст управление интерпретатору PHP
DNS-server has this
Получение и первичный domain name is in it’s
формы, отображение
сообщения об ошибке
отправка запроса на
сервер
анализ запроса
MIME-типа
запрошенного файла и
соединений) own DNS-table?
Loop begin:
его доступности find the ip-address
Yes
Upper DNS-server
Получение и Инициализация движка either finds the IP
отображение Передача файла на Загрузка и анализ приложения, in his table, or
Конец, Да Нет В файле есть Нет Обнаружены Нет
сообщения об ошибке Есть ошибки? исполнение файла, являющегося подключение внешних sends the request
загрузка не выполнена ошибки? ошибки?
(здесь до корректной обработки ситуации
приложением может не дойти – будет
интерпретатору PHP «точкой входа» PHP-Файлов, создание further to his upper
просто сообщение от веб-сервера) объектов и т.д. DNS-server: up to
Да Да root DNS-servers.

Получение и Передача сообщения Генерация корректного Loop end:

Конец, отображение Отправка ответа об ошибке сообщения об ошибке find the ip-address
сообщения об ошибке Отмена приёма файла (в хорошем случае – корректного (если ошибка может быть обработана
загрузка не выполнена (сообщение либо сгенерировано движком
браузеру сообщения от движка приложения, в движком сайта, в противном случае –
приложения, либо получено от плохом случае – стандартного просто генерация сообщения об ошибке
интерпретатора PHP) сообщения от интерпретатора PHP) от интерпретатора PHP)
Yes No
The IP-
address found

Восстановление
Проверка логина,
сессии, Проверка типа, размера
Обнаружены Нет Установление пароля, хоста- Обнаружены Нет
инициализация и иных параметров
ошибки? соединения с СУБД инициатора, иных ошибки? DNS-server returns the
необходимых файла DNS-server returns the
параметров IP-address of the
данных error message to client
requested domain
Да Да

Получение и Передача результатов Повторная генерация

отображение веб-серверу страницы загрузки Client sends request to Client displays an
Конец, Отправка ответа Генерация корректного
сообщения об ошибке Отмена приёма файла (в этот момент веб-сервер может ещё файла с только что given IP-address. error message
загрузка не выполнена браузеру продолжать приём файла – зависит от сообщения об ошибке
(сообщение либо сгенерировано движком
приложения, либо получено от способа подключения PHP К серверу и полученным The request contains URL and any
интерпретатора PHP) множества иных условий) сообщением об ошибке additional info needed (GET/POST/
COOKIES/etc).

END
No Client displays
Формирование запроса Firewall checks if this
Предобработка и Финальная подготовка и an error END
к СУБД, проверяющего
подстановка выполнение запроса к
Обработка запроса от Здесь при connection is allowed
message
наличие такого файла приложения
на сервере
параметров запроса СУБД обработке Yes
запросов тоже
No Server OS sends Client displays
реализуется Server OS checks if this
connection is allowed an error message an error
Получение и
отображение
Передача результатов
Формирования полного
много стадий to client message
Конец, Отправка ответа веб-серверу Да Этот файл уже Нет Формирование хэша
загрузка не выполнена
сообщения об ошибке
браузеру
Отмена приёма файла (в этот момент веб-сервер может ещё
продолжать приём файла – зависит от есть? имени файла
имени файла для контроля и Yes
или страницы с хранения на сервере
сообщением об ошибке
способа подключения PHP К серверу и
множества иных условий) проверок – они Web-server receives the
END
опущены для request

упрощения
Перемещение Установка необходимых
Формирование запроса
к СУБД, добавляющего
Финальная подготовка и
Обработка запроса от
схемы
принятого файла в прав в файловой выполнение запроса к No Web-server sends Client displays
место хранения системе
информацию о
СУБД
приложения алгоритма Web-servers checks if the
requested domain is listed in an error message an error
принятом файле it’s virtual hosts table
to client message

Yes

Yes
Web-server END
Получение и Корректировка Web-servers checks if URL preprocesses URL
Конец, отображение страницы Отправка ответа Получение данных от Нет Файл сохранён Да данных сессии и preprocessing is needed with appropriate
загрузка не выполнена с сообщением об браузеру приложения успешно? объектов module
ошибке приложения
No

Web-server analyses the

request
(extracts GET/POST/
Конец,
Получение и
отображение новой
Отправка ответа Получение данных от Отправка результатов Подготовка новой Алгоритм, раскрывающий COOKIES/etc data and the
requested file name)
загрузка выполнена браузеру приложения веб-серверу страницы приложения
страницы суть этого блока, см. ниже
Server OS reports an
Server OS checks if the No error to web-server, Client displays
requested object exists and web-server sends the an error
can be accessed error message (f.e. message
404, 403, etc) to client
Yes

STATIC Web-server reads END

Web-server checks the
content type the file and sends it
content to client

DYNAMIC
Web-server starts some Client analyses, parses
the content, performs
external software (ES) some other actions (starts
and sends all needed JS, flash-plugins, etc)
data to it

Client displays
ES analyses and parses the the content to
file (containing a script written
user
in some programming
language) or executes byte-
coded script
END

No
ES returns an error Web-server sends this Client displays an
Is script OK?
message to web-server error message to client error message

Yes
END
ES executes the script

ES establishes a
Yes
Is connection to connection to DBMS and
DBMS reuqired? sends a query to some
database
No
ES reacts at this message
No in some way, f.i. generates
DBMS checks if this DBMS returns an error
some special error page
connection is allowed message to DCGS and returns it to web-
server
Yes

No Web-server sends the

DBMS checks if the
given query can be result of DCGS work to
executed client

Yes

DBMS executes the given query and Client displays the

returns data to DCGS given information

ES assembles all needed END

data, generates the
DYNAMIC content and
sends it to web-server

Client analyses,
parses the content, Client displays
Web-server send this
performs some other the content to
data to client actions (starts JS, user
flash-plugins, etc)

END

8
Basic security testing

9
Areas of Test
Automation High
Efficiency – Part 1
Software Testing Introduction
Areas of Test
Automation High
Efficiency – Part 2
Software Testing Introduction
General overview

Regression testing Performance testing

Multi-environment testing Smoke test for large systems
Configuration testing Applications without human interface
Combinatory techniques Technical and low-level operations
Unit testing Long routine operations
Some cases of integration testing Standard functionality
Basic security testing Classic well-automated areas

2
Performance testing

3
Smoke test for large systems

4
Applications without human interface

DBMS
Web-service
Web-server
Server side of client-server application
… and so on …

5
Technical and low-level operations

6
Long routine operations

7
Standard functionality

Cart
Payments
Registration
Settings management
… and so on …

8
Classic well-automated areas

Link-checking
Backups
System monitoring
Updates
… and so on …

9
But…

Beware of!
Complex user interface
Huge unstable technology domain
Usability and other human-related things
Unstable requirements
…

10
Areas of Test
Automation High
Efficiency – Part 2
Software Testing Introduction
Tips on Test
Automation
Software Testing Introduction
With test cases to be automated…

More details
1. Standard search page is loaded. 1. The search page is loaded:
• TITLE = “Search page”
• There is a form with fields:
• input type=”text”
• input type=“submit” value=“Go!”
• There is the logo (logo.jpg)
• There are no pictures except the logo.
• …

2
With test cases to be automated…

Miscellaneous tools/systems
1. Click the “Search” link. 1. Click the “Search” link.
2. Use clickAndWait for timing 2. Wait for the new page to load.
synchronization.

1. Sent the WM_CLICK message to any 1. Set focus to any non-minimized app window.
visible app window. 2. Simulate left mouse click (or tap) for this
focused window.

3
With test cases to be automated…

Timing synchronization
1. Click the “Refresh” button. 1. Click the “Refresh” button.
2. Check that the “Results” grid is empty. 2. Wait for all background AJAX queries to
complete (once they are, the “In progress” bar
becomes disabled).
3. Check that the “Results” grid is empty.

4
With test cases to be automated…

No hard-coding!
1. Open http://ourcoolapplication.com/ 1. Open {START_URL}.

5
With test cases to be automated…

Test independence
1. With the previously created file… … … 1. Create a file following the next rules: … … …
2. With the file created in Step 1… … …

6
With test cases to be automated…

Automation is programming
if ($date_value == '2012.08.22') if ($date_value == date('Y.m.d'))
{ {
… Hardcoding …
} }

if ($status = 734) if ($status == POWER_USER)

{ {
Magic
… number
…
} }

7
Understand your tool, perform meaningful actions!

Understand your tool

Imagine that someone has to check with Selenium IDE 2.x that some
web-page contains some text in some defined place.
Using verifyTextPresent command is Using verifyText command is OK, as it gets the
WRONG as it does not consider the location containing element locator and takes it into
of text. account.

8
Understand your tool, perform meaningful actions!

Actions and checks are NOT THE SAME!

Action Check
Enter the X value into the Y field. Check that the Y field contains the X value.
Click the A link on the B page. Check that the B page contains the A link.

Fill all fields of the form K on the page M with Check that the page M contains the form K.
valid values.
Click the N button. Check that the N button has the Z label.

9
Use proper data sources

A lot of data…
Random values
Values generation with some algorithm
Trusted external data sources (web-services, DB, etc)
Real existing user data
Manual generation

10
Tips on Test
Automation
Software Testing Introduction
The Record and Playback
Technology

Software Testing Introduction

The Record and Playback Technology

The automation tool …

Records all interactions with application under test

Presents the recording in some script language

Allows a tester to edit and improve the script

Executes the script and logs the results

2
The Record and Playback Technology: Pros and Cons

Pros Cons

Easy to learn No “ifs”, “cycles”, “subroutines”

Quick test skeleton creation EVERY action is recorded

Auto-gathering of technical details Hardcoding

Routine actions auto-recording Meaningless variable names

3
The Record and Playback Technology: Conclusion

So, the record and playback technology…

Is an easiest way to try automation for beginners

Helps even experienced testers to speed up automation

Still is not a universal solution, often fails

Still leaves much manual work

4
Selenium IDE: General Overview

Selenium IDE is an integrated development environment for

Selenium scripts. It is implemented as a Chrome and Firefox
extension, and allows you to record, edit, and debug tests.
Get the latest version here:
https://www.seleniumhq.org

Selenium IDE is one of the most

simple tools that uses the Record
and Playback technology
5
Selenium IDE: General Overview

Here goes live demo…

6
The Record and Playback
Technology

Software Testing Introduction

Selenium IDE
Overview
Software Testing Introduction
Selenium IDE: General Overview Read and remember!

Selenium IDE is an integrated development environment for

Selenium scripts. It is implemented as a Chrome and Firefox
extension, and allows you to record, edit, and debug tests.
Get the latest version here:
https://www.seleniumhq.org

2
Selenium Ecosystem Overview

Selenium WebDriver can drive a browser natively either

locally or on remote machines.

Selenium Grid takes Selenium Remote Control to another

level by running tests on many servers at the same time.

Selenium IDE is a Chrome and Firefox extension that makes

it easy to record and playback tests in the browser.

Selenium Remote Control is a client/server system that

allows you to control web browsers locally or remotely.
3
Selenium IDE: Quick Preview of the Main Idea

Imagine we want to automate the following test case:

1. Open https://www.seleniumhq.org
2. Click the “Projects” link
3. Click the “Selenium IDE” link
4. Check that the page contains the following text: “Selenium IDE has
plugin support”

4
Selenium IDE: Quick Preview of the Main Idea

Here is the result in Selenium IDE

5
Selenium IDE: Quick Preview of the Main Idea

Here goes live demo…

6
Selenium IDE: Quick Preview of the Main Idea

And one more – a little more complex:

1. Open https://www.seleniumhq.org
2. Enter “Legacy” search phrase into the search field
3. Click the “Go” button
4. Check that the page contains the link with the following text:
“Legacy-Selenium-IDE — Selenium Documentation”

7
Selenium IDE: Quick Preview of the Main Idea

Here is the result in Selenium IDE

8
Selenium IDE: Quick Preview of the Main Idea

Here goes live demo…

9
Selenium IDE
Overview
Software Testing Introduction
Selenium IDE
Interface
Software Testing Introduction
Selenium IDE Interface
Record, playback, save, etc. tools

The list of
currently
loaded test
cases
Comment/uncomment Current test
current command case

Current
command
details
(editable)
Test cases
execution
log

2
Selenium IDE Interface

The details for currently selected

command are loaded in the
corresponding fields

3
Selenium IDE Interface

The Description field is tricky: once

filled it displays its content instead of
the command details (hover mouse
pointer over the command in the
main list to see the details)

4
Selenium IDE Interface

The Reference tab shows short

documentation summary

5
 Selenium IDE Interface

Recording
Run all test cases on/off
Change test case Disable breakpoints
Run current execution speed (toggle a breakpoint Pause on
test case only Run current by clicking on exceptions
test case step command line
by step number)

6
Selenium IDE Interface

Find target on page (check if

Select target on page Target value is OK)

7
Selenium IDE Interface

Here goes live demo…

8
Selenium IDE
Interface
Software Testing Introduction
Selenium IDE
Commands
Software Testing Introduction
Main Selenium IDE Commands Read and remember!

Actions Manipulate the state of the application (like “click this link”
and “select that option”). If an Action fails, or has an error, the
execution of the current test is stopped.
Accessors Examine the state of the application and store the results in
variables, e.g. “storeTitle”. They are also used to automatically
generate Assertions.

Assertions … are like Accessors, but verify that the state of the
application conforms to what is expected (e.g., “make sure
the page title is X” and “verify that this checkbox is checked”).

Flow operators Allow to use such common programming operators as “if”,

“while”, etc.
2
Main Selenium IDE Commands: examples

Actions Manipulate the state of the application (like “click this link”
and “select that option”). If an Action fails, or has an error, the
execution of the current test is stopped.

3
Main Selenium IDE Commands: examples

Accessors Examine the state of the application and store the results in
variables, e.g. “storeTitle”. They are also used to automatically
generate Assertions.

4
Main Selenium IDE Commands: examples

Assertions Verify that the state of the application conforms to what is

expected (e.g., “make sure the page title is X” and “verify that
this checkbox is checked”).

5
Main Selenium IDE Commands

assert When an “assert” fails, the test is aborted. Useful

to check some major condition, failure of which
leaves other actions meaningless.

verify When a “verify” fails, the test will continue

execution, logging the failure. Useful to check
multiple independent conditions.

Assertions … are like Accessors, but verify that the state of the
application conforms to what is expected (e.g., “make sure
the page title is X” and “verify that this checkbox is checked”).

6
Main Selenium IDE Commands: examples

Flow operators Allow to use such common programming operators as “if”,

“while”, etc.

Warning! Flow control operators are in active development stage for now. Some
strange things may happen. Read the manual carefully!
7
What can we do with one or another HTML element?

Sorry, this course is about simple automation with Selenium IDE. If you are not
familiar with HTML, please begin here:
https://www.w3schools.com/html/

8
Selenium IDE
Commands
Software Testing Introduction
Selenium IDE
Locators
Software Testing Introduction
Selenium IDE Locators Overview

For many Selenium IDE commands, a target is required. This target identifies
an element in the content of the web application, and consists of the location
strategy followed by the location in the format locatorType=location.

2
Selenium IDE Locators Overview

Selenium IDE supports the following locator types…

Context-insensitive Context-sensitive

id css

name xpath

linkText

partialLinkText

3
Selenium IDE Locators Overview

Context-insensitive Context-sensitive

Based on elements attributes values Based on document structure

Document
<html>
<body>
Root Element
<form id="loginForm">
<input name="username" type="text" /> <html>
<input name="password" type="password" />
<input name="continue" type="submit" value="Login" /> Element Element
<input name="continue" type="button" value="Clear" /> <head> <body>
</form>
</body> Element Element
</html> <title> <a>

Text
"Test" Attribute
<href>

Text
"Trainings"

4
Selenium IDE Locators: id

With id locator the first element with the id attribute value matching the
location will be used.

<html>
<body>
<form id="loginForm">
<input name="username" type="text" />
<input name="password" type="password" />
<input name="continue" type="submit" value="Login" />
<input name="continue" type="button" value="Clear" />
</form>
</body>
</html>

id=loginForm

5
Selenium IDE Locators: name

The name locator type will locate the first element with a matching name
attribute. If multiple elements have the same value for a name attribute, then
you can use filters to further refine your location strategy.
<html>
<body>
<form id="loginForm">
<input name="username" type="text" />
<input name="password" type="password" />
<input name="continue" type="submit" value="Login" />
<input name="continue" type="button" value="Clear" />
</form>
</body>
</html>

name=username
name=continue type=submit
name=continue value=Clear

6
Selenium IDE Locators: linkText and partialLinkText

The linkText and partialLinkText locator type will locate the first element with a
matching text value of an <a> attribute (or matching part of this text value for
partialLinkText).
<html>
<body>
<div id="someLinks">
<a href="https://epam.com" />EPAM</a>
<a href="https://training.by" />EPAM Training Center</a>
<a href="https://learn.by" />EPAM Open Trainings</a>
</div>
</body>
</html>

linkText=EPAM
partialLinkText=Center
partialLinkText=Open Trainings

7
Selenium IDE Locators: css

The css locators use CSS selectors for binding style properties to elements in
the document.
<html>
<body>
<form id="loginForm">
<input class="required" name="username" type="text" />
<input class="required passfield" name="password" type="password" />
<input name="continue" type="submit" value="Login" />
<input name="continue" type="button" value="Clear" />
</form>
</body>
</html>

css=form#loginForm
css=input[name="username"]
css=input.required[type="text"]
css=input.passfield
css=#loginForm input[type="button"]
css=#loginForm input:nth-child(2)
8
Selenium IDE Locators: xpath

The xpath locators use xPath syntax to locate elements. (It works because once
processed by browser the HTML document looks similar to XML document.)
<html>
<body>
<form id="loginForm">
<input class="required" name="username" type="text" />
<input class="required passfield" name="password" type="password" />
<input name="continue" type="submit" value="Login" />
<input name="continue" type="button" value="Clear" />
</form>
</body>
</html>

xpath=/html/body/form[1]
xpath=//form[1]
xpath=//form[@id='loginForm']
xpath=//input[@name='username']
xpath=//form[@id='loginForm']/input[1]
xpath=//input[@name='continue'][@type='button']
9
Selenium IDE Locators: WARNING!

You should NEVER use “view source” to find any locator! NEVER!

Use “Inspect Element” (in Firefox) or “Inspect” (in Chrome) instead!

Browsers use correcting parsing algorithms, so the source page code and parsing result may differ!

10
Selenium IDE Locators: WARNING!

<!DOCTYPE html> You should NEVER use “view

<html>
source” to find any locator!
<body>
<table> NEVER!
<tr>
<td>Test</td>
</tr> Use “Inspect Element” (in Firefox)
</table> or “Inspect” (in Chrome) instead!
</body>
</html>

Browsers use correcting parsing algorithms, so the source page code and parsing result may differ!

11
Selenium IDE Locators: xpath and css

xpath locators css locators

More commonly used (more “classic”) Work faster
More easy to understand (for More short and clear
programmers)
Provide wide range of capabilities Provide specific unique capabilities (e.g.,
pseudo-classes like :visited)
Support functions Require good CSS knowledge
May work with DOM both “downwards”, May work with DOM “downwards” only
and “upwards”
Support text search capability Do not support text search capability

12
Selenium IDE Locators: what to choose?

Each Selenium IDE command works with ANY locator, but…

Context-insensitive Context-sensitive

id css

name xpath

linkText
While xpath locators are more
partialLinkText common, css locators are
usually more convenient.

13
Selenium IDE
Locators
Software Testing Introduction
Selenium IDE CSS
Locators
Software Testing Introduction
Selenium IDE CSS Locators Overview

The css locators use CSS selectors for binding style properties to elements in
the document.

In CSS, selectors are patterns used to select the element(s) you want to work
with (usually, to apply some style while composing a web-page).

Disclaimer: this is NOT a complete CSS guide for frontend developers, this is
just a short overview of basic principles!

You may also find this link useful:

https://www.w3schools.com/cssref/css_selectors.asp

2
Selenium IDE CSS Locators: universal selector (*)

The universal selector (*) selects all elements (or all elements inside another
element). (Rarely used on its own.)

HTML+CSS Example Selenium IDE Example

* {margin:0; padding:0;} css=ul * a

css=ul a
Looks for a elements inside any
elements inside ul elements.
<ul>
<li>Item <a href="site.com">1</a></li>
<li><p>Item 2</p></li>
</ul>

3
Selenium IDE CSS Locators: element

The element selector selects all elements with the specified element name.
(Rarely used on its own.)

HTML+CSS Example Selenium IDE Example

p {font-family: Arial, Calibri;} css=li p

Looks for p elements inside li

elements.

<ul>
<li>Item <a href="site.com">1</a></li>
<li><p>Item 2</p></li>
</ul>

4
Selenium IDE CSS Locators: class

The class selector selects elements with a specific class attribute. Write a
period (.) character, followed by the name of the class.

HTML+CSS Example Selenium IDE Example

.red {color:red;} css=.red
.green {color:green;}
Looks for any elements with .red
class.

css=p.red
<ul>
<li>Item <a class="red" href="site.com">1</a></li>
<li><p class="red">Item 2</p></li>
Looks for p elements with .red
<li><p class="green">Item 3</p></li>
</ul>
class.

5
Selenium IDE CSS Locators: id

The id selector selects elements with a specific id attribute. Write a number

sign (#) character, followed by the id value.

HTML+CSS Example Selenium IDE Example

#marked {background-color:yellow} css=#marked
Looks for any element with
#marked id.

css=p#marked
<ul>
<li>Item <a class="red" href="site.com">1</a></li>
<li><p id="marked" class="red">Item 2</p></li>
Looks for p element with #marked
<li><p class="green">Item 3</p></li>
</ul>
id.

6
Selenium IDE CSS Locators: attribute

The attribute selector selects elements with the specified attribute,

attribute=value selects elements with the specified attribute value.

HTML+CSS Example Selenium IDE Example

a[target]{color:red} css=*[target]
a[href="_self"]{color:green}
a[href="http://site3.com"]{color:magenta} Looks for any element with target
attribute.

css=a[target=_blank]
<a target="_self" href="http://site0.com">Site 0</a><br>
<a
<a
target="_blank" href="http://site1.com">Site 1</a><br>
href="http://site2.com">Site 2</a><br>
Looks for a element with target
<a href="http://site3.com">Site 3</a><br>
attribute that has _blank value.

7
Selenium IDE CSS Locators: descendant

The descendant selector (AKA element element) selects elements within other
elements (it works for any hierarchy depth).

HTML+CSS Example Selenium IDE Example

ol li {color:green} css=ol li
span span span {color:magenta}
Looks for li elements inside ol
elements.

css=span span span

<ol><span><li>Green text.</li></span></ol>
<span>Text iside span-1, Looks for span elements inside span
<span>inside span-2
<span>inside span-3</span></span></span>. elements inside span elements.

8
Selenium IDE CSS Locators: child

The child selector (AKA element>element) selects elements with a specific

direct parent (one hierarchy level ONLY).

HTML+CSS Example Selenium IDE Example

p > i {color:red;} css=p > i
Looks for i element that is DIRECT
child of p element.

This i element does NOT match

<p><i>Italic</i> in paragraph.</p>
<p><span><i>Italic</i> in span in paragraph. this criteria.
</span></p>

9
Selenium IDE CSS Locators: immediately after

The immediately after selector (AKA element1+element2) selects elements2

that are placed immediately after elements1.

HTML+CSS Example Selenium IDE Example

p + i {color:red;} css=p + i
Looks for i elements immediately
after p elements.

<p>Paragraph.</p> This i element does NOT match

<i>Italic after paragraph.</i><br>
<i>Italic after italic after paragraph.</i> this criteria.

10
Selenium IDE CSS Locators: preceded by

The preceded by selector (AKA element1~element2) selects elements2 that

are placed after elements1 (within one hierarchy level).

HTML+CSS Example Selenium IDE Example

p ~ i {color:green;} css=p ~ i
Looks for i elements preceded by
p element.

<i>Italic before paragraph.</i><br>

<p>Paragraph.</p>
<i>Italic after paragraph.</i><br>
<i>Italic after italic after paragraph.</i>

11
Selenium IDE CSS Locators: pseudo-class

The pseudo-class selector selects elements with a specific state (defined by the
variety of CSS pseudo-classes).

HTML+CSS Example Selenium IDE Example

input:focus {border: 1px solid green;} css=input:focus
Looks for a focused element with
the input type. In this example any
of two inputs may have (or do not
have) the focus.
<form>
<input type="text"><br />
<input type="text"><br />
</form>

12
Selenium IDE CSS Locators: pseudo-element

The pseudo-element selector selects elements with a specific property

(defined by the variety of CSS pseudo-elements).

HTML+CSS Example Selenium IDE Example

p:first-letter {color:red} Not supported. May have some
p:first-line {color:green} tricky JS-based workaround.

<p>Just a <br> paragraph.</p>

13
Selenium IDE CSS Locators: some real-life examples

Take a pause and name elements identified by the following locators

h1
div.tagline
li#intro a
div.result > a
li ~ ul
a[href$=".pdf"]
span.alpha div#id > a:visited

14
Selenium IDE CSS
Locators
Software Testing Introduction
Selenium IDE xPath
Locators
Software Testing Introduction
Selenium IDE xPath Locators Overview

xPath is a syntax for defining parts of an XML document. xPath uses path
expressions to navigate in XML documents. This approach works for locating
HTML elements because DOM-tree meets all necessary XML document criteria.

Let’s take a closer look…

2
Selenium IDE xPath and HTML

<html> Document
<head>
<title>Test</title>
</head> Root Element
<body> <html>
<a href="training.by">Trainings</a>
</body>
</html> Element Element
<head> <body>

Element Element
<title> <a>

Text
"Test" Attribute
<href>

Text
"Trainings"

3
Selenium IDE xPath Locators: how it works

Imagine HTML
document like a
set of nested
rectangles. So,
xPath is a path
through that
rectangles.

4
Selenium IDE xPath Locators: how it works

//html/body/table[1]

5
Selenium IDE xPath Locators: how it works

//html/body/table/tbody/tr[1]
Wait, wait! Where
is tbody?! It’s
there, trust me .

6
Selenium IDE Locators: WARNING!

You should NEVER use “view source” to find any locator! NEVER!

Use “Inspect Element” (in Firefox) or “Inspect” (in Chrome) instead!

Browsers use correcting parsing algorithms, so the source page code and parsing result may differ!

7
Selenium IDE Locators: WARNING!

<!DOCTYPE html> You should NEVER use “view

<html>
source” to find any locator!
<body>
<table> NEVER!
<tr>
<td>Test</td>
</tr> Use “Inspect Element” (in Firefox)
</table> or “Inspect” (in Chrome) instead!
</body>
</html>

Browsers use correcting parsing algorithms, so the source page code and parsing result may differ!

8
Selenium IDE xPath Locators: how it works

//html/body/table/tbody/tr[2]/td[1]

9
Selenium IDE xPath Locators: how it works

//html/body/table/tbody/tr[2]/td/form/div/input

10
Selenium IDE xPath Locators: how it works

//div[2]

11
Selenium IDE xPath Locators: good approach

<tr>
<td>lalala</td>
<td> //html/body/table/tbody/tr[2]/td[2]/div/ul/li[2]
<div style="border: 1px solid blue">
<ul id="ono">
<li>One</li>
<li>Two</li>
<li>Three</li>
//li[2]
</ul>
</div>
</td>
<td>tratata</td> //ul[@id='ono']/li[2]
</tr>

Find your element and move upwards by DOM-tree

until you find some element with id or unique class or
any other unique set of properties.
12
Selenium IDE xPath Locators: some real-life examples

Take a pause and name elements identified by the following locators

//button[@id='submit']
//div[not(@style='display:none')]/a
//*[text()='the visible text' and @class = 'class1']
//li[contains(@value, 'choice1')]
//span[count(div)>=3]
//*[@class='search-result']/descendant-or-self::*[@id='training']
//a[ends-with(@href, '.pdf')]

13
Selenium IDE xPath
Locators
Software Testing Introduction
Selenium IDE
Usage Sample
Software Testing Introduction
General Idea

Imagine we have a web-application we

want to test with Selenium IDE.
Let it be a simple calculator (I know that
everybody hate testing calculators, but
still)…

Find the application here:

http://svyatoslav.biz/testlab/calc/

2
Check list to transform in Selenium IDE test cases

1. All four operations with correct numbers.

2. Division by zero (exp: “Division by zero” message).
3. Fraction digits (and rounding correctness).
4. “Save / clear / surprise me” behavior.
4a. “Save”: all fields save their values.
4b. “Clear”: all fields return to the default state.
4c. “Surprise me”: all fields (except for “Next”) get random values.
5. Too large / small numbers handling (exp: “+Infinity” or “-Infinity”, “Machine
zero”).

Take a pause and re-write this check-list (improve it)

3
Test 1: all four operations with correct numbers

echo Prepare variables

For maximum store 1 min

effectiveness we store
executeScript
1000
return Math.floor(Math.random() * (${max} - ${min} + 1)) + ${min}
max
a
generate random echo A = ${a}
executeScript return Math.floor(Math.random() * (${max} - ${min} + 1)) + ${min} b
numbers here. echo B = ${b}
executeScript return +${a} + +${b} c
echo Expected = ${c}
echo Main logic here
Following this open /calc.php
type name=a ${a}
sample, create type name=b ${b}

tests for type

click
name=f
xpath=(//input[@name='op'][@value='+'])
10

subtraction, select name=next label=Save

click css=input[type="submit"]
multiplication, verifyElementPresent xpath=//*[contains(text(), '${a} + ${b} = ${c}')]
and division.

4
Test 2: division by zero (exp: “Division by zero” message)

echo Prepare variables

Here we may omit store 1 min
random number store 1000 max

generation as we need executeScript

return Math.floor(Math.random() * (${max} - ${min} + 1)) +
${min}
a

the divider to be zero echo A = ${a}

and we don’t care about echo

open
Main logic here
/calc.php
anything else. But let’s type name=a ${a}

for now leave this script type

type
name=b
name=f
0
10
as is. click xpath=(//input[@name='op'][@value='/'])
select name=next label=Save
Re-write the script click css=input[type="submit"]
verifyElementPresent xpath=//*[contains(text(), 'Division by zero')]
eliminating random
number generation.

5
Test 3: fraction digits (and rounding correctness)

echo Prepare variables

Unlike with previous test store 10 a
this one really needs store 57 b
store 0.175439 c
more complex algorithm store 6 f

to cover more typical echo Expected = ${c}

echo Main logic here
usage situation. For now open /calc.php

it works, but it may be type

type
name=a
name=b
${a}
${b}
improved. type name=f ${f}
click xpath=(//input[@name='op'][@value='/'])

Re-write the script with select name=next label=Save

click css=input[type="submit"]
more complex approach verifyElementPresent xpath=//*[contains(text(), '${a} / ${b} = ${c}')]
to cover more verifyElementPresent xpath=//*[contains(text(), 'with ${f} fraction digits')]

situations.

6
Test 4c: “Surprise me”, all fields (except for “Next”) get random values

echo Main logic here

The idea behind this open

type
/calc.php
name=a 1

script is to refresh the type

type
name=b
name=f
2
1
click xpath=(//input[@name='op'][@value='+'])

page several times select

click
name=next
css=input[type="submit"]
label=Surprise me!

waiting for all necessary store

store
10
false
counter
success
while (${counter} > 0) && (${success} == 'false')
fields to change their echo
storeValue
Attempts left = ${counter}
name=a old_a

values. storeValue
storeValue
name=b
name=f
old_b
old_f
operations = document.getElementsByName("op"); old_op = '?'; if (operations[0].checked) {old_op = '+';} if
executeScript (operations[1].checked) {old_op = '-';} if (operations[2].checked) {old_op = '*';} if (operations[3].checked) old_op
{old_op = '/';} return old_op;
click css=input[type="submit"]
storeValue name=a new_a
Cases ‘a’ (“Save”) and ‘b’ storeValue
storeValue
name=b
name=f
new_b
new_f

(“Clear”) intentionally executeScript

operations = document.getElementsByName("op"); new_op = '?'; if (operations[0].checked) {new_op = '+';} if
(operations[1].checked) {new_op = '-';} if (operations[2].checked) {new_op = '*';} if (operations[3].checked) new_op
{new_op = '/';} return new_op;
left for your own self- executeScript
if ((${old_a} != ${new_a}) && (${old_b} != ${new_b}) && (${old_f} != ${new_f}) && (${old_op} != ${new_op}))
success
{ return 'true'; } else { return 'false'; }
study. executeScript
end
return +${counter} - 1 counter

assert success true

7
Test 5: too large / small numbers handling

This test is (probably) open /calc.php

the easiest one: we take type name=a 10e500
two huge numbers and type name=b 10e500
multiply them to get an type name=f 10
enormous number.
xpath=(//input[@name=
click
'op'][@value='*'])
select name=next label=Save
Two other cases (“- click
css=input[type="submit"
Infinity”, “Machine ]
zero”) intentionally left xpath=//*[contains(text(
verifyElementPresent
for your own self-study. ), '+Infinity')]

8
Selenium IDE
Usage Sample
Software Testing Introduction