Software Testing and

Quality Assurance

Lecture 2:
Foundations of
Software Testing
SFE 4030 1
Terminology:
Failure

“Deviation of the component or system from its expected delivery,

service or result”

“Manifested inability of a system

to perform required function”

2
3
Windows failure leads to “blue screen of death”.
4
Terminology: Defect /
Fault
“Flaw in component or system
that can cause the component or system
to fail to perform its required function”

“A defect, if encountered during execution,

may cause a failure of the component or system”

Synonym: Fault

5
6
Fault in Apple’s Secure Socket Layer code
7

https://avandeursen.com/2014/02/22/gotofail-
security/
Terminology:
Error

“A human action
that produces an incorrect result”

Synonym: Mistake

8
Faults, Failures, and
Bugs
• Failure:
• Manifested inability of a system
to perform required function.
• Defect (fault):
• missing / incorrect code
“bug”
• Error (mistake)
• human action producing fault
• And thus:
• Testing: Attempt to trigger
failures
• Debugging: Attempt to find faults given a failure
9
Principles of Testing
#1
• Testing shows the presence of defects

• Testing does not show the absence of defects!

• “no test team can achieve 100% defect detection

effectiveness” (Black et al)

10
 Principles of Testing #2:
Exhaustive Testing is ….....
……….

11
Principle #2: Exhaustive Testing is
Impossible • A simple program: 3 inputs, 1 output
• a,b,c: 32 bit integers
• trillion test cases / s.
All oceans dry
All plants dead Tests done (2.5 bill. y)

26
Principles of Testing #3: Test
Early
• Start testing as early as possible
• To let tests guide design
• To get feedback as early as possible
• To find bugs when they are cheapest to fix
• To find bugs when have caused least damage

13
Faults can be introduced
at any moment in the
software development
process

Finding faults in different

development phases may
require different types of
testing

14
Cost to Repair:
Early Discovery Pays
Off 16

4
2
1

Requirement Design Unit test Acceptance Post release

15
test
 Principles of Testing #4: If you find a bug, keep

Defects are likely to be on searching in its

‘neighborhood’

Clustered
• “Hot” components requiring frequent
change, bad habits, poor developers,
tricky logic, business uncertainty,
innovative, size, ….

• Pareto Principle / Law of vital few: “Pareto Diagram”

• 80% of effects come from 20% of causes for an IBM system:
70% of defects caused by
2 components.
• Use to focus test effort

30
Principles of Testing #5:
Is there one best test method for my project?

The pesticide paradox:

Every method you use to prevent or find

bugs leaves a residue of subtler bugs
against which those methods are ineffectual.

• Re-running the same test suite again and again on

a changing program gives a false sense of security

• We need variation in testing

17
Principles of Testing #6:
Is there a single test method for any project?

• Testing is context-dependent

18
Principles of Testing #7
Absence-of-errors
Fallacy
There is more to success than absence of errors

Thorough understanding of business value is necessary

“Building the software right versus building the right software.”

“Finding and removing defects is not a way to improve the overall

quality or performance of a system” – Russ Ackoff

https://embeddedartistry.com/blog/2019/2/5/beyond-continual-improvement 33
Revisiting the
Principles?
1. Testing cannot show absence of bugs
2. Exhaustive testing is impossible
3. Testing needs to start early
4. Defects tend to be clustered
5. Pesticide paradox yields test methods ineffective
6. Testing is context-dependent
7. There is more to quality than absence of defects

20
Psychology of
Testing
• We are all biased

• Independence of testing required:

• From self-testing to external approval
• Testers with different backgrounds

• Good knowledge of rigorous techniques and procedures required

21
Cognitive
Bias
• “System 1”: Fast, instinctive,
emotional.
• "System 2”: slower, more
deliberative, and more logical.

• System 2 requires effort and is

happy to let System 1 do the
work

22
Example of Cognitive
Bias
• “What You See is All There Is” (WYSIATI)
• Being satisfied with the evidence you see.

• Key problem in software engineering:

• “It works on my machine”
• “I’ve tried it, and it works”
• “All 100 tests pass, we can ship”

23
ACM Software EngineeringCode of
Ethics
1. PUBLIC - Software engineers shall act consistently with the public interest.
2. CLIENT AND EMPLOYER - Software engineers shall act in a manner that is in the best
interests of their client and employer consistent with the public interest.
3. PRODUCT - Software engineers shall ensure that their products and related
modifications meet the highest professional standards possible.
4. JUDGMENT - Software engineers shall maintain integrity and independence
in their professional judgment.
5. MANAGEMENT - Software engineering managers and leaders shall subscribe to and
promote an ethical approach to the management of software development and
maintenance.
6. PROFESSION - Software engineers shall advance the integrity and reputation of the
profession consistent with the public interest.
7. COLLEAGUES - Software engineers shall be fair to and supportive of their colleagues.
8. SELF - Software engineers shall participate in lifelong learning regarding the practice
of their profession and shall promote an ethical approach to the practice of the 24
profession.
 25
https://www.acm.org/code-of-ethics
 26
https://www.acm.org/code-of-ethics
Question!

Which test level corresponds to testing a system to check if individual

component are fulfilling functionalities?

A. Acceptance Testing
B. Integration Testing
C. System Testing
D. Unit Testing

27
Question!

Which test level corresponds to testing a system to check if individual

component are fulfilling functionalities?

A. Acceptance Testing
B. Integration Testing
C. System Testing
D. Unit Testing

28
Software Life Cycle
• Period of time that
• begins when a software system is conceived
• ends when the system is no longer available for use.

• Phases: concept development, requirements, design,

implementation, test, installation, retirement

• Phases may overlap and be performed iteratively

29
30
Verification vs validation

Verification:
"Are we building the product right”.
The software should conform to its specification.

Validation:
"Are we building the right product”.
The software should do what the user really requires.

Chapter 8 Software testing 32

 V & V confidence
Aim of V & V is to establish confidence that the system is ‘fit for purpose’.

Depends on system’s purpose, user expectations and marketing

environment

33
 V & V confidence

• Software purpose
• The level of confidence depends on how critical the software is to an
organisation.

• User expectations
• Users may have low expectations of certain kinds of software.

• Marketing environment
• Getting a product to market early may be more important than finding
defects in the program.
 Validation and defect testing

The first goal leads to validation testing

You expect the system to perform correctly using a given
set of test cases that reflect the system’s expected use.

The second goal leads to defect testing

The test cases are designed to expose defects. The test
cases in defect testing can be deliberately obscure and need
not reflect how the system is normally used.

Chapter 8 Software testing/Sommerville 35

 Testing Process Goals
Validation testing
To demonstrate to the developer and the system customer that
the software meets its requirements
A successful test shows that the system operates as intended.

Defect testing
To discover faults or defects in the software where its behavior is
incorrect or not in conformance with its specification
A successful test is a test that makes the system perform
incorrectly and so exposes a defect in the system.

Chapter 8 Software testing 36

 Inspections and testing

Software inspections Concerned with analysis of

the static system representation to discover problems (static verification)
May be supplement by tool-based document and code analysis.

Software testing Concerned with exercising and

observing product behaviour (dynamic verification)
The system is executed with test data and its operational behaviour is
observed.

Chapter 8 Software testing 37

Inspections and testing

Chapter 8 Software testing 38

 Software inspections
These involve people examining the source representation with the aim of discovering
anomalies and defects.

Inspections not require execution of a system so may be used before implementation.

They may be applied to any representation of the system (requirements, design,
configuration data, test data, etc.).

They have been shown to be an effective technique for discovering program errors.

Chapter 8 Software testing 39

 Advantages of Inspections

During testing, errors can mask (hide) other errors. Because inspection is a
static process, you don’t have to be concerned with interactions between
errors.

Incomplete versions of a system can be inspected without additional costs. If

a program is incomplete, then you need to develop specialized test harnesses
to test the parts that are available.

As well as searching for program defects, an inspection can also consider

broader quality attributes of a program, such as compliance with standards,
portability and maintainability.

Chapter 8 Software testing 40

 Inspections and testing
 Inspections and testing are complementary and not opposing
verification techniques.

 Both should be used during the V & V process.

 Inspections can check conformance with a specification but not

conformance with the customer’s real requirements.

 Inspections cannot check non-functional characteristics such as

performance, usability, etc.

Chapter 8 Software testing 41

A model of the software
testing process

Chapter 8 Software testing 42

 Stages of testing
Development testing, where the system is tested during
development to discover bugs and defects.

Release testing, where a separate testing team test a complete

version of the system before it is released to users.

User testing, where users or potential users of a system test the

system in their own environment.

Chapter 8 Software testing 43

Test
Levels
• Component (unit) testing.
Units in isolation
• Integration testing
Interaction
between units
• System testing
System-level
properties
Test levels in right leg of V model
• Acceptance
testing
Focus on user 44
Graham, D., Black, R., & Van Veenendaal, E. (2019). Foundations of
needs software testing. Cengage.
Systems Thinking / Russ
Ackoff
• The defining properties of a system are properties of the whole which
none of its parts have

• A system is not the sum of the behavior of its parts, it is a product of

their interactions

• The performance of a system depends on how the parts fit, not how
they act taken separately

https://embeddedartistry.com/blog/2019/2/5/beyond-continual-improvement
45
Test
Types

Group of test activities

Aimed at testing a component or system
Focused on a specific test objective

46
Test
Types
• Testing of Function
• Functional testing, black box testing

• Testing of software product characteristics

• Non-functional testing

• Testing of software structure / architecture

• Structural testing, white box testing

• Testing related to changes

• Confirmation vs Regression testing 59
Regression
Testing
• Testing of a previously tested program
• Following modification
• To ensure that defects have not been introduced
• In unchanged areas of the software
• As a result of the changes made.

• Performed when the software or its environment is changed.

• Continuous delivery? Automate regression testing

60
Testing As Software
Evolves
• The norm in software development!
• Many updates per day in modern web apps
• Current cars (wireless updates)
• Formula 1 (every two weeks)
• Aircraft (737 max compatibility)

• Dependency and compatibility management are key

• Direct test efforts to changed software

• Confirm change works as expected, without regressions.
• Automated regression testing where possible
61
Maintenance Testing
• Testing after a system is stable and deployed
• Test changes to an operational system
• Test impact of changed environment to an operational system
(e.g. security updates of libraries used)

• Impact analysis:
• Determine which parts are affected by change
• Conduct regression testing for those.

62
Key Points
• Software development follows life cycle activities
• The V-Model helps reason about verification vs validation,
decomposition vs composition, and construction vs testing.
• We test at different levels
• Different test types target specific test objectives
• Iterative projects do continuous regression testing and thus maximize
test automation

64