Basics of Softw are Testing

(Unit-I) Ms.Mamatha.H.R Assistant Professor Dept. of Information ScienceE&ngg. PES Institute of Technology Ref: Adithya P Mathur

Introduction

The purpose of this introductory chapter is to get familiarize with basic concepts related to software testing

Mamatha.H.R

Errors
Errors are a part of our daily life. Humans make errors in their thoughts, actions, and in the products that might result from their actions. Errors occur wherever humans are involved in taking actions and making decisions.
These fundamental facts of human existence make testing an essential activity.
Mamatha.H.R 3

Errors: Examples

Mamatha.H.R

Testing

Testing is a process used to determine whether there are any errors in our thoughts, actions and the products generated The primary goal of testing is to determine if the thoughts, actions and products are as desired, that is they conform to the requirements.

Mamatha.H.R 5

Testing of thoughts is usually designed to determine if a concept or method has been understood satisfactorily. Ex: An instructor administers a test to determine how well the students have understood what the instructor wanted to convey. Here there is an attempt by the tester to determine if the human thoughts behave as desired.

Mamatha.H.R 6

Testing of actions is designed to check if a skill that results in the actions has been acquired satisfactorily. Ex: A tennis coach administers a test to determine how well the understudy makes a serve. Here there is an attempt by the tester to determine if the actions behave as desired.

Mamatha.H.R 7

Testing of a product is designed to check if the product behaves as desired. Ex: A software developer tests the program developed to determine if it behaves as desired. Here there is an attempt by the tester to determine if the products behave as desired.

Mamatha.H.R 8

Most modern compilers are able to detect syntactic errors. Testing focuses on semantic errors, also known as faults. Ex Sorting There may be a situation where the tester made a mistake (an error) that led to his incorrect interpretation (perception) of the behavior of the program (the product)

Mamatha.H.R 9

Err or, faults, failures

Mamatha.H.R

10

An Error occurs in the process of writing a program. A Fault is the manifestation of one or more errors. A Failure occurs when a faulty piece of code is executed leading to an incorrect state that propagates to the programs output.

Mamatha.H.R 11

The programmer might misinterpret the requirements and consequently write incorrect code. Upon execution, the program might display behavior that does not match with the expected behavior, implying thereby that a failure has occurred.

Mamatha.H.R 12

A fault in the program is also commonly referred to as baug or a defect. The terms error and bug are by far the most common ways of referring to something wrong in the program text that might lead to a failure.
Mamatha.H.R 13

A N ecessary Evil

All project managers know that they must do some testing The basic questions are;
How much? What sort? By whom? When and by when? All difficult questions.

Why do we test?

Provide confidence in the system Identify areas of weakness Establish the degree of quality Establish the extent that the requirements have been met, i.e. what the users asked for is what they got not what someone else though they wanted To provide an understanding of the overall system To prove it is both usable and operable To provide sufficient information to allow an objective decision on applicability to deploy
Mamatha.H.R 15

First up How much?

How good does the product need to be?

How do we measure it? How do we make a decision? What industry are you in? Time and resource [human and machine]

Are there any legal or compliance issues?

What will it cost?

Are we capable? We know what we need to do but can we do it?

Mamatha.H.R 16

How do we decide?

What matters to our users?

What does the system do? Risk Requirements Money

Test Techniques- Coverage

Static Dynamic

Mamatha.H.R

17

Secondly What sort?

Unit Interface Functionality Performance Usability Security Availability Operability

Mamatha.H.R

18

Do w e need tools to help?

Automated test tools are often seen as a silver bullet Some tests cannot be successfully executed or analysed without them 80% of test tools end up as shelfware through:

Abandonment Neglect Banishment

Mamatha.H.R 19

Thirdly By whom?

Users: those who will ultimately use the system Customers: those who define the syste m are they the same as the users? Testers Analysts Developers

Mamatha.H.R

20

Are w e capable of testing?

Is an experienced tester better than an experienced user at finding faults? How can testers help themselves and users?

Working with the users to understand their systems Providing testing skills transfer Attending testing industry conferences Attaining industry recognised software testing qualifications
Mamatha.H.R 21

Lastly When?
Once the code is complete? As soon as the architecture is defined? Once the system delivery is complete? During development? Once the business requirements have been defined? As soon as the project is given the- go ahead?

Mamatha.H.R 22

By when?

Project deadline driven by:

Legislation Arbitrary date Financial year Calendar date Seasonal Marketing window Safety Criticality Money

Mamatha.H.R

23

Soft ware testing

Software testingis anempirical investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software Testing also provides an objective, independent view of the software to allow the business to appreciate and understand the risks at implementation of the software. The process of executing a program or application with the intent of findin sogftware bugs .

Mamatha.H.R 24

Software Testing can also be stated as the process of validating and verifying that a software program/application/product: meets the business and technical requirements that guided its design and development; works as expected; and can be implemented with the same characteristics.

Software Testing, depending on the testing method employed, can be implemented at any time in the development process. However, most of the test effort occurs after the requirements have been defined and the coding process has been completed.

Every software product has a target audience. For example, the audience for video game software is completely different from banking software. Therefore, when an organization develops or otherwise invests in a software product, it can assess whether the software product will be acceptable to its end users, its target audience, its purchasers, and other stakeholders. Software testingis the process of attempting to make this assessment.

A study conducted bNy IST ,The National Institute of Standards and Technology in 2002 reports that software bugs cost the U.S. economy $59.5 billion annually. More than a third of this cost could be avoided if better software testing was performed.

History

The separation of debugging from testing was initially introduced byGlenford J. Myers in 1979.Although his attention was on breakage testing ("a successful test is one that finds a bug") it illustrated the desire of the software engineering community to separate fundamental development activities, such as debugging, from that of verification. Dave Gelperin and William C. Hetzel classified in 1988 the phases and goals in software testing in the following stages: Until 1956- Debugging oriented 19571978 - Demonstration oriented 19791982 - Destruction oriented 19831987 - Evaluation oriented 19882000 - Prevention oriented

1. 2. 3. 4. 5.

Scope

A primary purpose for testing is to detect software failurtehsatso defects may be uncovered and corrected. This is a-trnivoinal pursuit. Testing cannot establish that a product functions properly unaldler conditions but can only establish that it does not functionerplyrop under specific conditions. The scope of software testing often includes examination ofe caosd well as execution of that code in various environments and conditions as well as examining the aspects of code: doeswithadto it is supposed to do and do what it needs to do. In the current culture of software development, a testing organization may be separate from the development team. Theere ar various roles for testing team members. Information derived from software testing may be used to correct the process by which software is developed.

certi fications

Testing certifications Certified Associate in Software Testing (CAST) offered by tQheAI)( CATe offered by theInternational Institute for Software Testing Certified Manager in Software Testing (CMST) offered by the I)(QA Certified Software Tester (CSTE) offered by the (QAI) Certified Software Test Professional (CSTP) offered by Itnhternational Institute for Software Testing (Australian Version) offered byK. J. Ross & Associates ISEB offered by thIenformation Systems Examinations Board ISTQB Certified Tester, Foundation Level (CTFL) offered byInthteernational Software Testing Qualification Board ISTQB Certified Tester, Advanced Level (CTAL) offered byInthteernational Software Testing Qualification Board TMPF [dubious discuss ] Next Foundation offered by tEhexamination Institute for Information Science Quality assurance certifications CMSQ offered by thQe uality Assurance Institute(QAI) CSQA offered by thQe uality Assurance Institute(QAI) CSQE offered by thAe merican Society for Quality(ASQ)

The importance of software testing increases as software pervades more and more into our daily lives

1. 2. 3.

Software testing consists of a large number of related and intertwined activities. Some of these are administrative technical routine

Technical activities include test case and oracle design at the unit, subsystem, integration, system and regression levels. Administrative activities include manpower planning, budgeting and reporting

Planning activities include test planning, quality assessment and control and manpower allocation.

Several test-related activities are product specific. Example Testing of a device driver often includes tasks such as writing a device simulator.

Simulators include heart simulator in testing cardiac pacemakers, an USB port simulator useful in testing I/O drivers, and an airborne drone simulator used in testing control software for airborne drones.

All these simulator activities are extremely important for effective testing and test automation But they often require a significant development effort.

Writing a device simulator and testing is both a development and a testing activity.

Test Automation

There is a tremendous need for automating testing tasks. Because execution of many tests can be tiring as well as err-oprrone.

Mamatha.H.R

39

Many programming groups are relying more and more on automated testing , especially groups that use Test -driven development . There are many frameworks to write tests in, and Continuous Integrationsoftware will run tests automatically every time code is checked into a version control system.
Mamatha.H.R 40

The process of test automation cannot be generalized . Example: Automating regression tests for an embedded device such as a pacemaker is quite different from that for an I/O device driver that connects to the USB port of a PC.

Mamatha.H.R 41

Testing Tools
Program testing and fault detection can be aided significanytlytesbting tools and debuggers . Testing/debug tools include features such as: Program monitors, permitting full or partial monitoring of parmogrcode including:
Instruction Set Simulato ,r permitting complete instruction level monitoring and trace facilities Program animation , permitting step-by-step execution and conditiona bl reakpoint at source level or minachine code Code coveragereports

Formatted dump or Symbolic debugging , tools allowing inspection of program variables on error or at chosen points Automated functional GUI testing tools are used to repeat sy-sleteveml tests through the GUI Benchmarks , allowing run-time performance comparisons to be made Performance analysis(or profiling tools) that can help to highlhigohtt spots and resource usage Some of these features may be incorporated into Inatengrated

Mamatha.H.R 42

Development Environment(IDE).

Mamatha.H.R

43

General-purpose tools

GUI testing Eggplant, Marathon andPounder Performance or load testing eLoadExpert,DBMonster,JMeter,Dieseltest, WAPT,LoadRunner and Grinder

Regression testing Echelon, TestTube, WinRunner and XTest

Developer and Tester as two roles

A developer is one who writes code A tester is one who tests code.

Software quality

Static quality attributes: refetro the actual code and related documentation. Dynamic quality attributes: relateto the behavior of the application in use.

Soft ware quality

Static quality attribute:s structured, maintainable, testable code as well as the availability of correct and complete documentation. Dynamic quality attribute:s software reliability, correctness, completeness, consistency, usability, and performance

Mamatha.H.R

47

Reliability refers to the probability of failufrreee operation. Correctness refers to the correct operation of an application and is always with reference to some artifact.

For a tester, correctness is with respect to the requirements For a user, it is often with respect to a user manual.

Software quality (contd.)

Completeness refers to the availability of all features listed in the requirements, or in the user manual. An incomplete software is one that does not fully implement all features required.

Software quality (contd.)

Consistency refers to adherence to a common set of conventions and assumptions. For example, all buttons in the user interface might follow a common color coding convention. An example of inconsistency would be when a database application displays the date of birth of a person in the database.
Mamatha.H.R 50

Software quality (contd.)

Usability refers to the ease with which an application can be used. This is an area in itself and there exist techniques for usability testing. Psychology plays an important role in the design of techniques for usability testing.
Mamatha.H.R 51

Software quality (contd.)

Performance refers to the time the application takes to perform a requested task. It is considered as a non-functional requirement. It is specified in terms such as ``This task must be performed at the rate of X units of activity in one second on a machine running at speed Y, having Z gigabytes of memory."
Mamatha.H.R 52

Requ irements, behavior, correctness

Requirements leading to two different programs:

Requirement 1:It is required to write a program that inputs two integers and outputs the maximum of these. Requirement 2: It is required to write a program that inputs a sequence of integers and outputs the sorted version of this sequence.

Mamatha.H.R

53

Requ irements: Incompleteness

Suppose that programmax is developed to satisfy Requirement 1. The expected output ax when the input integers are 13 and 19 omf can be easily determined to be 19.

Suppose now that the tester wants to know if the two inteegetros ar be input to the program on one line followed by a roertu carriagne, r on two separate lines with a carriage return typed in after each

number. The requirement as stated above fails to provide anearnsw to this question.

Requ irements: Ambiguity

Requirement 2 is ambiguous. It is not clear whether the input sequence is to sorted in ascending or in descending order. The behavior ofsort program, written to satisfy this requirement, will depend on the decision taken by the programmer while writing sort.

Input domain (Input space)

The set of all possible inputs to a progPramis known as the input domain or input space, Po.f Using Requirement 1 above we find the input domaimn aoxf to be the set of all pairs of integers where each elementpianir the integers is in the rang-e32,768 till 32,767. Using Requirement 2 it is not possible to find the input domain for the sort program.

Input domain (Continued)

Modified Requirement 2: It is required to write a program that inputs a sequence of integers and outputs the integers in this sequence sorted in either ascending or descending order. The order of the output sequence is determined by an input request character which should be ``A'' when an ascending sequence is desired, and ``D'' otherwise. While providing input to the program, the request character is input first followed by the sequence of integers to be sorted; the sequence is terminated with a period.
Mamatha.H.R 57

Input domain (Continued)

Based on the above modified requirement, the input domain for sort is a set of pairs. The first element of the pair is a character. The second element of the pair is a sequence of zero or more integers ending with a period.

Mamatha.H.R

58

Valid/ Invalid Inputs

The modified requirement for sort mentions that the request characters can be ``A'' and ``D'', but fails to answer the question ``What if the user types a different character ? When usingsort it is certainly possible for the user to type a character other than ``A'' and ``D''. Any character other than ``A' and ``D'' is considered as invalid inputsotrot. The requirement forsort does not specify what action it should take when an invalid input is encountered.
Mamatha.H.R 59

Corre ctness vs. Reliability

Though correctness of a program is desirable, it is almost never the objective of testing. To establish correctness via testing would imply testing a program on all elements in the input domain. In most cases that are encountered in practice, this is impossible to accomplish. Thus correctness is established via mathematical proofs of programs
Mamatha.H.R 60

Corre ctness and Testing

While correctness attempts to establish that the program is error free, testing attempts to find if there are any errors in it. Thus completeness of testing does not necessarily demonstrate that a program is error free.

Testing, debugging, and the error removal processes together increase our confidence in the correct functioning of the program under test.

Software reliability: two definitions Software reliability A[ NSI/IEEE Std 729-1983]: is the probability of failure free operation of software over a given time interval and under given conditions. Software reliabilityis the probability of failure free operation of software in its intended environment.

Operational profile An operational profileis a numerical description of how a program is used. Consider a sort program which, on any given execution, allows any one of two types of input sequences. Sample operational profiles for sort follow.

Opera tional profile

Mamatha.H.R

64

Opera tional profile

Mamatha.H.R

65

Testing and debugging

Testing is the process of determining if a program has any errors. When testing reveals an error, the process used to determine the cause of this error and to remove it, is known as debugging.
Mamatha.H.R 66

A test/debu cycle

~.[I]pU

d~Jta,

Op~mt~orUi :p:mfile

I
Debu~~he
se 8 & 10'[l Tter~~ n: :~ffig.ra~l:[Jl

_J

:Fi e. te:!i,t

Mamatha.H.R

67

Test plan A test cycle is often guided btyesat plan. Example: Thesort program is to be tested to meet the requirements given earlier. Specifically, the following needs to be done. Execute sort on at least two input sequences, one with ``A'' and the other

with ``D'' as request characters.

Test plan (contd.) Execute the program on an empty input sequence. Test the program for robustness against erroneous inputs such as ``R'' typed in as the request character All failures of the test program should be recorded in a suitable file using the Company Failure Report Form.

Test c ase/data
A test caseis a pair consisting of test data bteo input to the program and the expected output. The test data is a set of values, one for each input variable. A test setis a collection of zero or more test cases. Sample test case fosrort:

Test data: <''A '12 -29 32 > Expected output:-29 12 32

Program behavior

Can be specified in several ways: plain natural language, a state diagram, formal mathematical specification, etc. A state diagramspecifies program states and how the program changes its state on an input sequence. inputs.

Progr am behavior: Example

Consider a menu driven application.

Program behavior: Example (contd.)

Mamatha.H.R

73

Behavior: observation and analysis

In the first step onoebserves the behavior. In the second step onaenalyzes the observed behavior to check if it is correct or not. Both these steps could be quite complex for large commercial programs. The entity that performs the task of checking the correctness of the observed behavior is known as anoracle
Mamatha.H.R 74

Oracl e: Example

Mamatha.H.R

75

Oracle: Programs
Oracles can also be programs designed to check the behavior of other programs. For example, one might use a matrix multiplication program to check if a matrix inversion program has produced the correct output. In this case, the matrix inversion program inverts a given matrix A and generates B as the output matrix.
Mamatha.H.R 76

Ora cle: Construction

Construction of automated oracles, such as the one to check a matrix multiplication program or a sort program, requires the determination ofinput-output relationship. In general, the construction of automated oracles is a complex undertaking.

Mamatha.H.R

77

Ora cle construction: Example

Mamatha.H.R

78

Tes ting and verification

Program verificationaims at proving the correctness of programs by showing that it contains no errors. This is very different fromtesting that aims at uncovering errors in a program.

Program verification and testing are best considered as complementary techniques. In practice, one can shed program verification, but not testing.
Mamatha.H.R 79

Tes ting and verification (contd.)

Testing is not a perfect technique in that a program might contain errors despite the success of a set of tests. Verification might appear to be perfect technique as it promises to verify that a program is free from errors. However, thenperso who verified a program might have made mistake in the verification process; there might be an incorrect assumption on the input conditions; incorrect assumptions might be made regarding the components that interface with the program, and so on.
Verified and published programs have been shown Mamatha.H.R to be incorrect.
80

Test Metrics

The term metric refers to a standard of measurement.

Mamatha.H.R

81

Types of metrics used in software testing and their relationships

Test metrics

Product Organizational Project Process Static Dynamic

Mamatha.H.R

82

Organization Establishes test processes Used in projects To test products

Four general core areas that assist in the design of metrics

Schedule-related metrics measure actual completion times of various activities and compare these with estimated time to completion. Quality-related metricsmeasure quality of a product or a process

Resource-related metricsmeasure items such as cost in dollars, manpower and tests executed. Size-related metricsmeasure size of various objects such as the source code and number of tests in a test suite.

Organizational Metrics
Metrics at the level of an organization are useful in overall project planning and management. Some of these metrics are obtained by aggregating compatible metrics across multiple projects.

Organizational-level metrics allow senior management to monitor the overall strength of the organization and points to areas of weakness. Thus these metrics help senior management in setting new goals and plan for resources needed to realize these goals.

Project metrics
Project metrics relate to a specific project. Example :the I/O device testing project or a compiler project. These are useful in the monitoring and control of a specific project. The ratio of actu- -planned system at test effort is one project metric.

Proc ess Metrics

The goal of a process metric is to assess the goodness of the process. Ex: unit test, integration test and system test. The defect that is found later is costlier to fix. Hence, a metric that classifies defects according to the phase in which they are found assists in evaluating the process itself.

Product Metrics: Generic

Product metrics relate to a specific product such as compiler for a programming language. These are useful in making decisions related to the product. Ex: Should this product be released for use by the customer.

Types of product metrics

The cyclomatic complexity Halstead metric

Cycl omatic complexity

Proposed by Thomas McCabe in 1976 Based on the control flow of a program Given the CFG G of a program P, The cyclomatic complexity is computed as: V(G) = E N+2p Where V(G) =cyclomatic complexity E = Edges N= Nodes p = Connected procedures of a program P

p counts only procedures that are reachable from the main function V(G) is the complexity of a CFG G that corresponds to a procedure reachable from the main procedure. V(G) is not the complexity of the entire program instead it is the complexity of a procedure in P that corresponds to G.

Larger value of V(G) tend to imply higher program complexity and hence the program more difficult to understand V(G) of the values 5 or less are recommended.

Halstead metric

Published by Prof. Maurice Halstead. Using the program size (S) and effort (E), the following estimator has been proposed for the number of errors (B) 0 .667 0.333 B = 7.6 E S

Halstead measures of program complexity and effor t

Measure Operator count Operand Count Unique Operators Unique operands Notation Definition Number of operators N1
in a program

N2 n1 n2

Number of operands in a program Number of unique operators in a program Number of unique operands in a program

Measure Program vocabulary

Notation n

Definition n1+n2 N1+N2 N* Log2 D*V n

Program size N Program volume Difficulty Effort V D E

Mamatha.H.R

(2/n1) * (n2/N2)
97

Advantage of using an estimator such as B is that it allows the management to plan for testing resources. Ex: A larger value of number of expected errors will lead to a larger number of testers and testing resources to complete the test process over a given duration.
Mamatha.H.R 98

Software science metrics are not suitable to object oriented languages like java & C++. Specially devised metrics are used for the OO languages.

Mamatha.H.R

99

A sample of product metrics: OO S oftware

Metric
Reliability

Meaning
Probability of failure of a software product w.r.t a given operational profile in a given environment. Number of defects per KLOC Distribution of defects by their level of severity Fraction of testableitems,e.g.basic blocks,covered.also a metric for test adeqMaumathca.yH.R or goodness of tests.
100

Defect density Defect severity Test coverage

Cyclomatic complexity

Measures complexity of a program based on its CFG

Weighted methods n n=1 ci, ci is the complexity of method i in per class the class under consideration Class coupling Measures the number of classes to which a given class is coupled Set of all methods that can inbveoked,directly and indirectly,when a message is sent to object O

Response set

Number of children Number of immediate descendants of a class in the class hierarchy

Mamatha.H.R 101

Progress Monitoring and Trends

Metrics are often used for monitoring progress. This requires making measurements on a regular basis over time. Such measurements offer trends

Ex one could measure the cumulative number of defects found and plot these over time. such a plot will rise over time.

Static Metrics
Static metrics are those computed without having to execute the product. Ex number of testable entities in an application is a static product metric The average number of testers working on a project is a static project metric.

Dyn amic Metrics

Dynamic metrics require code execution Ex the number of testable entities actually covered by a test suite is a dynamic product metric. Number of defects remaining to be fixed could be treated as dynamic metric as it can be computed accurately only after a code change has been made and the product retested.

Testability

Testability is the degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have met.

Categories of testability metrics

Static testability metrics Dynamic testability metrics

Static testability metrics

Ex software complexity The more complex an application, the lower the testability, i.e., the higher the effort required to test it.

Dynamic testability metrics

Ex cod-ebased coverage criteria A program for which it is difficult to generate tests that satisfy the statement coverage criterion is considered to have low testability than one for which it is easier to construct such tests.

High testability is a desirable goal. It is recommended that features to be tested and how they are to be tested must be identified during the requirements stage. This information is carried over to coding phase through design phase.

Example A pplication E to control the operation of an elevator

Variety of tests has to be performed. One test checks if the elevator hoist motor and the brakes are working correctly (can be done when hardware is available). To improve the testability of E, A component that allows to communicate with a hoist motor and brake simulator and display the status of the simulated hardware devices is required.

Another test requirement for E is that it must allow a tester to experiment with various scheduling algorithms.

Testability is a concern in both hardware and software designs. In hardware design, testability implies that three exist tests to detect any fault w.r.t a fault model in a finished product. Thus, the aim is to verify the correctness of a finished product.

Testability in software focuses on the verification of design and implementation.