CHAPTER 1 Testing - Page 1

Every profession has its own vocabulary.To learn a profession, the first and crucial step is to master its vocabulary.The entire knowledge of a profession is compressed and kept it in its vocabulary. Take our own software testing profession, while communicating with our collegues, we frequently use terms like 'regression testing', 'System testing', now imagine communicating the same to a person who is not in our profession or who doesn't understand our testing vocabulary, we need to explain in detail each and every term . ommunication becomes so difficult and painful.To speak the language of testing, you need to learn its vocabulary. !ind below a huge collection of testing vocabulary

Affinity Diagram" # group process that takes large amounts of language data, such as developing by brainstorming, and divides it into categories Audit: This is an inspection$assessment activity that verifies compliance with plans, policies and procedures and ensures that resources are conserved. Baseline:# quantitative measure of the current level of performance. Benchmarking" omparing your company's products, services or processes against best practices or competitive practices, to help define superior performance of a product,service or support processes. Black-b ! Testing: # test technique that focuses on testing the functionality of the program component or application against its specifications without knowlegde of how the system constructed. B undary "alue analysis: # data selection technique in which test data is chosen from the %boundaries% of the input or output domain classes, data structures and procedure parameters. hoices often include the actual minimum and maximum boundary values, the maximum value plus or minus one and the minimum value plus or minus one. Branch Testing: # test method that requires that each possible branch on each decision be executed on at least once. &rainstorming" # group process for generating creative and diverse ideas. Bug: # catchall term for all software defects or errors. Certificati n testing: #cceptance of software by an authori'ed agent after the software has been validated by the agent or after its validity has been demonstrated to the agent. Check# int$ r "erificati n # int%: Expected behaviour of the application which must be validated with the actual behaviour after certain action has been performed on the application.

Client: The customer that pays for the product received and receives the benefit from the use of the product. C nditi n C "erage: # white(box testing technique that measures the number of or percentage of decision outcomes covered by the test cases designed.)**+ condition coverage would indicate that every possible outcome of each decision had been executed at least once during testing. C nfigurati n &anagement T ls Tools that are used to keep track of changes made to systems and all related artifacts. These are also known as version control tools. C nfigurati n testing: Testing of an application on all supported hardware and software platforms.This may include various combinations of hardware types, configuration settings and software versions. C m#leteness: # product is said to be complete if it has met all requirements. C nsistency: #dherence to a given set of rules. C rrectness: The extent to which software is free from design and coding defects. ,t is also the extent to which software meets the specified requirements and user ob-ectives. C st f 'uality: .oney spent above and beyond expected production costs to ensure that the product the customer receives is a quality product. The cost of quality includes prevention, appraisal, and correction or repair costs. C n"ersi n Testing: /alidates the effectiveness of data conversion processes, including field( field mapping and data translation. Cust mer: The individual or organi'ation, internal or external to the producing organi'ation that receives the product. Cycl matic c m#le!ity: The number of decision statements plus one.

Testing - Page (
Debugging: The process of analysing and correcting syntactic, logic and other errors identified during testing. Decisi n C "erage: # white(box testing technique that measures the number of ( or percentage ( of decision directions executed by the test case designed. )**+ 0ecision coverage would indicate that all decision directions had been executed at least once during testing. #lternatively each logical path through the program can be tested. Decisi n Table # tool for documenting the unique combinations of conditions and associated results in order to derive unique test cases for validation testing.

Defect Tracking T ls Tools for documenting defects as they are found during testing and for tracking their status through to resolution. Desk Check: # verification technique conducted by the author of the artifcat to verify the completeness of their own work. This technique does not involve anyone else. Dynamic Analysis: #nalysis performed by executing the program code.0ynamic analysis executes or simulates a development phase product and it detects errors by analy'ing the response of the product to sets of input data. Entrance Criteria: 1equired conditions and standards for work product quality that must be present or met for entry into the next stage of the software development process. E)ui"alence Partiti ning: # test technique that utili'es a subset of data that is representative of a larger class. This is done in place of undertaking exhaustive testing of each value of the larger class of data. Err r r defect: ).# discrepancy between a computed, observed or measured value or condition and the true, specified or theortically correct value or conditon 2.3uman action that results in software containing a fault 4e.g., omission or misinterpretation of user requirements in a software specification, incorrect translation or omission of a requirement in the design specification5 Err r *uessing: Test data selection techniques for picking values that seem likely to cause defects. This technique is based upon the theory that test cases and test data can be developed based on intuition and experience of the tester. E!hausti"e Testing: Executing the program through all possible combination of values for program variables. E!it criteria: Standards for work product quality which block the promotion of incomplete or defective work products to subsequent stages of the software development process. +l ,chart 6ictorial representations of data flow and computer logic. ,t is frequently easier to understand and assess the structure and logic of an application system by developing a flow chart than to attempt to understand narrative descriptions or verbal explanations. The flowcharts for systems are normally developed manually, while flowcharts of programs can be produced. + rce +ield Analysis # group technique used to identify both driving and restraining forces that influence a current situation. + rmal Analysis Technique that uses rigorous mathematical techniques to analy'e the algorithms of a solution for numerical properties, efficiency, and correctness. +uncti nal Testing Testing that ensures all functional requirements are met without regard to the final program structure.

Testing - Page Hist gram # graphical description of individually measured values in a data set that is organi'ed according to the frequency or relative frequency of occurrence. # histogram illustrates the shape of the distribution of individual values in a data set along with information regarding the average and variation. .ns#ecti n # formal assessment of a work product conducted by one or more qualified independent reviewers to detect defects, violations of development standards, and other problems. ,nspections involve authors only when specific questions concerning deliverables exist. #n inspection identifies defects, but does not attempt to correct them. #uthors take corrective actions and arrange follow(up reviews as needed. .ntegrati n Testing This test begins after two or more programs or application components have been successfully unit tested. ,t is conducted by the development team to validate the interaction or communication$flow of information between the individual components which will be integrated. /ife Cycle Testing The process of verifying the consistency, completeness, and correctness of software at each stage of the development life cycle. Pass0+ail Criteria 0ecision rules used to determine whether a software item or feature passes or fails a test. Path Testing # test method satisfying the coverage criteria that each logical path through the program be

tested. 7ften, paths through the program are grouped into a finite set of classes and one path from each class is tested. Perf rmance Test /alidates that both the online response time and batch run times meet the defined performance requirements. P licy .anagerial desires and intents concerning either process 4intended ob-ectives5 or products 4desired attributes5. P #ulati n Analysis #naly'es production data to identify, independent from the specifications, the types and frequency of data that the system will have to process$produce. This verifies that the specs can handle types and frequency of actual data and can be used to create validation tests. Pr cedure The step(by(step method followed to ensure that standards are met. Pr cess ). The work effort that produces a product. This includes efforts of people and equipment guided by policies, standards, and procedures. 2. # statement of purpose and an essential set of practices 4activities5 that address that purpose. Pr f f C rrectness The use of mathematical logic techniques to show that a relationship between program variables assumed true at program entry implies that another relationship between program variables holds at program exit. 'uality # product is a quality product if it is defect free. To the producer, a product is a quality product if it meets or conforms to the statement of requirements that defines the product. This statement is usually shortened to" quality means meets requirements. !rom a customer8s perspective, quality means 9fit for use.: 'uality Assurance $'A% 0eals with 'prevention' of defects in the product being developed.,t is associated with a process.The set of support activities 4including facilitation, training, measurement, and analysis5 needed to provide adequate confidence that processes are established and continuously improved to produce products that meet specifications and are fit for use. 'uality C ntr l $'C% ,ts focus is defect detection and removal. Testing is a quality control activity 'uality .m#r "ement To change a production process so that the rate at which defective products 4defects5 are produced is reduced. Some process changes may require the product to be changed.

Testing - Page 1
Rec "ery Test Evaluates the contingency features built into the application for handling interruptions and for returning to specific points in the application processing cycle, including checkpoints, backups, restores, and restarts. This test also assures that disaster recovery is possible. Regressi n Testing Testing of a previously verified program or application following program modification for extension or correction to ensure no new defects have been introduced. Risk &atri! Shows the controls within application systems used to reduce the identified risk, and in what segment of the application those risks exist. 7ne dimension of the matrix is the risk, the second dimension is the segment of the application system, and within the matrix at the intersections are the controls. !or example, if a risk is 9incorrect input: and the systems segment is 9data entry,: then the intersection within the matrix would show the controls designed to reduce the risk of incorrect input during the data entry segment of the application system. 2catter Pl t Diagram # graph designed to show whether there is a relationship between two changing variables. 2tandards The measure used to evaluate products and identify nonconformance. The basis upon which adherence to policies is measured. 2tatement f Re)uirements The exhaustive list of requirements that define a product. 2tatement Testing # test method that executes each statement in a program at least once during program testing. 2tatic Analysis #nalysis of a program that is performed without executing the program. ,t may be applied to the requirements, design, or code. 2tress Testing This test sub-ects a system, or components of a system, to varying environmental conditions that defy normal expectations. !or example, high transaction volume, large database si'e or restart$recovery circumstances. The intention of stress testing is to identify constraints and to ensure that there are no performance problems. 2tructural Testing # testing method in which the test data is derived solely from the program structure.

2tub Special code segments that when invoked by a code segment under testing, simulate the behavior of designed and specified modules not yet constructed. 2ystem Test 0uring this event, the entire system is tested to verify that all functional, information, structural and quality requirements have been met. Test Case Test cases document the input, expected results, and execution conditions of a given test item. Test Plan # document describing the intended scope, approach, resources, and schedule of testing activities. ,t identifies test items, the features to be tested, the testing tasks, the personnel performing each task, and any risks requiring contingency planning. Test 2cri#ts # tool that specifies an order of actions that should be performed during a test session. The script also contains expected results. Test scripts may be manually prepared using paper forms, or may be automated using capture$playback tools or other kinds of automated scripting tools. Test 2uite &anager # tool that allows testers to organi'e test scripts by function or other grouping. 3nit Test Testing individual programs, modules, or components to demonstrate that the work package executes per specification, and validate the design and technical quality of the application. The focus is on ensuring that the detailed logic within the component is accurate and reliable according to pre(determined specifications. Testing stubs or drivers may be used to simulate behavior of interfacing modules. 3sability Test The purpose of this event is to review the application user interface and other human factors of the application with the people who will be using the application. This is to ensure that the design 4layout and sequence, etc.5 enables the business functions to be executed as easily and intuitively as possible. This review includes assuring that the user interface adheres to documented ;ser ,nterface standards, and should be conducted early in the design stage of development. ,deally, an application prototype is used to walk the client group through various business scenarios, although paper copies of screens, windows, menus, and reports can be used. 3ser Acce#tance Test ;ser #cceptance Testing 4;#T5 is conducted to ensure that the system meets the needs of the organi'ation and the end user$customer. ,t validates that the system will work as intended by the user in the real world, and is based on real world business scenarios, not system requirements. Essentially, this test validates that the right system was built. 4alidati n 0etermination of the correctness of the final program or software produced from a development pro-ect with respect to the user needs and requirements.

4erificati n ). The process of determining whether the products of a given phase of the software development cycle fulfill the requirements established during the previous phase. 2. The act of reviewing, inspecting, testing, checking, auditing, or otherwise establishing and documenting whether items, processes, services, or documents conform to specified requirements. 5alkthr ughs 0uring a walkthrough, the producer of a product 9walks through: or paraphrases the products content, while a team of other individuals follow along. The team8s -ob is to ask questions and raise issues about the product that may lead to defect identification. 5hite-b ! Testing # testing technique that assumes that the path of the logic in a program unit or component is known. <hite(box testing usually consists of testing paths, branch by branch, to produce predictable results. This technique is usually used during tests executed by the development team, such as ;nit or omponent testing.

CHAPTER ( 5hat is 'uality6

5hat is )uality6 r Define )uality6 =ot of quality pioneers defined quality in different ways # quality product is defined as the one that meets product requirements &ut >uality can only be seen through customer eyes.So the most important definition of quality is meeting customer needs or ;nderstanding customer requirements, expectations and exceeding those expectations. ustomer must be satisfied by using the product, then its a quality product. 5hats the difference bet,een meeting #r duct re)uirements and meeting cust mer needs6 Aren7t cust mer needs tranlsated int #r duct re)uirements6 ?ot always.Though our aim is to accurately capture customer needs into requirements and build a product that satisfies those needs, we sometimes fail to do so because of the following reasons ( ustomers fail to accurately communicate their exact needs (captured requirements can be misinterpreted

Can7t ,e define a )uality #r duct as the ne that c ntains n bugs0defects6 >uality is much more than absence of defects$bugs. onsider this, though the product may have 'ero defects, but if the usability sucks i.e it is difficult to learn and operate the product, then its not a quality product. .f the #r duct has s me defects8 can it be still called a )uality #r duct6 ,t depends on the nature of those bugs.&ut in some cases, even though a product has bugs, it can be still called a quality product. ;nless the product is very critical, aiming for 'ero defects is not cost effective always.<e should aim for )**+ defect 'detection', but given the budget, time and resources constraints, we can still release the product with some unfixed or open bugs. ,f the open bugs cause no loss to the customer,then it can be still called a quality product. .s )uality nly testers res# nsiblity6 ?o. >uality is everybody's responsibility including the customer.<e, testers identify the deviations and report them, thats it.There are many factors that impact the quality such as maintainabiltiy, reusability, flexibility, portabilty which the testers can't validate. Testers can only validate the correctness, reliability, usability and interoperability of a product and report the deviations. 5hen is the right time t catch a bug6 #s soon as possible.The cost of fixing the bug will keep on increasing exponentially as the product development progresses.!or example, the cost of fixing a design bug identified in system testing is much more than fixing it, if it had been identified during design phase itself because now you not only have to rectify the design but also the code, the corresponding documents and code that is dependent on this code. Are there any ther )uality c ntr l #ractices a#art fr m testing6 @es.,nspections, design and code walkthroughs, reviews etc. ,hat are s ft,are )uality fact rs6 software quality factors are attributes of the software that, if they are wanted and not present, pose a risk to the success of the software. There are )) main factors and their definitions are given below. The priority and importance of the these attributes keeps changing from product to product.=ike if the product being developed needs to be changed quite frequently, then flexibility and reusability of the product needs to be given priority. The following are the quality factors Correctness" Extent to which a program satisfies its requirements Reliability" Extent to which a program can be expected to perform its intended function with required precision. Efficiency" The amount of computing resources and code required by a program to perform a function. Integrity" Extent to which access to software or data by unauthori'ed persons can be controlled. Usability" Effort required learning, operating, preparing input, and interpreting output of a program. Maintainability" Effort required locating and fixing an error in an operational program. Testability" Effort required testing a program to ensure that it performs its intended function.

Flexibility" Effort required modifying an operational program. Portability" Effort required to transfer software from one configuration to another. Reusability" Extent to which a program can be used in other applications A related to the packaging and scope of the functions that programs perform. Interoperability" Effort required to couple one system with another. H , t reduce the am unt s#end t ensure and build )uality6 r H , t reduce the c st f )uality6 cost of quality includes the total amount spent on preventing errors, identifying and correcting errors. oming to reducing this cost.Try to build a product that has less defects or no defects even before it goes to testing phase and to achieve this you should spend more money and effort on tyring to prevent errors from going into the product.@ou must concentrate greatly on building efficient and effective processes and keep on continuously improving them by identifying weakness in them.@ou many not reap great benefits immediately but over a long run you can make significant savings by reducing the cost of quality. H , t reduce the c st f fi!ing a bug6 atch it as early as possible. #s the development process progresses,the cost of fixing a bug keep on increasing exponentially. 6ractice life cycle testing.

10

CHAPTER /ife Cycle Testing r 4 testing

,n traditional waterfall model, testing comes at the fag end of the development process.?o testing is done during requirements gathering phase, design phase and development phase. 0efects identified during this disconnected testing phase are very costly to fix which is this model's biggest disadvantage. =ife cycle testing or / testing aims at catching the defects as early as possible and thus reduces the cost of fixing them.,t achieves this by continuously testing the system during all phases of the development process rather than -ust limiting testing to the last phase. The life cycle testing can be best accomplished by the formation of a separate test team. when the pro-ect starts both the system development process and system test process begins. The team that is developing the system begins the systems development process and the team that is conducting the system test begins planning the system test process.&oth teams start at the same point using the same information.The systems development team has the and document the requirements for developmental purposes. The test team will likewise use those same requirements, but for the purpose of testing the system. #t appropriate points during the developmental process, the test team will test the developmental process in an attempt to uncover defects. The following is the software testing process which follows life cycle testing Re)uirements *athering #hase: /erify whether the requirements captured are true user needs /erify that the requirements captured are complete, unambiguous, accurate and non conflicting with each other Design #hase: /erify whether the design achieves the ob-ectives of the requirements as well as the design being effective and efficient /erification Techniques" 0esign walkthroughs, 0esign ,nspections

C ding #hase: /erify that the design is correctly translated to code /erify coding is as per company's standards and policies /erification Techniques" ode walkthroughs, code ,nspections /alidation Techniques" ;nit testing and ,ntegration techniques 2ystem Testing #hase: Execute test cases =og bugs and track them to closure

11

3ser Acce#tance #hase: ;sers validate the applicability and usability of the software in performing their day to day operations. &aintenance #hase: #fter the software is implemented, any changes to the software must be thoroughly tested and care should be taken not to introduce regression issues.

The life cycle testing is also called / testing. The pro-ect8s 0o and heck procedures slowly converge from start to finish 4see above figure5, which indicates that as the 0o team attempts to implement a solution, the heck team concurrently develops a process to minimi'e or eliminate the risk. ,f the two groups work closely together, the high level of risk at a pro-ect8s inception will decrease to an acceptable level by the pro-ect8s conclusion.

CHAPTER 1 Ty#es f Testing - Page 1

Black b ! testing ( not based on any knowledge of internal design or code. Tests are based on requirements and functionality. 5hite b ! testing ( based on knowledge of the internal logic of an application's code. Tests are based on coverage of code statements, branches, paths, conditions.

12

3nit testing ( ;nit is the smallest compilable component. # unit typically is the work of one programmer.This unit is tested in isolation with the help of stubs or drivers.Typically done by the programmer and not by testers.
.ncremental integrati n testing ( continuous testing of an application as new functionality is addedB requires that various aspects of an application's functionality be independent enough to work separately before all parts of the program are completed, or that test drivers be developed as neededB done by programmers or by testers. .ntegrati n testing ( testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client$server and distributed systems. +uncti nal testing ( black(box testing aimed to validate to functional requirements of an applicationB this type of testing should be done by testers. 2ystem testing ( black(box type testing that is based on overall requirements specificationsB covers all combined parts of a system. End-t -end testing ( similar to system testing but involves testing of the application in a environment that mimics real(world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate. Even the transactions performed mimics the end users usage of the application. 2anity testing ( typically an initial testing effort to determine if a new software version is performing well enough to accept it for a ma-or testing effort. !or example, if the new software is crashing systems every C minutes, bogging down systems to a crawl, or destroying databases, the software may not be in a 'sane' enough condition to warrant further testing in its current state. 2m ke testing ( The general definition 4related to 3ardware5 of Smoke Testing is" Smoke testing is a safe harmless procedure of blowing smoke into parts of the sewer and drain lines to detect sources of unwanted leaks and sources of sewer odors. ,n relation to software, the definition is Smoke testing is non(exhaustive software testing, ascertaining that the most crucial functions of a program work, but not bothering with finer details. 2tatic testing ( Test activities that are performed without running the software is called static testing. Static testing includes code inspections, walkthroughs, and desk checks Dynamic testing ( test activities that involve running the software are called dynamic testing. Regressi n testing ( Testing of a previously verified program or application following program modification for extension or correction to ensure no new defects have been introduced.#utomated testing tools can be especially useful for this type of testing. Acce#tance testing ( final testing based on specifications of the end(user or customer, or based on use by end(users$customers over some limited period of time. / ad testing (=oad testing is a test whose ob-ective is to determine the maximum sustainable load the system can handle. =oad is varied from a minimum 4'ero5 to the maximum level the system can sustain without running out of resources or having, transactions suffer 4application( specific5 excessive delay.

13

2tress testing ( Stress testing is sub-ecting a system to an unreasonable load while denying it the resources 4e.g., 1#., disc, mips, interrupts5 needed to process that load. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not expected to process the overload without adequate resources, but to behave 4e.g., fail5 in a decent manner 4e.g., not corrupting or losing data5. The load 4incoming transaction stream5 in stress testing is often deliberately distorted so as to force the system into resource depletion.

Ty#es f Testing - Page (

Perf rmance testing ( /alidates that both the online response time and batch run times meet the defined performance requirements. 3sability testing ( testing for 'user(friendliness'. learly this is sub-ective, and will depend on the targeted end(user or customer. ;ser interviews, surveys, video recording of user sessions, and other techniques can be used. 6rogrammers and testers are usually not appropriate as usability testers. .nstall0uninstall testing ( testing of full, partial, or upgrade install$uninstall processes. Rec "ery testing ( testing how well a system recovers from crashes, hardware failures, or other catastrophic problems. 2ecurity testing ( testing how well the system protects against unauthori'ed internal or external access, willful damage, etcB may require sophisticated testing techniques. C m#atibility testing ( testing how well software performs in a particular hardware$software$ operating system$network$etc. environment. E!#l rat ry testing ( often taken to mean a creative, informal software test that is not based on formal test plans or test casesB testers may be learning the software as they test it. Ad-h c testing ( similar to exploratory testing, but often taken to mean that the testers have significant understanding of the software before testing it. & nkey testing:-monkey testing is a testing that runs with no specific test in mind. The monkey in this case is the producer of any input data 4whether that be file data, or input device data5. Deep pressing some keys randomely and check whether the software fails or not. 3ser acce#tance testing ( determining if software is satisfactory to an end(user or customer. C m#aris n testing ( comparing software weaknesses and strengths to competing products.

14

Al#ha testing ( testing of an application when development is nearing completionB minor design changes may still be made as a result of such testing. Typically done by users within the development team. Beta testing ( testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by end(users or others, not by programmers or testers. &utati n testing ( a method for determining if a set of test data or test cases is useful, by deliberately introducing various code changes 4'bugs'5 and retesting with the original test data$cases to determine if the 'bugs' are detected. 6roper implementation requires large computational resources Cr ss br ,ser testing ( application tested with different browser for usablity testing E compatiblity testing C ncurrent testing ( .ulti(user testing geared towards determining the effects of accessing the same application code, module or database records. ,dentifies and measures the level of locking, deadlocking and use of single(threaded code and locking semaphores etc.

9egati"e testing ( Testing the application for fail conditions,negative testing is testing the tool with improper inputs.for example entering the special characters for phone number

CHAPTER : 2 ft,are Testing Techni)ues

Testing techniques can be used to effectively design efficient test cases.These techniques can be grouped into black(box and white(box techniques. !ind below some of the techniques

15

Black-B ! Testing techni)ues

<hen creating black(box test cases, the input data used is critical. Three successful techniques for managing the amount of input data required include" E)ui"alence Partiti ning #n equivalence class is a subset of data that is representative of a larger class.Equivalence partitioning is a technique for testing equivalence classes rather thanundertaking exhaustive testing of each value of the larger class. !or example, aprogram which edits credit limits within a given range 4),*** ( ),C**5 would have three equivalence classes" F ),*** 4invalid5 &etween ),*** and ),C** 4valid5 G ),C** 4invalid5 B undary Analysis # technique that consists of developing test cases and data that focus on the input and output boundaries of a given function. ,n same credit limit example, boundary analysis would test" =ow boundary H$( one 4III and ),**)5 7n the boundary 4),*** and ),C**5 ;pper boundary H$( one 4),JII and ),C*)5 Err r *uessing Test cases can be developed based upon the intuition and experience of the tester. !or example, in an example where one of the inputs is the date, a tester may try !ebruary 2I, 2***

5hite-B ! Testing techni)ues

<hite(box testing assumes that the path of logic in a unit or program is known. <hite(box testing consists of testing paths, branch by branch, to produce predictable results. The following are white(box testing techniques" 2tatement C "erage Execute all statements at least once. Decisi n C "erage Execute each decision direction at least once. C nditi n C "erage Execute each decision with all possible outcomes at least once. Decisi n0C nditi n C "erage Execute all possible combinations of condition outcomes in each decision. Treat all iterations as two(way conditions exercising the loop 'ero times and one time.

16

CHAPTER ; Testing &etrics

<hile testing a product, test manager has to take a lot of decisions like when to stop testing or when is the application ready for production, how to track testing progress, how to measure the quality of a product at a certain point in the testing cycleKTesting metrics can help to take better and accurate decisions =ets start by defining the term '.etric' # metric is a mathematical number that shows a relationship between two variables. 2 ft,are metrics are measures used to quantify status or results.

H , t track testing #r gress6

The best way is to have a fixed number of test cases ready before test execution cycle begins.Then the testing progress is measured by the total number of test cases executed. < C m#leti n L 4?umber of test cases executed5$4Total number of test cases5 ?ot only the testing progress but also the following metrics are helpful to measure the quality of the product < Test cases Passed L 4?umber of test cases 6assed5$4?umber of test cases executed5 < Test cases +ailed L 4?umber of test cases 6assed5$4?umber of test cases executed5 Note: A test case is Failed when atleast one bug is found while executing it, otherwise Passed

H , many r unds r cycles f testing sh uld be d ne6 r 5hen t st # testing6

=ets discuss few approaches

17

A##r ach 1"This approache requires, that you have a fixed number of test cases ready before test execution cycle.,n each testing cycle you execute all test cases.@ou stop testing when all the test cases are 6assed or + failure is very very less in the latest testing cycle. A##r ach (".ake use of the following metrics &ean Time Bet,een +ailure: The average operational time it takes before a software system fails. C "erage metrics" the percentage of instructions or paths executed during tests. Defect density: defects related to si'e of software such as 9defects$)*** lines of code: 7pen bugs and their severity levels, ,f the coverage of code is good, .ean time between failure is quite large, defect density is very ow and not may high severity bugs still open, then 'may' be you should stop testing. 'Mood', 'large', 'low' and 'high' are sub-ective terms and depends on the product being tested.!inally, the risk associated with moving the application into production, as well as the risk of not moving forward, must be taken into consideration.

CHAPTER =

18

Test Plan Tem#late

Test Planning: is the selection of techniques and methods to be used to validate the product
against its approved requirements and design.,n this activity we assess the software application risks, and then develop a plan to determine if the software minimi'es those risks.<e document this planning in a Test 6lan document.

E!#lanati n f different secti ns in the tem#late

D cument 2ign ff: ;sually a test plan document is a contract between testing team and all the other teams involved in developing the product including the higher management folks. &efore signoff all interested parties thoroughly reviews the test plan and gives feedback, raises issues or concerns, if any.7nce everybody is satisfied with the test plan, they signoff the document and which is a green signal for the testing team to start executing the test plan. Change Hist ry: ;nder this section, you specify, who changed what in the document and when, along with the version of the document which contain the changes. Re"ie, and A##r "al Hist ry: This captures who reviewed the document and whether they #pproved the test plan or not. The reviewer may suggest some changes or comments4if any5 to be incorporated in the test plan. D cument References: #ny additional documents that will help better understand the test plan like design documents and$or 1equirements document etc. D cument 2c #e: ,n this section specify what the test plan covers and who its intended audience is. Pr duct 2ummary: ,n this section describe briefly about the product that is to be tested. Pr duct 'uality * als: ,n this section describe important quality goals of the product. !ollowing are some of the typical quality goals (1eliability, proper functioning as specified and expected. (1obustness, acceptable response to unusual inputs, loads and conditions. (Efficiency of use by the frequent users (Easy to use even for the less frequent users Testing >b?ecti"es: ,n this section specify the testing goals that need to be accomplished by the testing team. The goals must be measurable and should be prioriti'ed. The following are some example test ob-ectives. /erify functional correctness Test product robustness and stability. .easure performance Nhot spots8 4locations or features that are problem areas5. Assum#ti ns: ,n this section specify the expectations, which if not met could have negative impact on this test plan execution. Some of the assumptions can be on the test budget that must be allocated, resources needed etc. Testing 2c #e: ,n this section specify Nwhat will be covered in testing8 and Nwhat will not be covered8.

19

Testing 2trategy: ,n this section specify different testing types used to test the product. Tools needed to execute the strategy are also specified. Testing 2chedule: ,n this section specify, first the entire pro-ect schedule and then detailed testing schedule. Res urces: ,n this section specify all the resources needed to execute the plan successfully

C mmunicati n A##r ach: ,n this section specify how the testing team will report the bugs to the development, how it will report the testing progress to management, how it will report issues and concerns to higher ups.

CHAPTER @ Test Case Tem#late

Test >utline: This document is written before writing test cases.This is a planning document
in which the flows or scenarios are written at a high level. These flows or scenarios are later expanded to test cases, in which they are written in detail.#lso the biggest advantage of writing this document, before going to test cases is the 'traceability matrix', where you ensure that the pro-ect$feature is sufficiently or thoroughly covered by the individual test cases.

E!#lanati n f different secti ns in the tem#late

Change Hist ry: ;nder this section, you specify, who changed what in the document and when, along with the version of the document which contain the changes. Re"ie, and A##r "al Hist ry: This captures who reviewed the document and whether they #pproved the test outline or not. ,f approved, the reviewer will specify the review comments4if any5 to be incorporated in the test outline.There is a review template at the end of the testcaseOtemplate.doc, which can be used to specify the comments for test outline also.,f the test

20

outline document is '?ot #pproved', then either the scenarios mentioned are not sufficient or the scenarios are in a very bad shape4not in a state to be reviewed5 etc. D cument References: #ny additional documents that will help better understand the test outline document like design documents or 1equirements document etc. Pr ?ects C "ered in Test >utline: 6ro-ects can be features of the product or modules which are covered in the test outline document. Traceability &atri!: This .atrix is filled after finishing writing all scenarios in the outline.This is to ensure that all requiremnts or features are sufficiently covered by the test cases and none are missing.So you map the requirement or feature and subfeature to the test case that will be covering it. The following ,0s uniquely identify the requirements or feature and subfeature.@ou can add your own ,0s based on the need 1E>O,0 L 1equirement ,0 from the S1S document 00O,0 L 0etailed 0esign ,0 from the 0etailed 0esign document 2etu# Re)uirements: #ny setup that has to be done in the application being tested, prior to executing this test case, should be mentioned here.!or example, if the test case needs certain login ,0s with certain settings to begin, which are not created as part of the test case, then such things need to mentioned in this section. Test >b?ecti"es: Specify at a very high level, what the test case is intended to achieve or verify. Test Case /imitati ns: 0oes the test case achieve the above mentioned test ob-ective completely or are there any exceptionsKThese exceptions need to be specified in this section.!or example, test case has to verify 'something' on type #, type & and type P, but because of some reason it could ?7T verify that 'something' on type P, then its a limitation. Test Case De#endencies 0 Assum#ti ns: 6rior to executing this test case, any other test case needs to be runK #ll those dependencies need to mentioned here. Pr cess +l ,: ,n this section, we specify at a high level what the flow of the test case is.Suppose there are multiple users in the test case, then a process flow can look like user1: does something user2: does something else user1: does again something user2: says good bye Test >utline Table c lumn - 73ser7: <ho has to perform the action. Suppose in a application, there are two roles '&uyer' and 'Supplier', then user can be those role names. Test >utline Table c lumn - 7Acti n7: ;nder #ction you specify the following !low ?ame ( # high level name given to action performed by the user.Suppose &uyer has to create certain purchase orders in the applications, then the flow name can be ' reate 6urchase 7rders' 0escription ( The following things should be mentioned here at a high level 0escription of what actions should be performed <hat is the type or characteristics of data to be used. <hat should be verified or checked after performing the action. Eff rt Estimates: ,n this section you specify the effort needed to write each test case and the effort needed to execute them.

21

CHAPTER A Test Case Tem#late

E!#lanati n f different secti ns in the tem#late
Change Hist ry: ;nder this section, you specify, who changed what in the document and when, along with the version of the document which contain the changes. Re"ie, and A##r "al Hist ry: This captures who reviewed the document and whether they #pproved the test case or not. ,f approved, the reviewer will specify the review comments to be incorporated in the test case.There is a review template at the end of the template document, which can be used to specify the comments.,f the test case document is '?ot #pproved', then either the test case is not necessary4redundant5 or it is in a very bad shape4not in a state to be reviewed5 D cument References: #ny additional documents that will help better understand the test case like test oulines or design documents or 1equirements document etc. .ntr ducti n0>"erall Test >b?ecti"es: Specify at a very high level, what the test case is intended to achieve or verify. Test Case /imitati ns: 0oes the test case achieve the above mentioned test ob-ective completely or are there any exceptionsKThese exceptions need to be specified in this section.!or example, test case has to verify something on type #, type & and type P, but because of some reason it could ?7T verify that something on type P, then its a limitation. Test Case De#endencies 0 Assum#ti ns: 6rior to executing this test case, any other test case needs to be runK #ll those dependencies need to mentioned here. 2etu# Re)uirements: #ny setup that has to be done in the application being tested, prior to executing this test script should be mentioned here.!or example, if the test case needs certain =ogin ,0s with certain settings to begin, which are not created as part of the test case, then such things need to mentioned in this section Pr cess +l ,: ,n this section, we mention who does what in the test case. Suppose there are multiple users in the test case, then a process flow can look like user1: does something user2: does something else user1: does again something user2: says good bye

22

Test Case: The actual test case begins in section C, which can be further divided into subsections upon convenience and need.!or example, if the test case is for an integrated application, then everytime we login to a new application, we can have a new subsection. !ollowing is the example of how a test case looks like Step Num: 1 Step escription: chec! login Path and Action: "nter user name, "nter pwd, clic! #ogin $est ata: abcd, abcd "xpected %esults: &erify error message is thrown that username and password entered are wrong A##endi!: This section contain any additional data that the test case refers.!or example if your test case has large amounts of 'Test 0ata' which is difficult to put under the column 'Test 0ata' for each step, then you can use the appendix section to hold the data and in the test case, can give reference to appendix. Test Case Re"ie, Tem#late: This template can be used by the reviewers to provide their review comments.They can classify the comments based on their severity.The Test Engineer who incorporates the comments in the test case, should specify the action taken by him in the template and then ' lose' the comment.

CHAPTER 1B /ife Cycle f a 2 ft,are Bug

7nce a bug4defect or error5 is found, it should be communicated to the developers who can fix it. 7nce the bug is fixed$resolved, the fix should be verified by the testers and should be closed. The following topics are discussed in this page Bug .nf rmati n:,nformation that should be captured in the bug so that developers can clearly

23

understand the bug and fix it. /ist f Bug statuses: /ifecycle f s me ty#es f bugs: Analysis f bugs"&ugs logged during a testing phase a invaluable source to improve the existing testing processes.

Bug inf rmati n

The following information should be captured in the bug so that developers can clearly understand the bug, get an idea of it's severity, and reproduce it if necessary.#lso the developer should mention in the bug, the cause of the problem, steps he has taken to fix it$fix description, steps he has taken to verify the fix and any information that helps to prevent such issues in future. Bug .D : # unique identifier4number5 of the bug Bug status: ,n the long road between logging bug and fixing it, the status of a bug communicates where it is.Eg" ?ew,#ssigned,fixed,closed etc. # list of different bug statuses are mentioned below along with their descriptions. A##licati n details: 0etails of the application like application name, version, ;1=, database details etc. C m# nent and0 r subc m# nent: The part of the application in which the bug was found by tester En"ir ment details: Such as 7perating system, hardware platform etc. 2e"erity0Criticality: Pri rity: !or bugs of same severity, this field can be used to decide which one's to fix first. Test case name0number0identifier: 2ub?ect: 7ne(line description of the bug Bug Descri#ti n: # detailed description of the bug Re#r ducible ste#s" # step by step description to reproduce the bug Data used: Additi nal inf rmati n: !ile excerpts,error messages,log file excerpts, screen shots that would be helpful in finding the cause of the problem or fix it. Tester name: Tester c ntact details: Bug re# rting date and time: Assigned t : De"el #er t ,hich the bug is assignedC

24

Descri#ti n f #r blem cause: Descri#ti n f fi!: C de secti n0file0m dule0class0meth d that ,as fi!ed: Date f fi!: 4ersi n f the file that c ntains the fi!:

/ist f Bug statuses

9e,: <hen a bug is found, the tester logs the bug and the status of N?ew8 is assigned to the bug. Assigned: The development team verifies if the bug is valid. ,f the bug is valid, development leader assigns it to a developer to fix it and a status of N#ssigned8 is set to it. Additi nal inf rmati n Re)uested: <hen developer$devOlead needs more information from tester to understand the bug 9 t Re#r ducible: <hen dev lead could not reproduce the bug 9 t a Bug: ,nvalid bug4a bug that does not require any code fix5 Du#licate Bug: #lready a bug is logged for the same issue 0eferred" The fix of the bug is postponed to some future release. +i!ed but n t #atched: The bug is resolved but the fix is yet to pushed to testing instance. Ready f r retesting : The fix is pushed to testing instance and ready for retesting by tester Cl sed8fi! "erified: The tester verifies the fix and the bug is resolved completely Cl sed89 t a bug: The tester verifies the bug and finds the bug does not require code fix Cl sed8Du#licate bug: Re #ened: The tester verifies and finds the bug is not fixed4either completely or partially5

/ifecycle f s me ty#es f bugs

4alid bug: ?ew (G #ssigned (G !ixed but not patched (G 1eady for retesting (G losed,fix verified .n"alid bug: ?ew (G ?ot a &ug (G losed,?ot a bug Du#licate bug: ?ew (G 0uplicate &ug (G losed,0uplicate bug Re #ened bug: ?ew (G #ssigned (G !ixed but not patched (G 1eady for retesting (G 1eopened (G !ixed but not patched (G 1eady for retesting (G losed,fix verified

25

Analysis f bugs
&ugs logged during a testing phase a invaluable source to improve the existing testing processes.The holygrail for any testing team is 'ero customer bugs.7nce a product is released, ma-ority of the customer bugs come within Qmonths to ) year of product usage. &ut immediately after a testing of product is over the following can be done. (Testing Team should analy'e all the invalid$duplicate$couldOnotObeOreproduced bugs and come up with measures to reduce their count in future testing efforts. 7nce customer bugs start pouring in the following can be done. (Testing Team should analy'e each and every customer bug,find out why they have missed them in their testing effort and take appropriate measures.

26