IT2032 Software Testing

 Unit – 4 :- Test Management

 People and organizational issues in testing
 Test Goal, Policies, Planning, Management, Execution
and Reporting
 The Test Organization
 Where we are…
 we know how to define what the project is about (scope
planning)
 we know how to assess what the risks are (risk
management) [… and actually explained how to deal with
them during the project]

… today we will look at issues related to selecting the “best people in

town” (here we are between scoping and planning) and issues
related to managing people (analogously to risk management, we
will dig a little deeper and explore topics related to managing people
during project execution and monitoring)
 People in the process
People are an organisation’s most important
assets.
The tasks of a manager are essentially people-
oriented. Unless there is some understanding of
people, management will be unsuccessful.
Poor people management is an important
contributor to project failure.
 Common People Issues
 Perceptions and Misconceptions about testing
Perception 1 :-
Testing is not technically challenging
Perception 2:-
Testing does not provide me a career path or growth
Perception 3:-
I am put in testing – what is wrong with me?
Perception 4:-
These folks are my adversaries.
 Common People Issues
Perception 5:-
Testing is what I can do in the end if I get time
Perception 6:-
There is no sense of ownership in testing
Perception 7:-
Testing is only destructive
 Comparison between Testing &
Development functions
Testing is frequently said to be a crush time
function.
More elastically is given in the projects within the
earlier phase.
Testing functions are arguably the most difficult
ones to staff
Testing functions hold several external
dependencies when compared with development
functions.
 Providing carrier paths for testing
professionals
Technical challenges
Learning opportunities
Increasing responsibility & authority
Increasing independence
Ability to have a significant influences on an
organizations success
Rewards and recognition
 Three Stages to go for an individual
in testing group
Follow Stage
People who starts their job in an organization.
Formulation Stage
Independent decision making, Clarity communication to
assign work for others
Test lead stage
 The individual role acts as a role model.
 Filling high quality defects
Categorizing of matrix related to testing
 Achieve the function
Knowledge:-
What to do in various situations.
Skill:-
 How to do the work
Attitude:-
It pertains to being motivated and wanting to do the right
things.
 The role of ecosystem and a call for
action

 Role of education system

 Role of senior management
 Role of the community
 Organization structures for testing
teams
 Dimensions of organization structures ( 2 dimensions)
 Organization type
Product organization
 Service organization
 Geographic distribution
 Single site ( All members are located at one place )
 Multi site ( Entire teams is spread across many locations )
 Organization Structures for Testing
Developer’s test their own code
Development team responsibility (buddy testing)
Tester(s) in the development team
A dedicated team of testers
Internal test consultants providing advice to
projects
A separate test organization
 Organization Structures for Testing
Independence
Who does the Testing?
Component testing – developers
System testing – independent test team
Acceptance testing – users
Independence is more important during test design
than during test execution
The use of structured test techniques increases
the independence
A good test strategy should mix the use of
developers and independent testers
 Separate testing teams for different
phase of testing
Generally testing is a single & homogenous activity
 Black box testing
 System testing
 Performance testing
 Integration testing
 White box testing
 Testing Services
It is out sourced to geographically distributed
teams.
 It has got significant acceptance in the industry.
 As a process, testing perfectly defined
 Many test automation tolls which are maximize the
challenges in testing
 Typical roles and responsibilities of
testing services organization
 Near site team :-
 It is a small team . This is kept near to the customer site.
 Remote team :-
It is large in size because it has to lot of work.
 Challenges & issues of testing service organization
 Outside effect and estimation of recourse
 Domain expertise
 Privacy and customer isolation issues
 Maintaining a bench
 Test Planning
 preparing the reader to address two fundamental
maturity goals at level 2 of the TMM:
(i) developing organizational goals/ policies relating to
testing and debugging,
(ii) test planning.
These maturity goals are managerial in nature.
They are essential to support testing as a managed
process.
According to R. Thayer, a managed process is one
that is planned, monitored, directed, staffed, and
organized .
Simple examples of the three types of goals
mentioned are shown below.
1. Business goal: to increase market share 10% in
the next 2 years in the area of financial software.
2. Technical goal: to reduce defects by 2% per
year over the next 3 years.
3. Business/technical goal: to reduce hotline calls
by 5% over the next 2 years.
4. Political goal: to increase the number of women
and minorities in high management positions by 15%
in the next 3 years
 Test planning
Test planning requires the planner to articulate the
testing goals for
a given project,
to select tools and techniques needed to achieve the
goals,
to estimate time and resources needed for testing tasks
so that testing is effective,
on time,
within budget,
 consistent with project goals.
 Testing and debugging goals and
policies
A goal can be described as (i) a statement of
intent, or (ii) a statement of a accomplishment that
an individual or an organization wants to achieve.
Goal statements can express expectations in
quantitative terms or be more general in nature.
For the testing goals below, goals 1 and 2 express
what is to be achieved in a more quantitative
manner than goals 3 and 4.
 Testing Goals
1. One-hundred percent of testing activities are
planned.
2. The degree of automation for regression testing is
increased from 50% to 80% over the next 3 years.
3. Testing activities are performed by a dedicated
testing group.
4. Testing group members have at least a bachelor-
level degree and have taken a formal course in
software testing.
 Testing Policy
A policy can be defined as a high-level statement
of principle or course of action that is used to
govern a set of activities in an organization .
Because a policy provides the vision and
framework for decision making, it is important to
have the policy formally adopted by the
organization, documented, and available for all
interested parties.
 Testing Policy cont.
A policy statement should be formulated by a team
or task force consisting of upper management,
executive personnel, and technical staff.
Testing policy statements reflect, integrate, and
support achievement of testing goals.
These goals in turn often target increasing
software quality and improving customer
satisfaction.
 Test Planning
A plan is a document that provides a framework or
approach for achieving a set of goals.
In the software domain, plans can be strictly
business oriented, for example, long-term plans to
support the economic growth of an organization, or
they can be more technical in nature, for example,
a plan to develop a specific software product.
Test planning is an essential practice for any
organization that wishes to develop a test
process that is repeatable and manageable.
Pursuing the maturity goals embedded in the
TMM structure is not a necessary precondition
for initiating a test-planning process.
Milestones are tangible events that are expected
to occur at a certain time in the project’s lifetime.
Managers use them to determine project status.
The planner usually includes the following essential
high-level items.
1. Overall test objectives. As testers, why are we
testing, what is to be achieved by the tests, and what
are the risks associated with testing this product?
2. What to test (scope of the tests). What items,
features, procedures, functions, objects, clusters, and
subsystems will be tested?
3. Who will test. Who are the personnel responsible for
the tests?
4. When to test. What are the schedules for tests?
What items need to be available?
5. How to test. What strategies, methods, hardware,
software tools, and techniques are going to be
applied? What test documents and deliverable should
be produced?
6. When to stop testing. It is not economically feasible
or practical to plan to test until all defects have been
revealed. This is a goal that testers can never be sure
they have reached. Because of budgets, scheduling,
and customer deadlines, specific conditions must be
outlined in the test plan that allow testers/managers to
decide when testing is considered to be complete.
 A hierarchy of test plans
Software Quality Assurance(V&V) Plan

Master test plan Review plan, Inspections

and walkthroughs

Unit test Integration System test Acceptance

plan test plan plan test plan
 Test plan components
1 .Test Plan Identifier:- Each test plan should have a
unique identifier so that it can be associated with a
specific project and become a part of the project history.
2 .Introduction:- Test planner gives an overall
description of the project, the software system being
developed or maintained, and the soft ware items and/or
features to be tested.
3 .Items to Be Tested:- This is a listing of the entities to
be tested and should include names, identifiers, and
version/revision numbers for each entity. The items
listed could include procedures, classes, modules,
libraries, subsystems, and systems.
 Test plan components cont.
4 . Features to Be Tested:- Features may be described
as distinguishing characteristics of a software
component or system.
5 .Approach :- The planner should also include for each
feature or combination of features, the approach that
will be taken to ensure that each is adequately tested.
Tools and techniques necessary for the tests should be
included. 6 .Item
Pass/Fail Criteria :- the tester must have a set of
criteria to decide on whether the test has been passed
or failed upon execution. The master test plan should
provide a general description of these criteria.
 Test plan components cont.
7 .Suspension and Resumption Criteria:-Test plan,
criteria to suspend and resume testing are described.
In the simplest of cases testing is suspended at the end
of a working day and resumed the following morning.
8 .Test Deliverables:- Execution-based testing has a
set of deliverables that includes the test plan along with
its associated test design specifications, test
procedures, and test cases.
9 .Testing Tasks:- A Work Breakdown Structure is a
hierarchical or treelike representation of all the tasks
that are required to complete a project.
 Test plan components cont.
10.The Testing Environment:- Here the test planner
describes the software and hardware needs for the
testing effort.
11. Responsibilities:- The staff who will be responsible
for test-related tasks should be identified. This includes
personnel who will be:
• transmitting the software-under-test;
• developing test design specifications, and test
cases;
• executing the tests and recording results;
• tracking and monitoring the test efforts;
 Test plan components cont.
• checking results;
• interacting with developers;
• managing and providing equipment;
• developing the test harnesses;
• interacting with the users/customers.
This group may include developers, testers,
software quality assurance staff, systems analysts, and
customers/users
 Test plan components cont.
12.Staffing and Training Needs:-The test planner should
describe the staff and the skill levels needed to carry out
test-related responsibilities such as those listed in the
section above. 13.
Scheduling :-Task durations should be established and
recorded with the aid of a task networking tool. 14.
Risks and Contingencies:- These risks should be: (i)
identified,
(ii) evaluated in terms of their probability of occurrence,
(iii) prioritized,
(iv) contingency plans should be developed that can be
activated if the risk occurs.
15. Testing Costs:-Test costs that should included in
the plan are:
• costs of planning and designing the tests;
• costs of acquiring the hardware and software
necessary for the tests (includes development of the
test harnesses);
• costs to support the test environment;
• costs of executing the tests;
• costs of recording and analyzing test results;
• tear-down costs to restore the environment.
 Testing cost
The nature of the organization; its testing maturity
level, and general maturity.
The nature of the software product being developed.
The scope of the test requirements.
The level of tester ability.
Knowledge of the project problem domain.
The level of tool support.
Training requirements
 Some approaches to test cost
estimation
Test impact items Test tasks test
Test cost drivers WBS

Delphi method
Module and
Expert judgment
heuristics Test cost
estimates
Developer/tester
ratios

Historical
project/ test
database
 Test plan attachments
Test plan were principally managerial in nature:
tasks, schedules, risks, and so on.
The reader may be puzzled as to where in the test
plan are the details needed for organizing and
executing the tests.
For example, what are the required inputs,
outputs, and procedural steps for each test; where
will the tests be stored for each item or feature;
 Example of entries in a requirements
traceability matrix
Requirement Requirement Priority Review Test ID
identifier description (Scale 1 - Status
10)
SR – 25 – Displays 8 Yes TC – 25 – 2
13.5 opening TC – 25 – 2
screens
SR – 25 – Checks the 9 Yes TC – 25 – 18
52.2 validity of TC – 25 – 23
user
password
 Test Design Specifications
To develop test design specifications many documents
such as the requirements, design documents, and user
manual are useful.
Test Design Specification Identifier
Features to Be Tested
Approach Refinements
Test Case Identification
Pass/Fail Criteria
 Test Case Specifications

Test Case Specification Identifier

Test Items
Input Specifications
Output Specifications
Special Environmental Needs
Special Procedural Requirements
Intercase Dependencies
 Test Procedure Specifications
A procedure in general is a sequence of steps
required to carry out a specific task.
Test Procedure Specification Identifier
Each test procedure specification should be assigned a
unique identifier.
Purpose
Describe the purpose of this test procedure and
reference any test cases it executes.
Specific Requirements
List any special requirements for this procedure, like
software, hardware, and special training.
Procedure Steps
setup: to prepare for execution of the procedure;
start: to begin execution of the procedure;
proceed: to continue the execution of the procedure;
measure: to describe how test measurements related to
outputs will be made;
 shut down: to describe actions needed to suspend the
test when unexpected events occur;
restart: to describe restart points and actions needed to
restart the procedure from these points;
stop: to describe actions needed to bring the procedure
to an orderly halt;
wrap up: to describe actions necessary to restore the
environment;
 Locating test items
Suppose a tester is ready to run tests on an item
on the date described in the test plan.
(i) version/revision number of the item;
(ii) location of the item;
(iii) persons responsible for the item (e.g., the
developer);
(iv) references to item documentation and the test plan
it is related to;
(v) status of the item;
(vi) approvals—space for signatures of staff who
approve the transmittal
 Reporting test results
The test plan and its attachments are test-
related documents that are prepared prior to
test execution.
There are additional documents related to
testing that are prepared during and after
execution of the tests.
 Test Log
The test log should be prepared by the person
executing the tests. It is a diary of the events that
take place during the test.
The test log is valuable for
Regression testing that takes place in the development of
future releases of a software product,
Circumstances where code from a reuse library is to be
reused.
In all these cases it is important that the exact conditions
of a test run are clearly documented so that it can be
repeated with accuracy.
Test Log Identifier
Each test log should have a unique identifier.
Description
In the description section the tester should identify
the items being tested, their version/revision
number, and their associated Test Item/Transmittal
Report. The environment in which the test is
conducted should be described including hardware
and operating system details.
Activity and Event Entries
The tester should provide dates and names of test
log authors for each event and activity. This section
should also contain:
1. Execution description: Provide a test procedure identifier
and also the names and functions of personnel involved in
the test.
2. Procedure results: For each execution, record the results
and the location of the output. Also report pass/fail status.

3. Environmental information: Provide any environmental

conditions specific to this test.
4. Anomalous events: Any events occurring before/after an
unexpected event should be recorded. If a tester is unable
to start or compete a test procedure, details relating to
these happenings should be recorded (e.g., a power
failure or operating system crash).

5. Incident report identifiers: Record the identifiers of incident

 Test Incident Report
1. Test Incident Report identifier: to uniquely identify this
report.
2. Summary: to identify the test items involved, the test
procedures, test cases, and test log associated with this
report.
3. Incident description: this should describe time and date,
testers, observers, environment, inputs, expected outputs,
actual outputs, anomalies, procedure step, environment, and
attempts to repeat. Any other information useful for the
developers who will repair the code should be included.
4. Impact: what impact will this incident have on the testing
effort, the test plans, the test procedures, and the test cases?
A severity rating should be inserted here.
 Test Summary Report
This report is prepared when testing is complete.
1. Test Summary Report identifier: to uniquely
identify this report.
2. Variances: these are descriptions of any
variances of the test items from their original design.
Deviations and reasons for the deviation from the test
plan, test procedures, and test designs are discussed.
3.Summary of results: the document author
summarizes the testing results. All resolved incidents
and their solutions should be described. Unresolved
incidents should be recorded.
 4.Comprehensiveness assessment: the document author
discusses the comprehensiveness of the test effort as
compared to test objectives and test completeness criteria as
described in the test plan. Any features or combination of
features that were not as fully tested as was planned should
be identified.
5.Evaluation: in this section the author evaluates each test
item based on test results. Did it pass/fail the tests? If it failed,
what was the level of severity of the failure?
6. Summary of activities: all testing activities and events
are summarized
7. Approvals: the names of all persons who are needed to
approve this document are listed with space for signatures and
dates.
 Test summary Report
Test Planning

Software Project Master Detailed test

Management Plan Test Plan plans for level

Test Case
Specification

Test Design
Specification
Test Procedure
Specification
 Test Case
Specification
Execution Test item
transmittal plan
Test Procedure
Specification
Test Logs Test Incident
Reports
Completion

Test Summary Report

 The role of three groups in test
planning and policy development
Recall that in the TMM framework three groups
were identified as critical players in the testing
process.
They all work together toward the evolution of a
quality testing process.
These groups were managers, developers/ testers,
and users/clients.
The manager‘s view involves commitment and
support for those activities and tasks related to
improving testing process quality.
The developer/tester‘s view encompasses the
technical activities and tasks that when applied,
constitute best testing practices.
The user/client view is defined as a cooperating or
supporting view.
The developers/testers work with client/user
groups on quality-related activities and tasks that
concern user-oriented needs.
 Process and the engineering
disciplines
Computer science students are now being
introduced to the fundamentals of software
engineering.
As the field matures, they will be able to obtain a
degree and be certified in the area of software
engineering As members of this emerging profession
we must realize that one of our major focuses as
engineers is on designing, implementing, managing,
and improving the processes related to software
development.
Testing is such a process.
You can serve as the change agent, using your
education in the area of testing to form a process
group or to join an existing one.
You can initiate the implementation of a defined
testing process by working with management and
users/clients toward achievement of the technical and
managerial-oriented maturity goals at TMM level 2.
Minimally you can set an example on a personal level
by planning your own testing activities. If the project
manager receives effective personal test plans from
each developer or test specialist, then the quality of
the overall test plan will be improved.
Finally, you can become proficient in, and
consistently apply, black and white box testing
techniques, and promote testing at the unit,
integration, and system levels.
You need to demonstrate the positive impact of these
practices on software quality, encourage their
adaptation in the organization, and mentor your
colleagues, helping them to appreciate, master, and
apply these practices.
 Introducing the test specialist
maintenance and application of test policies;
development and application of test-related
standards;
participating in requirements, design, and code
reviews;
test planning;
test design;
test execution;
mentoring and training of new test personnel;
 Introducing the test specialist
test measurement;
test monitoring (tasks, schedules, and costs);
defect tracking, and maintaining the defect
repository;
acquisition of test tools and equipment;
identifying and applying new testing techniques,
tools, and methodologies;
test reporting.
 Specialist Skills Needed in Testing
 Test managers (test management, reporting, risk analysis)
 Test analyst (test case design)
 Test automation experts (programming test scripts)
 Test performance experts (creating test models)
 Database administrator or designer (preparing test data)
 User interface experts (usability testing))
 Test environment manager
 Test methodology experts
 Test tool experts
 Domain experts
 The technical level testers need to
have:
 an education that includes an understanding of general
software engineering principles, practices, and
methodologies;
 strong coding skills and an understanding of code structure
and behavior;
 a good understanding of testing principles and practices;
 a good understanding of basic testing strategies, methods,
and techniques;
 the ability and experience to plan, design, and execute test
cases and test procedures on multiple levels (unit, integration,
etc.);
 a knowledge of process issues;
 Building a testing group
It was mentioned that organizing, staffing, and
directing were major activities required to manage a
project and a process. These apply to managing the
testing process as well. Staffing activities include
filling positions, assimilating new personnel,
education and training, and staff evaluation.
 They describe four categories for employees based
on their performance:
(i) retain, (ii) likely to retain, (iii) likely to release, (iv) and
release.
 Steps in forming a test group

Upper management support for Title salary

testing group location
Qualification
Establish test group organization,
Duties
career path
Define education and skills levels
Develop job description
Interview Candidates

Select test group members

 Structure of test group
It is important for a software organization to have
an independent testing group. The group should
have a formalized position in the organizational
hierarchy. A reporting structure should be
established and resources allocated to the group.
A test organization is expensive, it is a strategic
commitment. Given the complex nature of the
software being built, and its impact on society,
organizations must realize that a test organization
is necessary and that it has many benefits.
 The Test Manager
The test manager is usually responsible for test
policy making, customer interaction, test planning,
test documentation, controlling and monitoring of
tests, training, test tool acquisition, participation in
inspections and walkthroughs, reviewing test work,
the test repository, and staffing issues such as
hiring, firing, and evaluation of the test team
members.
 The Test Lead
He or she may be responsible
for duties such as test
planning, staff supervision, and
status reporting. The test lead
also participates in test design,
test execution and reporting,
technical reviews, customer
interaction, and tool training.
 The Test Engineer & The Junior Test
Engineer
The Test Engineer
The test engineers design, develop, and execute tests,
develop test harnesses, and set up test laboratories and
environments. They also give input to test planning and
support maintenance of the test and defect repositories.
The Junior Test Engineer
The junior test engineers are usually new hires. They
gain experience by participating in test design, test
execution, and test harness development. They may also
be asked to review user manuals and user help facilities
defect and maintain the test and defect repositories.