SOFTWARE TESTING

Experiment 1
Introduction to Software Testing,
Identifying and Categorizing Software
bugs

CLO 1 : Construct experiments/projects of varying complexities.

CLO 2: Use modern tools and languages.
CLO 3: Demonstrate an original solution of problem under
discussion.
CLO 4: Work individually as well as in teams
Software Testing:
Software Testing is the process of identifying the correctness and quality of software
program. The purpose is to check whether the software satisfies the specific requirements,
needs and expectations of the customer. In other words, testing is executing a system or
application in order to find
• bugs,
• defects or
• errors

Difference between defect, error, bug, failure:

“A mistake in coding is called error, error found by tester is called

defect, defect accepted by development team then it is called bug,
build does not meet the requirements then it is failure.”

1-Examples of Program bugs :

TYPE: Accidental

for (i=0; i<numrows; i++)

for (j=0; j<numcols; j++);

pixels++;

Reason: Caused by a stray ";" on line 2. Accidental bugs are often caused by stray
characters, etc. While "minor" in their fix, they can be the devil to find!

Note: if used correctly, a "prettyprinter" or auto-indenter would help you spot this one.

EXAMPLE 2:

TYPE: Missing or improper initialization

int minval(int *A, int n)
{ int currmin;
for (int i=0; i<n; i++)
if (A[i] < currmin)
currmin = A[i];
return currmin;
}
Reason: Since currmin was never initialized, it could easily start out as the minimum
value. Some compilers spot no-initialization errors. Note that an improper initialization,
while rarer, is even harder to spot than a missing one!

EXAMPLE 3:

TYPE:Dyslexic

int minval(int *A, int n)

{
int currmin = MAXINT;
for (int i=0; i<n; i++)
if (A[i] > currmin)
currmin = A[i];

return currmin;

Reason: Here, the ">" on line 5 should be "<". Even people who are not normally dyslexic
are subject to these types of errors.

EXAMPLE 4:

TYPE: Mis-copy bug

switch (i)
{ case 1:
do_something(1);
break;
case 2:
do_something(2);
break;
case 3:
do_something(1);
break;
case 4:
 do_something(4);
break;
default:
break; }

Reason: The cases were generated by copying case 1. Under case 3, the values were not
changed as appropriate for the case. Code reuse is good -- but this form of code copying has
its dangers!

EXAMPLE 5:

TYPE: Accidental
if (foo = 5)
foo == 7;

Reason: Two bugs in one. These are usually caused by accident rather than
misunderstanding. The "=" of line 1 should probably be "==" (this one will always evaluate
to true), while the "==" of line 2 should almost certainly be "=" (it has no effect). A
syntactic weakness in C/C++, neither of these statements is syntactically wrong. Many
compilers will warn you about both of these.

EXAMPLE 6:

TYPE: Abused global

int i=5;
int j;

int foo(int j)
{
for (i=0; i<j; i++)
do_nothing();
return j;
}

void ineedj(void)
{
cout << "j is " << j << "\n";
}
main()
{
int j;
j =foo(i);
ineedj();
}
Reason: This illustrates some fun with global/local variables. In function foo, j is local
and i is global. Since i is being used as a loop variable, this is almost certainly wrong.
Making j local here may or may not be logically correct, but it is certainly stylistically
incorrect since the semantic meaning of j is being used in two distinct ways (once as a
global, once as a local, which by definition must be inconsistent). In main, j is local. So,
when it gets set by the call to foo, the global value is not being set. So, ineedj is out of
luck -- the value is still undefined.

Note: If the variable is global, never use that name for anything else.

2- Common Categories of Software Errors:

1) Functionality Errors:
Functionality is a way the software is intended to behave. Software has a functionality error
if something that you expect it to do is hard, awkward, confusing, or impossible.
Check this screenshot:
Expected Functionality for Cancel button is that the ‘Create new project’ window should
close and none of the changes should be saved (i.e. no new project must be created). If the
Cancel button is not clickable then it is a functionality error.

2) Communication Errors:
These errors occur in communication from software to end-user. Anything that the end user
needs to know in order to use the software should be made available on screen.
Few examples of communication errors are – No Help instructions/menu provided, features
that are part of the release but are not documented in the help menu, a button named ‘Save’
should not erase a file etc.
3) Missing command errors:

This happens to occur when an expected command is missing. See this screenshot:

This window allows the user to create a new project. However, there is no option for the
user to exit from this window without creating the project. Since ‘Cancel’ option/button is
not provided to the user, this is a missing command error.

#4) Syntactic Error:

Syntactic errors are misspelled words or grammatically incorrect sentences and are very
evident while testing software GUI. Please note that we are NOT referring to syntax
errors in code. The compiler will warn the developer about any syntax errors that occur
in the code Note the misspelled word ‘Cancel’:
Note the grammatically incorrect message:

5) Error handling errors:

Any errors that occur while the user is interacting with the software needs to be handled in a
clear and meaningful manner. If not, it is called as an Error Handling Error.
Take a look at this image. The error message gives no indication of what the error actually
is. Is it missing mandatory field, saving error, page loading error or is it a system error?
Hence, this is an ‘Error Handing Error’.

When possible, further steps should be listed for the user to follow.
If the software has certain mandatory fields that need to be filled before they can save the
information on a form, the validation messages should be clear and indicative of the action
that is required by the user. Here are other examples:
6) Calculation Errors:
These errors occur due to any of the following reasons:
• Bad logic
• Incorrect formulae
• Data type mismatch
• Coding errors
• Function call issues , etc.

In 1999, NASA lost its Mars climate orbiter because one of the subcontractors NASA
employed had used English units instead of the intended metric system, which caused the
orbiter’s thrusters to work incorrectly. Due to this bug, the orbiter crashed almost
immediately when it arrived at Mars.
7) Control flow errors:
The control flow of a software describes what it will do next and on what condition.
For example, consider a system where user has to fill in a form and the options available to
user are: Save, Save and Close, and Cancel. If a user clicks on ‘Save and Close’ button, the
user information in the form should be saved and the form should close. If clicking on the
button does not close the form, then it is a control flow error.
3-Software Defect Taxonomies
In software test design we are primarily concerned with taxonomies of defects, ordered lists
of common defects we expect to encounter in our testing.
Beizer's Taxonomy
One of the first defect taxonomies was defined by Boris Beizer in Software Testing
Techniques. It defines a four-level classification of software defects. The top two levels are
shown here.

1xxx Requirements
11xx Requirements incorrect
12xx Requirements logic
13xx Requirements,
completeness
14xx Verifiability
15xx Presentation, documentation
16xx Requirements changes

2xxx Features and Functionality

21xx Feature/function correctness
22xx Feature completeness
23xx Functional case completeness
24xx Domain bugs
25xx User messages and diagnostics
26xx Exception conditions mishandled

3xxx Structural Bugs

31xx Control flow and sequencing
32xx Processing

4xxx Data
41xx Data definition and structure
42xx Data access and handling

5xxx Implementation and Coding

51xx Coding and typographical
52xx Style and standards violations
53xx Documentation
 6xxx Integration
61xx internal interfaces
62XX External interfaces, timing, throughput

7XXX System and Software Architecture

71XX O/S call and use
72XX Software architecture
73XX Recovery and accountability
74XX Performance
75XX Incorrect diagnostics, exceptions
76XX Partitions, overlays
77XX System, environment

8XXX Test Definition and Execution

81XX Test design bugs
82XX Test execution bugs
83XX Test documentation

4- Bug Categories and Taxonomies

• Why categorises bugs?

–Improve brainstorming
• What bugs might there be in our product?
–Improve process review
• Do we have ways to find all the kinds of bugs we might get?
–Measurement
5- Common Beliefs about Bugs

Programmers (and testers) sometimes have optimistic views about bugs. Here are some of
them:
• Bug locality hypothesis
–Belief: Bug discovered in module A probably only affects A because my program
is well-structured
–Reality: Subtle bugs show up far away (both in space and time) from the cause
 • Good Language Constructs
–Belief: Good programming languages and practices, such as strong typing or
structured coding, prevent most bugs
–Reality: True to an extent, but no statistical evidence in big systems

• Control-flow bug dominance

–Belief: Errors in the program’s flow control are the most frequent type of bugs
–Reality: Data-related errors (data-flow and data-structure) are at least as frequent

• Code/Data separation
–Belief: Bugs tend to respect the separation between code and data
–Reality: Well, if you assume your computer will never execute your data as code
accidentally modify an instruction by incrementing its op code then pray when
executing your programs

• Corrections stay
–Belief: A corrected bug remains corrected
–Reality: First of all, how can you be sure that you have actually corrected a bug in
the first place?

Silver bullets
–Belief: X (a language, methodology, technique, etc) grants immunity against bugs
–Reality: In you trust X blindly, you’re in for a hard time

• Smartness suffices
–Belief: If you are smart you’ll catch most bugs
–Reality: Maybe true for fairly easy bugs. Subtle bugs also need methodology,
techniques and rigour

Lab Tasks
• Make a list of at least 5 examples of specific bugs
• In what software? What was the failure? What fault caused it?
– Use your own experience
– Order them from most to least important (critical, frequent)
• Try to group the bugs (yours and others) into categories
– Try to define broad categories (maybe aim for 8)
– (hint: think of the whole software engineering process)

• Look at Beizer’s taxonomy (handout)

– Compare it with our categories
Most frequent bugs in Beizer’s taxonomy

– Tick the two categories that you think contain the most frequent bugs