Chapter-04-Empirical Investigation
Chapter-04-Empirical Investigation
Empirical Investigation
Empirical theory
SE Investigation:
Examples
the code?
Experiment to initiate novel practices
Would it be better to start OO design with UML?
Would the use of SRE improve software quality?
SE Investigation: Why?
To improve (process and/or product)
To evaluate (process and/or product)
To prove a theory or hypothesis
To disprove a theory or hypothesis
To understand (a scenario, a situation)
To compare (entities, properties, etc.)
For better processes, tools, and languages to gather and
analyze requirements, model designs, develop, test, and
evolve applications.
SE Investigation:
What?
developers performance
Tool’s performance
Person’s perceptions
Tool’s usability
Document’s understandability
Data evaluation
Data interpretation
preferred technique.
If replication is not possible at higher levels, then experiment is
not possible.
If the cost of replication is low, then we can consider experiment.
2. Stating Goal & hypothesis:
The first step is deciding what to investigate.
The goal for the research can be expressed as a hypothesis in
quantifiable terms that is to be tested.
The test result (the collected data) will confirm or refute the
hypothesis.
The goal for your research can be expressed as a hypothesis that
you want to test.
The hypothesis is the tentative idea that you think explains the
behavior you want to explore.
Stating Goal & hypothesis(cont.)
Examples:
Eg1. Can integrated development and testing tools
improve our productivity?
Eg2. Does Cleanroom software development produce
better-quality software than using the conventional
development methods?
Eg3. Does code produced using Agile software
development have a lower number of defects per KLOC
than code produced using the conventional methods?
Eg4.“Using Scrum produces better quality software than
using the Extreme Programming method.”
Stating Goal & hypothesis(cont.)
The method used is to first evaluate a null hypothesis, which
states that the proposed relationship does not hold.
variables)
Example: Effect of “programming language” on the “quality”
of resulting code.
Programming language is an independent and
quality is a dependent variable.
3. Maintaining Control over Variables(cont.)
There are many ways that a study can provide misleading results.
Potential problems with empirical studies are classified as
categories of threats to validity.
There are four categories of threats to validity:
i.Conclusion validity- Using the wrong statistical tests, having too
small a sample, Searching for relationships between too many
variables
ii.Construct validity- use meaningful measures that have
been validated in the narrow sense
4. Threats to Validity(cont.)
5. Analysis
The analysis phase has two parts
i. Review all the measurements taken to make sure that they are
valid and useful.
ii. Analyze the sets of data according to the statistical principles
6. Dissemination & decision making
i. Documenting conclusions to duplicate experiments & confirm
conclusions in a similar setting.
ii. To support decisions how to develop or maintain software in the
future
Key Experimental Design Concepts
Simple designs - make the experiment practical, minimize time,
money, personnel, easier to analyze & experimental resources.
The three key experimental design concepts: Replication,
Randomization, & Local control.
Replication:
Involves repeating an experiment under identical conditions,
rather than repeating measurements on the same experimental
unit.
⁃ To know how much confidence we can place in the results of the experiment.
⁃ Replication enables us to estimate the mean effect of any experimental factor.
Key Experimental Design
Concepts(cont.)
II. Randomization:
The random assignment of subjects to groups or of
treatments to experimental units, so that we can assume
independence (conclusion & internal validity) of results.
III. Local control:
II. Local control refers to the control that you have over the
placement of subjects in experimental units and the
organization of those units.
Blocking means allocating experimental units to groups so that the units
within a block are relatively homogeneous.
Balancing is assignment of treatments – an equal number of subjects is
assigned to each treatment.
Types of Experimental Designs
An independent variable is called a factor in the experimental
design
Various values or classifications for each factor are called the
levels of the factor. Levels can be continuous or discrete,
quantitative or qualitative.
Eg. A study to determine the effect of experience and language
on the productivity of programmers has two factors: experience
and language. The dependent variable is productivity.
Most designs in software engineering research are based on
two simple relations between factors: Crossing and Nesting
Types of Experimental
Designs(cont.)
1. Crossing:
2. Expressing the design in terms of factors, called the
factorial design
3. How many different treatment combinations are required.
4. Two factors, A and B, in a design are crossed if each level of each
factor appears with each level of the other factor. This relationship is
denoted as A × B
Types of Experimental
Designs(cont.)
2. Nesting:
Factor B is nested within factor A if each meaningful level of
B occurs in conjunction with only one level of factor A.
The relationship is depicted as B(A), where B is the nested
factor and A is the nest factor.
Empirical Research Guidelines
Contents
1. Experimental context
2. Experimental design
3. Data collection
4. Analysis
5. Presentation of results
6. Interpretation of results
1. Experimental Context
Goals:
Ensure that the objectives of the experiment
defined
Monitor the data collection and watch for