SPC 2308 Simulation &

Modeling
Chapter 8:
Measurement and statistics

1
 Measurement and Statistics
Purpose:

This section is about the methodology of measurement. What goes into designing an
experiment, gathering some numbers, interpreting the results, and presenting those
results to management in a way that allows them to make the necessary decisions.

Warm-up Experiment:

Divide into teams and measure the length of an object in the classroom. To do so you
will need to make team decisions about tools, techniques, and reporting metrics.

Upon completion, discuss what can be learned from this experiment.

2
 Measurement and Statistics
FUNDAMENTAL QUESTIONS ABOUT MEASUREMENT:

i.What kind of accuracy can you expect from a computer (or any other)
measurement?

ii.When you make a measurement, can you believe the result? How sure are you
of the result?

iii.How should you state the result of an experiment? How do you reflect your
belief in its accuracy?

iv.Can one number represent the performance of a product?

v.When have you measured enough?

vi.How do you extrapolate from what you know to what you'd like to know?

3
 Measurement and Statistics
FUNDAMENTAL QUESTIONS ABOUT MEASUREMENT:

i.How do you know what tools to use?

ii.Is everything in a computer measurable?

iii.How do you know what to measure?

iv.Should you always know the result of a measurement before you make it?

v.How do you figure out dependencies; how does one variable depend on
another?

vi.So after all this talk about the details of measurement, how do you actually
design an experiment?

4
 Measurement and Statistics
1. What kind of accuracy can you expect from a computer (or any
other) measurement?

Associated Questions are:

• What are some sources of uncertainty when measuring a

computer and its software.
• Is a computer deterministic? (What is the meaning of
deterministic? Do a detour on predictable, deterministic,
stochastic and chaotic.)
• What are the pros and cons of taking all the variation out of an
environment. Repeatability vs. believability.

5
Here are some factors that lead to experimental variation:

i. System/Component/Molecule/Atom – how granular is the

measurement.
ii. Background Activity
iii. End effects and incomplete cycle effects. Measurement error.
iv. Randomness doesn't mean equality (stochastic process).
v. Randomness from resource contention ( stochastic process
vi. Changing hardware.
vii. Tool granularity

6
 Measurement and Statistics
1. What kind of accuracy can you expect from a computer (or any other)
measurement?

Example: You write a program that measures time in seconds. What

percentage accuracy can you get from your experiment.

Example: You want to measure the time required to execute a routine and
have available a system call named get_time_of_day. get_time_of_day
returns time in units of 1/65535 seconds = 16 microseconds. The time
required to execute the get_time_of_day routine itself is 100
microseconds. What is the shortest routine that can be measured with
this tool? How would you do it?

Bottom Line: Never believe a real system number to better than 5 - 10%.
Artificial numbers can sometimes be repeated to 1 - 2%, but are
susceptible to spurious factors.

7
 Measurement and Statistics
2. When you make a measurement, can you believe the result? How
sure are you of the result? ?

Suppose you make several determinations of some measure. If you can answer yes to
the following questions, then you can have some faith in your measurement:

• Can you explain why the numbers vary? (“Handwaving” isn't allowed here, but “statistics”
may be a valid answer.)

• If variations are greater than 10%, can you figure out what's causing the variation and could
you eliminate it if time allowed?

• If the granularity of your tool is greater than the measurement variations, is that acceptable?
(Your granularity then becomes your uncertainty.)

But How Much Do You Trust It?

To answer this we need a brief digression into some math.

Suppose we've taken a number of measurements

m1 , m2 ,...mn 8
 Measurement and Statistics
2. When you make a measurement, can you believe the result? How
sure are you of the result? ?

Then the mean and standard deviation are:

M E m 1 / n i 1 mi

n

 ( M  mi ) 2

SD  E[m 2 ]  E[m]2 0.5  i

{N  1}
s =  = variance = SD2

The first form of the Standard Deviation is the form of the underlying data. The second form is that of
the measured data. They are the same for an infinite amount of data and close enough for a large set
of numbers.

NOTE: Use of these equations assumes that the measurements are independent of each other.

9
Measurement and Statistics
2. When you make a measurement, can you believe the result? How sure
are you of the result?

COMPARING TWO SETS OF MEASUREMENTS:

You’ve just measured the Performance of the latest release of your product. The numbers are better
than they were when you measured them on the last release. But what does “better” mean.
How do you show that two sets of numbers, with lots of uncertainty in each of the sets, really
have one set better than the other.

First of all, here’s the easy way. With your two sets, calculate their means and their confidence
intervals (the % confidence you use is up to you.) Visually plot these results as show in the
three examples below:

A B C

A. Here the confidence The results are such that The confidence intervals
intervals don’t overlap. The the mean of one set is overlap but the means are not
results are different from each within the confidence inside the CI of the other set.
other. interval of the other set. Need to do a more complex
The two sets are NOT
different.
test. 10
 Measurement and Statistics
3. How should you state the result of an experiment? How do
you reflect your belief in its accuracy?
Pat has developed a new product, "rabbit" about which she wishes to determine performance.
There is special interest in comparing the new product, rabbit to the old product, turtle, since
the product was rewritten for performance reasons. (Pat had used Performance Engineering
techniques and thus knew that rabbit was "about twice as fast" as turtle.) The measurements
showed:

Performance Comparisons

Product Transactions / second Seconds/ transaction Seconds to process transaction

Turtle 30 0.0333 3
Rabbit 60 0.0166 1

Which of the following statements reflect the performance comparison of rabbit and turtle?

o Rabbit is 100% faster than turtle. o Rabbit takes 200% less time than turtle.
o Rabbit is twice as fast as turtle. o Turtle is 50% as fast as rabbit.
o Rabbit takes 1/2 as long as turtle. o Turtle is 50% slower than rabbit.
o Rabbit takes 1/3 as long as turtle. o Turtle takes 200% longer than rabbit.
o Rabbit takes 100% less time than turtle. o Turtle takes 300% longer than rabbit.
11
 Measurement and Statistics
3. How should you state the result of an experiment? How do
you reflect your belief in its accuracy?
• The guiding principle in stating a result is to keep it simple.

• State the accuracy using the same methods we've just discussed. Use Means, Standard
Deviations, and Confidence Intervals.

• It goes without saying that reflecting your belief in the accuracy presupposes you’ve done the
experiment correctly. Some simple guidelines:

A. You always do the experiment wrong the first five times. Through experience you learn to look critically
at your result to see if it makes sense. If not, then you go figure out what went wrong. Usually it’s some
parameter that wasn’t controlled.

B. Only vary one parameter at a time.

C. Watch out for interactions between parameters. The result of changing one parameter results in some
other parameter changing as well.

D. Don’t do too many or too few experiments.

E. Get someone else to check your results – by the time you finish a measurement you have too much
invested in it and are very likely to miss something obvious.

12
 Measurement and Statistics
4. Can one number represent the performance of a product?

Answer: No, but you'll be asked to do it anyway.

Recall:
n
Mean or Expected Value: Mean  E ( x)  pi xi xf ( x)dx
i 1

Median That value for which there’s an equal probability of being above it and below it.
Mode The most likely value. The value with the highest probability.

Mode

Median

Mean

13
 Measurement and Statistics
5. When have you measured enough?
This is really two questions:

a) When have you measured enough to get the accuracy of answer that
management expects at this time?

This is a matter of setting the correct expectations before you start. Many times
the answer is in response to a “what if” question – you can get the
appropriate accuracy in one hour. Other times you’ll need weeks of
design/setup/measurement/analysis to get the expected accuracy.

NOTE: Only a small amount of the total experimental time is in the measurement.
Most time goes for design and elimination of unwanted factors. So this
question could be stated as “How complicated should an experiment be?”

b) When have you measured enough to get the degree of accuracy you
expected for the experiment?

You can use the confidence measures. In essence, confidence is

1
Confidence 
n 14
 Measurement and Statistics
5. When have you measured enough?

The relationship between the number of required samples and experimental parameters is:

 100 zs 
n  
 rxmean 

Here n = number of samples required

z = the number of deviations of the desired confidence
s = Standard Deviation
r = The desired accuracy in percent.
xmean = The mean of the measurement

NOTE: See that the more accuracy you want (s), the more measurements you need.
NOTE: If your numbers all come out the same, stop. Measurement uncertainty is not the largest part
of the error in your metric.

15
 Measurement and Statistics
6. How do you extrapolate from what you know to what you'd like to
know?
Often we need a result that is unmeasurable, or would require eons to determine. Is it legal to guess?

Answer:
Sure - as long as you also estimate the uncertainty of your guess.

Here are a few practice situations that will help you improve your powers of estimation. Remember,
there is no RIGHT answer.

1. Estimate how many people will come to this class next week. More important than the answer is
the assumptions you use for your answer.

2. Approximately how many cars were in the parking lot outside this building when you came in
tonight? How many are there now?

3. What is the probability that you will be killed in a car accident?

4. I recently saw a lawn service truck that had printed on its side “Over 7 trillion blades cut.” Is this
a reasonable claim for them to make?

16
 Measurement and Statistics
7. How do you know what tools to use?

• We'll do a lot more on tools later, but for right now, the best answer
is to measure the simplest way possible.

• Usually tools are easier to come by than environments.

• Make sure the tool is less granular than the required uncertainty.

17
 Measurement and Statistics
8. Is everything in a computer measurable? No

• Some electrical signals may not be available.

• The place to make a measurement is in code not under your control.

• We have a very poor sense of typical/normal. We don't know what

our users typically do with the machine.

• The measurement may perturb the system and destroy what we

wanted to know.

• Available measurements may not relate to what I want to know. For

instance, which disk blocks are being accessed by each of the
processes on a system.

18
 Measurement and Statistics
9. How do you know what to measure?

• This is the hardest question of all. To know what to measure you must have a picture or
model of your product. Most of the rest of this course will deal with various kinds of
pictures.

• Often an adequate model is a causal one: first procedure A executes; this causes
hardware B to produce an effect; then interrupt code handles the hardware result; etc.

• Things to keep in mind include:

• Interaction between variables – do you expect a change in X to produce a change in

Y? You should have a guess as to the result before you make the measurement.
• Changing one variable at a time, and measuring it at 10 different values, can be
extremely wasteful and time consuming.
• Change only the variables that matter. If you don’t know, try changing something,
just once, and see what happens

• Example: You wish to design an experiment that will measure the time required to
execute a program on various Intel processors.. What parameters would you need to vary
to try different processors and configurations? DESIGN THE TESTS TO BE RUN.

19
 Measurement and Statistics
10. Should you always know the results of a measurement before you
make it?

• You should always have a guess so you can tell if your

result is way off.
• That guess should be the result of a model/theory of how
the mechanism you are measuring is working

20
 Measurement and Statistics
11. How do you figure out dependencies; how does one variable
depend on another ?

This whole topic is something called linear regression. It says that if you can plot two
variables, x and y, and there’s a simple relationship between the variables, then you can
define the dependency between them.

Good SIMPLE Model. Good Complicated Model BAD Model

A linear regression means that we can fit a curve of the form y = a + bx. The quality of the
fit (error) can be defined as the sum of the y distances between the fitting-curve and the
experimental data.

21
 Measurement and Statistics
12. So after all this talk about the details of measurement, how do you actually
design an experiment?

1. State Goals and Define The System

a. What is it you hope to accomplish? Why is it worth doing?
b. What is the hardware and software (the system) that you will use to achieve these
goals?

2. List Services and Outcomes

a. For the system you’ve chosen, what are the services provided. For instance, if you’re
studying a disk subsystem, it can absorb data (write) or present you with data (read) or
give an error.
b. By outcomes here are meant very high level statements. The outcome of a disk read is
DATA. It’s not a performance or quantifiable answer expected here.

3. Select Metrics
a. What are the criteria you want to use to compare performance? This is still not a
quantifiable value, but simply what it is you will measure. This could be a speed metric, or
an accuracy metric.

22
 Measurement and Statistics
4. List Parameters
a. What parameters affect performance? If you’re measuring disks, then the model of disk
determines it’s seek time, it’s rotational latency, etc. This is a system parameter.
b. The kind of test you use, determined by the workload you use, can also define parameters.
These might be requested IO’s per second, random or sequential blocks, etc.

5. Select Factors to Study

a. A factor is a parameter that you vary.
b. So, for the parameters you’ve just listed – all of which you COULD vary, which ones will
you actually modify during the course of the experiment?

6. Select Evaluation Technique

a. You could do this experiment by modeling. You would mathematically represent the
system under study and modify parameters in this model.
b. You could do this experiment by simulation. You would write a program that represented
the system. Again you could modify parameters and look at results.
c. You could do this experiment by measurement. Here you have a real system, drive it with
some kind of workload, and get the results.
d. In practice, in industry, only measurements are valued. It’s generally cheaper to use the
real system than it is to build a mathematical or simulated system.
23
 Measurement and Statistics
7. Select Workload
a. How will you drive the system under test?
b. It depends on the Evaluation Technique. With a simulation you may
have collected some data that you can feed into your program.
c. For a measurement evaluation, you will have some kind of software
that drives the system you’re testing. You will need to find a workload
that tickles the parameter of interest to you.

8. Design Experiments
a. What experiments will you do to collect the data you want?
b. This means selecting the actual values to be used as factors. If one
of your factors is the type/model of disk, then how many different
disks will you use?

24
 Measurement and Statistics
9. Make A Guess What The Result Will Be
a. Many people take a measurement and say “Oh, that must be right.” The best way to be able
to make that statement is to have understood what should happen and then either get what
you expected or not.
b. If you get what’s expected, then you can be confident that:
 You understand a picture of how the system is working.
 You did your measurements correctly.
c. If you DON’T get what’s expected, then you can be confident that:
 You didn’t understand the system and so you need to form a new picture.
 You did the measurement wrong – there’s some experimental error.

10. Conduct the Measurement, Analyze and Interpret Data

a. Now actually do the measurement, simulation, or whatever you’ve designed.
b. It’s rare that you just get a number and you’re all done.
c. There is always interpretation to be done:
 What does the data mean?
 Is this the result I would expect?
d. There are always statistics to be done:
 Is the data valid?
 What is the uncertainty in the measurements? 25
Measurement and Statistics
11. Figure Out What You Want To Talk About
a. Know your audience. Are they management types (who
want only an overview) or are they technical people (who
want all the details.) Proper targeting is important!
b. Choose from all the data you have, those pieces that are
most relevant. Don’t forget to make it interesting!

12. Present Final Results

c. As you know, in the real world, it’s not what you do, it’s
what others think you do.
d. Presentation is everything.

26