S TAT IS T IC S SUR V I VA L SERIES

Essential Dos and

Don’ts of Statistics

STATIST
ICS 101
CHAPTER 1
The Importance of Statistics in Science. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

CHAPTER 2
Don’t Start Without A Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

CHAPTER 3
Don’t be a P-Hacker. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

CHAPTER 4
Don’t Add Subjects Until You Hit Significance.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

CHAPTER 5
Statistical Analysis with GraphPad Prism. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Essential Dos and Don’ts of Statistics 2

CHAPTER 1

The Importance As a scientist, research is the

foundation of your work. You

of Statistics in begin with insightful questions,

develop hypotheses, and then

Science collect and analyze data from

which you make your conclusions.
This allows you to solve meaning-
ful problems in the world.
This research process, however, isn’t always that
simple. No matter how carefully you perform your
experiments, research always contains uncer-
tainty and error. That is where statistics comes
in. Statistics help you quantify uncertainty in your
research. This provides credibility to your process
No matter how carefully you
and conclusions. That’s why it’s essential in every
perform your experiments, area of science.
research always contains However, being a scientist doesn’t make you a
uncertainty and error. That statistician. The truth is many scientists have
is where statistics comes only taken introductory stats courses. As a result,
analyzing data from even relatively simple exper-
iments can be overwhelming. Even worse, when
done incorrectly, your interpretations of the
results may be invalid.

The following guide is designed to help. It outlines

ways you may be inadvertently—and negatively—
impacting your research. From skipping important
questions as part of your experimental design, to
influencing the data collection process, review
these essential dos and dont’s and improve the
accuracy of your research.

Essential Dos and Don’ts of Statistics 3

 CHAPTER 2

Don’t Start Without a Plan

One of the most common mistakes people

make in research is collecting a bunch of
data without having thought through what
questions they are trying to answer, what
specific hypotheses they want to test, and
what statistical tests they can use to test
these hypotheses.
Analyzing data requires many important decisions: Parametric
or nonparametric test? Eliminate outliers or not? Transform
the data first? Normalize to external control values? Adjust for
covariates? Use weighting factors in regression?

All these decisions (and more) should be part of your experi-

mental design.

When decisions about statistical analysis are made after

inspecting the data, it is easy for statistical analysis to become
a high-tech Ouija board—a method to produce results you hoped
to see, as opposed to an objective method of analyzing data. The
new name for this is p-hacking, and we will get into that in much
more detail in the next chapter. DO
However, having a plan does more than just help you avoid making Before you start data collection, have
critical mistakes in the analysis it can also save you time. Imagine an experimental design planned out.
after weeks of data collection, you examine your data and realize This includes how you plan to collect
you should have added another variable or measured something data, parameters to your data collec-
differently. Changing the way the data is analyzed after you’ve tion, as well as an idea of how you plan
seen the results of your work can and will impact the validity of the on analyzing the data.
statistical results. Many researchers have fallen into this trap.

Essential Dos and Don’ts of Statistics 4

 CHAPTER 3

Don’t Be a P-Hacker

The results from data collected this way cannot be

We must address p-hacking.
interpreted at face value. Even if there really is no
The act of p-hacking is a pervasive problem, and difference (or no effect), the chance of finding a “sta-
occurs when you influence the data collection pro- tistically significant” result purely by chance after
cess or statistical analyses performed in order to this process exceeds 5%.
produce a statistically significant result, whether you
The problem is that you introduce bias when you
mean to or not.
choose to collect more data (or analyze the data dif-
This is important: Statistical results can only be ferently) only when the P value is greater than 0.05. If
interpreted at face value when every choice in data the P value was less than 0.05 in the first analysis, it
analysis was performed exactly as planned and docu- might be larger than 0.05 after collecting more data
mented as part of the experimental design. or using an alternative analysis. But you would never
see this if you only collected more data or tried dif-
Unfortunately, p-hacking occurs quite frequently, ferent data analysis strategies when the first P value
and often you may not even realize it’s happening. was greater than 0.05.
Imagine this scenario: You develop a hypothesis,
define test parameters, and then collect and analyze
the data. You’ve decided to use the traditional value
for your significance level of α=0.05 for your statisti-
cal tests (i.e. P values less than 0.05 are considered
“statistically significant”). The results you obtain
aren’t statistically significant but show a difference
Unfortunately, p-hacking
or trend in the direction you expected, so you collect
some more data and reanalyze. Or perhaps you try occurs quite frequently,
a different way to analyze the data: remove a few and often you may not even
outliers; transform to logarithms; try a nonparamet-
realize it’s happening.
ric test; redefine the outcome by normalizing; use a
method to compare one variable while adjusting for
differences in another—the list of possibilities is end-
less. The point is that you keep trying until you obtain
a statistically significant result.

Essential Dos and Don’ts of Statistics 5

 CHAPTER 3

Important concepts:
Hypotheses and P values
Most statistical tests work by
generating not one, but two hypoth-
eses: the Null Hypothesis and the Three Types of P-Hacking
Alternative hypothesis. Before you
perform an experiment and record
your observations, you should 1 Changing the values analyzed.
understand these terms: The first kind of P-hacking involves changing the actual
values analyzed. Examples include ad hoc sample size
∙ Alternative Hypothesis (Ha): the
selection, switching to an alternate control group (if you
hypothesis that the observations
don’t like the first results and your experiment involved
are due to some real effect
two or more control groups), trying various combinations
∙ Null Hypothesis (H 0): the hypothe- of independent variables to include in a multiple regres-
sis that the observations are due sion (whether the selection is manual or automatic), trying
to random chance. analyses with and without outliers, and analyzing various
subgroups of the data.
∙ Significance level (α): the prob-
ability of rejecting H0 when it’s
actually true
2 Reanalyzing a single data set with
different statistical tests.
What is a P value?
Examples: Trying a parametric and then a nonparametric
Generally, the reason you perform test. Analyzing the raw data, then analyzing the loga-
an experiment is because you’re rithms of the data.
interested in Ha (for example, that

a treatment improves outcomes or

that two groups of measurements
3 Inadvertently choosing the
have different means). However, what wrong analysis.
you test is H0 (that the treatment has This happens when researchers performed a reasonable
no effect, or that the two groups have analysis given their assumptions and their data, but would
the same means. The P value is the have done other analyses that were just as reasonable had
probability of obtaining an outcome the data turned out differently.
at least as extreme as the outcome
you observed if H0 were true.

What does “statistically

significant” mean? DO
Once you’ve calculated a P value, The bottom line is that exploring your
you can test for statistical signifi- data can be a very useful way to gener-
cance. If your P value is smaller than ate hypotheses and make preliminary
α, it’s unlikely that you would have conclusions. When you do so, make sure
obtained your results if H0 were all such analyses are clearly labeled, and
true. Therefore, you can reject H0, then retested with new data.
stating that the effect is “statisti-
cally significant.”
Essential Dos and Don’ts of Statistics 6
 Repeat to
increase sample
size (n)

The Path to P-Hacking Analyze only a

subset of the data

Include more
Stop variables in
Reporting the model.
Results
Adjust data
(e.g. divide by
body weight)
YES

Transform
Do one or more
Begin Analyze data P < 0.05? NO the data
of the following
(i.e. logarithms)

Remove
suspicious
outliers

Pick a different
group to use as
outliers

Compare a
different outcome
variable

Use a different
statistical test

Essential Dos and Don’ts of Statistics 7

 CHAPTER 4

Don’t Add Subjects Until

You Hit Significance

Adding subjects until you Simulations to Demonstrate the Problem

hit significance may be The graph below illustrates this point via simulation. We began by
simulating two groups of data by drawing values from a Gaussian
tempting, but it is also distribution (mean=40, SD=15, but these values are arbitrary).
misleading. Both groups were simulated using exactly the same distribution,
and so have the exact same true mean value. We picked N=5
Here’s a common scenario. Rather than in each group and computed an unpaired t test (comparing the
choosing a sample size before beginning means of two groups) and recorded the P value. Then we added
a study, you simply repeat the statistical one subject to each group (so N=6) and recomputed the t test
analyses as you collect more data, and and P value. We repeated this until N=100 in each group. Then we
then: repeated the entire simulation three times. Because these sim-
ulations were done comparing two groups with identical popula-
∙ If the result is not statistically signif-
tion means, any “statistically significant” result we obtain must
icant, collect some more data, and
be a coincidence -- a Type I error.
reanalyze
The graph plots P value on the Y axis vs. sample size (per group)
∙ If the result is statistically significant,
on the X axis. The green shaded area at the bottom of the graph
stop the study
shows P values less than 0.05, so deemed “statistically significant”.
The problem with this approach is that
you’ll keep going if you don’t like the result,
but stop if you do like the result. The con- 

sequence is that the chance of obtaining




a “significant” result if the null hypothesis  

were true is a lot higher than 5%.


 











  




Essential Dos and Don’ts of Statistics 8

CHAPTER 4

Experiment 1 (purple) reached a P value of its curve. We would have stopped the
less than 0.05 when N=7, but the P value second (blue) experiment when N=61, and
is higher than 0.05 for all other sample the third (orange) experiment when N=92.
sizes. Experiment 2 (blue) reached a P In all three cases, we would have declared
value less than 0.05 when N=61 and also at the results to be "statistically significant".
N=88 and 89. Experiment 3 (orange) curve
Since these simulations were created for
hit a P value less than 0.05 when N=92
values where the true mean in both groups
and remained lower than this value until
was identical, any declaration of "statis-
N=100.
tical significance" is a Type I error. If the
If we followed the sequential approach, null hypothesis is true (the two population
we would have declared the results in means are identical) we expect to see this
all three experiments to be "statistically kind of Type I error in 5% of experiments (if
significant". We would have stopped when we use the traditional definition of α=0.05
N=7 in the first (purple) experiment, so so P values less than 0.05 are declared to
would never have seen the dotted parts be significant).

Carried out long enough, this kind of

sequential approach will always result
in a Type I error. In other words, if you
extended any experiment long enough
Since these simulations were created for (infinite N), they would all eventually reach
values where the true mean in both groups statistical significance. Of course, in
some cases you would eventually give up
was identical, any declaration of “statistical even without "statistical significance".
significance” is a Type I error. But this sequential approach will produce
"significant" results in far more than 5% of
experiments, even if the null hypothesis
were true, and so this approach is invalid.

DO
It is important that you choose a sample size and stick with it. You’ll fool yourself if you
stop when you like the results, but keep going when you don’t. The alternative is using
specialized sequential or adaptive methods that take into account the fact that you
analyze the data as you go. To learn more about these techniques, research ‘sequential’
or ‘adaptive’ methods in advanced statistics books.

Essential Dos and Don’ts of Statistics 9

 CHAPTER 5

Statistical Analysis with

GraphPad Prism
GraphPad Prism is the world’s
leading data analysis and
graphing solution purpose-
built for scientific research.
750,000 of the world’s leading scientists use Prism
to save time performing statistical analyses, make
more accurate analysis choices, and elegantly graph
and present their scientific research.

Download a free trial today—no credit cards, no

commitments— and you will be on your way to sharing
your research with the world!

www.graphpad.com

FOR MAC AND WINDOWS

Essential Dos and Don’ts of Statistics 10