Nephrol Dial Transplant (2010) 25: 1388–1393

doi: 10.1093/ndt/gfp732
Advance Access publication 12 January 2010

CME Series

Sample size calculations: basic principles and common pitfalls

Marlies Noordzij1, Giovanni Tripepi2, Friedo W. Dekker3, Carmine Zoccali2, Michael W. Tanck4
and Kitty J. Jager1

Downloaded from https://academic.oup.com/ndt/article-abstract/25/5/1388/1842407 by guest on 03 July 2019

1
ERA-EDTA Registry, Department of Medical Informatics, Academic Medical Center, University of Amsterdam, Amsterdam, The
Netherlands, 2CNR–IBIM, Clinical Epidemiology and Pathophysiology of Renal Diseases and Hypertension, Renal and
Transplantation Unit, Ospedali Riunti, Reggio Calabria, Italy, 3Department of Clinical Epidemiology, Leiden University Medical
Centre, Leiden, The Netherlands and 4Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Academic Medical
Center, University of Amsterdam, Amsterdam, The Netherlands
Correspondence and offprint requests to: Marlies Noordzij; E-mail: [email protected]

Abstract sensitive to errors because small differences in selected

One of the most common requests that statisticians get from parameters can lead to large differences in the sample size.
investigators are sample size calculations or sample size In this paper, we explain the basic principles of sample
justifications. The sample size is the number of patients size calculations by means of examples from the nephrol-
or other experimental units included in a study, and deter- ogy literature. In addition, we discuss the most common
mining the sample size required to answer the research pitfalls in sample size calculations and comment on how
question is one of the first steps in designing a study. to report these calculations.
Although most statistical textbooks describe techniques
for sample size calculation, it is often difficult for investiga- Why sample size calculations?
tors to decide which method to use. There are many formu- The main aim of a sample size calculation is to determine
las available which can be applied for different types of data the number of participants needed to detect a clinically rel-
and study designs. However, all of these formulas should be evant treatment effect. Pre-study calculation of the required
used with caution since they are sensitive to errors, and sample size is warranted in the majority of quantitative
small differences in selected parameters can lead to large studies. Usually, the number of patients in a study is re-
differences in the sample size. In this paper, we discuss stricted because of ethical, cost and time considerations.
the basic principles of sample size calculations, the most However, if the sample size is too small, one may not be
common pitfalls and the reporting of these calculations. able to detect an important existing effect, whereas samples
that are too large may waste time, resources and money. It is
Keywords: epidemiology; nephrology; power; sample size; statistics
therefore important to optimize the sample size. Moreover,
calculating the sample size in the design stage of the study
is increasingly becoming a requirement when seeking eth-
Introduction ical committee approval for a research project.
The sample size is the number of patients or other experi- Components of sample size calculations
mental units included in a study, and one of the first practical
steps in designing a trial is the choice of the sample size In order to calculate the sample size, it is required to have
needed to answer the research question. Also in the critical some idea of the results expected in a study. In general, the
appraisal of the results of published trials, evaluating the greater the variability in the outcome variable, the larger the
sample size required to answer the research question is an sample size required to assess whether an observed effect is
important step in interpreting the relevance of these results. a true effect. On the other hand, the more effective (or
It is therefore not surprising that one of the most frequent harmful!) a tested treatment is, the smaller the sample size
requests that statistical consultants get from investigators needed to detect this positive or negative effect. Calculating
are sample size calculations or sample size justifications. the sample size for a trial requires four basic components:
Techniques for sample size calculations are described in
most conventional statistical textbooks. However, the wide 1. The type I error (alpha). Clinical studies are usually
range of formulas that can be used for specific situations and performed in a sample from a population rather than in the
study designs makes it difficult for most investigators to de- whole study population. In research, we are testing hypoth-
cide which method to use. Moreover, these calculations are eses to determine whether (results in) particular samples

© The Author 2010. Published by Oxford University Press on behalf of ERA-EDTA. All rights reserved.
For Permissions, please e-mail: [email protected]
Sample size calculations 1389
Table 1. Overview of errors in clinical research fect that is present in a population using a test based on a
sample from that population (true positive). The power is
Population the complement of beta: 1-beta. So, in case of a beta of
Difference 0.20, the power would be 0.80 or 80%, representing the
does not exist probability of avoiding a false-negative conclusion, or
the chance of correctly rejecting a null hypothesis.
Difference does not exist False negative result
Type II error (beta)
3. The smallest effect of interest. The smallest effect of
Difference exists False positive result interest is the minimal difference between the studied
Power (1-beta)
Type I error (alpha) groups that the investigator wishes to detect and is often
referred to as the minimal clinically relevant difference,
sometimes abbreviated as MCRD. This should be a differ-
differ from each other. On the one hand, the null hypoth- ence that the investigator believes to be clinically relevant
esis (H0) hypothesizes that the groups of subjects (sam- and biologically plausible. For continuous outcome vari-

Downloaded from https://academic.oup.com/ndt/article-abstract/25/5/1388/1842407 by guest on 03 July 2019

ples) that are being compared are not different, that is ables, the minimal clinically relevant difference is a numer-
they come from the same source population. The alterna- ical difference. For example, if body weight is the outcome
tive hypothesis (H1), on the other hand, hypothesizes that of a trial, an investigator could choose a difference of 5 kg
these groups are different and that therefore they seem to as the minimal clinically relevant difference. In a trial with
be drawn from different source populations. Sample size a binary outcome, for example the effect of a drug on the
calculations are needed to define at what number of sub- development of a myocardial infarction (yes/no), an inves-
jects it becomes quite unlikely that adding more subjects tigator should estimate a relevant difference between the
will change the conclusion. event rates in both treatment groups and could choose,
In the process of hypothesis–testing, two fundamental er- for instance, a difference of 10% between the treatment
rors can occur. These errors are called type I and type II er- group and the control group as minimal clinically relevant
rors, and an overview of these errors is presented in Table 1. difference. Even a small change in the expected difference
The type I error (alpha) measures the probability that, giv- with treatment has a major effect on the estimated sample
en the H0 that the samples come from the same source size, as the sample size is inversely proportional to the
population, the differences found are likely to happen. In square of the difference. For instance, if one would need
other words, the alpha represents the chance of a falsely 1000 subjects to detect an absolute difference of 4.8%,
rejecting H0 and picking up a false-positive effect. The al- 4000 subjects per treatment group would be required to de-
pha is most commonly fixed at 0.05, which means that the tect a 2.4% difference.
researcher desires a <5% chance of drawing a false-posi-
tive conclusion. 4. The variability. Finally, the sample size calculation is
based on using the population variance of a given outcome
2. Power. Instead of a false-positive conclusion, investi- variable that is estimated by means of the standard devia-
gators can also draw a false-negative conclusion. In such tion (SD) in case of a continuous outcome. Because the
cases, they conclude that there is no difference between variance is usually an unknown quantity, investigators of-
two groups or treatments when in reality there is, or in oth- ten use an estimate obtained from a pilot study or use in-
er words, they falsely accept the H0 that the compared formation from a previously performed study. For example,
samples come from the same source population. This is in an echocardiography substudy of the Australian Initiat-
called a type II error (beta). Conventionally, the beta is ing Dialysis Early And Late (IDEAL) Study, Cooper et al.
set at a level of 0.20, meaning that the researcher desires aim to determine whether the timing of dialysis initiation
a <20% chance of a false-negative conclusion. For the cal- has an effect on left ventricular mass. For their sample size
culation of the sample size, one needs to know the power calculation, the investigators used recent data from another
of a study. The power reflects the ability to pick up an ef- laboratory indicating that the mean left ventricular mass in

Table 2. Summary of the components for sample size calculations

Component Definition

Alpha (type I error) The probability of falsely rejecting H0 and detecting a statistically significant difference when the groups in reality are not
different, i.e. the chance of a false-positive result.
Beta (type II error) The probability of falsely accepting H0 and not detecting a statistically significant difference when a specified difference
between the groups in reality exists, i.e. the chance of a false-negative result.
Power (1-beta) The probability of correctly rejecting H0 and detecting a statistically significant difference when a specified difference between
the groups in reality exists.
Minimal clinically The minimal difference between the groups that the investigator considers biologically plausible and clinically relevant.
relevant difference

Variance The variability of the outcome measure, expressed as the SD in case of a continuous outcome.

Abbreviations: H0, null hypothesis; i.e. the compared samples come from the same source population (the compared groups are not different from each
other); SD, standard deviation.
1390 M. Noordzij et al.
2
renal failure patients in Australia is 140 g/m with an SD of value 0.842 should be filled in for b in the formula.
60 g/m2. These multipliers for conventional values of alpha and
Sometimes, the minimal clinically relevant difference beta can be found in Table 3.
and the variability are combined and expressed as a mul- Suppose the investigators consider a difference in SBP
tiple of the SD of the observations; the standardized differ- of 15 mmHg between the treated and the control group
ence. The standardized difference is also referred to as the (μ1 – μ2) as clinically relevant and specified such an effect to
effect size and can be calculated as: be detected with 80% power (0.80) and a significance level

difference between the means in the two treatment groups

Standardized difference ¼
population standard deviation

A summary of all components of sample size calcula- alpha of 0.05. Past experience with similar experiments,

Downloaded from https://academic.oup.com/ndt/article-abstract/25/5/1388/1842407 by guest on 03 July 2019

tions is presented in Table 2. with similar measuring methods and similar subjects, sug-
gests that the data will be approximately normally distribut-
How to calculate the sample size for randomized ed with an SD of 20 mmHg. Now we have all of the
controlled trials specifications needed for determining sample size using
the approach as summarized in Box 1. Entering the values
Formulas for sample size calculation differ depending on in the formula yields: 2 × [(1.96 + 0.842)2 × 202] / 152 =
the type of study design and the studies outcome(s). These 27.9, this means that a sample size of 28 subjects per group
calculations are particularly of interest in the design of ran- is needed to answer the research question.
domized controlled trials (RCTs). In RCTs, a lot of money is
invested, and it is therefore important to be sufficiently sure
that enough patients are included in the study arms in order SBP as a binary outcome
to find as statistically significant a difference that we as- For a study with a binary outcome, calculating the required
sume there is in the population. In general, sample size cal- sample size is slightly more complicated. It should be cal-
culations are performed based on the primary outcome of culated based on the number of events rather than on the
the study. number of people in the study (Box 2). The number of
Based on two examples, we will now demonstrate how events can be increased either by choosing higher risk pa-
to calculate sample size using the simplest formulas for an tients, by increasing the follow-up time, or by increasing
RCT comparing two groups of equal size. Suppose one the sample size.
wished to study the effect of a new hypertensive drug on
(i) systolic blood pressure (SBP) as a continuous outcome Box 2
and (ii) SBP as a binary outcome, i.e. below or above 140 Simplest formula for a binary outcome and equal sample
mmHg (hypertension yes/no). These situations are illus- sizes in both groups, assuming: alpha = 0.05 and power =
trated in Box 1 and Box 2, respectively [1]. 0.80 (beta = 0.20).

SBP as a continuous outcome n = the sample size in each of the groups

p1 = proportion of subjects with hypertension in treat-
Box 1 ment Group 1
Simplest formula for a continuous outcome and equal q1 = proportion of subjects without hypertension in
sample sizes in both groups, assuming: alpha = 0.05 treatment Group 1 (= 1− p1)
and power = 0.80 (beta = 0.20) [1]. p 2 = proportion of with hypertension in treatment
Group 2
n = the sample size in each of the groups q2 = proportion of subjects without hypertension in
μ1 = population mean in treatment Group 1 treatment Group 2 (= 1− p2)
μ2 = population mean in treatment Group 2 x = the difference the investigator wishes to detect
μ1 − μ2 = the difference the investigator wishes to detect a = conventional multiplier for alpha = 0.05
σ2 = population variance (SD) b = conventional multiplier for power = 0.80
a = conventional multiplier for alpha = 0.05
b = conventional multiplier for power = 0.80 ½ða + bÞ2 ðp1 q1 + p2 q2 Þ
n=
x2
n = 2 ½ða + bÞ2 σ2 
ðμ1 −μ2 Þ2
In this case, we suppose that the investigators consider a
difference in event rate of 10% (0.10) as clinically relevant.
When the significance level alpha is chosen at 0.05, like Based on recently published findings from studies with a
in these examples, one should enter the value 1.96 for a similar design, they expect that the proportion of subjects
in the formula. Similarly, when beta is chosen at 0.20, the with hypertension in the treated group will be ∼20% (p1 =
Sample size calculations 1391
Table 3. Multipliers for conventional values of alpha and beta For some simple clinical trials, nomograms or graphs
can be used to estimate the sample size required for the
Multipliers for conventional values of alpha study. An example of such a nomogram published by Alt-
man is presented in Figure 1 [2]. However, one should
Alpha Multiplier
0.05 1.96
keep in mind that, although these graphical methods work
0.01 2.58 well, they often make assumptions about the type of data
and statistical tests to be used.
Multipliers for conventional values of beta

Beta Multiplier Other outcome types

0.20 0.842
0.10 1.28 In many trials, the outcomes may not be continuous or bi-
0.05 1.64 nary as above, but instead may be survival (e.g. time to
0.01 2.33 event). In these cases, the details of calculation differ,
but using the four aforementioned components, persist

Downloaded from https://academic.oup.com/ndt/article-abstract/25/5/1388/1842407 by guest on 03 July 2019

through calculations with other types of outcomes. Howev-
0.20) and in the control group, ∼30% (p2 = 0.30). This au- er, other assumptions can be necessary.
tomatically means that q1 and q2 are 0.80 and 0.70, respec-
tively. Again, the investigators assume a power of 80%
Other study designs than RCTs
(0.80) and an alpha of 0.05, which means that the value
1.96 should be filled in for a and the value 0.842 should In this paper, we focus on sample size calculations for
be filled in for b. We can now enter all values in the for- RCTs, but also for studies with another design such as
mula presented in Box 2: [(1.96+0.842)2 × (0.20 × 0.80 + case-control or cohort studies, sample size calculations
0.30 × 0.70)] / 0.102 = 290.5, this means that a sample size are sometimes required. Although the calculation of sam-
of 291 subjects per group is needed to answer the research ple size is based on the same principles for all parallel
question. study designs, the formulas for sample size calculations
Many of the formulas to calculate sample size are not for other study designs than RCTs often need some adap-
straightforward, and it is recommendable to ask for the tations. For example, published formulas for case-control
help of a statistician in all but the most basic studies. designs provide sample sizes required to determine that

Fig. 1. Nomogram for the calculation of sample size or power (adapted from Altman 1982) [2].
1392 M. Noordzij et al.
Table 4. Approximate relative sample size for different levels of alpha In most studies, investigators estimate the difference of
and power interest and the standard deviation based on results from a
pilot study, published data or on their own knowledge and
Alpha (type I error)
opinion. This means that the calculation of an appropriate
0.05 0.01 0.001 sample size partly relies on subjective choices or crude es-
timates of certain factors which may seem rather artificial
Power (1-beta) to some. Unless the pilot study was large, using informa-
0.80 100 149 218 tion from a pilot study often results in unreliable estimates
0.90 134 190 266
0.99 234 306 402 of the variability and the minimal clinically relevant differ-
ence. By definition, pilot studies are underpowered, and
the observed difference in a pilot study is therefore an im-
precise estimate of the difference in the population. Not
an odds ratio is significantly different from one, after ad- accounting for this sampling error will lead to underpow-
justment for potential confounders [3]. Also, sample size ered studies [7]. Also published reports could provide an

Downloaded from https://academic.oup.com/ndt/article-abstract/25/5/1388/1842407 by guest on 03 July 2019

calculations for special types of RCTs, like cluster-ran- estimate of the outcome in the control group. Although
domized trials, in which health interventions are allocated they often incorporate a lot of differences with the study
randomly to intact clusters, communities, health centres or one aims to perform, such as dissimilar eligibility criteria,
practices rather than to individual subjects, need an alter- endpoints and treatments, some information on the control
native approach [4]. The same holds true for trials with a group usually exists [8]. Finally, another approach is to
crossover design, because these studies compare the results survey experts in the field to determine what difference
of two treatments on the same group of patients. The sample would need to be demonstrated for them to adapt a new
size calculated for a crossover study can also be used for a treatment in terms of costs and risks [9].
study that compares the value of a variable after treatment After mentioning these pitfalls, it may seem useless
with its value before treatment [5]. Finally, sample size cal- to perform a sample size calculation. However, even if
culations for clinical trials testing the equivalence rather based on estimates and assumptions, a sample size cal-
than the superiority between two treatments need another culation is considerably more useful than a completely
approach. These equivalence or non-inferiority trials usually arbitrary choice.
demand higher sample sizes [6]. The aforementioned alter-
native calculations are less common and more complicated Post hoc sample calculations. Sometimes, published
and will in most cases require statistical assistance. studies wrongfully report their ‘power’ instead of 95%
confidence intervals (CIs). The power should always be
calculated prior to a study to determine the required sam-
Common pitfalls
ple size, since it is the pre-study probability that the study
The calculation of the sample size is troubled by a large will detect a minimum effect regarded as clinically signif-
amount of imprecision, because investigators rarely have icant. After the study is conducted, one should not perform
good estimates of the parameters necessary for the calcu- any ‘post hoc’ power calculations. Once the effect of the
lation. Unfortunately, the required sample size is very sen- study is known, investigators should use the 95% CI to ex-
sitive to the choice of these parameters. press the amount of uncertainty around the effect estimate.

The effects of selecting alpha and the power. In most cases,

Reporting of sample size calculations
the conventional choices of an alpha of 0.05 and a power of
0.80 are adequate. However, dependent on the topic studied, According to the CONSORT statement, sample size calcu-
other assumptions can be made. Different assumptions of lations should be reported and justified in all published
alpha and the power will directly influence the sample size, RCTs [10]. These calculations provide important informa-
as is illustrated by Table 4. A lower alpha and a higher power tion. Firstly, they specify the primary endpoint, which safe-
will both lead to a larger sample size and as a result to higher guards against changing the planned outcome and
costs. To be aware of the influence of changes in these para- claiming a large effect on an outcome which was not the
meters, it can be helpful to perform sample size calculations original primary outcome. Secondly, knowing the planned
for different values of the parameters (sensitivity analyses). size alerts readers to potential problems, like problems
In case of doubt, one should generally choose the largest with the recruitment or an early stop of the trial [8]. Read-
sample size. ers of a published trial should be able to find all assump-
tions underlying the sample size calculation; the alpha, the
Estimating the difference and SD. Studies often aim to power, the event rate in the control group and the treatment
determine parameters like event rates in the treated group effect of interest (or the event rate in the treated group).
and the control group. Needing to estimate these para- Many studies only include statements like ‘we calculated
meters before the start of the study therefore seems strange that the sample size in each treatment group should be
for many investigators. It is, however, important to realize 250 at an alpha of 0.05 and a power of 0.80’. However,
that the parameters they are estimating in order to calculate such a statement is almost meaningless because it neglects
the required sample size are not the population parameters the estimates for the effect of interest and the variability.
as such, but the treatment effects they consider biologically Based on an example from the IDEAL Study, we can illus-
plausible and clinically relevant. trate that a better way to report these calculations would
Sample size calculations 1393
be: ‘A clinically significant effect of 10% or more over the costs related to performing an RCT, we recommend to per-
3 years would be of interest. Assuming 3-year survival form the calculations with caution or to ask for statistical
rates in the control group and the intervention group of advice during the designing phase of the study.
64% and 74% respectively, with a two-sided significance
of 0.05 and a power of 0.8, a total of 800–1000 patients
Conflict of interest statement. The results presented in this paper have not
will be required’ [11]. been published previously in whole or part, except in abstract format.
Although, ideally, all four components conventionally
required for sample size calculation should be published,
Charles et al. [12] recently showed that of 215 published References
reports of RCTs, 10 (5%) did not report any sample size
1. Florey CD. Sample size for beginners. BMJ 1993; 306: 1181–1184
calculation, and 92 (43%) did not report all the required
2. Altman DG. Statistics and ethics in medical research: III How large a
parameters. Moreover, a Danish study by Chan et al. sample? Br Med J 1980; 281: 1336–1338
[13] demonstrated that, also in study protocols, sample 3. Lui KJ. Sample size determination in case-control studies. J Clin Epi-
size calculations are often poorly reported, and that explicit demiol 1991; 44: 609–612

Downloaded from https://academic.oup.com/ndt/article-abstract/25/5/1388/1842407 by guest on 03 July 2019

discrepancies between protocols and published papers are 4. Hayes RJ, Bennett S. Simple sample size calculation for cluster-ran-
common. They found that only 11 of 62 (18%) identified domized trials. Int J Epidemiol 1999; 28: 319–326
studies described existing sample size calculations fully 5. Giraudeau B, Ravaud P, Donner A. Sample size calculation for cluster
randomized cross-over trials. Stat Med 2008; 27: 5578–5585
and consistently in both the protocol and the publication. 6. Christensen E. Methodology of superiority vs. equivalence trials and
non-inferiority trials. J Hepatol 2007; 46: 947–954
Further reading 7. Wittes J. Sample size calculations for randomized controlled trials.
Epidemiol Rev 2002; 24: 39–53
Methods for sample size calculations are described in sev- 8. Schulz KF, Grimes DA. Sample size calculations in randomised
eral general statistics textbooks, such as Altman (1991) [14] trials: mandatory and mystical. Lancet 2005; 365: 1348–1353
or Bland (2000) [15]. Specialized books which discuss sam- 9. Brasher PM, Brant RF. Sample size calculations in randomized trials:
common pitfalls. Can J Anaesth 2007; 54: 103–106
ple size determination in many situations were published by 10. Rothman KJ. Epidemiology: An Introduction. New York: Oxford
Machin et al. (1997) [16] and Lemeshow et al. (1990) [17]. University Press, 2002
In addition, there are different software programs that 11. Cooper BA, Branley P, Bulfone L et al. The Initiating Dialysis Early
can assist in sample size calculations. Examples of validat- and Late (IDEAL) Study: study rationale and design. Perit Dial Int
ed and user-friendly programmes that can be applied to 2004; 24: 176–181
calculate the sample size for several types of data and 12. Charles P, Giraudeau B, Dechartres A et al. Reporting of sample size
calculation in randomised controlled trials: review. BMJ 2009; 338:
study designs are nQuery Advisor, PASS and ‘Power and
b1732
Precision’. For those programmes, a paid license is re- 13. Chan AW, Hrobjartsson A, Jorgensen KJ et al. Discrepancies in
quired. There are also a number of websites that allow free sample size calculations and data analyses reported in randomised
sample size calculations. However, those programmes are trials: comparison of publications with protocols. BMJ 2008; 337:
not always reliable. An example of a reliable, freely avail- a2299
able website is: http://www.stat.uiowa.edu/∼rlenth/Power/ 14. Altman DG. Practical Statistics for Medical Research. London, UK:
index.html [18]. Chapman & Hall, 1991
15. Bland M. An Introduction to Medical Statistics. Oxford, UK: Oxford
University Press, 2000, 3rd edn
16. Machin D, Campbell M, Fayers P et al. Sample Size Tables for Clin-
Conclusions ical Studies. London, UK: Blackwell Science, 1997, 2nd edn
17. Lemeshow S, Levy PS. Sampling of Populations: Methods and Ap-
In conclusion, the calculation of the sample size is one of plications. New York, US: John Wiley & Sons Inc, 1999, 3rd edn
the first and most important steps in designing a study. Al- 18. Lenth R. V. Java applets for power and sample size (computer
software) 2006–9. Retrieved October 12, 2009, from http://www.stat.
though techniques for sample size calculations are de-
uiowa.edu/∼rlenth/Power
scribed in many statistical textbooks, performing these
calculations can be complicated. Because sample size cal-
culations are sensitive to errors and because of the high Received for publication: 15.10.09; Accepted in revised form: 3.12.09