Key statistical ideas for research students

If you are unsure of how you will analyse your data, consult a statistician before you collect any data, not (just)
after you have collected your data to reduce the likelihood that you don’t collect the right data or enough data.

The basis of statistical Because population And a non-random Population variability also means that random samples
variability means a sub-group is only provide estimates of the population parameters
analysis is population single case is not of interest. In the example, the red diamonds are the
unlikely to be
variability. representative of the representative sample means of two different random samples of 20,
population. either. while the red line is the population mean.

Population variability Statistical uncertainty is Sample means (blue dots) and

95% CIs (error bars) for samples
means there is a estimated with confidence Sample means (blue dots) and
with 𝑛 = 10. The red dashed
certain amount of intervals (CIs). A 95% CI is line is the real population mean. 95% CIs (error bars) for samples
such that if the experiment Note how the error bars are not with 𝑛 = 50. Note that more data
statistical uncertainty was repeated many times, means more precision (error bars
all the same size because their
or sampling error in then 95% of the time the sizes are also estimates from are smaller), but still 1 time in 20
any sample estimate real population parameter the samples. the real mean will lie outside the
(e.g. mean) would lie within error bars (red bars).
of a population
the CI (i.e. within the error
parameter. bars), or in other words, on
(Figure simulations calculated with
average, 1 time in 20 the
https://rpsychologist.com/d3/ci/)
real population parameter
would lie outside the CI
(red bars in figures).

“Standard error (of the While it is common practice to give the “standard Non-overlap of SEM error bars does not
mean)” error bars can error of the mean (SEM)” as the error bar, since necessarily imply a statistically significant
these are an approximately 68% CI, there is almost difference. In the example, 𝑝 = 0.092.
be misleading about
the level of precision a 1 in 3 chance that the “real answer” will lie
outside the error bars.
obtained.
As a rough rule of thumb, the 95% CI is about
double the SEM CI.

Statistical significance • Suppose the data opposite is from a teaching

is a measure of how experiment where the control group received
traditional teaching and the treatment group received
confident we can be Control
a new approach.
that an observed • Factors other than the teaching approach
difference is not just (“confounds”) also affect students’s marks: some pull
sampling error. marks up (e.g. higher IQ, higher effort, extra tutoring,
(This is not a precise …), while others pull students’ marks down (e.g. lower
definition of what statistical IQ, lower effort, extended illness, …): hence the spread
significance actually in the marks.
• We compare sample means to average out these Treatment
measures, but it is how the
result is used to interpret confounds to determine the average impact of the
findings.) different teaching methods alone.
• This approach only works if the treatment and control “A one-sided t-test revealed that the treatment
groups are random samples. group on average scored statistically significantly
• A statistically signifiant difference means we are better than the control group (mean difference =
reasonably confident the difference is a real effect and 0.88, 95% CI: [0.27,1.48], p = 0.003).”
not just due to sampling error. I.e. we estimate that on average students do 8.8%
• Not statistically significant means that we cannot be better with the new teaching approach, but due to
confident that the observed difference is not just statistical uncertainty, we are 95% confident the
gain is between 2.7% and 14.8%. p = 0.003 means
sampling error (which is not the same as concluding
we are very confident this gain is not just sampling
the difference is just sampling error). error (assuming also sound methods).

D.R. Rowland, Learning Development, Student Services, The University of Queensland 1

“Statistically Statistical significance is only a measure of our confidence that a difference or relationship is
significant” does not not just the result of random variation / sampling error, it is not a measure of the practical
necessarily mean importance of the result. Cf. “guilty beyond reasonable doubt” does not tell one whether the
“important” or of crime committed was a minor misdemeanor or a major felony.
“practical value”. E.g. Rasmussen et al.1 found a very highly statistically significant
relationship between the walking speed of 45 year olds and
And thus it is essential you
do not report a result as their score on a perceptual reasoning test, but the relationship
being “significant” when was quite weak, and hence unlikely to be of any predictive value
what you mean is that is in a clinical setting (𝑟 = 0.29, 95%CI: [0.22, 0.35], 𝑝 < 0.001).
“statistically significant”.

The size of an effect or Common examples: • The strength of the relationship

strength of a • To characterise how different two groups are, between two random,
relationship can be compare the between groups variability (i.e. the continusous variables is
usefully characterized difference in the means) to the within group characterised by the correlation
variability (a pooled standard deviation), giving coefficient (visualisations).
by an effect size.
Cohen’s d (visualisations). • For regression analyses, 𝑅2 gives
(See also Effect sizes primer) • Important effect size measures in health statistics the proportion of the total
include the relative risk, odds ratio and number variance explained by the model.
needed to treat.

Best practice for 1. How confident are you that the observed difference or relationship is “real” and not just a result
reporting of random variation? Report a p-value.
inferential 2. What is your best estimate of the size of the difference or strength of the relationship? Report an
statistics answers effect size.
3. How much uncertainty is there in your estimate of the size of the effect? Report confidence
four questions:
intervals.
(See journal 4. What is the practical importance (“clinical significance”) of the size of the effect you have found?
guidelines.)

However, the cutoff of 𝑝 = 0.05 for statistical For discussions of alternative approaches, see Lakens (2017) and
significance is arbitrary, so consider alternatives. Coulson et al. (2010).

Always plot your data Because not all relationships Because an outlier could Because not all data is normally
to check that what you are doing are linear. E.g. the light make a correlation seem distributed and highly skewed data
output of glow worms has an much stronger than it really can bias means. E.g. the means
is sensible. below are 21.6 and 8.4 (a large
s-shaped dependence on is. With outlier, 𝑟 = 0.67;
For a statistical test to provide temperature. without outlier, 𝑟 = 0.18. difference), while the medians are
valid results, your data must have 5.0 and 3.4 (a small difference).
the properties assumed by the
statistical test applied to it.
Choose the right test.
Check assumptions:
• For regression analyses
• For group differences
Choose the best chart.

Always plot your data, but The correlation below appears to be The data beow is a random sample
note that random variation moderately strong (𝑟 = 0.47), but from a uniform distribution, despite
can look like real relationships, the data is actually random and r is the second group looking like it is
not statistically significantly different quite different from the rest.
especially for small data sets, from zero (𝑝 = 0.18).
so always test for statistical
significance.
(Help making a good chart
choice.)

1
Source: https://doi.org/10.1001/jamanetworkopen.2019.13123
D.R. Rowland, Learning Development, Student Services, The University of Queensland 2
Check your data meets Tests of statistical significance are probability calculations, and so to make these calculations
the assumptions we need to make assumptions about the properties of the data. Thus, if these assumptions are
not at least approximately true, the results may be quite misleading.
required for the
Guide to choosing the right test based on research question and type of data.
statistical tests applied
Other conditions for valid use: (a) curve fitting; and (b) other types of analyses.
to it.

Learn more
Discussions of issues
• Reinhart (2015). Statistics done wrong: The woefully complete guide. (San Francisco: no starch press).
• Blog: The 20% Statistician: http://daniellakens.blogspot.com/
• Blog: R Psychologist: http://rpsychologist.com/
• Journal guidelines for best practice in the analysis and reporting of statistics are provided in:
Curtis, M. J., Bond, R. A., Spina, D., Ahluwalia, A., Alexander, S. P. A., Giembycz, M. A., et al. (2015). Experimental
design and analysis and their reporting: new guidance for publication in BJP. British Journal of Pharmacology,
172(14), 3461–3471.
Data management issues and discussions
• UQ Library guide and tool: https://web.library.uq.edu.au/library-services/services-researchers/manage-research-data
• Good data “hygiene” to reduce the chances of errors or data loss: Data Organization in Spreadsheets
https://www.tandfonline.com/doi/full/10.1080/00031305.2017.1375989
• Ten Simple Rules for Reproducible Computational Research:
https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285
Conceptual overviews of elementary statistics
• Statistics Every Writer Should Know: A simple guide to understanding basic statistics, for journalists and other
writers who might not know math. http://www.robertniles.com/stats/
• H. Motulsky (2010). Intuitive Biostatistics: A nonmathematical guide to statistical thinking. (New York: Oxford
University Press).
• D. Rowntree, Statistics without tears: a primer for non-mathematicians, (Scribner, New York, 1981). QA276.12
.R68 1981
• G. Kranzler and J. Moursund, Statistics for the Terrified 2nd ed., (Prentice Hall, Upper Saddle River, 1999).
QA276.12.K73
• D.B. Orr, Fundamentals of Applied Statistics and Surveys, (Chapman and Hall, New York, 1995). QA276.12.O77
• Visualising key statistical concepts with interactive simulations to aid understanding:
o Magnusson, K. (2014b). Interpreting Cohen's d effect size: an interactive visualization. Retrieved from
http://rpsychologist.com/d3/cohend/
o Magnusson, K. (2014c). Interpreting confidence intervals: an interactive visualization. Retrieved from
http://rpsychologist.com/d3/CI/
o Magnusson, K. (2015). Understanding statistical power and significance testing: an interactive visualization.
Retrieved from http://rpsychologist.com/d3/NHST/

Elementary statistics
• The Khan Academy (https://www.khanacademy.org/math/statistics-probability)
Instructional videos on essentially an entire introductory level course of probability and statistics.
• Online Statistics Education: A Multimedia Course of Study (http://onlinestatbook.com/). Project Leader: David M.
Lane, Rice University. Also available to download as a pdf file.
• McDonald, J.H. 2014. Handbook of Biological Statistics (3rd ed.). Sparky House Publishing, Baltimore, Maryland.
Available online at http://www.biostathandbook.com/index.html
• Medical Statistics at a Glance – by Aviva Petrie, Caroline Sabin (Library e-text)
Elementary and more advanced statistics
• The online GraphPad Prism software guide is a great source of conceptual explanations, and in addition to
explaining how to use the software to conduct various analyses, also explains how to interpret the output of
analyses and provides analysis checklists to ensure you are conducting appropriate analyses. See:
https://www.graphpad.com/guides/prism/8/statistics/index.htm (This software is available to UQ staff and
students – https://my.uq.edu.au/information-and-services/information-technology/software-and-web-

D.R. Rowland, Learning Development, Student Services, The University of Queensland 3

 apps/software-uq/graphpad-prism – but check it has the functionality you need before making it your choice of
software.)
• The course notes for Penn State University’s entire undergraduate and postgraduate statistics programs are
available online. See: https://onlinecourses.science.psu.edu/statprogram/programs and
https://onlinecourses.science.psu.edu/statprogram/node/106 To access the undergraduate course notes, Google
“PSU <course code>” (E.g. “PSU STAT 100”).
• InfluentialPoints.com provides critical discussions of statistics:
o Elementary section: https://influentialpoints.com/Training/basic_statistics_introduction.htm
o More advanced section:
https://influentialpoints.com/Training/statistical_mistakes_in_research_use_and_misuse_of_statistics_i
n_biology.htm
Guides on what test to use when
– https://stats.idre.ucla.edu/other/mult-pkg/whatstat/ (also has software package guides)
– http://www.socr.ucla.edu/Applets.dir/ChoiceOfTest.html
– http://www.graphpad.com/support/faqid/1790/
– Assumptions for (a) curve fitting; and (b) other types of analyses such as group differences.
– Warning: Use generic guides to get you going. Find a good text which discusses the recommended test in enough
detail that you can understand what it is for and when it is and is not valid to use.
Research design in the social sciences
• Trochim, W. M. K. (2006). Research methods knowledge base. Retrieved from
http://www.socialresearchmethods.net/kb/index.php
• Social Science Research Design: http://www.socialresearchmethods.net/kb/index.php
• Recommendations for best practice methodological planning in social psychology:
https://doi.org/10.1016/j.jesp.2016.03.004 (see Appendix A for a checklist of things to do). Although aspects are
particularly for social psychology, the basic framework would be valuable in a range of fields.
Research design in medicine and epidemiology
• Advantages and disadvantages of different designs: Centre for Evidence Based medicine:
https://www.cebm.ox.ac.uk/resources/ebm-tools/study-designs
• Statistics Toolkit – by Rafael Perera, Carl Heneghan, and Douglas Badenoch (Library e-text)

Critical appraisal tools, including levels of evidence

• Centre for Evidence Based medicine: https://www.cebm.ox.ac.uk/resources
Data visualisation
• Comprehensive online book: Fundamentals of Data Visualization: A Primer on Making Informative and Compelling
Figures https://serialmentor.com/dataviz/index.html
• A guide to choosing the right chart type, depending on what you are trying to show:
https://www.clips.edu.au/wp-content/uploads/choosing-a-good-graph.pdf (A good starter, but as with any
summary sheet, missing lots of important detail.)
• A compendium of options for R users: http://shinyapps.org/apps/RGraphCompendium/index.php
• A brief but insightful instructional video (https://www.youtube.com/watch?v=6lm4wJ1qm0w) and associated
summary sheet (http://www.principiae.be/book/pdfs/TM&Th-4.0-summary.pdf)
• Fundamental Statistical Concepts in Presenting Data: Principles for Constructing Better Graphics:
http://biostat.mc.vanderbilt.edu/wiki/pub/Main/RafeDonahue/fscipdpfcbg_currentversion.pdf (makes some
excellent points, but a bit longwinded in places).

D.R. Rowland, Learning Development, Student Services, The University of Queensland 4