8/18/2018 15 Statistical Hypothesis Tests in Python (Cheat Sheet)

 Navigation

Start Here       Blog       Books       FAQ       About       Contact

Search... 

Need help with Statistics? Take the FREE Mini-Course

15 Statistical Hypothesis Tests in Python (Cheat Sheet)

by Jason Brownlee on August 15, 2018 in Statistics

   

Quick-reference guide to the 15 statistical hypothesis tests that you need in

applied machine learning, with sample code in Python.
Although there are hundreds of statistical hypothesis tests that you could use, there is only a small subset that you may need to
use in a machine learning project.

In this post, you will discover a cheat sheet for the most popular statistical hypothesis tests for a machine learning project with
examples using the Python API.

Each statistical test is presented in a consistent way, including:

The name of the test.

What the test is checking.
The key assumptions of the test.
How the test result is interpreted.
Python API for using the test.

Note, when it comes to assumptions such as the expected distribution of data or sample size, the results of a given test are likely
to degrade gracefully rather than become immediately unusable if an assumption is violated.

Generally, data samples need to be representative of the domain and large enough to expose their distribution to analysis.

In some cases, the data can be corrected to meet the assumptions, such as correcting a nearly normal distribution to be normal
by removing outliers, or using a correction to the degrees of freedom in a statistical test when samples have differing variance, to
name two examples.

Finally, there may be multiple tests for a given concern, e.g. normality. We cannot get crisp answers to questions with statistics;
instead, we get probabilistic answers. As such, we can arrive at different answers to the same question by considering the
question in different ways. Hence the need for multiple different tests for some questions we may have about data.

Let’s get started.

Your Start in Machine Learning

https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/ 1/11
8/18/2018 15 Statistical Hypothesis Tests in Python (Cheat Sheet)

Your Start in Machine ×

Learning
You can master applied Machine Learning 
without the math or fancy degree.
Find out how in this free and practical email
course.

Statistical Hypothesis Tests in Python Cheat Sheet

Photo by davemichuda, some rights reserved. Email Address

Tutorial Overview START MY EMAIL COURSE

This tutorial is divided into four parts; they are:

1. Normality Tests
2. Correlation Tests
3. Parametric Statistical Hypothesis Tests
4. Nonparametric Statistical Hypothesis Tests

1. Normality Tests
This section lists statistical tests that you can use to check if your data has a Gaussian distribution.

Shapiro-Wilk Test
Tests whether a data sample has a Gaussian distribution.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Interpretation

H0: the sample has a Gaussian distribution.

H1: the sample does not have a Gaussian distribution.

Python Code

1 from scipy.stats import shapiro

2 data1 = ....
3 stat, p = shapiro(data)

More Information

scipy.stats.shapiro
Shapiro-Wilk test on Wikipedia

D’Agostino’s K^2 Test

Tests whether a data sample has a Gaussian distribution.
Your Start in Machine Learning

https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/ 2/11
8/18/2018 15 Statistical Hypothesis Tests in Python (Cheat Sheet)
Assumptions

Observations in each sample are independent and identically distributed (iid).

Interpretation

H0: the sample has a Gaussian distribution.

H1: the sample does not have a Gaussian distribution.

Python Code

1 from scipy.stats import normaltest

2 data1 = ....
3 stat, p = normaltest(data)

More Information Your Start in Machine ×

Learning
scipy.stats.normaltest
D’Agostino’s $K$-squared test on Wikipedia You can master applied Machine Learning 
without the math or fancy degree.
Anderson-Darling Test Find out how in this free and practical email
course.
Tests whether a data sample has a Gaussian distribution.

Assumptions Email Address

Observations in each sample are independent and identically distributed (iid).

START MY EMAIL COURSE

Interpretation

H0: the sample has a Gaussian distribution.

H1: the sample does not have a Gaussian distribution.

Python Code

1 from scipy.stats import anderson

2 data1 = ....
3 result = anderson(data)

More Information

scipy.stats.anderson
Anderson-Darling test on Wikipedia

2. Correlation Tests
This section lists statistical tests that you can use to check if two samples are related.

Pearson’s Correlation Coefficient

Tests whether two samples have a monotonic relationship.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample are normally distributed.
Observations in each sample have the same variance.

Interpretation

H0: the two samples are independent.

H1: there is a dependency between the samples.

Python Code

1 from scipy.stats import pearsonr

2 data1, data2 = ...
3 corr, p = pearsonr(data1, data2) Your Start in Machine Learning

https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/ 3/11
8/18/2018 15 Statistical Hypothesis Tests in Python (Cheat Sheet)
More Information

scipy.stats.pearsonr
Pearson’s correlation coefficient on Wikipedia

Spearman’s Rank Correlation

Tests whether two samples have a monotonic relationship.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample can be ranked.

Interpretation Your Start in Machine ×

Learning
H0: the two samples are independent.
H1: there is a dependency between the samples. You can master applied Machine Learning 
without the math or fancy degree.
Python Code Find out how in this free and practical email
course.
1 from scipy.stats import spearmanr
2 data1, data2 = ...
3 corr, p = spearmanr(data1, data2) Email Address
More Information
START MY EMAIL COURSE
scipy.stats.spearmanr
Spearman’s rank correlation coefficient on Wikipedia

Kendall’s Rank Correlation

Tests whether two samples have a monotonic relationship.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample can be ranked.

Interpretation

H0: the two samples are independent.

H1: there is a dependency between the samples.

Python Code

1 from scipy.stats import kendalltau

2 data1, data2 = ...
3 corr, p = kendalltau(data1, data2)

More Information

scipy.stats.kendalltau
Kendall rank correlation coefficient on Wikipedia

Chi-Squared Test
Tests whether two categorical variables are related or independent.

Assumptions

Observations used in the calculation of the contingency table are independent.

25 or more examples in each cell of the contingency table.

Interpretation

H0: the two samples are independent.

Your Start in Machine Learning

https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/ 4/11
8/18/2018 15 Statistical Hypothesis Tests in Python (Cheat Sheet)
H1: there is a dependency between the samples.

Python Code

1 from scipy.stats import chi2_contingency

2 table = ...
3 stat, p, dof, expected = chi2_contingency(table)

More Information

scipy.stats.chi2_contingency
Chi-Squared test on Wikipedia

3. Parametric Statistical Hypothesis Tests

This section lists statistical tests that you can use to compare data samples.
Your Start in Machine ×
Learning
Student’s t-test
You can master applied Machine Learning 
Tests whether the means of two independent samples are significantly different. without the math or fancy degree.
Find out how in this free and practical email
Assumptions course.

Observations in each sample are independent and identically distributed (iid).Email Address
Observations in each sample are normally distributed.
Observations in each sample have the same variance.
START MY EMAIL COURSE
Interpretation

H0: the means of the samples are equal.

H1: the means of the samples are unequal.

Python Code

1 from scipy.stats import ttest_ind

2 data1, data2 = ...
3 stat, p = ttest_ind(data1, data2)

More Information

scipy.stats.ttest_ind
Student’s t-test on Wikipedia

Paired Student’s t-test

Tests whether the means of two paired samples are significantly different.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample are normally distributed.
Observations in each sample have the same variance.
Observations across each sample are paired.

Interpretation

H0: the means of the samples are equal.

H1: the means of the samples are unequal.

Python Code

1 from scipy.stats import ttest_rel

2 data1, data2 = ...
3 stat, p = ttest_rel(data1, data2)

More Information

scipy.stats.ttest_rel Your Start in Machine Learning

https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/ 5/11
8/18/2018 15 Statistical Hypothesis Tests in Python (Cheat Sheet)
Student’s t-test on Wikipedia

Analysis of Variance Test (ANOVA)

Tests whether the means of two or more independent samples are significantly different.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample are normally distributed.
Observations in each sample have the same variance.

Interpretation

H0: the means of the samples are equal. Your Start in Machine ×
H1: one or more of the means of the samples are unequal. Learning
Python Code You can master applied Machine Learning 
without the math or fancy degree.
1 from scipy.stats import f_oneway
Find out how in this free and practical email
2 data1, data2, ... = ...
3 stat, p = f_oneway(data1, data2, ...) course.

More Information
Email Address
scipy.stats.f_oneway
Analysis of variance on Wikipedia START MY EMAIL COURSE

Repeated Measures ANOVA Test

Tests whether the means of two or more paired samples are significantly different.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample are normally distributed.
Observations in each sample have the same variance.
Observations across each sample are paired.

Interpretation

H0: the means of the samples are equal.

H1: one or more of the means of the samples are unequal.

Python Code

Currently not supported in Python.

More Information

Analysis of variance on Wikipedia

4. Nonparametric Statistical Hypothesis Tests

Mann-Whitney U Test
Tests whether the distributions of two independent samples are equal or not.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample can be ranked.

Interpretation

H0: the distributions of both samples are equal. Your Start in Machine Learning

https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/ 6/11
8/18/2018 15 Statistical Hypothesis Tests in Python (Cheat Sheet)
H1: the distributions of both samples are not equal.

Python Code

1 from scipy.stats import mannwhitneyu

2 data1, data2 = ...
3 stat, p = mannwhitneyu(data1, data2)

More Information

scipy.stats.mannwhitneyu
Mann-Whitney U test on Wikipedia

Wilcoxon Signed-Rank Test

Tests whether the distributions of two paired samples are equal or not. Your Start in Machine ×
Learning
Assumptions
You can master applied Machine Learning 
Observations in each sample are independent and identically distributed (iid).
without the math or fancy degree.
Observations in each sample can be ranked. Find out how in this free and practical email
Observations across each sample are paired. course.

Interpretation
Email Address
H0: the distributions of both samples are equal.
H1: the distributions of both samples are not equal. START MY EMAIL COURSE

Python Code

1 from scipy.stats import wilcoxon

2 data1, data2 = ...
3 stat, p = wilcoxon(data1, data2)

More Information

scipy.stats.wilcoxon
Wilcoxon signed-rank test on Wikipedia

Kruskal-Wallis H Test
Tests whether the distributions of two or more independent samples are equal or not.

Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample can be ranked.

Interpretation

H0: the distributions of all samples are equal.

H1: the distributions of one or more samples are not equal.

Python Code

1 from scipy.stats import kruskal

2 data1, data2, ... = ...
3 stat, p = kruskal(data1, data2, ...)

More Information

scipy.stats.kruskal
Kruskal-Wallis one-way analysis of variance on Wikipedia

Friedman Test
Tests whether the distributions of two or more paired samples are equal or not.
Your Start in Machine Learning

https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/ 7/11
8/18/2018 15 Statistical Hypothesis Tests in Python (Cheat Sheet)
Assumptions

Observations in each sample are independent and identically distributed (iid).

Observations in each sample can be ranked.
Observations across each sample are paired.

Interpretation

H0: the distributions of all samples are equal.

H1: the distributions of one or more samples are not equal.

Python Code

1 from scipy.stats import friedmanchisquare

2 data1, data2, ... = ... Your Start in Machine ×
3 stat, p = friedmanchisquare(data1, data2, ...)
Learning
More Information
You can master applied Machine Learning 
scipy.stats.friedmanchisquare without the math or fancy degree.
Friedman test on Wikipedia Find out how in this free and practical email
course.

Further Reading
Email Address
This section provides more resources on the topic if you are looking to go deeper.

A Gentle Introduction to Normality Tests in Python START MY EMAIL COURSE

How to Use Correlation to Understand the Relationship Between Variables
How to Use Parametric Statistical Significance Tests in Python
A Gentle Introduction to Statistical Hypothesis Tests

Summary
In this tutorial, you discovered the key statistical hypothesis tests that you may need to use in a machine learning project.

Specifically, you learned:

The types of tests to use in different circumstances, such as normality checking, relationships between variables, and
differences between samples.
The key assumptions for each test and how to interpret the test result.
How to implement the test using the Python API.

Do you have any questions?

Ask your questions in the comments below and I will do my best to answer.

Did I miss an important statistical test or key assumption for one of the listed tests?
Let me know in the comments below.

Get a Handle on Statistics for Machine Learning!

Develop a working understanding of statistics
…by writing lines of code in python

Discover how in my new Ebook:

Statistical Methods for Machine Learning

It provides self-study tutorials on topics like:

Hypothesis Tests, Correlation, Nonparametric Stats, Resampling, and much more…

Discover how to Transform Data into Knowledge

Skip the Academics. Just Results.
Your Start in Machine Learning

https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/ 8/11
8/18/2018 15 Statistical Hypothesis Tests in Python (Cheat Sheet)
Click to learn more.

Your Start in Machine ×

Learning
You can master applied Machine Learning 
without the math or fancy degree.
    Find out how in this free and practical email
course.

Email Address
About Jason Brownlee
Jason Brownlee, Ph.D. is a machine learning specialist who teaches developers how to get results with modern machine
learning methods via hands-on tutorials. START MY EMAIL COURSE

View all posts by Jason Brownlee →

 How to Reduce Variance in a Final Machine Learning Model A Gentle Introduction to SARIMA for Time Series Forecasting in Python 

2 Responses to 15 Statistical Hypothesis Tests in Python (Cheat Sheet)

REPLY 
Jonathan dunne August 17, 2018 at 7:17 am #

hi, the list looks good. a few omissions. fishers exact test and Bernards test (potentially more power than a fishers exact
test)

one note on the anderson darling test. the use of p values to determine GoF has been discouraged in some fields .

REPLY 
Jason Brownlee August 17, 2018 at 7:43 am #

Excellent note, thanks Jonathan.

Indeed, I think it was a journal of psychology that has adopted “estimation statistics” instead of hypothesis tests in reporting
results.

Leave a Reply

Your Start in Machine Learning

https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/ 9/11
8/18/2018 15 Statistical Hypothesis Tests in Python (Cheat Sheet)

Name (required)

Email (will not be published) (required)

Your Start in Machine ×

Website Learning
You can master applied Machine Learning 
without the math or fancy degree.
SUBMIT COMMENT Find out how in this free and practical email
course.

Welcome to Machine Learning Mastery Email Address

Hi, I'm Jason Brownlee, Ph.D.

START MY EMAIL COURSE
My goal is to make developers like YOU awesome at applied machine learning.

Read More

Statistics for Machine Learning

Skip the academic textbooks!

Finally understand stats by writing code in Python.

Click o Get Started With Statistics Today!

POPULAR

How to Develop a Deep Learning Photo Caption Generator from Scratch

NOVEMBER 27, 2017

How to Use Word Embedding Layers for Deep Learning with Keras
OCTOBER 4, 2017

How to Develop a Neural Machine Translation System from Scratch

JANUARY 10, 2018 Your Start in Machine Learning

https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/ 10/11
8/18/2018 15 Statistical Hypothesis Tests in Python (Cheat Sheet)

How to Define an Encoder-Decoder Sequence-to-Sequence Model for Neural Machine Translation in Keras
OCTOBER 26, 2017

How to Develop an Encoder-Decoder Model for Sequence-to-Sequence Prediction in Keras

NOVEMBER 2, 2017

How to Reshape Input Data for Long Short-Term Memory Networks in Keras
AUGUST 30, 2017

Your Start in Machine

How to Develop an Encoder-Decoder Model with Attention for Sequence-to-Sequence Prediction in Keras ×
OCTOBER 17, 2017
Learning
How to Use the Keras Functional API for Deep Learning You can master applied Machine Learning 
OCTOBER 27, 2017
without the math or fancy degree.
Find out how in this free and practical email
How to Develop Word Embeddings in Python with Gensim course.
OCTOBER 6, 2017

Email Address
How to Develop a Word-Level Neural Language Model and Use it to Generate Text
NOVEMBER 10, 2017
START MY EMAIL COURSE

You might also like…

Your First Machine Learning Project in Python

Your First Neural Network in Python
How to Setup a Python for Machine Learning
Your First Classifier in Weka
Your First Model for Time Series Forecasting

© 2018 Machine Learning Mastery. All Rights Reserved.

Privacy | Disclaimer | Terms of Service | Search | Newsletter | Contact | About

Your Start in Machine Learning

https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/ 11/11