Performing t-tests

HYPOTHESIS TESTING IN PYTHON

James Chapman
Curriculum Manager, DataCamp
Two-sample problems
Compare sample statistics across groups of a variable
converted_comp is a numerical variable

age_first_code_cut is a categorical variable with levels ( "child" and "adult" )

Are users who first programmed as a child compensated higher than those that started as
adults?

HYPOTHESIS TESTING IN PYTHON

Hypotheses
H0 : The mean compensation (in USD) is the same for those that coded first as a child and
those that coded first as an adult.

H0 : μchild = μadult

H0 : μchild − μadult = 0

HA : The mean compensation (in USD) is greater for those that coded first as a child
compared to those that coded first as an adult.

HA : μchild > μadult

HA : μchild − μadult > 0

HYPOTHESIS TESTING IN PYTHON

Calculating groupwise summary statistics
stack_overflow.groupby('age_first_code_cut')['converted_comp'].mean()

age_first_code_cut
adult 111313.311047
child 132419.570621
Name: converted_comp, dtype: float64

HYPOTHESIS TESTING IN PYTHON

Test statistics
Sample mean estimates the population mean

x̄ - a sample mean
x̄child - sample mean compensation for coding first as a child
x̄adult - sample mean compensation for coding first as an adult
x̄child − x̄adult - a test statistic
z-score - a (standardized) test statistic

HYPOTHESIS TESTING IN PYTHON

Standardizing the test statistic
sample stat − population parameter
z=
standard error
difference in sample stats − difference in population parameters
t=
standard error
(x̄child − x̄adult ) − (μchild − μadult )
t=
SE(x̄child − x̄adult )

HYPOTHESIS TESTING IN PYTHON

Standard error
SE(x̄child − x̄adult ) ≈ √
s2child s2adult
+
nchild nadult

s is the standard deviation of the variable

n is the sample size (number of observations/rows in sample)

HYPOTHESIS TESTING IN PYTHON

Assuming the null hypothesis is true
(x̄child − x̄adult ) − (μchild − μadult )
t=
SE(x̄child − x̄adult )
(x̄child − x̄adult )
H0 : μchild − μadult = 0 → t=
SE(x̄child − x̄adult )
(x̄child − x̄adult )
t=
√
s2child s2adult
+
nchild nadult

HYPOTHESIS TESTING IN PYTHON

Calculations assuming the null hypothesis is true
xbar = stack_overflow.groupby('age_first_code_cut')['converted_comp'].mean()

adult 111313.311047
child 132419.570621
Name: converted_comp, dtype: float64 age_first_code_cut

s = stack_overflow.groupby('age_first_code_cut')['converted_comp'].std()

adult 271546.521729
child 255585.240115
Name: converted_comp, dtype: float64 age_first_code_cut

n = stack_overflow.groupby('age_first_code_cut')['converted_comp'].count()

adult 1376
child 885
Name: converted_comp, dtype: int64

HYPOTHESIS TESTING IN PYTHON

Calculating the test statistic
(x̄child − x̄adult )
t=
√
s2child s2adult
+
nchild nadult

import numpy as np
numerator = xbar_child - xbar_adult
denominator = np.sqrt(s_child ** 2 / n_child + s_adult ** 2 / n_adult)
t_stat = numerator / denominator

1.8699313316221844

HYPOTHESIS TESTING IN PYTHON

 Let's practice!
HYPOTHESIS TESTING IN PYTHON
 Calculating p-values
from t-statistics
HYPOTHESIS TESTING IN PYTHON

James Chapman
Curriculum Manager, DataCamp
t-distributions
t statistic follows a t-distribution
Have a parameter named degrees of
freedom, or df
Look like normal distributions, with fatter
tails

HYPOTHESIS TESTING IN PYTHON

Degrees of freedom
Larger degrees of freedom → t-distribution
gets closer to the normal distribution

Normal distribution → t-distribution with

infinite df

Degrees of freedom: maximum number of

logically independent values in the data
sample

HYPOTHESIS TESTING IN PYTHON

Calculating degrees of freedom
Dataset has 5 independent observations
Four of the values are 2, 6, 8, and 5

The sample mean is 5

The last value must be 4

Here, there are 4 degrees of freedom

df = nchild + nadult − 2

HYPOTHESIS TESTING IN PYTHON

Hypotheses
H0 : The mean compensation (in USD) is the same for those that coded first as a child and
those that coded first as an adult

HA : The mean compensation (in USD) is greater for those that coded first as a child
compared to those that coded first as an adult

Use a right-tailed test

HYPOTHESIS TESTING IN PYTHON

Significance level
α = 0.1

If p ≤ α then reject H0 .

HYPOTHESIS TESTING IN PYTHON

Calculating p-values: one proportion vs. a value
from scipy.stats import norm
1 - norm.cdf(z_score)

SE(x̄child − x̄adult ) ≈ √
s2child s2adult
+
nchild nadult

z-statistic: needed when using one sample statistic to estimate a population parameter

t-statistic: needed when using multiple sample statistics to estimate a population parameter

HYPOTHESIS TESTING IN PYTHON

Calculating p-values: two means from different groups
numerator = xbar_child - xbar_adult
denominator = np.sqrt(s_child ** 2 / n_child + s_adult ** 2 / n_adult)
t_stat = numerator / denominator

1.8699313316221844

degrees_of_freedom = n_child + n_adult - 2

2259

HYPOTHESIS TESTING IN PYTHON

Calculating p-values: two means from different groups
Use t-distribution CDF not normal CDF

from scipy.stats import t

1 - t.cdf(t_stat, df=degrees_of_freedom)

0.030811302165157595

Evidence that Stack Overflow data scientists who started coding as a child earn more.

HYPOTHESIS TESTING IN PYTHON

 Let's practice!
HYPOTHESIS TESTING IN PYTHON
 Paired t-tests
HYPOTHESIS TESTING IN PYTHON

James Chapman
Curriculum Manager, DataCamp
 US Republican presidents dataset
state county repub_percent_08 repub_percent_12
0 Alabama Hale 38.957877 37.139882
1 Arkansas Nevada 56.726272 58.983452
2 California Lake 38.896719 39.331367
3 California Ventura 42.923190 45.250693
.. ... ... ... ...
96 Wisconsin La Crosse 37.490904 40.577038
97 Wisconsin Lafayette 38.104967 41.675050
98 Wyoming Weston 76.684241 83.983328
99 Alaska District 34 77.063259 40.789626

[100 rows x 4 columns]

100 rows; each row represents county-level votes in a presidential election.

1 https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/VOQCHQ

HYPOTHESIS TESTING IN PYTHON

Hypotheses
Question: Was the percentage of Republican candidate votes lower in 2008 than 2012?

H0 : μ2008 − μ2012 = 0

HA : μ2008 − μ2012 < 0

Set α = 0.05 significance level.

Data is paired → each voter percentage refers to the same county

Want to capture voting patterns in model

HYPOTHESIS TESTING IN PYTHON

From two samples to one
sample_data = repub_votes_potus_08_12
sample_data['diff'] = sample_data['repub_percent_08'] - sample_data['repub_percent_12']

import matplotlib.pyplot as plt

sample_data['diff'].hist(bins=20)

HYPOTHESIS TESTING IN PYTHON

Calculate sample statistics of the difference
xbar_diff = sample_data['diff'].mean()

-2.877109041242944

HYPOTHESIS TESTING IN PYTHON

Revised hypotheses
Old hypotheses: x̄diff − μdiff
t=
√
H0 : μ2008 − μ2012 = 0 s2dif f
HA : μ2008 − μ2012 < 0 ndiff

df = ndif f − 1

New hypotheses:
H0 : μdiff = 0

HA : μdiff < 0

HYPOTHESIS TESTING IN PYTHON

Calculating the p-value
x̄diff − μdiff
n_diff = len(sample_data) t=
√
s2diff
100
ndiff
s_diff = sample_data['diff'].std()
df = ndiff − 1
t_stat = (xbar_diff-0) / np.sqrt(s_diff**2/n_diff)

-5.601043121928489 from scipy.stats import t

p_value = t.cdf(t_stat, df=n_diff-1)

degrees_of_freedom = n_diff - 1
9.572537285272411e-08

99

HYPOTHESIS TESTING IN PYTHON

 Testing differences between two means using ttest()
import pingouin
pingouin.ttest(x=sample_data['diff'],
y=0,
alternative="less")

T dof alternative p-val CI95% cohen-d \

T-test -5.601043 99 less 9.572537e-08 [-inf, -2.02] 0.560104

BF10 power
T-test 1.323e+05 1.0

1Details on Returns from pingouin.ttest() are available in the API docs for pingouin at https://pingouin-
stats.org/generated/pingouin.ttest.html#pingouin.ttest.

HYPOTHESIS TESTING IN PYTHON

ttest() with paired=True
pingouin.ttest(x=sample_data['repub_percent_08'],
y=sample_data['repub_percent_12'],
paired=True,
alternative="less")

T dof alternative p-val CI95% cohen-d \

T-test -5.601043 99 less 9.572537e-08 [-inf, -2.02] 0.217364

BF10 power
T-test 1.323e+05 0.696338

HYPOTHESIS TESTING IN PYTHON

Unpaired ttest()
pingouin.ttest(x=sample_data['repub_percent_08'],
y=sample_data['repub_percent_12'],
paired=False, # The default
alternative="less")

T dof alternative p-val CI95% cohen-d BF10 \

T-test -1.536997 198 less 0.062945 [-inf, 0.22] 0.217364 0.927

power
T-test 0.454972

Unpaired t-tests on paired data increases the chances of false negative errors

HYPOTHESIS TESTING IN PYTHON

 Let's practice!
HYPOTHESIS TESTING IN PYTHON
 ANOVA tests
HYPOTHESIS TESTING IN PYTHON

James Chapman
Curriculum Manager, DataCamp
Job satisfaction: 5 categories
stack_overflow['job_sat'].value_counts()

Very satisfied 879

Slightly satisfied 680
Slightly dissatisfied 342
Neither 201
Very dissatisfied 159
Name: job_sat, dtype: int64

HYPOTHESIS TESTING IN PYTHON

Visualizing multiple distributions
Is mean annual compensation different for
different levels of job satisfaction?

import seaborn as sns

import matplotlib.pyplot as plt
sns.boxplot(x="converted_comp",
y="job_sat",
data=stack_overflow)
plt.show()

HYPOTHESIS TESTING IN PYTHON

Analysis of variance (ANOVA)
A test for differences between groups

alpha = 0.2

pingouin.anova(data=stack_overflow,
dv="converted_comp",
between="job_sat")

Source ddof1 ddof2 F p-unc np2

0 job_sat 4 2256 4.480485 0.001315 0.007882

0.001315 <α
At least two categories have significantly different compensation

HYPOTHESIS TESTING IN PYTHON

Pairwise tests
μvery dissatisfied ≠ μslightly dissatisfied μslightly dissatisfied ≠ μslightly satisfied
μvery dissatisfied ≠ μneither μslightly dissatisfied ≠ μvery satisfied
μvery dissatisfied ≠ μslightly satisfied μneither ≠ μslightly satisfied
μvery dissatisfied ≠ μvery satisfied μneither ≠ μvery satisfied
μslightly dissatisfied ≠ μneither μslightly satisfied ≠ μvery satisfied

Set significance level to α = 0.2.

HYPOTHESIS TESTING IN PYTHON

pairwise_tests()
pingouin.pairwise_tests(data=stack_overflow,
dv="converted_comp",
between="job_sat",
padjust="none")

Contrast A B Paired Parametric ... dof alternative p-unc BF10 hedges

0 job_sat Slightly satisfied Very satisfied False True ... 1478.622799 two-sided 0.000064 158.564 -0.192931
1 job_sat Slightly satisfied Neither False True ... 258.204546 two-sided 0.484088 0.114 -0.068513
2 job_sat Slightly satisfied Very dissatisfied False True ... 187.153329 two-sided 0.215179 0.208 -0.145624
3 job_sat Slightly satisfied Slightly dissatisfied False True ... 569.926329 two-sided 0.969491 0.074 -0.002719
4 job_sat Very satisfied Neither False True ... 328.326639 two-sided 0.097286 0.337 0.120115
5 job_sat Very satisfied Very dissatisfied False True ... 221.666205 two-sided 0.455627 0.126 0.063479
6 job_sat Very satisfied Slightly dissatisfied False True ... 821.303063 two-sided 0.002166 7.43 0.173247
7 job_sat Neither Very dissatisfied False True ... 321.165726 two-sided 0.585481 0.135 -0.058537
8 job_sat Neither Slightly dissatisfied False True ... 367.730081 two-sided 0.547406 0.118 0.055707
9 job_sat Very dissatisfied Slightly dissatisfied False True ... 247.570187 two-sided 0.259590 0.197 0.119131

[10 rows x 11 columns]

HYPOTHESIS TESTING IN PYTHON

As the number of groups increases...

HYPOTHESIS TESTING IN PYTHON

Bonferroni correction
pingouin.pairwise_tests(data=stack_overflow,
dv="converted_comp",
between="job_sat",
padjust="bonf")

Contrast A B ... p-unc p-corr p-adjust BF10 hedges

0 job_sat Slightly satisfied Very satisfied ... 0.000064 0.000638 bonf 158.564 -0.192931
1 job_sat Slightly satisfied Neither ... 0.484088 1.000000 bonf 0.114 -0.068513
2 job_sat Slightly satisfied Very dissatisfied ... 0.215179 1.000000 bonf 0.208 -0.145624
3 job_sat Slightly satisfied Slightly dissatisfied ... 0.969491 1.000000 bonf 0.074 -0.002719
4 job_sat Very satisfied Neither ... 0.097286 0.972864 bonf 0.337 0.120115
5 job_sat Very satisfied Very dissatisfied ... 0.455627 1.000000 bonf 0.126 0.063479
6 job_sat Very satisfied Slightly dissatisfied ... 0.002166 0.021659 bonf 7.43 0.173247
7 job_sat Neither Very dissatisfied ... 0.585481 1.000000 bonf 0.135 -0.058537
8 job_sat Neither Slightly dissatisfied ... 0.547406 1.000000 bonf 0.118 0.055707
9 job_sat Very dissatisfied Slightly dissatisfied ... 0.259590 1.000000 bonf 0.197 0.119131

[10 rows x 11 columns]

HYPOTHESIS TESTING IN PYTHON

More methods
padjust : string

Method used for testing and adjustment of pvalues.

'none' : no correction [default]

'bonf' : one-step Bonferroni correction

'sidak' : one-step Sidak correction

'holm' : step-down method using Bonferroni adjustments

'fdr_bh' : Benjamini/Hochberg FDR correction

'fdr_by' : Benjamini/Yekutieli FDR correction

HYPOTHESIS TESTING IN PYTHON

 Let's practice!
HYPOTHESIS TESTING IN PYTHON