ANOVA for Machine Learning

ANOVA is useful when we need to compare more than two groups and determine whether their means are significantly different. Suppose you're trying to understand which ingredients in a recipe affect its taste. Some ingredients, like spices might have a strong influence while others like a pinch of salt might not change much.

In machine learning, features act like these ingredients they contribute differently to the final prediction. Instead of guessing, we need a way to measure which features matter most. This is where ANOVA (Analysis of Variance) comes in. It helps us determine if differences in feature values lead to meaningful changes in the target variable, guiding us in selecting the most relevant features for our model.

Understanding ANOVA with a Real-World Example

Let’s say we have three schools: School A, School B and School C. We collect test scores from students in each school and calculate the average score for each group. The key question is:

Do students from at least one school perform significantly differently from the others?

To answer this ANOVA uses hypothesis testing:

• Null Hypothesis (H₀): There is no significant difference between the mean scores of the three schools.
• Alternative Hypothesis (H₁): At least one school’s mean score is significantly different from the others.

ANOVA does not tell us which group is different it only tells us a difference exists. If the p-value from the ANOVA test is less than 0.05 we reject the null hypothesis and conclude that at least one group has a significantly different mean score.

Key Assumptions of ANOVA

For ANOVA to work effectively three important assumptions must be met:

1. Independence of Observations:
   • Each data point should be independent of others.
   • In our example one student’s test score should not influence another student’s score.
2. Homogeneity of Variances (Equal Variance):
   • The variation in scores across all groups should be roughly the same.
   • If one school’s scores vary widely while the others have similar scores ANOVA results may be unreliable.
3. Normal Distribution:
   • The data within each group should follow a normal distribution.
   • If the data is highly skewed it can not work well.

How ANOVA Test Works?

To understand how ANOVA works let's go through it step by step focusing on key concepts with the help of a example.

Step 1. Calculate Group Means

First we calculate the mean for each group. Let's say you are comparing smartphone prices from three brands: Brand A, Brand B, and Brand C. Let's assume the following data for the smartphone prices:

• Brand A: [200, 210, 220, 230, 250]
• Brand B: [180, 190, 200, 210, 220]
• Brand C: [210, 220, 230, 240, 250]

Now we calculate the mean for each brand:

• Mean of Brand A = (200 + 210 + 220 + 230 + 250) / 5 = 222
• Mean of Brand B = (180 + 190 + 200 + 210 + 220) / 5 = 200
• Mean of Brand C = (210 + 220 + 230 + 240 + 250) / 5 = 230

Step 2. Calculate Overall Mean

Next we calculate the overall mean.

Overall mean = (200 + 210 + 220 + 230 + 250 + 180 + 190 + 200 + 210 + 220 + 210 + 220 + 230 + 240 + 250) / 15 = 215

Step 3. Calculate variances:

There are basically two methods to calculate the variance of the data:

1. Within-group variance: This measures how much the scores in a group differ from the group’s average. If scores are close to the average, the variance is small. If scores are spread out, the variance is large. The formula for calculation is :

Within-group variance = \frac{1}{n_i - 1} \sum_{j=1}^{n_i} (X_{ij} - \bar{X_i})^2

Where:

• X_i​ = individual prices
• \bar{X} = mean of the group
• n = number of prices in the group

For Brand A: Prices: [200, 210, 220, 230, 250] and Mean: \bar{X}=222

The squared differences are:

• (210−230)²=(−20)²=400
• (220−230)²=(−10)²=100
• (230−230)²=(0)²=0
• (240−230)²=(10)²=100
• (250−230)²=(20)²=400

Sum of squared differences = 484+ 144+ 4+ 64+ 784=1480

Now calculate the variance for Brand A:

• Variance for Brand A = \frac{1480}{5-1} = \frac{1480}{4} = 370

similarly we will calculate for both Brand B and Brand C and we get:

• Variance for Brand B = \frac{1000}{5-1} = \frac{1000}{4} = 250
• Variance for Brand C = \frac{1000}{5-1} = \frac{1000}{4} = 250

2. Between-group variance: It measures how much the group means differ from the overall mean. If the group means are far apart then the variance will be large. If the group means are close to each other the variance will be small. To calculate this we use the formula:

Between-group variance =\frac{1}{k - 1} \sum_{i=1}^{k} n_i (\bar{X_i} - \bar{X})^2

Where:

• n_i​ is the number of data points in each group (5 in each group),
• \bar{X_i}​ is the mean of each group,
• \bar{X} is the overall mean.

Step-by-step Calculation:

• For Brand A: (\bar{X_A} - \bar{X})^2 = (222 - 215)^2 = (7)^2 = 49

Contribution to between-group variance: 5 \times 49 = 245

• For Brand B:(\bar{X_B} - \bar{X})^2 = (200 - 215)^2 = (-15)^2 = 225

Contribution to between-group variance: 5 \times 225 = 1125

• For Brand C: (\bar{X_C} - \bar{X})^2 = (230 - 215)^2 = (15)^2 = 225

Contribution to between-group variance: 5 \times 225 = 1125

Sum of contributions:\text{Between-group variance} = \frac{245 + 1125 + 1125}{3-1} = \frac{2495}{2} = 1247.5

Step 4. F-Ratio Calculation

Once we have the within-group and between-group variances we calculate the F-ratio by dividing the between-group variance by the within-group variance:

F = \frac{\text{Between-group variance}}{\text{Within-group variance}} = \frac{1247.5}{290} \approx 4.3

• A high F-ratio suggests that the between-group variance is much larger than the within-group variance. This means that the groups are significantly different from each other.
• A low F-ratio indicates that the groups are not very different from each other.

Step 5. Interpreting the F-Ratio

To understand the results of the F-ratio we compare it to a critical value from the F-distribution table.

• If the F-ratio is greater than the critical value it indicates that there is a significant difference between at least one group’s mean and the others and we reject the null hypothesis.
• On the other hand if the F-ratio is small we fail to reject the null hypothesis means there is not enough evidence to say that the group means are different.

The F-ratio is 4.3 which we can compare to a critical value from the F-distribution table based on the degrees of freedom:

• Degrees of freedom for the numerator ( 𝑑 𝑓 _ {between} ): 𝑘 − 1 = 3 − 1 = 2 k−1=3−1=2)
• Degrees of freedom for the denominator {\text{}df_{within}}: n−k=15−3=12

If the calculated F-ratio is greater than the critical value from the table (which depends on the significance level usually 0.05),we reject the null hypothesis and conclude that there are significant differences between the group means

Types of ANOVA Tests

ANOVA has two main types: one-way and two-way depending on how many independent variables are involved.

1. One-Way ANOVA

This test is used when we have one independent variable with two or more groups. It helps check if at least one group is different from the others. Imagine we are comparing the average prices of smartphones from three brands: Brand A, Brand B, and Brand C and we have Independent variable: Brand (A, B, and C) and Dependent variable is Smartphone price.

Firstly We set up two hypotheses:

• Null Hypothesis (H₀): All brands have the same average price.
• Alternative Hypothesis (H₁): At least one brand has a different average price.

ANOVA helps determine if the price differences are due to real variation between brands or just random chance. However it only considers one factor (brand) at a time. If we want to check multiple factors we use two-way ANOVA.

2. Two-Way ANOVA

A two-way ANOVA is used when we have two independent variables which allow us to analyze their individual effects and their interaction.

For example suppose we want to see how brand and storage capacity (64GB, 128GB, 256GB) affect smartphone prices.

• Factor 1: Brand (A, B, C)
• Factor 2: Storage capacity
• Dependent variable: Price

Using two-way ANOVA, we test:

• Does brand affect price?
• Does storage size affect price?
• Does the effect of storage size depend on the brand? (interaction effect)

If there’s an interaction, it means one factor’s effect changes depending on the other. For example, Brand A’s prices rise with more storage, but Brand C’s prices stay the same.

In machine learning, detecting interactions can help create new features (like brand × storage) to improve predictions. This helps us understand how brand and storage together influence price.

ANOVA for Feature Selection in Machine Learning

ANOVA is also used in machine learning for feature selection. When building a model, not all features help predict the target. ANOVA helps find important numerical features when the target is categorical (like "Yes" or "No"). Feature selection makes the model simpler, faster, and more accurate.

For example, a teacher wants to know if study hours, assignments, or attendance impact student grades (A, B, C, D). The ANOVA F-test (like Scikit-learn’s f_classif) checks if the average values of a feature differ across target groups.

How it works:

• The F-test checks if the feature’s means differ across groups (e.g., study hours across grades).
• If there’s a big difference, the feature is important; if not, it’s less important.

The test gives an F-statistic and a p-value:

• Low p-value (< 0.05) = important feature
• High p-value = less important, can be removed

This helps pick the best features for the model.

Difference between Oneway Anova and Two way Anova

The difference between the Oneway Anova and Two way anova is given below:

ANOVA helps compare multiple groups to check if their means differ significantly. It prevents multiple t-tests and reduces errors. but when we need to analyze multiple dependent variables at once we use MANOVA (Multivariate Analysis of Variance). which we will study in next article.