Lab-3: Exercise set on Hypothesis Testing

(IIT Delhi - Abu Dhabi: AAIL100 Intro. to Data Science)

In this exercise set, we will be working on a real dataset (reference to be provided after the deadline) collected to understand the impact of job training on earning outcomes. This dataset comes from a real randomized controlled trial of a U.S. job-training program conducted in the mid-1970s. Individuals were randomly assigned either to receive job training (treatment) or not (control). The dataset records pre-treatment characteristics (age, education, race, marital status, prior earnings) and post-treatment earnings outcomes. This dataset is provided to you in a CSV file named data.csv.

Let us start with loading the data using our usual Python commands.

import numpy as np
import pandas as pd
from matplotlib.pyplot import subplots

data = pd.read_csv('data.csv')
data.head(6) # This outputs the first 6 rows of the tabular dataframe.

	ID	treat	age	educ	black	hispan	married	nodegree	pre1	pre2	post
0	NSW1	1	37	11	1	0	1	1	0.0	0.0	9930.0460
1	NSW2	1	22	9	0	1	0	1	0.0	0.0	3595.8940
2	NSW3	1	30	12	1	0	0	0	0.0	0.0	24909.4500
3	NSW4	1	27	11	1	0	0	1	0.0	0.0	7506.1460
4	NSW5	1	33	8	1	0	0	1	0.0	0.0	289.7899
5	NSW6	1	22	9	1	0	0	1	0.0	0.0	4056.4940

1. Permutation tests

We typically start with EDA (Exploratory Data Analysis) to see whether there is some evidence that training increases earnings. Answer the questions below by writing Python commands and giving appropriate explanations. You can add markdown or code cells as appropriate. For code cells, please ensure you comment your code.

Task 1.1 (Questions): [5 points]

1. What is the sample size? How many treated vs control?
2. Compute and report the mean of post by treatment group.
3. Based only on group means, does training appear to increase earnings?

# This cell is left empty for reference. You can create more of these cells as per requirement. 

Task 1.2: (Training and earnings) [15 points]

Perform a permutation test as discussed in class to determine if job training resulted in higher earnings. Show all the steps, including the null hypothesis/alternate hypothesis, test statistic, appropriate plots, and the empirical p-value. Interpret the p-value and give a concluding statement.

# This cell is left empty for reference. You can create more of these cells as per requirement. 

Task 1.3: (Training and employability) [15 points]

Perform a permutation test as discussed in class (e.g., RCT for lower back pain) to determine if job training affected employability post-training. Employability is a binary outcome, where a non-zero earning indicates employment. Show all the steps, including the null hypothesis/alternate hypothesis, test statistic, appropriate plots, and the empirical p-value. Interpret the p-value and give a concluding statement. Compare the results to the earning outcomes in the previous task. Do they tell the same story?

# This cell is left empty for reference. You can create more of these cells as per requirement. 

2. Rank-based permutation test

When the outcome distribution is skewed, a permutation test comparing the means of the treatment and control groups may not accurately reflect the true picture, since the average is sensitive to extreme values. In such cases, a rank-based test may be more appropriate. In a rank-based test, all data points are assigned ranks based on the outcome (e.g., the person with the least earnings is given rank 1, and so on). Then the average ranks in the two groups can be compared. See Section 17.2.1 in the [LDS] book for a detailed discussion (https://learningds.org/ch/17/inf_pred_gen_HT.html). Let us first check if the outcomes indeed have a skewed distribution. To check this, let us plot the histogram of the earnings.

fig, ax = subplots(figsize = (8, 4))
ax.hist(data['post'], bins=50);

Task 2.1: (Training and earnings using rank test) [15 points]

Perform a rank test to determine if job training has a positive effect on earnings. Show all the steps, including the null hypothesis/alternate hypothesis, test statistic, appropriate plots, and the empirical p-value. Interpret the p-value and give a concluding statement.

# This cell is left empty for reference. You can create more of these cells as per requirement. 

3. Multiple Testing

Multiple testing usually arises because analysts examine:

• many outcomes
• many subgroups
• many model variants

Here we’ll generate 24 tests by combining:

• 3 outcomes × 8 subgroups = 24 hypotheses.

All hypotheses are based on pre-treatment subgroup definitions, so they are valid subgroup analyses (though still subject to multiplicity).

Notes

• The dataset is typically treated as a completely randomized experiment.
• We use permutation tests to get p-values for each subgroup/outcome effect.
• Multiple-testing corrections are still needed because we are testing many hypotheses.

Cautions

• Subgroups must be defined using pre-treatment variables only.
• Avoid "shopping" for subgroups after looking at outcomes.
• Report the full family of tests (or pre-register a limited set) to avoid selective reporting.

Below, we define a family of 24 hypotheses for our dataset (i.e., 24 tests).

Outcomes (3)

1. post (earnings in 1978; continuous)
2. employed = 1(post > 0) (binary)
3. growth = post - pre1 (earnings growth; continuous)

Subgroups (8 total)

We define 4 subgroup dimensions, each with 2 levels (8 subgroup labels):

• Race: black = 1 vs 0
• Education: nodegree = 1 vs 0
• Age: age <= 30 vs age > 30
• Prior earnings: pre1 == 0 vs pre1 > 0

For each outcome × subgroup, we test whether the treatment effect differs from 0 within that subgroup.

Sanity checks

Before running 24 tests, verify that each subgroup has enough treated and control observations.

Task 3.1: [5 points]

Print the number of treatment and control observations in each of the 8 subgroups defined above. You must use appropriate Python commands to do this.

# This cell is left empty for reference. You can create more of these cells as per requirement. 

How many observations are good many? Typically, around 30 observations in each of the control and treatment groups is considered sufficient (the number 30 is not an ad hoc number, there is a theory backing this up that we'll see soon in the class).

Task 3.2: [15 points]

For each (outcome × subgroup):

• Subset to the subgroup
• Compute effect estimate (difference in means/proportions)
• Compute permutation empirical p-value with num_repetitions=5000
• Store results in a table and print this table below.

Tip: Use a different seed per test (e.g., based on a counter) so results are reproducible but not identical.

# This cell is left empty for reference. You can create more of these cells as per requirement. 

We will implement the four methods discussed in class to make decisions for rejecting null hypotheses.

• Naive: Reject if p-value $\leq \alpha$ (we'll use $\alpha = 0.05$)
• Bonferroni (FWER): reject if $p \leq \alpha/m$
• Holm step-down (FWER): sequentially reject smallest p-values with adjusted thresholds
• Benjamini–Hochberg (BH) (FDR): reject if $p_{(k)} ≤ (k/m) q$

Task 3.3: [30 points]

Implement the above four methods and produce a table that includes the list of all 24 tests and whether the null hypotheses are rejected or not as per each of these four methods. Write a concluding statement for your multiple testing.

# This cell is left empty for reference. You can create more of these cells as per requirement.