Lab-5 (Parameter Estimation)

(Intro. to Data Science: IIT Delhi - Abu Dhabi)

In this lab, we will focus on parameter estimation using the bootstrap method. Let us first load the data that we shall be working with. The dataset is an "air-quality" dataset, which can be found here. These are readings of air-quality sensors taken over a period of time. Please note that the missing sensor values have been replaced with -200. We shall ignore these values in all our calculations. In the first part of this assignment, you will be asked to calculate the 95% confidence intervals for all the non-categorical parameters in the data using the bootstrap method on a sample of size 100 from the dataset. Let us first load the dataset.

import pandas as pd
import numpy as np
from matplotlib.pyplot import subplots

population_data = pd.read_csv('AirQualityUCI.csv')
population_data.head()

	Date	Time	CO(GT)	PT08.S1(CO)	NMHC(GT)	C6H6(GT)	PT08.S2(NMHC)	NOx(GT)	PT08.S3(NOx)	NO2(GT)	PT08.S4(NO2)	PT08.S5(O3)	T	RH	AH
0	10/03/04	18:00:00	2.6	1360	150	11.9	1046	166	1056	113	1692	1268	13.6	48.9	0.7578
1	10/03/04	19:00:00	2.0	1292	112	9.4	955	103	1174	92	1559	972	13.3	47.7	0.7255
2	10/03/04	20:00:00	2.2	1402	88	9.0	939	131	1140	114	1555	1074	11.9	54.0	0.7502
3	10/03/04	21:00:00	2.2	1376	80	9.2	948	172	1092	122	1584	1203	11.0	60.0	0.7867
4	10/03/04	22:00:00	1.6	1272	51	6.5	836	131	1205	116	1490	1110	11.2	59.6	0.7888

1. 95% confidence intervals based on a sample

We shall work with a sample of size 100. To make sure that everyone has the same random sample, we us work with a fixed random seed for the random number generator. This is an important concept from the perspective of reproducibility of experiments.

# Use a dedicated Generator instance for better practice
rng = np.random.default_rng(seed = 54321)

# Pick a sample of size 100 from the population
sample_data = population_data.sample(100, replace = False, random_state = rng).reset_index(drop=True)
sample_data

	Date	Time	CO(GT)	PT08.S1(CO)	NMHC(GT)	C6H6(GT)	PT08.S2(NMHC)	NOx(GT)	PT08.S3(NOx)	NO2(GT)	PT08.S4(NO2)	PT08.S5(O3)	T	RH	AH
0	06/03/05	19:00:00	2.5	1055	-200	6.2	821	321	797	194	942	949	8.6	38.1	0.4282
1	28/06/04	0:00:00	1.2	991	-200	7.9	896	69	857	78	1616	926	27.2	38.9	1.3821
2	16/11/04	7:00:00	0.9	867	-200	3.3	669	271	1069	119	981	779	10.1	44.0	0.5434
3	25/04/04	21:00:00	1.4	940	95	5.1	769	100	1103	90	1310	731	15.5	37.0	0.6447
4	02/11/04	9:00:00	4.4	1296	-200	25.2	1435	562	504	112	1899	1489	20.8	57.9	1.4070
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
95	31/12/04	1:00:00	-200.0	930	-200	4.8	752	-200	889	-200	981	820	6.9	50.3	0.5029
96	16/06/04	18:00:00	2.3	1048	-200	12.6	1070	111	807	106	1737	842	31.3	25.4	1.1415
97	25/11/04	13:00:00	2.0	1104	-200	11.4	1028	279	696	109	1263	1227	15.4	43.0	0.7483
98	23/09/04	23:00:00	1.7	1071	-200	8.9	934	179	763	94	1504	942	22.5	53.7	1.4486
99	10/09/04	0:00:00	1.2	1031	-200	7.3	871	147	950	96	1393	972	25.2	38.1	1.2049

100 rows × 15 columns

1.1 Task 1 (15 points)

Use the bootstrap method on the sampled data to report the 95% confidence intervals on mean, median, and standard deviation of all the non-categorical parameters in the data. The code you write in the code cell below should output all the confidence intervals. Please ensure that you comment your code adequately and report the quantities with appropriate labels. Also clearly mention any assumptions/simplifications you are using.

# Please write the code for Task 1 here. You may create more cells for you work if needed.

1.2 Task 2 (5 points)

Since we have the population data, we can compute the mean, median, and standard deviation for all non-categorical parameters and check whether the 95% intervals that you calculated in Task 1 indeed contain the population statistics. Calculate and report these findings in below

# Write the code related to task 2 here. Create more code/markdown cell as appropriate.

1.3 Task 3 (15 points)

In the class, we checked the validity of the confidence intervals by sampling from the population repeatedly (say 50 times), computing the 95% confidence interval, and then checking the fraction of times (out of the 50 repetitions) our confidence intervals contain the true population parameter. For this task, you are asked to repeat those experiments and give the "stacked up intervals" plot we did in class for all the non-categorical parameters in this data.

2. Confidence interval for hypothesis testing

We briefly mentioned how confidence intervals can be used for hypothesis testing. Let us work on a case study to see this. This is from the CIT book that we have been using. You can follow along with the material while you work on the tasks that we define below.

We will study this in the context of data that are a subset of the information gathered in a randomized controlled trial about treatments for Hodgkin’s disease. Hodgkin’s disease is a cancer that typically affects young people. The disease is curable but the treatment can be very harsh. The purpose of the trial was to come up with dosage that would cure the cancer but minimize the adverse effects on the patients.

hodgkins_data = pd.read_csv('hodgkins.csv')
hodgkins_data.head()

	height	rad	chemo	base	month15
0	164	679	180	160.57	87.77
1	168	311	180	98.24	67.62
2	173	388	239	129.04	133.33
3	157	370	168	85.41	81.28
4	160	468	151	67.94	79.26

This table hodgkins contains data on the effect that the treatment had on the lungs of 22 patients. The columns are:

• Height in cm
• A measure of radiation to the mantle (neck, chest, under arms)
• A measure of chemotherapy
• A score of the health of the lungs at baseline, that is, at the start of the treatment; higher scores correspond to more healthy lungs.
• The same score of the health of the lungs, 15 months after treatment

We will compare the baseline and 15-month scores. As each row corresponds to one patient, we say that the sample of baseline scores and the sample of 15-month scores are paired - they are not just two sets of 22 values each, but 22 pairs of values, one for each patient.

At a glance, you can see that the 15-month scores tend to be lower than the baseline scores – the sampled patients’ lungs seem to be doing worse 15 months after the treatment. This is confirmed by the mostly positive values in the column drop, the amount by which the score dropped from baseline to 15 months.

hodgkins_data1 = hodgkins_data.assign(Drop = (hodgkins_data['base'] - hodgkins_data['month15']))
hodgkins_data1

	height	rad	chemo	base	month15	Drop
0	164	679	180	160.57	87.77	72.80
1	168	311	180	98.24	67.62	30.62
2	173	388	239	129.04	133.33	-4.29
3	157	370	168	85.41	81.28	4.13
4	160	468	151	67.94	79.26	-11.32
5	170	341	96	150.51	80.97	69.54
6	163	453	134	129.88	69.24	60.64
7	175	529	264	87.45	56.48	30.97
8	185	392	240	149.84	106.99	42.85
9	178	479	216	92.24	73.43	18.81
10	179	376	160	117.43	101.61	15.82
11	181	539	196	129.75	90.78	38.97
12	173	217	204	97.59	76.38	21.21
13	166	456	192	81.29	67.66	13.63
14	170	252	150	98.29	55.51	42.78
15	165	622	162	118.98	90.92	28.06
16	173	305	213	103.17	79.74	23.43
17	174	566	198	94.97	93.08	1.89
18	173	322	119	85.00	41.96	43.04
19	173	270	160	115.02	81.12	33.90
20	183	259	241	125.02	97.18	27.84
21	188	238	252	137.43	113.20	24.23

hodgkins_data1['Drop'].hist(bins = np.arange(-20, 100, 20));

We can calculate the average drop:

print('Average drop:', hodgkins_data1['Drop'].mean())

Average drop: 28.615909090909096

In the sample, the average drop is about 28.6. But could this be the result of chance variation? The data are from a random sample. Could it be that in the entire population of patients, the average drop is just 0?

To answer this, we can set up two hypotheses:

• Null hypothesis: In the population, the average drop is 0.
• Alternative hypothesis: In the population, the average drop is not 0.

To test this hypothesis with a 1% cutoff for the p-value, let’s construct an approximate 99% confidence interval for the average drop in the population. (Note that this is the case where the population data is not given to you. All you have is the sample data of 22 patients. So, the bootstrap method for computing the average drop truly makes sense in this scenario.)

2.1 Task 4 (10 points)

Write code below to compute the 99% confidence interval on the average drop. Check whether 0 falls within the 99% confidence interval. Based on this, decide whether you should reject the null hypothesis given that the cutoff for the p-value is 1%.

# Write the code related to task 4 here. Create more code/markdown cell as appropriate.