Lab-10 (Logistic Regression)

(Intro. to Data Science: IIT Delhi - Abu Dhabi)

In this lab, we will build a large-scale binary classifier for click-through-rate (CTR) prediction using logistic regression. The assignment is based on the public Criteo click logs data, a classic large sparse dataset with:

• 1 binary label
• 13 integer features
• 26 categorical features
• many missing values
• very high-cardinality sparse predictors (sparse means few non-zero values)

This is an important success case for logistic regression: even though the model is linear, it often performs very well on large, sparse data with many indicator-like features.

What you will learn

By the end of this lab, you should be able to:

1. explain why logistic regression can work well on large, sparse datasets,
2. preprocess mixed numeric and categorical features,
3. use feature hashing for high-cardinality categorical variables,
4. train a scalable logistic model with stochastic gradient descent,
5. evaluate probability predictions using log loss, ROC-AUC, and threshold-based metrics,
6. compare baselines against a stronger sparse linear model.

# Library functions we'll use
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from scipy import sparse
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.feature_extraction import FeatureHasher
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import (
    accuracy_score,
    roc_auc_score,
    log_loss,
    precision_recall_fscore_support,
    confusion_matrix,
    classification_report,
)

# N_ROWS denotes the number of observations we'll use from the 
# dataset that has 1M observations. For the first round of experimentation
# we work with the subset of size 10000. You should make this 1M at the end.
N_ROWS = 10000

# This is the seed we will use for various randomized procedures
RANDOM_STATE = 42

# We create convenient names of features and target
num_cols = [f"I{i}" for i in range(1, 14)]
cat_cols = [f"C{i}" for i in range(1, 27)]
all_cols = ["label"] + num_cols + cat_cols

# Read the click feedback data in a dataframe
df = pd.read_csv(
    'Criteo.csv',
    sep=",",
    nrows = N_ROWS,
    compression="infer",
    low_memory=False,
)
# Changing feature names and target
df.columns = all_cols
df.head(3)

	label	I1	I2	I3	I4	I5	I6	I7	I8	I9	...	C17	C18	C19	C20	C21	C22	C23	C24	C25	C26
0	0	1.0	1	5.0	0.0	1382.0	4.0	15.0	2.0	181.0	...	e5ba7672	f54016b9	21ddcdc9	b1252a9d	07b5194c	NaN	3a171ecb	c5c50484	e8b83407	9727dd16
1	0	2.0	0	44.0	1.0	102.0	8.0	2.0	2.0	4.0	...	07c540c4	b04e4670	21ddcdc9	5840adea	60f6221e	NaN	3a171ecb	43f13e8b	e8b83407	731c3655
2	0	2.0	0	1.0	14.0	767.0	89.0	4.0	2.0	245.0	...	8efede7f	3412118d	NaN	NaN	e587c466	ad3062eb	3a171ecb	3b183c5c	NaN	NaN

3 rows × 40 columns

You must have noticed weird alphanumeric strings as values of the categorical features. What do you think they represent? Try finding out the purpose of having these alphanumeric values from the site that made the data available.

Let us first do some exploratory data analysis.

# Let us find the click rate. This is the fraction of observations labeled 1
click_rate = df["label"].mean()
print("Click rate:", click_rate)

# If you print the first few rows of the data, you will find that some of the entries are 
# not defined. Let us try to get a sense of the number of missing entries by features.
print('\nMissing entries for different deatures in sorted order:')
missing_counts = df.isna().sum().sort_values(ascending=False)
print(missing_counts.head(15))

# Let us see the statistics of the numerical columns
display(df[num_cols].describe())

Click rate: 0.2182

Missing entries for different deatures in sorted order:
C22    8182
I12    7735
C20    4496
C19    4496
C26    4496
C25    4496
I10    4481
I1     4481
I6     2511
I3     2037
I4     1978
I13    1978
C6     1332
I5      494
I7      492
dtype: int64

	I1	I2	I3	I4	I5	I6	I7	I8	I9	I10	I11	I12	I13
count	5519.000000	10000.000000	7963.000000	8022.000000	9.506000e+03	7489.000000	9508.000000	9990.000000	9508.000000	5519.000000	9508.000000	2265.000000	8022.000000
mean	3.379598	110.864300	46.949642	8.013588	1.708226e+04	143.394979	14.525768	13.081882	124.675011	0.577097	2.347602	0.894040	10.909250
std	9.419810	443.486036	893.981495	11.027284	6.256866e+04	365.587339	52.580114	17.567965	277.391841	0.656325	4.563452	3.805947	20.051778
min	0.000000	-2.000000	0.000000	0.000000	0.000000e+00	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
25%	0.000000	0.000000	2.000000	2.000000	2.720000e+02	9.000000	0.000000	2.000000	10.000000	0.000000	0.000000	0.000000	2.000000
50%	1.000000	3.000000	7.000000	4.000000	2.480000e+03	40.000000	3.000000	7.000000	41.000000	0.000000	1.000000	0.000000	5.000000
75%	3.000000	38.000000	21.000000	10.000000	1.049125e+04	129.000000	10.000000	20.000000	123.000000	1.000000	2.000000	1.000000	13.000000
max	214.000000	18522.000000	65535.000000	280.000000	1.618112e+06	12167.000000	1658.000000	682.000000	7335.000000	6.000000	79.000000	98.000000	770.000000

Next, we create a train:validation:test split. We will use 70% of the data for training. Out of the remaining, we shall use half for validation (choosing the hyperparameters, etc.) and the remaining for the final test.

y = df["label"].astype(int).values

# We collect the indices of the training set
train_idx, temp_idx = train_test_split(
    np.arange(len(df)),
    test_size=0.30,
    random_state=RANDOM_STATE,
    stratify=y,
)

# We collect the indices of the validation and test set
valid_idx, test_idx = train_test_split(
    temp_idx,
    test_size=0.50,
    random_state=RANDOM_STATE,
    stratify=y[temp_idx],
)

# Let us examine the split
print('train:validation:test: ', len(train_idx), len(valid_idx), len(test_idx))

train:validation:test:  7000 1500 1500

Evaluation

Before we start training our logistic regression-based predictor, let us understand the metrics used to evaluate it. Recall that in logistic regression for binary classification, we try to model the probability that the label of a given observation $X$ is $1$ (i.e., $\mathbf{Pr}[y|X]$). This means that our trained model outputs a number between 0 and 1 for an input observation $X$. Since the final output is a label (i.e., 0 or 1), not a value between 0 and 1, a threshold of 0.5 is typically used. This means that if the number is $\geq 0.5$ for an observation, the output is $1$; otherwise, it is $0$. We know how to evaluate a binary classifier, i.e., one that outputs discrete values 0/1. The typical metrics are accuracy, precision, recall, and f1-score.

Log-loss: Recall that our loss function was defined in terms of the probability of the prediction. We defined $$L(\theta) = \sum_i (p_i)^{y_i} \cdot (1-p_i)^{1-y_i}, \textrm{ where } p_i \textrm{ is the probability of predicting 1}.$$ We consider the log likelihood function $\ell(\theta)$, which is basically the logarithm of $L(\theta)$. Note that this will be a negative number. We consider $- \ell(\theta)$, which is known as log-loss. So, for any classification algorithm that models the probabilities $Pr[y|X]$, we can use the above formula to calculate the log-loss and use it to evaluate the predictor (the smaller the log-loss, the better). Logistic regression is one such classification technique that models $Pr[y|X]$ and the probability is given by $p_i = \frac{1}{1+e^{-(\theta \cdot X_i)}}$.

ROC-AUC: In logistic regression, the model outputs a probability $p_i = Pr[y=1 | X = X_i]$. For example, it may predict:

• Student A: 0.92, Student B: 0.73, Student C: 0.41, Student D: 0.08.

These are not final class labels yet. To turn them into labels, you choose a threshold such as $0.5$.

• predict class 1 if $p_i \geq 0.5$ and predict $0$ is $p_i < 0.5$.

ROC-AUC is about what happens when you vary that threshold. ROC stands for Receiver Operating Characteristic. The ROC curve plots TPR against FPR for all possible thresholds (TPR and FPR are defined below):

• True Positive Rate (also called recall or sensitivity): $TPR = \frac{TP}{TP + FN}$
• False Positive Rate: $FPR = \frac{FP}{FP + TN}$.

So for logistic regression, you sort examples by predicted probability and see how well the model separates positives from negatives as the threshold moves from 1 down to 0.

So, why is this useful for logistic regression?

Logistic regression naturally gives a score/probability, not just a yes/no answer. ROC-AUC measures how well those scores rank positive examples above negative ones. That is the key idea:

• A good logistic regression model gives higher probabilities to true positives than to true negatives
• ROC-AUC measures how often that ranking is good

Meaning of AUC: AUC means Area Under the ROC Curve.

• $AUC = 1$: Perfect separation, i.e., all positive items are ranked higher than all negative ones.
• $AUC = 0.5$: No better than random guessing.
• $AUC < 0.5$: worse than random, meaning the ranking is reversed

An important interpretation of $AUC$ is: $$AUC = \mathbf{Pr}[\textrm{a random positive gets a higher score than a random negative}]$$

Let's consider a small example:

Example	True class	Predicted Probability
A	1	0.90
B	1	0.80
C	0	0.70
D	1	0.60
E	0	0.40
F	0	0.20

This is fairly good because most positives have higher scores than negatives. Compare every positive with every negative:

• Positives: 0.90, 0.80, 0.60
• Negatives: 0.70, 0.40, 0.20

There are $3 \times 3 = 9$ positive-negative pairs. Count how many times positive score $>$ negative score:

• 0.90 beat all three negative scores
• 0.80 beat all three negative scores
• 0.60 beats 0.40 and 0.20, but not 0.70 So, the number of correct orderings is 8 out of 9. So, $AUC = 8/9$.

ROC-AUC vs accuracy: Accuracy depends on one chosen threshold, often $0.5$. ROC-AUC does not depend on one threshold. It measures the model’s performance across all thresholds. That is why two logistic regression models can have similar accuracy but different ROC-AUC. A model with a higher ROC-AUC usually has better ranking ability.

# We will use the following evaluator that calculates the metrics discussed above
def evaluate_binary_classifier(y_true, y_prob, threshold=0.5):
    y_pred = (y_prob >= threshold).astype(int)
    p, r, f1, _ = precision_recall_fscore_support(
        y_true, y_pred, average="binary", zero_division=0
    )
    return {
        "accuracy": accuracy_score(y_true, y_pred),
        "log_loss": log_loss(y_true, y_prob, labels=[0, 1]),
        "roc_auc": roc_auc_score(y_true, y_prob),
        "precision": p,
        "recall": r,
        "f1": f1,
    }

# Collect the train/validation/test sets
y_train = y[train_idx]
y_valid = y[valid_idx]
y_test = y[test_idx]


# Baseline A: predict class 0 always
majority_prob_valid = np.zeros_like(y_valid, dtype=float)

# Evaluate Baseline A
baseline_A_metrics = evaluate_binary_classifier(y_valid, majority_prob_valid, threshold=0.5)
print('\nEvaluation of Baseline A (probability 0 for everything):')
print(baseline_A_metrics)

# Baseline B: predict the training click rate always
train_click_rate = y_train.mean()
constant_prob_valid = np.full_like(y_valid, fill_value=train_click_rate, dtype=float)
# Remember, the click rate is the fraction of samples labeled 1
print("Training click rate:", train_click_rate)

# Evaluate Baseline B
baseline_B_metrics = evaluate_binary_classifier(y_valid, constant_prob_valid, threshold=0.5)
print('\nEvaluation of Baseline B (probability=click_rate for everything):')
print(baseline_B_metrics)

Evaluation of Baseline A (probability 0 for everything):
{'accuracy': 0.7813333333333333, 'log_loss': 7.881545541086952, 'roc_auc': np.float64(0.5), 'precision': 0.0, 'recall': 0.0, 'f1': 0.0}
Training click rate: 0.21814285714285714

Evaluation of Baseline B (probability=click_rate for everything):
{'accuracy': 0.7813333333333333, 'log_loss': 0.5252160226158614, 'roc_auc': np.float64(0.5), 'precision': 0.0, 'recall': 0.0, 'f1': 0.0}

Our initial baselines (A and B above) are trivial predictors. One sets the probabilities to 0 (Baseline A), whereas the other sets probabilities as the click rate, $0.218$. With the threshold sets as 0.5, this effectively predicts 0 as the label for all items. The accuracy of these predictors is high, 78.13%. However, this is due to the trivial fact that 78.13% of the samples have a true label of 0. So, accuracy is not a good metric in this scenario when there is significant skew in the label distribution. The better metrics are log-loss (low), precision (0), recall (0), f1 (0), and roc_auc (0.5).

Let us now build better predictors with higher precision, recall, and roc_auc (which, as we noted earlier, indicates better ranking ability). Before we set up our optimization problem to maximize log-loss, we must preprocess the numerical and categorical features. For numerical features, we use the standard scaler, ensuring missing values are handled appropriately before applying it. We shall also use the log1p function (i.e., $\log{(1+x)}$) that is a useful transformation for nonnegative, skewed data, especially when zeros are common. I'll leave it as an exercise to learn about the log1p transformation and the contexts in which it is effective in data analysis.

def build_numeric_block(df_part, num_cols, scaler=None, fit=False):
    X_num = df_part[num_cols].copy()

    # convert to numeric just in case.
    # 'coerce' means that if a value cannot be converted to 
    # number, then it is converted to nan.
    for c in num_cols:
        X_num[c] = pd.to_numeric(X_num[c], errors="coerce")
        
    # The fillna() method in pandas is used to replace missing (null) values, 
    # such as NaN or None, with specified data.
    X_num = X_num.fillna(0.0)

    # Apply log1p on nonnegative values
    X_num = np.log1p(np.maximum(X_num, 0))

    # Apply scaling
    if scaler is None:
        scaler = StandardScaler()
    if fit:
        X_num_scaled = scaler.fit_transform(X_num)
    else:
        X_num_scaled = scaler.transform(X_num)

    return X_num_scaled, scaler

Processing the categorical features requires a few considerations. One way to encode the categorical features for regression is one-hot encoding, which we have seen in earlier labs. In this lab, we would like to introduce another technique that is very effective in regression tasks. This is known as feature hashing. Let's try to understand this using an example. Consider a dataset with two categorical features (relevant to our current context of click prediction), city and browser. There may be various values of these features, e.g., city=Delhi, city = Abu Dhabi, city = Sydney, browser = Chrome, browser = Firefox. Suppose these are all the possible feature values, and that we perform one-hot encoding. Then, three 0/1-valued features will be created for city and two for browser. An observation $[Delhi, Chrome]$ will get encoded to $[1, 0, 0, 1, 0]$. Now, just imagine that there are $n$ features with every feature taking values from a set of $m$ values, then the one-hot encoding will create a feature vector of size $nm$ (i.e., the encoded data will have $mn$ columns), which may be very large (just imagine doing a one-hot encoding of URLs). Another way to encode is to fix the number of columns, say $HASH\_DIM$. Then we will hash the feature-category pairs to one of the $HASH\_DIM$ buckets and add the contributions from hashing values of a data point. This will create a feature vector for the data point. For example, let us say that $HASH\_DIM = 3$ for our city and browser dataset. For a datapoint $[Delhi, Chrome]$, we shall hash $city=Delhi$ and $browser=Chrome$, and suppose both land in the first bucket. Then the feature vector for this datapoint is $[2, 0, 0]$. This is an example of a collision, which makes the feature vector less interpretable than in one-hot encoding. However, if $HASH\_DIM$ is sufficiently large, collisions are rare, and we also gain efficiency advantages. We use feature hashing in the code below. Please review the code and learn about some of the finer points of the function calls, which were omitted in the above discussion.

def row_to_cat_dict(row, cat_cols):
    feats = {}
    for c in cat_cols:
        val = row[c]
        if pd.notna(val) and val != "":
            feats[f"{c}={val}"] = 1.0
    return feats


def build_categorical_block(df_part, cat_cols, hasher):
    generator = (row_to_cat_dict(row, cat_cols) for _, row in df_part[cat_cols].iterrows())
    return hasher.transform(generator)

# We keep the HASH_DIM sufficiently large to minimize collisions
HASH_DIM = 2**18

df_train = df.iloc[train_idx].reset_index(drop=True)
df_valid = df.iloc[valid_idx].reset_index(drop=True)
df_test  = df.iloc[test_idx].reset_index(drop=True)

# Preparing the Numeric block
X_train_num, scaler = build_numeric_block(df_train, num_cols, scaler=None, fit=True)
# Recall that we need to scale using the scaler used with the training set
X_valid_num, _ = build_numeric_block(df_valid, num_cols, scaler=scaler, fit=False)
X_test_num, _ = build_numeric_block(df_test, num_cols, scaler=scaler, fit=False)


# Preparing the Categorical block
hasher = FeatureHasher(n_features=HASH_DIM, input_type="dict")
X_train_cat = build_categorical_block(df_train, cat_cols, hasher)
X_valid_cat = build_categorical_block(df_valid, cat_cols, hasher)
X_test_cat  = build_categorical_block(df_test, cat_cols, hasher)

# Combined block. Converting to a sparse format helps save space.
# Please learn about the sparse function.
X_train = sparse.hstack([sparse.csr_matrix(X_train_num), X_train_cat]).tocsr()
X_valid = sparse.hstack([sparse.csr_matrix(X_valid_num), X_valid_cat]).tocsr()
X_test  = sparse.hstack([sparse.csr_matrix(X_test_num),  X_test_cat]).tocsr()

print("X_train shape:", X_train.shape)
print("X_valid shape:", X_valid.shape)
print("X_test shape:", X_test.shape)
# Print number of non-zero entries
print("X_train nnz:", X_train.nnz)

X_train shape: (7000, 262157)
X_valid shape: (1500, 262157)
X_test shape: (1500, 262157)
X_train nnz: 252154

The data is now prepared for fitting. As we discussed in class, log-loss is a convex function of the model parameters (i.e., $\theta$). Even though there is no closed-form analytical solution, we can run gradient descent to find optimal parameters. For the current context, we will use sklearn's SGDClassifier that runs stochastic gradient descent with regularization (either l1 or l2). Please review the code below and learn about some of the other setup parameters for SGDClassifier.

def fit_logistic_sgd(X_train, y_train, alpha=1e-5, penalty="l2", random_state=42):
    model = SGDClassifier(
        loss="log_loss",
        penalty=penalty,
        alpha=alpha,
        max_iter=1000,
        tol=1e-3,
        random_state=random_state,
    )
    model.fit(X_train, y_train)
    return model

One of the parameters used in setting up the SGDClassifier is the regularization parameter, which we have seen in earlier labs when we discussed ridge and lasso regression. We will use the validation set to find a good choice for the regularization parameter $\alpha$ and choice of l1-vs-l2.

# We run a small hyperparameter sweep
results = []

for alpha in [1e-6, 1e-5, 1e-4]:
    for penalty in ["l2", "l1"]:
        model = fit_logistic_sgd(X_train, y_train, alpha=alpha, penalty=penalty, random_state=RANDOM_STATE)
        valid_prob = model.predict_proba(X_valid)[:, 1]
        metrics = evaluate_binary_classifier(y_valid, valid_prob, threshold=0.5)
        metrics["alpha"] = alpha
        metrics["penalty"] = penalty
        results.append(metrics)

# We sort the result based on log-loss and then on roc_auc
results_df = pd.DataFrame(results).sort_values(["log_loss", "roc_auc"], ascending=[True, False])
results_df

	accuracy	log_loss	roc_auc	precision	recall	f1	alpha	penalty
4	0.756667	0.577680	0.684719	0.409756	0.256098	0.315197	0.000100	l2
5	0.746000	0.768294	0.659085	0.402214	0.332317	0.363940	0.000100	l1
2	0.742000	1.417620	0.636014	0.368889	0.253049	0.300181	0.000010	l2
3	0.742000	6.771917	0.619796	0.368889	0.253049	0.300181	0.000010	l1
0	0.736000	7.508400	0.644054	0.376812	0.317073	0.344371	0.000001	l2
1	0.744667	8.727804	0.613822	0.373272	0.246951	0.297248	0.000001	l1

# pick the best model and retrain if necessary

best_row = results_df.iloc[0]
best_alpha = float(best_row["alpha"])
best_penalty = best_row["penalty"]

best_model = fit_logistic_sgd(
    X_train, y_train,
    alpha=best_alpha,
    penalty=best_penalty,
    random_state=RANDOM_STATE
)

print("Best alpha:", best_alpha)
print("Best penalty:", best_penalty)

Best alpha: 0.0001
Best penalty: l2

# We now evaluate on the test set

test_prob = best_model.predict_proba(X_test)[:, 1]
test_metrics_05 = evaluate_binary_classifier(y_test, test_prob, threshold=0.5)
test_metrics_05

{'accuracy': 0.7666666666666667,
 'log_loss': 0.552779809308781,
 'roc_auc': np.float64(0.70889613656924),
 'precision': 0.4474885844748858,
 'recall': 0.2996941896024465,
 'f1': 0.358974358974359}

# We can choose a second threshold and compare

SECOND_THRESHOLD = 0.2   # edit this after inspecting results
test_metrics_alt = evaluate_binary_classifier(y_test, test_prob, threshold=SECOND_THRESHOLD)

comparison_df = pd.DataFrame(
    [test_metrics_05, test_metrics_alt],
    index=["threshold=0.5", f"threshold={SECOND_THRESHOLD}"]
)
comparison_df

	accuracy	log_loss	roc_auc	precision	recall	f1
threshold=0.5	0.766667	0.55278	0.708896	0.447489	0.299694	0.358974
threshold=0.2	0.677333	0.55278	0.708896	0.351044	0.565749	0.433255

# confusion matrix for threshold 0.5
y_test_pred = (test_prob >= 0.5).astype(int)
cm = confusion_matrix(y_test, y_test_pred)
cm

array([[1052,  121],
       [ 229,   98]])

# We can see the result of using logistic regression on only the numeric features

X_train_num_only = sparse.csr_matrix(X_train_num)
X_valid_num_only = sparse.csr_matrix(X_valid_num)
X_test_num_only = sparse.csr_matrix(X_test_num)

num_only_model = fit_logistic_sgd(
    X_train_num_only, y_train,
    alpha=best_alpha,
    penalty=best_penalty,
    random_state=RANDOM_STATE
)

num_only_valid_prob = num_only_model.predict_proba(X_valid_num_only)[:, 1]
num_only_metrics = evaluate_binary_classifier(y_valid, num_only_valid_prob, threshold=0.5)
print('Numeric only Metrics:')
print(num_only_metrics)

Numeric only Metrics:
{'accuracy': 0.78, 'log_loss': 0.5059201915895021, 'roc_auc': np.float64(0.6459785752934321), 'precision': 0.4642857142857143, 'recall': 0.039634146341463415, 'f1': 0.07303370786516854}

# The final comparison table

comparison_rows = []

# Baseline B on validation
baseline_B_metrics_row = dict(baseline_B_metrics)
baseline_B_metrics_row["model"] = "Constant probability baseline"
comparison_rows.append(baseline_B_metrics_row)

# Numeric-only logistic regression on validation
num_only_metrics_row = dict(num_only_metrics)
num_only_metrics_row["model"] = "Numeric-only logistic regression"
comparison_rows.append(num_only_metrics_row)

# Full model on validation
full_valid_prob = best_model.predict_proba(X_valid)[:, 1]
full_valid_metrics = evaluate_binary_classifier(y_valid, full_valid_prob, threshold=0.5)
full_valid_metrics["model"] = "Numeric + hashed categorical logistic regression"
comparison_rows.append(full_valid_metrics)

summary_df = pd.DataFrame(comparison_rows)
summary_df = summary_df[
    ["model", "accuracy", "log_loss", "roc_auc", "precision", "recall", "f1"]
].sort_values("log_loss")
summary_df

	model	accuracy	log_loss	roc_auc	precision	recall	f1
1	Numeric-only logistic regression	0.780000	0.505920	0.645979	0.464286	0.039634	0.073034
0	Constant probability baseline	0.781333	0.525216	0.500000	0.000000	0.000000	0.000000
2	Numeric + hashed categorical logistic regression	0.756667	0.577680	0.684719	0.409756	0.256098	0.315197

Project: Logistic Regression on Massive Datasets (100 points)

In the discussion above, we worked on a small subset of 10,000 points from a very large dataset with 100's of millions of observations. In this project, you are asked to train and test your predictor on the full dataset. When working on such large datasets, one has to understand that, since the dataset size runs into GBs, we cannot hope to run vanilla batch versions of the training procedures we discussed above, which assume the entire dataset fits into memory. In this project, you are asked to develop an out-of-core logistic regression method that trains a little at a time on data "chunks", instead of loading the full dataset into memory (from secondary storage) at once. Note that the chunksize parameter in pandas read_csv allows you to read chunks of data at a time. There are several important issues you need to revisit when working in such a "streaming" manner: (i) scaling numerical features, (ii) finding optimal parameters, (iii) feature hashing for categorical features, (iv) creating train:validation:test split. For this project, you are asked to implement the following functions:

• create_split: This function must read data in chunks and randomly split every chunk into train:validation:test parts in ratio (70:15:15) and keep writing these three parts into three separate files. At the end of the procedure, your function must create three files: training, test, and validation sets.
• out_of_core_training: This function performs the training using the training and validation sets.
• Evaluate_predictor: This should produce all the evaluation metrics for the test set.

You are allowed to use the appropriate functions/classes from the sklearn library. You must give reasoning for each and every step of your training procedure and for your choice of off-the-shelf library functions. The key issue you must focus on is how to implement the ideas discussed in this lab in a streaming manner.

Dataset: You can download the entire dataset (~11.15GB) from the following website. Please download the "Kaggle Display Advertising Dataset." You will find two files: (i) train.txt, (ii) test.txt. Please use "train.txt" for this project.