Exercises

# ============================================================
# Drill 1 — Pick the train/test split
# ============================================================
# DATA: 500k rows of customer events. Goal: predict churn at month end.
# QUESTION: random KFold, GroupKFold, or TimeSeriesSplit?
#
# ANSWER: time + user.
# - Random KFold leaks future into the past AND a single customer into both folds.
# - GroupKFold alone fixes the customer leak but still time-leaks.
# - TimeSeriesSplit fixes time but a single customer can appear in train AND test.
#
# Solution: split by date FIRST, then enforce one customer per fold by group.

from sklearn.model_selection import GroupShuffleSplit

# Define training period < cutoff; testing strictly after
df['set'] = np.where(df['date'] < '2026-04-01', 'train', 'test')

# Within train, hold out the most recent month for validation
df.loc[(df['set'] == 'train') & (df['date'] >= '2026-03-01'), 'set'] = 'val'

# ============================================================
# Drill 2 — Ship a baseline FIRST
# ============================================================
# QUESTION: what is the dumb baseline for this churn problem?
#
# ANSWER: predict the majority class for everyone -> compute accuracy + PR-AUC.
# If the imbalance is 95% retained, accuracy is 95% with zero work.
# Any model worth shipping must beat that on PR-AUC, not on accuracy.

# Concrete code:
from sklearn.metrics import classification_report, average_precision_score

majority = y_train.mode()[0]
print(classification_report(y_test, [majority] * len(y_test)))
print("baseline PR-AUC:", average_precision_score(y_test, [0.05] * len(y_test)))

# ============================================================
# Drill 3 — Spot the leakage
# ============================================================
# A teammate's notebook reports val AUC = 0.98 on a churn task. Suspicious.
# QUESTION: list three plausible leakage sources.
#
# ANSWER:
# 1) 'last_login_at' is computed at the snapshot date, AFTER the churn label.
# 2) The target was joined back into the feature frame and StandardScaler was
#    fit on the FULL dataset (leak via mean/std).
# 3) Customers are duplicated across train/val (same id, different sessions).
# Fix: time-aware features only; fit transforms inside the pipeline; group split.

# ============================================================
# Drill 4 — Choose the right metric
# ============================================================
# Scenario: payment fraud detector. 1 fraud per 5000 transactions. False positives
# cause a chargeback escalation; false negatives lose money to the fraudster.
#
# QUESTION: pick a metric the team should optimise on.
#
# ANSWER: precision @ recall = 0.6 (or similar operating point).
# - ROC-AUC is misleading under extreme imbalance (lots of TNs).
# - PR-AUC is better but you still need an operating point for the actual cut-off.
# - Pick the recall the business can afford (60% of fraud caught) and report the
#   precision at that point. Then maximise precision at that recall.

# ============================================================
# Drill 5 — Cross-validation strategy
# ============================================================
# Dataset: 10k rows, balanced classes. 30 features.
#
# QUESTION: 5-fold stratified KFold, 10-fold KFold, or repeated KFold?
#
# ANSWER: 5-fold stratified KFold is enough for 10k rows + a modern model.
# Repeated KFold (5x5) makes sense if hyperparameters look unstable, but it
# multiplies compute. 10-fold reduces variance at the cost of fold size — use
# it only when each fold needs to be representative (rare classes, low n).
#
# Code:
from sklearn.model_selection import StratifiedKFold, cross_val_score
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipe, X, y, cv=cv, scoring='roc_auc', n_jobs=-1)
print(scores.mean(), '+/-', scores.std())

# ============================================================
# Bonus — what is the FIRST thing you do on a new dataset?
# ============================================================
# ANSWER: print head, info, describe, isna sums, target distribution, time range.
# Then sketch the leak model: 'what would I see if leak X existed?'
# THEN train a model.
#
# Skipping the bonus is how a 6-month project ends with a model that beats every
# baseline in the notebook and loses to the dumb baseline in production.

# Fill in: from sklearn.____ import train_test_split