Summary

# ===== 1) Frame the problem =====
# - What decision will this drive?
# - What is the unit of decision (per user, per request, per day)?
# - What is the success metric the business measures?
# - What metric tracks that ML-side (proxy for the business)?

# ===== 2) Data =====
# - Where does the label come from? (system event, manual labels, weak supervision)
# - How fresh is the label? Future leakage hides here.
# - Is the dataset balanced? Imbalanced changes the metric.
# - Are there groups (users, devices, locations) that must not split across train/test?

# ===== 3) Ship a baseline FIRST =====
# - Always-majority class (binary)
# - Mean / median (regression)
# - Naive seasonality (time series)
# - Heuristic rules from the business team
# Any model worth shipping must beat the baseline on the agreed metric.

# ===== 4) Pick the right split =====
# - i.i.d.:         random stratified KFold
# - grouped:        GroupKFold (split by user / device)
# - time-ordered:   TimeSeriesSplit (NEVER random for time-shaped data)
# - tiny dataset:   nested CV or repeated KFold
# Lock the test set BEFORE any tuning; never peek.

# ===== 5) Model choice cheat sheet =====
# Tabular < 10k rows:   logistic / decision tree / random forest
# Tabular > 10k:        XGBoost / LightGBM / CatBoost
# Text < 50k:           TF-IDF + logistic
# Text large:           fine-tune small transformer (DistilBERT)
# Image:                transfer-learn from a pretrained CNN/ViT
# Time series:          Holt-Winters, ARIMA, Prophet, or XGBoost on lags
# Tiny labelled set:    few-shot LLM, k-NN, rule heuristic

# ===== 6) Pick the right metric =====
# - balanced binary:   ROC-AUC + accuracy
# - imbalanced:        PR-AUC + F1 of minority + precision @ chosen recall
# - multiclass:        macro-F1
# - regression:        MAE (outliers matter) / RMSE (else)
# - ranking:           NDCG@k / MAP@k
# Always report a confidence interval (bootstrap).

# ===== 7) Leakage red flags =====
# - Feature derived AFTER the target
# - Per-user features fit on the full dataset, not per fold
# - Test-set rows present in training under another id
# - Time-shaped data split randomly
# - Target encoding without holdout

# ===== 8) Validation strategy =====
# - Cross validate WITHIN train
# - Final test ONCE
# - For prod, retrain quarterly or when drift detected
# - Hold a TIME-FORWARD validation slice if the world changes

# ===== 9) Production handoff =====
# - Wrap preprocessing + model in a single artifact (sklearn Pipeline / ONNX)
# - Version the artifact (semver + git SHA)
# - Log inputs + predictions for monitoring
# - Drift monitor: KS test or PSI on numeric features daily
# - Shadow-deploy before traffic flip; A/B vs the previous model

# ===== 10) Common pitfalls =====
# - No baseline -> you cannot tell if the model is helping
# - Optimising the wrong metric -> ROC-AUC looks great while precision is awful
# - Leakage -> notebook shines, production crashes
# - Model swap without metric -> 'just tried XGBoost' is not a result
# - Forgotten Postgres feature -> serving DB returns nulls that training never saw

# ===== 11) Tools =====
# - sklearn        for tabular pipelines + cross-validation
# - XGBoost        for tabular SOTA
# - PyTorch / TF   for deep learning
# - mlflow / W&B   for tracking
# - DVC            for versioning datasets
# - feast          for online/offline feature parity
# - WhyLabs / Evidently / Soda for drift monitoring

# Next: deep learning, time-series forecasting, MLOps deep dive, RecSys.