iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

Linear Regression

By · Reviewed & last updated · Part of Machine Learning Tutorial

Linear regression fits y = wx + b. Logistic regression squashes the same linear combination through a sigmoid for binary classification. Both are the right baseline for almost any tabular ML problem.

Fit, evaluate, regularise

EXAMPLE
from sklearn.linear_model    import LinearRegression, Ridge, Lasso, LogisticRegression
from sklearn.preprocessing   import StandardScaler
from sklearn.pipeline        import Pipeline
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics         import r2_score, mean_squared_error, classification_report
import numpy as np

# === Regression ===
# 1) Plain OLS
from sklearn.datasets import fetch_california_housing
X, y = fetch_california_housing(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=0)

lr = LinearRegression().fit(Xtr, ytr)
print(f'R²:   {r2_score(yte, lr.predict(Xte)):.3f}')
print(f'RMSE: {np.sqrt(mean_squared_error(yte, lr.predict(Xte))):.3f}')
print(dict(zip(['feat'+str(i) for i in range(X.shape[1])], lr.coef_.round(3))))

# 2) Ridge — L2 regularisation (default for noisy data)
ridge = Pipeline([
    ('scaler', StandardScaler()),
    ('model',  Ridge(alpha=1.0)),
]).fit(Xtr, ytr)
print(f'Ridge R²: {ridge.score(Xte, yte):.3f}')

# 3) Lasso — L1, sets some weights to 0 (feature selection)
lasso = Pipeline([
    ('scaler', StandardScaler()),
    ('model',  Lasso(alpha=0.01)),
]).fit(Xtr, ytr)

# === Classification ===
from sklearn.datasets import load_breast_cancer
Xc, yc = load_breast_cancer(return_X_y=True)
Xctr, Xcte, yctr, ycte = train_test_split(Xc, yc, test_size=0.2, stratify=yc, random_state=0)

logreg = Pipeline([
    ('scaler', StandardScaler()),
    ('model',  LogisticRegression(C=1.0, max_iter=500, class_weight='balanced')),
]).fit(Xctr, yctr)

print(classification_report(ycte, logreg.predict(Xcte)))
print('CV F1:', cross_val_score(logreg, Xc, yc, cv=5, scoring='f1_macro').mean().round(3))

# 4) Probability calibration (when probs matter)
from sklearn.calibration import CalibratedClassifierCV
calibrated = CalibratedClassifierCV(logreg, cv=5).fit(Xctr, yctr)
probs = calibrated.predict_proba(Xcte)[:, 1]

# 5) Coefficient interpretation
import pandas as pd
breast = load_breast_cancer(as_frame=True).frame
coefs = pd.Series(logreg[1].coef_[0], index=breast.columns[:-1]).sort_values()
print(coefs.tail(5))     # top 5 features pushing predictions toward malignant

Why it matters

Always START with linear / logistic regression. They’re fast, interpretable, robust, often state-of-the-art on small / wide tabular data. Move to trees / GBDT / NNs only when you can’t beat the linear baseline.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(Xtr, ytr)
print(model.coef_, model.intercept_)
print('R^2:', model.score(Xte, yte))
Try it Yourself »

Discussion

Loading…