iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

K-Nearest Neighbours

By · Reviewed & last updated · Part of Machine Learning Tutorial

k-Nearest Neighbours classifies a new point by majority vote of its k closest training points. No model is fitted — the training data IS the model. Simple, surprisingly strong, but slow at prediction time and sensitive to feature scaling.

Train, tune, scale, evaluate

EXAMPLE
import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score
from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report, ConfusionMatrixDisplay

# 1) Load data — split
data = load_wine()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target,
    test_size=0.2, stratify=data.target, random_state=42,
)

# 2) ALWAYS scale features — distances dominate otherwise
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('knn',    KNeighborsClassifier(n_neighbors=5, weights='distance')),
])

# 3) Cross-validated baseline
scores = cross_val_score(pipe, X_train, y_train, cv=5)
print('cv acc:', scores.mean(), '±', scores.std())

# 4) Tune k + weighting via grid search
param_grid = {
    'knn__n_neighbors': list(range(1, 31, 2)),
    'knn__weights':     ['uniform', 'distance'],
    'knn__p':           [1, 2],   # 1=manhattan, 2=euclidean
}
grid = GridSearchCV(pipe, param_grid, cv=5, n_jobs=-1)
grid.fit(X_train, y_train)
print(grid.best_params_, grid.best_score_)

# 5) Evaluate on the held-out test set
y_pred = grid.predict(X_test)
print(classification_report(y_test, y_pred, target_names=data.target_names))
ConfusionMatrixDisplay.from_estimator(grid, X_test, y_test)

# 6) KNN for regression — same logic, average instead of vote
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_absolute_error
hx, hy = fetch_california_housing(return_X_y=True)
rx_tr, rx_te, ry_tr, ry_te = train_test_split(hx, hy, test_size=0.2, random_state=42)
reg = Pipeline([
    ('scaler', StandardScaler()),
    ('knn',    KNeighborsRegressor(n_neighbors=10, weights='distance')),
]).fit(rx_tr, ry_tr)
print('MAE:', mean_absolute_error(ry_te, reg.predict(rx_te)))

# 7) Why KNN can fall over
#   • SLOW prediction — O(N * d) per query; use ball_tree / kd_tree / 'brute'
#   • Curse of dimensionality — every point looks equidistant in high-d
#   • Class imbalance — use weights='distance' or SMOTE upstream
#   • Memory — full training set sits in RAM forever

# 8) Approximate nearest neighbours scale further
# pip install faiss-cpu / annoy / hnswlib
# Index millions of points, sub-millisecond queries.

Why it matters

KNN is the “reach for it first” baseline when you have low-d, scaled features and don’t need fast inference. If it beats the fancy model you were building, you saved a week.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
from sklearn.neighbors import KNeighborsClassifier
clf = KNeighborsClassifier(n_neighbors=5, weights='distance').fit(Xtr, ytr)
print(clf.score(Xte, yte))
Try it Yourself »

Discussion

Loading…