k Nearest Neighbors Regression: Complete Guide

Summary — k nearest neighbors regression is a supervised learning method that predicts a numerical value by averaging the targets of the k observations closest to the point to be predicted. Unlike k-NN classification, where neighbors vote by majority for a class, regression computes a mean (possibly distance-weighted) of the target values. This simple yet powerful approach can model complex relationships without assuming a predefined functional form.

Mathematical Principle

The core idea of k nearest neighbors regression is remarkably straightforward. Given a new example x and a training set D = {(x₁, y₁), …, (xₙ, yₙ)}, where each yᵢ is a continuous value, we identify the k closest observations in D according to a distance metric (typically Euclidean distance).

Uniform Predictions

The prediction ŷ(x) is the arithmetic mean of the targets of the k nearest neighbors:

ŷ(x) = (1/k) × Σ yᵢ

where the sum is over the indices i corresponding to the k nearest neighbors of x.

Weighted Predictions

To give more influence to the closest neighbors, inverse-distance weighting is often used:

ŷ(x) = Σ (wᵢ × yᵢ) / Σ wᵢ

where wᵢ = 1 / d(x, xᵢ) and d(·, ·) is the chosen distance. A very close neighbor will have a very high weight, while a distant neighbor will have a low weight. This variant is particularly useful when local data exhibits abrupt variations.

Distance Metrics

The choice of distance is crucial because it determines which points are considered “similar”:

• Euclidean distance (L2): d(x, x′) = √(Σ (xⱼ − x′ⱼ)²) — the most common, corresponds to natural geometric intuition.
• Manhattan distance (L1): d(x, x′) = Σ |xⱼ − x′ⱼ| — more robust to outliers, useful in high dimensions.
• Minkowski distance (Lp): d(x, x′) = (Σ |xⱼ − x′ⱼ|ᵖ)^(1/p) — parametric generalization; p=2 gives Euclidean, p=1 gives Manhattan.
• Mahalanobis distance: takes correlations between variables into account, but requires inverting the covariance matrix.

It is essential to normalize or standardize variables before applying k-NN. Indeed, a variable with a scale ten times larger will dominate the distance calculation to the detriment of others.

Intuition

Imagine you want to estimate the price of an apartment in a neighborhood you don’t know well. You don’t have a sophisticated economic model, but you know recent sale prices in the area. The k nearest neighbors regression approach consists of finding the k apartments most similar to yours (same size, same number of bedrooms, same neighborhood) and averaging their sale prices. If two apartments are nearly identical to yours and a third is somewhat different, the first two “count” more in your estimation — this is the idea of distance weighting.

What distinguishes k nearest neighbors regression from parametric methods like linear regression is that it assumes no functional form a priori. The relationship between features and the target emerges directly from the data. If the true relationship is sine-wave shaped, k-NN will discover it on its own, provided k is well chosen and there is sufficient training data.

This property makes k-NN a nonparametric and lazy learning method: no learning step is performed in advance. The training data is simply memorized, and all the “work” is deferred to prediction time.

See Also

• Find the Smallest Multiple in Python: Complete Guide and Optimization Tips
• Implement the SHA-1 Hash Algorithm in Python: Step-by-Step Guide

Python Implementation

Installing Dependencies

pip install scikit-learn numpy matplotlib

Example 1: Noisy Sinusoidal Data

Let’s start with a visual example using noisy sinusoidal data:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

# Generate noisy sinusoidal data
np.random.seed(42)
X = np.sort(5 * np.random.rand(200, 1), axis=0)
y = np.sin(X).ravel() + np.random.normal(0, 0.15, X.shape[0])

# Train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

# k-NN with k=5
knn = KNeighborsRegressor(n_neighbors=5, weights="uniform")
knn.fit(X_train, y_train)

# Predictions
y_pred = knn.predict(X_test)
print(f"R² on test: {r2_score(y_test, y_pred):.4f}")
print(f"RMSE on test: {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")

# Visualization
X_plot = np.linspace(0, 5, 500).reshape(-1, 1)
y_true_plot = np.sin(X_plot)
y_pred_plot = knn.predict(X_plot)

plt.figure(figsize=(10, 6))
plt.scatter(X_train, y_train, c="steelblue", alpha=0.5, s=20, label="Training")
plt.scatter(X_test, y_test, c="orange", alpha=0.5, s=20, label="Test")
plt.plot(X_plot, y_true_plot, c="red", linewidth=2, label="True function (sin)")
plt.plot(X_plot, y_pred_plot, c="green", linewidth=2, label="k-NN prediction (k=5)")
plt.xlabel("x")
plt.ylabel("y")
plt.title("k Nearest Neighbors Regression on Sinusoidal Data")
plt.legend()
plt.tight_layout()
plt.show()

The k-NN model captures the sinusoidal shape without any prior knowledge of the sine function — it simply averages local neighbors.

Example 2: Comparison with Linear Regression

The strength of k-NN regression becomes clear when comparing it to a linear model on nonlinear data:

from sklearn.linear_model import LinearRegression

# Same problem
X_lin = np.sort(5 * np.random.rand(200, 1), axis=0)
y_lin = np.sin(X_lin).ravel() + np.random.normal(0, 0.1, X_lin.shape[0])

# k-NN
knn_comp = KNeighborsRegressor(n_neighbors=5, weights="distance")
knn_comp.fit(X_lin, y_lin)

# Linear Regression
lr = LinearRegression()
lr.fit(X_lin, y_lin)

X_plot = np.linspace(0, 5, 500).reshape(-1, 1)

plt.figure(figsize=(10, 6))
plt.scatter(X_lin, y_lin, c="lightgray", s=15, label="Data")
plt.plot(X_plot, np.sin(X_plot), c="red", linewidth=2, label="True curve")
plt.plot(X_plot, lr.predict(X_plot), c="blue", linewidth=2, label="Linear regression")
plt.plot(X_plot, knn_comp.predict(X_plot), c="green", linewidth=2, label="k-NN (k=5, distance)")
plt.xlabel("x")
plt.ylabel("y")
plt.title("k-NN vs Linear Regression on nonlinear data")
plt.legend()
plt.tight_layout()
plt.show()

Linear regression can only draw a straight line: it completely fails to capture the sine wave. k-NN, on the other hand, follows the curve thanks to its nonparametric nature.

Example 3: Impact of Choosing k — Underfitting and Overfitting

The parameter k is the main lever of the bias-variance tradeoff:

k_values = [2, 5, 15, 50]

fig, axes = plt.subplots(2, 2, figsize=(12, 10))
axes = axes.ravel()

for i, k in enumerate(k_values):
    knn_k = KNeighborsRegressor(n_neighbors=k, weights="distance")
    knn_k.fit(X, y)
    y_plot = knn_k.predict(X_plot)

    axes[i].scatter(X, y, c="lightgray", s=10, label="Data")
    axes[i].plot(X_plot, np.sin(X_plot), c="red", linewidth=2, label="True curve", linestyle="--")
    axes[i].plot(X_plot, y_plot, c="navy", linewidth=2, label=f"k={k}")
    axes[i].set_title(f"k = {k}")
    axes[i].legend()
    axes[i].set_xlabel("x")
    axes[i].set_ylabel("y")

fig.suptitle("Impact of k on fit quality — k nearest neighbors regression")
plt.tight_layout()
plt.show()

• k too small (k=2): the curve follows every noise fluctuation — this is overfitting. The model memorizes instead of generalizing.
• k optimal (k=5 to k=15): the prediction follows the sine wave while smoothing the noise — this is the ideal zone.
• k too large (k=50): the curve flattens and tends toward the global mean — this is underfitting. All local structure is lost.

To automate the choice of k, cross-validation is typically used:

from sklearn.model_selection import cross_val_score

k_range = range(1, 51)
cv_scores = []
for k in k_range:
    knn_cv = KNeighborsRegressor(n_neighbors=k, weights="distance")
    scores = cross_val_score(knn_cv, X, y, cv=5, scoring="neg_mean_squared_error")
    cv_scores.append(-scores.mean())

best_k = int(k_range[np.argmin(cv_scores)])
print(f"Best k according to cross-validation: {best_k}")

plt.figure(figsize=(8, 5))
plt.plot(list(k_range), cv_scores, marker="o")
plt.axvline(best_k, c="red", linestyle="--", label=f"Best k={best_k}")
plt.xlabel("k (number of neighbors)")
plt.ylabel("RMSE (cross-validation)")
plt.title("Optimal k selection by cross-validation")
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

Example 4: Multi-dimensional Synthetic Data

k nearest neighbors regression also works with several explanatory variables:

from sklearn.datasets import make_regression
from sklearn.preprocessing import StandardScaler

# Create a multi-dimensional dataset
X_multi, y_multi = make_regression(
    n_samples=1000, n_features=10, n_informative=5,
    noise=25, random_state=42
)

# Standardization — essential for k-NN
scaler = StandardScaler()
X_multi_scaled = scaler.fit_transform(X_multi)

X_tr, X_te, y_tr, y_te = train_test_split(
    X_multi_scaled, y_multi, test_size=0.25, random_state=42
)

# Comparison of uniform vs distance weighting
knn_uniform = KNeighborsRegressor(n_neighbors=7, weights="uniform")
knn_distance = KNeighborsRegressor(n_neighbors=7, weights="distance")

knn_uniform.fit(X_tr, y_tr)
knn_distance.fit(X_tr, y_tr)

r2_u = r2_score(y_te, knn_uniform.predict(X_te))
r2_d = r2_score(y_te, knn_distance.predict(X_te))

print(f"R² uniform:    {r2_u:.4f}")
print(f"R² distance:   {r2_d:.4f}")

Distance weighting often offers a slight gain in predictive quality, as it gives more confidence to the most similar neighbors.