K-Nearest Neighbors (KNN) 3D

Question 1

How does KNN classify a new data point?

Accepted Answer

By majority vote of its K nearest neighbors — KNN looks at the K closest training points and assigns the class that is most common among them (majority vote).

Question 2

What happens if K is too small (e.g., K=1)?

Accepted Answer

The model is sensitive to noise (overfitting) — A very small K (like K=1) makes the model highly sensitive to noise and outliers, leading to high variance (overfitting).

Question 3

K-Nearest Neighbors classifies a new point by…

Accepted Answer

A majority vote among k closest points — KNN finds the k nearest labeled examples and assigns the most common class among them (or averages for regression).

Question 4

KNN is called a 'lazy' learner because it…

Accepted Answer

Does no real training — There's no model-fitting phase; all computation (distance to every point) is deferred until a query arrives, making prediction expensive.

Question 5

What is the prediction-time cost of naive KNN with n training points?

Accepted Answer

A linear O(n) cost comparing every stored example — It compares the query to all n stored points, so each prediction is O(n·d). Spatial structures (KD-trees, ball trees) speed this up in low dimensions.

Question 6

Choosing k too SMALL (e.g. k=1) tends to cause…

Accepted Answer

Overfitting — k=1 follows individual (possibly noisy) points exactly: low bias, high variance. Increasing k smooths the decision boundary.

Question 7

Choosing k too LARGE tends to cause…

Accepted Answer

Underfitting — Very large k averages over too many points, blurring class structure: high bias, low variance. k tunes the bias-variance trade-off.

Question 8

Why is FEATURE SCALING crucial for KNN?

Accepted Answer

Large-range features dominate the distance calculation — Euclidean distance lets a feature with a big numeric range overwhelm others; standardizing/normalizing ensures each feature contributes fairly.

Question 9

The 'curse of dimensionality' hurts KNN because in high dimensions…

Accepted Answer

All pairwise distances converge to roughly equal values — As dimensions grow, distances concentrate and neighborhoods become sparse, degrading KNN — motivating dimensionality reduction (e.g. PCA) first.

Question 10

Which distance metric is the default for continuous KNN?

Accepted Answer

Euclidean (L2), standard for real-valued feature data — Euclidean distance is standard; Manhattan (L1), cosine, or Hamming may suit specific data (e.g. cosine for text embeddings).

KNN Parameters

Test Point Location

Display

Classify one point with KNN

Test your knowledge

How does KNN classify a new data point?

Interview Questions

References & Further Reading

Love this visualization?