Gaussian Mixtures & Expectation-Maximization

Question 1

Compared with k-means, a Gaussian mixture model additionally recovers each cluster's:

Accepted Answer

Size, shape, orientation and relative weight — k-means finds only the centres. EM finds the full covariance and the mixture weight for each component.

Question 2

The two steps of expectation-maximization are:

Accepted Answer

Estimate the responsibilities, then re-estimate the clusters weighted by them — The E step computes probabilities from the current parameters; the M step updates each cluster using all the data, weighted by those probabilities.

Question 3

The probability that an instance belongs to a given cluster is called that cluster's:

Accepted Answer

Responsibility for the instance — During the maximization step, each cluster's update is dominated by the instances it is most responsible for.

Question 4

Unlike k-means, EM assigns instances:

Accepted Answer

Softly, with a probability of belonging to each cluster — A point can be 70% one cluster and 30% another, and the M step weights it into both accordingly.

Question 5

Géron sets n_init=10 when creating a GaussianMixture because the default is:

Accepted Answer

1, and EM can converge to a poor solution — Just like k-means, EM needs several runs with only the best kept. The book flags the default in a warning.

Question 6

With covariance_type="spherical", clusters must be:

Accepted Answer

Spherical, but they may have different diameters — Different variances are allowed; different shapes are not.

Question 7

covariance_type="diag" restricts clusters to ellipsoids whose:

Accepted Answer

Axes must be parallel to the coordinate axes — The covariance matrices must be diagonal, so no cluster can tilt.

Question 8

covariance_type="tied" means that all clusters:

Accepted Answer

Share the same shape, size and orientation — They share one covariance matrix. The default, "full", lets each cluster take any shape, size and orientation.

Question 9

Why do the covariance constraints exist at all?

Accepted Answer

Because EM can struggle to converge with many dimensions, clusters or few instances — Constraining the covariances reduces the number of parameters EM has to learn, which is the practical fix.

Question 10

Training complexity with covariance_type "tied" or "full" is:

Accepted Answer

O(kmn² + kn³), so it will not scale to many features — The spherical and diagonal cases are O(kmn), assuming the data has a clustering structure.

A real EM loop

Expectation-Maximization

Test your knowledge

Compared with k-means, a Gaussian mixture model additionally recovers each cluster's:

Learn these first

Interview Questions

References & Further Reading

Love this visualization?