From K-means we know that:

  • K-means forces clusters to be spherical
  • In K-means clustering every point can only belong to one cluster

But sometimes it might be desirable to have elliptical clusters than spherical clusters. And what if there is a data point right in the center of two clusters?

Gaußian Mixture Model

With a random variable XX, the mixed Gaussian model can be expressed by:

p(x)=k=1KπkN(Xμk,Σk)p(x)=\sum_{k=1}^{K} \pi_{k} \mathcal{N}\left(X | \mu_{k}, \Sigma_{k}\right)

where N(Xμk,Σk)\mathcal{N}\left(X | \mu_{k}, \Sigma_{k}\right) is the kthk^{th} component of the mixture model.

Generalized Form

Then we can generate a generalized form:

p(XM,Σ,π)=i=1sk=1KπkN(xiμk,Σk)p(X | M, \Sigma, \pi)=\prod_{i=1}^{s} \sum_{k=1}^{K} \pi_{k} \mathcal{N}\left(x_{i} | \mu_{k}, \Sigma_{k}\right)

forN(xiμk,Σk)=1(2π)ndet(Σk)e12Σk1(xiμk),xiμk\text{for} \quad \mathcal{N}\left(x_{i} | \mu_{k}, \Sigma_{k}\right)=\frac{1}{\sqrt{(2 \pi)^{n} \operatorname{det}\left(\Sigma_{k}\right)}} e^{-\frac{1}{2}\left\langle\Sigma_{k}^{-1}\left(x_{i}-\mu_{k}\right), x_{i}-\mu_{k}\right\rangle}


X=(x1x2xs)Rn×s input data X=\left( \begin{array}{llll}{x_{1}} & {x_{2}} & {\dots} & {x_{s}}\end{array}\right) \in \mathbb{R}^{n \times s} \quad \text { input data }

M=(μ1μ2μK)Rn×K prototype vectors M=\left( \begin{array}{llll}{\mu_{1}} & {\mu_{2}} & {\dots} & {\mu_{K}}\end{array}\right) \in \mathbb{R}^{n \times K} \quad \text { prototype vectors }

Σ=(Σ1Σ2ΣK)Rn×n×K covariance matrices \Sigma=\left(\Sigma_{1} \quad \Sigma_{2} \quad \ldots \quad \Sigma_{K}\right) \in \mathbb{R}^{n \times n \times K} \quad \text { covariance matrices }

π=(π1π2πK)RK×1 mixing weights\pi=\left( \begin{array}{llll}{\pi_{1}} & {\pi_{2}} & {\dots} & {\pi_{K}}\end{array}\right) \in \mathbb{R}^{K \times 1} \quad \text { mixing weights}

 and πk0 for all k{1,,K} as well as k=1Kπk=1\text { and } \quad \pi_{k} \geq 0 \quad \text { for all } \quad k \in\{1, \ldots, K\} \quad \text { as well as } \sum_{k=1}^{K} \pi_{k}=1

Now the goal for the algorithm is: given XX , determine the parameters MM, ΣΣ and ππ (for example by maximizing the likelihood)

Model Iteration Illustration

gaussian mixture model