Introduction to the Dirichlet Distribution

The Dirichlet distribution is a fundamental probability distribution in statistics, particularly in Bayesian inference where it serves as the conjugate prior for the multinomial distribution.

In this post, we explore its definition, geometry, and behavior using interactive simulations.

What is a "Random Distribution"?

To understand the Dirichlet, it helps to start with an analogy.

Imagine a standard six-sided die. Rolling it produces a number from 1 to 6. But physical dice are never perfectly fair; tiny manufacturing imperfections mean the probabilities of each side aren't exactly $1/6$.

Now, imagine a bag of 100 dice.

• A bag of crude, handmade dice from 100 years ago might have wildly different biases.
• A bag of precision casino dice will have probabilities very close to uniform.

If you reach into the bag and pull out a die, you are drawing a random probability mass function (pmf). The Dirichlet distribution is a way to model this randomness—it is a distribution over pmfs.

Definition and Geometry

A probability mass function $q = [q_1, ..., q_k]$ has two constraints:

1. All probabilities are non-negative: $q_i \ge 0$.
2. They sum to one: $\sum_{i=1}^k q_i = 1$.

Geometrically, these constraints restrict $q$ to a shape called the $(k-1)$-dimensional probability simplex, denoted $\Delta_k$.

For $k=3$ (e.g., a 3-sided die), the simplex is an equilateral triangle living in 3D space. Every point inside this triangle represents a valid set of probabilities $(q_1, q_2, q_3)$.

• The vertices represent deterministic outcomes (e.g., $q=[1,0,0]$).
• The center represents the uniform distribution $q=[1/3, 1/3, 1/3]$.

The Density Function

A random vector $Q$ follows a Dirichlet distribution with parameter vector $\alpha = [\alpha_1, ..., \alpha_k]$ (where $\alpha_i > 0$) if its probability density is given by:

Visualizing the Density (Figure 1)

The parameter $\alpha$ controls the shape of the distribution. The simulation below replicates Figure 1 from our reference paper. It shows the "heat" (probability density) on the simplex triangle for different values of $\alpha$.

• Red indicates high density (high probability of drawing such a pmf).
• Blue indicates low density.

Try the different presets to see the behavior described in the text:

• $\alpha = [1, 1, 1]$: The density is perfectly flat (uniform). Every valid pmf is equally likely.
• $\alpha = [10, 10, 10]$: The density concentrates in the center. This models the "precision casino dice"—most dice drawn from this bag will be nearly fair.
• $\alpha = [0.1, 0.1, 0.1]$: The density explodes at the corners (vertices). This models a bag of "trick dice"—most dice will almost always roll a 1, or almost always a 2, etc., but rarely a mix.
• $\alpha = [2, 5, 15]$: The distribution is asymmetric, pulling the mass toward the third component ($q_3$), which corresponds to the corner for $\alpha_3=15$.

Generating Samples (Figure 2)

We can also visualize the distribution by drawing random samples from it. Each blue dot in the simulation below represents one random pmf $Q$ drawn from $Dir(\alpha)$. This replicates Figure 2 from the text.

How to Sample?

A robust way to generate these samples is using Gamma random variables. If we draw $k$ independent values $z_i \sim \text{Gamma}(\alpha_i, 1)$, then the normalized vector:

follows a Dirichlet distribution.

Properties

• Expectation: The mean of the distribution is the normalized parameter vector: $E[Q_i] = \alpha_i / \sum \alpha_j$.
• Aggregation: If you sum two components (e.g., $Q_1 + Q_2$), the resulting distribution is still Dirichlet (with parameters $\alpha_1 + \alpha_2$). This "fractal-like" property makes the Dirichlet distribution incredibly useful for consistent modeling across different levels of granularity.