Most people explain VAEs as "autoencoders with a twist." That framing gets the architecture right and the intuition completely wrong.
A VAE is not a better autoencoder. It is a generative model — a probabilistic machine that learns where your data lives in the universe of all possible data. The autoencoder part is almost incidental. It is the tool we reach for because the real thing we want is mathematically intractable, and this is our way around that.
Let me show you what the real thing is, why it is intractable, and how VAEs get there anyway.
Before VAEs, before encoders, there is one question: what is the probability of this data point?
Call the data point x. It could be an image, a sentence, an audio waveform. If I could write down p(x) — the true probability distribution over all data — I could do everything: generate new samples, detect anomalies, compress. The whole game is learning p(x) from data.
The problem is that p(x) is wildly complex. An image is 512×512×3 pixels. The distribution over that space has billions of dimensions and impossible geometry. A simple Gaussian will not cut it. Nothing simple will.
Fig. 1 — We want p_theta to match p_data. Training is the process of closing that gap.
So here is the idea every latent variable model starts from. What if we could explain the complexity of x through something simpler — something hidden?
Before we go any further, we need to understand why we model p(x) the way we do — and specifically, what we mean when we say we "train a model on data."
You have a dataset drawn from some real distribution p_data. You have a model p_theta you control. You want p_theta to be close to p_data. But how do you measure closeness between two distributions?
The answer is KL divergence. It measures how different one distribution is from another:
When this is zero, the two distributions are identical. The larger it is, the more different they are. So training should minimize D_KL(p_data || p_theta).
Here is the key observation: expanding that KL divergence gives you:
The first term is the entropy of p_data — it does not depend on theta at all. So minimizing the KL divergence with respect to theta is exactly the same as maximizing:
Which, if you approximate the expectation with your training data, is the average log-likelihood. Maximum likelihood estimation is just KL minimization in disguise. When you train a neural network to maximize log p_theta(x) on your dataset, you are pulling p_theta toward p_data.
Fig. 2 — Minimizing KL divergence between p_data and p_theta is mathematically equivalent to maximum likelihood training.
This is why we care about p(x). It is not an arbitrary choice. It is the most principled way to ask: "how well does my model explain the data I actually observe?"
A note on direction: D_KL(p_data || p_theta) is the "forward" KL, and minimizing it makes p_theta cover all the modes of p_data — it is mode-covering. The "reverse" KL, D_KL(p_theta || p_data), is mode-seeking and behaves quite differently. Maximum likelihood uses the forward direction. VAEs, as we will see, use the reverse KL in the ELBO to regularize the encoder.
Take a dataset of face images. Some people are smiling. Some are not. Some faces are turned left, some right. Some have glasses.
These are the real axes of variation in your data. But your raw pixels do not know that. A pixel at position (304, 217) has no idea it is part of an eye.
What if you introduced hidden variables z that did capture these things? Pose, expression, identity — abstract factors of variation that, once you know them, make the image far easier to describe.
The key insight: conditioning on z is clustering.
When you ask "given z, what does x look like?" — you are asking what a face looks like given that it is a frontal shot of a smiling woman with glasses. That is a very narrow cluster of images. The distribution inside that cluster is much simpler than the distribution over all images.
Fig. 3 — Conditioning on z turns a complex global distribution into many simple local ones. Each cluster can be modeled with a Gaussian.
In the limit, p(x|z) can be a simple Gaussian — just a mean and a covariance. That sounds absurdly reductive. It is not, because z is doing all the heavy lifting. The Gaussian is not trying to describe all possible faces. It is describing the small blob of variation around one specific configuration. Condition well enough, and even a Gaussian is enough.
This is the architecture of a latent variable model. z captures high-level structure. p(x|z) only needs to describe the residual variation once you know z. p(x) — the full distribution we actually want — comes from marginalizing over all possible z.
Early latent variable models like Mixture of Gaussians made z discrete. You had K clusters, and z just told you which one a data point belonged to. For images, you would need K to be astronomically large. The approach breaks down.
VAEs make z continuous. Instead of K clusters, you have an infinite mixture. z is sampled from a standard Gaussian:
The identity covariance I means each dimension of z is independent, with variance 1. This is the prior — the default distribution you sample from before seeing any data. It is fixed. You do not learn it.
Once you have z, you pass it through two neural networks that output the mean and covariance of another Gaussian:
All the learning is in θ. The networks learn to map z to the right distribution for that particular z. p(x) is now an integral over all z — an infinite mixture of Gaussians. Each Gaussian is simple. Their combination is extraordinarily flexible.
To evaluate p(x) for a single data point, you need:
For every possible z, weight how likely x is given that z by how likely z is a priori, and integrate. In high dimensions, you cannot do this analytically. Naïve Monte Carlo fails too.
Why? You sample random z from the prior p(z) and evaluate p(x,z) at each. But in high dimensions, almost every z you sample will give p(x|z) ≈ 0 — that z corresponds to a completely different kind of data than x. The only z values that matter are the ones that actually explain x. Sampling uniformly from the prior, you almost never hit those.
The needle-in-a-haystack problem. You know the needle exists. The prior has no idea where it is.
What you actually want to sample from is p(z|x) — the posterior. Given this specific x, which z values likely generated it? But computing p(z|x) requires p(x) via Bayes' rule. You need the thing you are trying to compute. That is the core difficulty.
Since you cannot compute log p(x) directly, you find something computable that is always below it, then maximize that. If the lower bound goes up, log p(x) goes with it.
Introduce any distribution q(z) over the latent variables. By Jensen's inequality — the log of an expectation is at least the expectation of the log — you get:
The right side is the Evidence Lower Bound (ELBO). You estimate it via Monte Carlo: sample some z from q, evaluate the expression, average. That is just forward passes and log evaluations.
The bound becomes an equality when q(z) = p(z|x). The closer q is to the true posterior, the tighter the bound — and the better the ELBO tracks the actual log likelihood.
Fig. 4 — The ELBO is always below log p(x). The gap is exactly the KL divergence between q and the true posterior p(z|x). Improving q tightens the bound from below.
Rewrite the ELBO and you see something cleaner:
Two terms. The first is reconstruction quality: how well can the decoder recover x from z sampled by q? The second is a regularizer: how far is q from the prior p(z)?
These two terms pull against each other. The reconstruction term wants q to encode x precisely. The KL term wants q to look like a standard Gaussian regardless of x. Their tension shapes the entire latent space.
The KL term pushing q(z|x) toward p(z) does not just prevent overfitting. It makes generation possible.
During training, you get z by running x through the encoder. The encoder knows x, so it produces a z tuned to reconstruct x well. Fine.
At generation time, you have no x. You want a new data point. So you sample z from p(z) — the standard Gaussian prior — and run it through the decoder.
If the encoder was free to place z anywhere in latent space during training, the decoder would only learn regions the training data happened to land in. The prior p(z) and the actual distribution of training z's would be completely misaligned. Sample from the prior at test time, land in territory the decoder has never seen, and you get noise.
The KL term prevents this. It forces the distribution of z's during training to stay close to the prior. At test time, sampling from the prior lands in familiar territory.
This is what makes the VAE a generative model. Without the KL term, you have a stochastic autoencoder with no clean way to sample.
One problem remains. The ELBO involves an expectation over z ~ q(z|x; φ), and we need gradients with respect to φ — the encoder parameters — to train the encoder. But you cannot backpropagate through a sampling operation. Sampling is stochastic. There is no gradient.
The fix: instead of sampling z directly from N(μ, σ²), sample ε from a fixed standard Gaussian, then compute:
The randomness is in ε, which has no parameters. The dependence on φ is in the deterministic transformation. Gradients flow through μ and σ without issue. Separate the noise from the parameters — that is the whole trick.
Here is the full picture. An encoder that maps x to a distribution over z. A reparameterization step that separates noise from parameters. A decoder that maps z back to x. And a loss that balances reconstruction against KL regularization.
Fig. 5 — The full VAE. Encoder maps x to μ and σ. Reparameterization samples z without blocking gradients. Decoder reconstructs x. The loss has two terms that pull against each other.
In principle, each data point x_i has its own optimal variational parameters — the best q(z|x_i) for that specific point. Optimizing per-point does not scale and does not generalize to new data.
VAEs instead learn a single encoder network that maps any x to the parameters of q(z|x). One forward pass replaces one optimization problem. This is amortized inference: you pay a fixed cost upfront (training the encoder) and inference on new points becomes cheap.
It also gives you a bonus. The generative model and the inference model are co-trained. They are consistent. After training, you can run the encoder on unseen data and get a good approximate posterior immediately — the decoder already knows what z values from this encoder look like.
VAEs produce blurry samples. This is not an implementation failure. It is a consequence of the Gaussian decoder: when the decoder outputs a Gaussian, the maximum likelihood solution for a set of possible x values is their average, and averages of images are blurry images.
You can push back on this with better decoders, flow-based priors, or hierarchical VAEs. But the underlying tension is real. VAEs trade sample sharpness for a principled latent space and tractable likelihood bounds.
For anomaly detection, interpolation, semi-supervised learning, and latent space exploration — VAEs are still one of the cleanest tools available. For photorealistic generation, they are not the right choice.
Knowing when to use what is most of the work.
The one-paragraph summary: We want p_theta to match p_data. Maximum likelihood training is exactly KL minimization in disguise — it pulls our model toward the true data distribution. But p(x) under a latent variable model is intractable to compute. We introduce a variational distribution q(z|x) and maximize the ELBO — a tractable lower bound on log p(x) — instead. The ELBO has two terms: a reconstruction term that makes q a good encoder, and a KL term that keeps the latent space Gaussian so we can generate freely. The reparameterization trick makes the whole thing differentiable. The encoder is amortized across all data points. That is a VAE.