But what is cross-entropy? | Compression is Intelligence Part 2
3Blue1Brown
Zipping languages together 0:00
A 2002 paper called Language Trees and Zipping showed that you can find the family structure of languages using nothing more than ordinary file compression, like gzip. Given a pile of text documents in many languages, with no built-in knowledge of linguistics, you can cluster them by language and even recover which languages are historically related. The trick is to append a small snippet of document B to document A, compress the combined file, and compare that size to compressing A alone. This difference tells you how well B's patterns fit a compressor tuned for A, and it turns out to be small when the languages are similar and large when they are not. The same idea can also be used to identify who wrote a given document, and the concept underlying all of this is cross-entropy.
Two goals for this video 2:02
Cross-entropy is also central to how modern language models are trained, which hints at an unexpected link between that training process and compression. The plan is to first build up cross-entropy from the fundamentals of compression, then separately walk through the basics of pretraining and distilling language models, where cross-entropy seems to appear for a completely different reason. Because the same formula shows up in both places, the goal by the end is to reframe how you think about training large language models, not as next-token prediction, but as a form of compression.
The robot instruction example 3:02
Picture sending a faraway robot instructions to move up, down, left, or right, where up has a 50 percent chance, down a 1 in 4 chance, and left and right each a 1 in 8 chance. The best possible code assigns 1 bit to up, 2 bits to down, and 3 bits each to left and right, matching the general rule that the ideal number of bits for a symbol is the negative log base 2 of its probability, a quantity Claude Shannon called information content. Adding up these information contents across a whole message gives you its total information content, and the optimal encoding length for the message is approximately that total, even when the numbers are not whole.
When the distribution shifts 5:00
Now imagine the space agency changes the mission so that up and down each become 1 in 8 likely, left becomes 1 in 4, and right becomes fully half of all instructions, but your encoder is still hard-coded for the old scheme. Using the old code on the new distribution costs an average of 2.625 bits per message, and this number is the cross-entropy of the original distribution relative to the new one. In general, if you label the new distribution P and the original one Q, an optimal code for Q spends negative log base 2 of Q bits per symbol, and the entropy of Q is the weighted sum of those costs using Q's own probabilities as weights, which you can picture as bars whose width is Q and whose height is negative log base 2 of Q. Cross-entropy is the same kind of sum, but weighted by the new probabilities P instead, so the bars now have width P and height negative log base 2 of Q.
Simple two-outcome examples 9:34
With a distribution over just two events, if Q is a 50-50 split, its entropy is exactly 1 bit, and because every symbol carries the same amount of information under Q, using it on a skewed distribution P still costs 1 bit, so cross-entropy equals entropy here. But if Q is the skewed 90-10 distribution and P is the even 50-50 one, Q's own entropy is less than 1 bit, yet a code optimized for that skew performs worse against a fair split, giving a cross-entropy around 1.74 bits. This shows that order matters: swapping which distribution is Q and which is P changes the answer, because the two play very different roles in the formula.
Cross-entropy is minimized at equality 10:32
Graphing cross-entropy as a function of Q while holding P fixed produces a curve with a clear minimum, and that minimum occurs exactly when Q equals P, which makes sense since a code performs best when it is optimized for the very distribution it is used on. Tracing that minimum as P itself varies produces a new curve, and this traced curve is precisely the entropy of P. The single fact worth remembering is that cross-entropy, viewed with P fixed and Q variable, always reaches its smallest possible value when Q equals P, and that smallest value equals the entropy of P.
Revisiting the zipping trick as cross-entropy 13:04
The gzip language-tree method from the opening is essentially an empirical version of cross-entropy, since comparing the compressed size of A plus a snippet of B against A alone is asking the same question cross-entropy asks: how well does a scheme optimized for one context handle another. It is not a perfect match, since it treats a document as a sample of a language rather than comparing two probability distributions directly, and gzip itself is far from an optimal compressor, being a fairly simple algorithm that replaces repeated sequences with pointers to earlier occurrences. Even so, this rough approximation of cross-entropy is enough to produce a useful distance measure between languages, and this same pattern, quantifying how different one setting's patterns are from another's, is what makes cross-entropy useful for training language models on the true patterns of language found in training data.
How language models are trained 15:32
A modern language model splits text into tokens, usually words or word pieces, and works as a function that takes in a sequence of tokens and outputs a probability distribution over what token could come next. Training requires a loss function that scores whether predictions are good, and while the true loss surface lives in the billions of dimensions matching a model's parameters, gradient descent and backpropagation still let you tune those parameters to walk downhill and shrink the loss. The specific loss used is the average information per token from the model's perspective: for each true next token in the training text, you take the probability the model assigned to it and compute its negative log, so a confident, correct model gets a low loss and a confused model gets punished steeply, and natural log rather than log base 2 is used mainly because it makes the calculus involved in gradient descent cleaner.
Why it is really called cross-entropy loss 20:33
This loss is called cross-entropy loss even though the earlier formula does not obviously appear, and the usual explanation, that you are computing cross-entropy between the model's output and a distribution fully concentrated on the one true token, feels unsatisfying because it does not explain why logarithms are the right choice at all. A more convincing route starts from an example like the phrase my name is blank, where common names appear often in training data and rare names appear less often. If you let Q be the model's predicted probabilities for possible names and P be the actual proportions of names appearing in the data, the total average loss across every occurrence of this pattern becomes a weighted sum, and choosing the per example loss function F to be negative log turns that sum into exactly the cross-entropy of the model's output relative to the data's true statistics. Because cross-entropy is minimized precisely when the model's distribution matches the data's distribution, and because it turns out that logarithm is essentially the only function with this property, the argument runs the other direction: requiring the loss to be minimized only when the model matches the true statistics forces the per example loss to be a logarithm, a result that can be shown rigorously using a mathematical technique called a Lagrange multiplier.
Why the negative log is forced 25:02
Working through the math of minimizing the loss subject to the model's probabilities summing to one leads to a condition where the derivative of the loss with respect to a probability must look like a constant divided by that probability, and only the logarithm has this property. This means that if you want a loss function that is minimized exactly when the model matches the true statistics of the data, you are forced into using the negative log, which is exactly the cross entropy formula.
Distillation as a softer training signal 26:02
Distillation trains a small, efficient model to approximate a larger, smarter one, which is useful once you already have strong performance from a big model and want something cheaper to run. Instead of comparing the small model's predicted distribution to a single true next token, you compare it to the full distribution predicted by the large model, using cross entropy between the two. This gives a richer signal than ordinary training, similar to having a stronger player talk through many good chess moves rather than just watching one game, so a single example can teach as much as many ordinary examples would.
Reconnecting cross entropy to compression 28:33
Cross entropy first appeared from compression, then reappeared from a separate branch of math called constrained optimization, and the video argues these two appearances must still be connected, much like pi always ties back to circles. The next step, saved for a following part, is showing how a predictor like a language model can be turned into an actual compressor whose bit usage matches the model's information content, which would prove that training with cross entropy loss is the same as training the best possible text compressor.
KL divergence and a closing challenge 31:35
KL divergence is the gap between cross entropy and entropy, representing how many extra bits per symbol you waste by using a mismatched code, and in machine learning it works as an asymmetric distance measure between distributions that is zero only when they match. The video also mentions a virtual career fair page, 3b1b.co/talent, featuring company interviews and puzzles, including one from a company called Dopple that happens to use the KL divergence formula. It closes with three things to ponder: showing a compact algebraic form of KL divergence is equivalent to the longer one, interpreting a diagram version of the expression, and considering what would change if distillation used KL divergence instead of cross entropy as its loss.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
