Concept
A bit-valued measure of average uncertainty in a probability distribution. Building on Nyquist and Hartley, Shannon’s 1948 theory treats source-coding and noisy-channel limits under distinct assumptions; it does not directly measure semantic depth or one file’s compressed size.
Understand it in one breath
Treat rarer symbols as more informative and measure average uncertainty in bits. A fair coin has entropy 1 bit; if 26 English letters were equally likely, one letter would have log₂26≈4.70 bits. Real language has unequal frequencies and context, so its value differs. Building on Nyquist’s bandwidth bounds and Hartley’s logarithmic measure, Shannon placed probability distributions and noisy channels in one framework. Entropy links lossless-compression limits, communication, and some learning objectives, but it does not measure depth of meaning, cryptographic safety, or the ZIP byte size of one file.
At a glance
p (probability of heads) | H(p) [bits] | Meaning |
|---|---|---|
| 0 | 0 | Certain (no surprise) |
| 0.1 | 0.47 | |
| 0.25 | 0.81 | |
| 0.5 | 1 | Maximum — greatest uncertainty |
| 0.75 | 0.81 | |
| 0.9 | 0.47 | |
| 1 | 0 | Certain |
Entropy is maximal at p=0.5 (a fair coin): exactly 1 bit.
Key formula
Worked examples
- 1
Q.Entropy of a fair six-sided die
Key moments
Nyquist and Hartley — bandwidth and logarithmic information
Nyquist related distinguishable signaling rates to bandwidth, while Hartley measured equiprobable alternatives logarithmically. These were distinct telegraph results, not yet Shannon entropy or noisy-channel capacity.
Shannon — A Mathematical Theory of Communication
At Bell Labs, Claude Shannon defined source entropy and channel capacity in a probabilistic communication model. The measure deliberately abstracts away semantic meaning.
Hamming — locating and correcting one flipped bit
Hamming codes turned redundancy into parity coordinates that identify a single-bit error. Hamming(7,4) is a specific model, not a promise to repair arbitrary damage.
Lempel–Ziv compression
The Lempel–Ziv family introduced adaptive dictionary compression without requiring a known source distribution. Later formats combine different LZ variants with additional coding; an LZ phrase count is not a ZIP byte count.
Polar codes and the road to 5G
Polar codes gave an explicit efficient construction that asymptotically achieves the symmetric capacity of binary-input memoryless channels. Improved finite-length forms were later selected for parts of 5G control signaling, not all of 5G.
Modern applications
The theoretical limits of ZIP, MP3, and JPEG compression; channel capacity in wireless networks; cross-entropy loss in machine learning; and password-strength estimates.
Beyond MathVoyage
Loading…