Departure question
Read signals from chancePort 7 of 11Where two intuitions collide
By ignoring meaning and measuring only surprise, the limits of compression and communication became calculable.
This voyage is an editorial path for understanding, not a claim of direct historical influence or sole invention.
Treat rarer symbols as more informative and measure average uncertainty in bits. A fair coin has entropy 1 bit; if 26 English letters were equally likely, one letter would have log₂26≈4.70 bits. Real language has unequal frequencies and context, so its value differs. Building on Nyquist’s bandwidth bounds and Hartley’s logarithmic measure, Shannon placed probability distributions and noisy channels in one framework. Entropy links lossless-compression limits, communication, and some learning objectives, but it does not measure depth of meaning, cryptographic safety, or the ZIP byte size of one file.
p (probability of heads) | H(p) [bits] | Meaning |
|---|---|---|
| 0 | 0 | Certain (no surprise) |
| 0.1 | 0.47 | |
| 0.25 | 0.81 | |
| 0.5 | 1 | Maximum — greatest uncertainty |
| 0.75 | 0.81 | |
| 0.9 | 0.47 | |
| 1 | 0 | Certain |
Entropy is maximal at p=0.5 (a fair coin): exactly 1 bit.
A bit-valued measure of average uncertainty in a probability distribution. Building on Nyquist and Hartley, Shannon’s 1948 theory treats source-coding and noisy-channel limits under distinct assumptions; it does not directly measure semantic depth or one file’s compressed size.
Q.Entropy of a fair six-sided die
Ports in time
Follow the scenes to see problems, notation, standards of proof, and applications changing across different times and places.
Nyquist related distinguishable signaling rates to bandwidth, while Hartley measured equiprobable alternatives logarithmically. These were distinct telegraph results, not yet Shannon entropy or noisy-channel capacity.
The recorded place matches a canonical map anchor
Continue this scene on the mapAt Bell Labs, Claude Shannon defined source entropy and channel capacity in a probabilistic communication model. The measure deliberately abstracts away semantic meaning.
The recorded place matches a canonical map anchor
Continue this scene on the mapHamming codes turned redundancy into parity coordinates that identify a single-bit error. Hamming(7,4) is a specific model, not a promise to repair arbitrary damage.
The recorded place matches a canonical map anchor
Continue this scene on the mapThe Lempel–Ziv family introduced adaptive dictionary compression without requiring a known source distribution. Later formats combine different LZ variants with additional coding; an LZ phrase count is not a ZIP byte count.
The recorded place matches a canonical map anchor
Continue this scene on the mapPolar codes gave an explicit efficient construction that asymptotically achieves the symmetric capacity of binary-input memoryless channels. Improved finite-length forms were later selected for parts of 5G control signaling, not all of 5G.
The recorded place matches a canonical map anchor
Continue this scene on the mapThe theoretical limits of ZIP, MP3, and JPEG compression; channel capacity in wireless networks; cross-entropy loss in machine learning; and password-strength estimates.
Curated sources and problems. Bring one discovery back from OEIS, Project Euler, MathOverflow, or arXiv.
No concept belongs to one person
These are not inventor credits. They are different ports: opening a problem, sharpening a language, or carrying it into another world.
Number lenses
These numbers are editorial lenses for the voyage, not required prerequisites.
Travel from dice and insurance to data, noise, and learning algorithms—measuring uncertainty and turning it into decisions.
Open the number voyage
Probabilities between zero and one distinguish the information carried by rare and common symbols.
Open the number voyage
Logarithmic information becomes a real-valued quantity that adds across independent messages.
Open the number voyage
Concept genealogy
Concepts arriving from before
Current port
Information Entropy
Concepts opened from here
No direct successor port is curated yet.
Only direct editorial links are shown; this is not a complete learning order or historical influence line.
