Entropy gets described as “disorder,” which is a shame, because the better word is ignorance. The entropy of a system is how much you don’t know about it given what you can measure — the log of the number of microstates consistent with the macrostate you observe.
Shannon borrowed the machinery for information: a message’s entropy is how surprised you should be, on average, by its next symbol.
H(X) = −Σ p(x) · log₂ p(x)A fair coin: 1 bit per flip. A coin that always lands heads: 0 bits — nothing to learn. English text: a bit more than 1 bit per letter, which is why compression works and why you can rd ths sntnce wtht mst f th vwls.
The connection to random walks is tight. A walker’s position diffuses; the distribution spreads; entropy increases. Diffusion is the second law, wearing comfortable shoes.
What I keep chewing on: writing is entropy reduction for the reader, paid for by the writer. A good essay takes a cloud of maybes and collapses it into one path. This blog is me paying that cost in public.