Information Theory

Elements and Ideal Forms

Information theory begins with two primitive elements:

• Message — the thing to be communicated; a sequence of symbols drawn from a source
• Channel — the medium of transmission, characterized by noise that corrupts the message

These give rise to the theory’s central ideal forms:

• Entropy H(X) — the irreducible uncertainty in a message source (surprise, measured in bits)
• Mutual information I(X;Y) — what the received signal says about the sent signal
• Channel capacity C — the maximum rate of reliable transmission
• Coding — the transformation of messages into channel-suitable form (compression or error protection)

Extended forms (new in this enrichment): random variable, source, codeword, redundancy, syndrome, error-correcting code, Kolmogorov complexity (algorithmic information of individual objects), rate-distortion, and belief propagation (message-passing decoding).

The key platonic insight, due to Shannon (1948): information is not meaning but surprise — the log-reciprocal of probability. A uniform source over n symbols carries log₂ n bits. A deterministic source carries zero. Entropy is the ideal form of uncertainty. Kolmogorov complexity supplies the ultimate, uncomputable lower bound for description length of individual strings.

Cross-links to probability (random variables, distributions), abstract algebra and linear algebra (linear codes over finite fields), and signal processing (noisy channels as waveforms) are explicit in the substrate.

Axioms and Provable Limits

Shannon entropy is uniquely determined (up to a constant) by three axioms — continuity, symmetry, and the chain rule:

H(X) = −∑ p(x) log₂ p(x)

Derived properties and the two fundamental theorems are proved from the definition alone (see YAML for exact statements).

The enrichment adds the link to Kolmogorov complexity (uncomputable but conceptually the “true” information content) and the algebraic structure of modern capacity-approaching codes (generator matrices, parity-check matrices, Tanner graphs).

Measurement and Cause/Effect

Entropy, capacity, bit-error rate, and compression ratio are directly measurable or estimable from data. The causal structure (noise ↑ → errors ↑ → coding ↑ → errors ↓) is a stable balancing loop that the systematic model captures.

New causal links emphasize redundancy and syndrome resolution as the mechanisms that close the loop.

Procedures

Huffman coding, belief-propagation decoding, and empirical entropy estimation are given with full steps in the YAML. These are the concrete, implementable realizations of the ideal forms.

Modern turbo/LDPC codes approach the Shannon limit within a fraction of a dB using exactly the belief-propagation procedure on sparse graphs.

Communication as a System

A communication system is a stock-and-flow system whose equilibrium is the Shannon limit. The enrichment strengthens the model with explicit residual_error stock and error-correction flow, plus the rate-distortion balancing loop.

The error-correction loop (noise → errors → coding → errors ↓) is the canonical balancing feedback of the field.

Coding and the Shannon Limit

The engineering problem remains: design real codes and protocols that approach the theoretical limits under tractability, latency, power, and hardware constraints.

The enrichment adds explicit constraints on block length, fixed-point arithmetic, and channel uncertainty, plus objectives for modern capacity-approaching codes.

Practical systems (cellular, Wi-Fi, deep-space, storage, QR codes) are all explicit negotiations of these objectives against these constraints.