Compression Economics

The simplest byte-mode reference scheme spends one decision bit per byte and eight literal bits for each miss. That means it beats raw bytes only when the miss rate is below seven eighths, which is an easy bar but not a meaningful end state.

A stronger representation compresses the decision stream toward its binary entropy and avoids raw literals when the correct unit is still near the top of the predictor ranking. A rank residual, sparse top-k residual, Huffman section, or range/ANS-style section coder can use more of the predictor distribution than a single hit bit.

The economics resemble a spelling test with hints. A perfect predictor costs almost nothing after the seed because every answer is a hit. A poor predictor degenerates toward storing the original text plus overhead. The format earns its keep when the predictor is often right, or at least wrong in a ranked way that can be encoded more cheaply than a full literal.

• Literal residual mode is easiest to audit and debug.
• Top-k residual mode can store predictor rank when the correct unit is near the top.
• Sparse top-k residual mode stores miss positions and compact residuals when hit rates are very high.
• Huffman and zstd section coding reduce overhead for decision and residual streams.