AIXC: A Deterministic Predictor-Based Text Compression Format

I

Motivation

AIXC is based on a compact premise: if both sides can run the same predictor, the archive does not need to store every next unit. It can store enough seed material to initialize context, then a decision stream saying whether the predictor was correct. On misses, the encoder stores the actual residual unit so decoding remains exact.

That premise turns compression into an explicit contract between a file and a predictor. The hard part is not packing bits. The hard part is making sure any conforming decoder can reproduce the same prediction sequence, tokenizer behavior, and residual interpretation without guessing.

This is not magic compression and it is not a claim that a language model makes arbitrary files free. The predictor must either be cheap enough to include, already installed at the decoder, or shared across many related archives. If every file has to carry a large model beside it, the model bytes can overwhelm the archive savings.

• Primary goal: exact round-trip text compression using a deterministic predictor contract.
• Reference mode: seed units, packed hit/miss bits, and literal residuals.
• Practical format property: entropy-coded decision streams and compact residual representations.

II

Reference Codec

The v0 loop is intentionally direct. The encoder writes the seed, asks the predictor for each next unit, writes a hit bit when the argmax matches, and writes a miss bit plus the actual unit otherwise. The decoder reads the same seed, runs the same predictor, and reconstructs the stream by choosing either the predictor output or the stored literal.

A crucial implementation detail is that both encoder and decoder feed the actual decoded unit back into the predictor state. They do not let an incorrect prediction become the next context item, so one miss cannot cascade into a permanently divergent decode.

III

Compression Economics

The simplest byte-mode reference scheme spends one decision bit per byte and eight literal bits for each miss. That means it beats raw bytes only when the miss rate is below seven eighths, which is an easy bar but not a meaningful end state.

A stronger representation compresses the decision stream toward its binary entropy and avoids raw literals when the correct unit is still near the top of the predictor ranking. A rank residual, sparse top-k residual, Huffman section, or range/ANS-style section coder can use more of the predictor distribution than a single hit bit.

The economics resemble a spelling test with hints. A perfect predictor costs almost nothing after the seed because every answer is a hit. A poor predictor degenerates toward storing the original text plus overhead. The format earns its keep when the predictor is often right, or at least wrong in a ranked way that can be encoded more cheaply than a full literal.

• Literal residual mode is easiest to audit and debug.
• Top-k residual mode can store predictor rank when the correct unit is near the top.
• Sparse top-k residual mode stores miss positions and compact residuals when hit rates are very high.
• Huffman and zstd section coding reduce overhead for decision and residual streams.

IV

Codec State Machine

A conforming implementation can be reconstructed as two lockstep state machines. The encoder and decoder both maintain a context buffer, a predictor handle, a unit index, and stream cursors. The encoder additionally observes the source unit at each index; the decoder additionally observes the next decision bit and residual cursor.

state E = {
  context: unit[]
  index: number
  predictor: Predictor
  decisions: bit[]
  residuals: unit[]
}

for source[index]:
  prediction = predictor.argmax(context)
  if prediction == source[index]:
    decisions.push(1)
  else:
    decisions.push(0)
    residuals.push(source[index])
  context.push(source[index])
  index += 1

state D = {
  context: seed
  index: number
  predictor: Predictor
  decisionCursor: number
  residualCursor: number
}

while output.length < originalLength:
  prediction = predictor.argmax(context)
  if decisions[decisionCursor] == 1:
    unit = prediction
  else:
    unit = residuals[residualCursor]
    residualCursor += 1
  output.push(unit)
  context.push(unit)
  decisionCursor += 1

V

Deterministic Decoding Contract

AIXC treats the predictor as part of the format, not as an informal dependency. A decoder needs to know exactly which model, tokenizer, runtime family, quantization setting, special-token policy, and tie-break rule were used. Otherwise two machines can agree on the file bytes but disagree on the next predicted unit.

For model-token compression, the tokenizer must also round-trip the source text exactly. If the tokenizer normalizes or drops information, the codec either has to reject the input or store a raw passthrough block. Byte-mode predictors avoid that tokenizer problem but usually give up the native advantage of off-the-shelf language models.

• Run predictors in evaluation mode with sampling disabled.
• Select the next unit by argmax, with a fixed lowest-id tie break.
• Pin tokenizer files, model weights, runtime versions, and quantization settings.
• Feed actual decoded units back into predictor state after both hits and misses.

VI

Binary Container

The repository implements an AIXC binary container with a fixed header and aligned sections for seed material, decision bits, residuals, and optional metadata. The header records archive mode, entropy codec, predictor backend, original byte length, unit counts, section offsets, and integrity flags.

This shape keeps parsing predictable. A decoder can validate the header, locate each section without scanning the whole file, verify optional CRC32 or SHA-256 body metadata, and confirm the predictor fingerprint before attempting a decode.

• Magic: AIXC.
• Header: 96-byte fixed prefix with little-endian offsets and lengths.
• Decision stream: LSB-first packed bits in the reference format.
• Residual stream: raw literals, ULEB128 token ids, top-k ranks, or section-coded bytes depending on mode.
• Optional section: TLV metadata for body integrity and predictor manifest fingerprints.

+----------------------+ 0
| magic/version/header |
+----------------------+
| seed section         |
+----------------------+
| packed decisions     |
+----------------------+
| residual section     |
+----------------------+
| optional metadata    |
+----------------------+
| integrity trailer    |
+----------------------+

VII

Header and Manifest Schema

The fixed header is the decoder's first trust boundary. It must contain enough information to locate every section, reject incompatible modes, allocate decode buffers, and validate predictor compatibility before any predictor call is made.

magic: 4 bytes = AIXC
version: u16
flags: u32
unitMode: enum(byte, token)
predictorKind: enum(toy, lzp, byte-model, hf-token)
entropyMode: enum(raw, huffman, zstd)
originalLength: u64
seedOffset, seedLength: u64, u64
decisionOffset, decisionLength: u64, u64
residualOffset, residualLength: u64, u64
metadataOffset, metadataLength: u64, u64
bodyCrc32: optional u32
manifestSha256: optional bytes32

The manifest is canonical JSON or an equivalent canonical binary map. Required fields include predictor kind, version, tokenizer identity when applicable, tie-break rule, unit mode, residual mode, entropy mode, training corpus identity for sidecar models, and any quantization or runtime switches that affect argmax.

VIII

Residual Coding Modes

AIXC separates prediction correctness from residual representation. The simplest residual is the literal unit, but a predictor distribution contains more information than a binary hit/miss flag. If the correct unit appears near the top of the ranked prediction list, the archive can store a rank or compact top-k code rather than a full literal.

The format can still fall back to literal residuals. This fallback is essential because the archive must be exact even when the predictor is wrong in an unhelpful way. The residual mode therefore defines a total decode rule, not merely an optimization hint.

predict next unit
  |
  +-- exact top prediction -> decision hit, no residual
  |
  +-- true unit in top-k -> decision miss, rank residual
  |
  +-- otherwise -> decision miss, literal residual

IX

Entropy-Coded Sections

Packed one-bit decisions are easy to inspect but not always economical. Decision streams are often biased and clustered; residual streams are often highly nonuniform. AIXC therefore treats section coding as an independent layer: after semantic streams are generated, each section can be encoded by a configured entropy backend.

Huffman coding is attractive in the reference implementation because it is deterministic, inspectable, and simple to decode. zstd-style section compression is useful for comparing predictor-specific savings against mature generic compression. The archive manifest records the selected coding mode so decoders do not infer it from bytes.

• Decision bits can be packed raw or entropy-coded as a binary stream.
• Residual units can be literal bytes, varint token IDs, rank codes, or compressed byte sections.
• Metadata remains separately parseable so integrity and predictor checks run before decode.
• The coding layer must be deterministic; adaptive decoders cannot depend on unstored model state.

X

Predictor Interface

AIXC does not require a specific model class. It requires a deterministic predictor interface. The minimum interface accepts a context and returns an ordered candidate list or an argmax unit. More advanced residual modes require a stable top-k ordering.

type Unit = number;

type Predictor = {
  manifest: CanonicalManifest;
  reset(seed: Unit[]): void;
  observe(unit: Unit): void;
  argmax(): Unit;
  topK?(k: number): Unit[];
};

The interface deliberately separates reset, prediction, and observation. This prevents an implementation from hiding state changes inside argmax calls and makes encoder/decoder equivalence easier to test.

XI

Predictor Families and Practical Meaning

The repository includes several predictor families because the format question is broader than any one model. The toy backend is for tests, the LZP byte backend is a fast deterministic full-file baseline, trained-byte models explore shared corpus priors, neural-byte models test learned next-byte prediction, and Hugging Face token models test the higher-overhead language-model path.

These predictors have different economics. LZP adapts from the file itself and has no large sidecar, but it cannot know a corpus before seeing it. Trained-byte and neural predictors can be much stronger when the model is shared, but their model files must be counted unless the deployment environment already provides them. Hugging Face models add tokenizer and runtime reproducibility problems on top of model size.

• Adaptive byte predictors are easiest to reproduce and benchmark.
• Shared trained predictors can make archives tiny, but only after the model cost is amortized.
• Neural predictors are useful research baselines, not currently the fastest or smallest path in the repo.
• Token-model predictors require exact tokenizer round trips and pinned runtime behavior.

XII

Implementation

The compressor implementation is structured as a reference codec plus predictor backends. The dependency-free toy-byte backend is useful for tests and deterministic examples, while the LZP byte backend provides fast full-file runs. Additional modes cover trained byte models, neural byte predictors, Hugging Face causal language models, and native acceleration through a C extension.

The implementation also includes a local web workbench. It can manage corpora, run compression jobs, inspect archive metadata, train byte or neural sidecar models, and regenerate benchmark reports. AIXC is therefore a laboratory for testing predictor contracts rather than a single codec script.

• aixc/: codec, predictors, entropy coding, CLI, UI server, and native C source.
• tests/: round-trip integrity, varint, bit packing, and backend contract tests.
• benchmarks/: repeatable benchmark scripts and generated report artifacts.
• compression-research-report.md: design notes, model recommendations, and file-format rationale.

XIII

Correctness Conditions

Lossless decoding depends on three independent equalities: the seed must reconstruct the initial context, the predictor must produce the same ordered candidates, and the residual stream must be consumed in the same positions. If any equality fails, a decoder can still parse the archive but cannot guarantee the original text.

By induction over positions, a hit reconstructs the original unit from the deterministic predictor, and a miss reconstructs it from the residual at pointer j. Feeding the reconstructed actual unit back into context preserves the induction hypothesis for the next position.

XIV

Benchmark Results and Use Cases

The benchmark workbook makes the tradeoff explicit. On the five selected LogHub datasets, native codecs remain the right answer for self-contained compression: the best native archive ratios range from about 0.025 on Zookeeper to about 0.047 on Linux and Proxifier, while the current full-log AIXC LZP profile lands between 0.050 and 0.109. It completes the lossless runs, but it is slower than native zstd/xz and usually larger when no shared model is allowed.

AIXC earns its best ratios when the predictor is shared out of band. With corpus-specific max-context byte models, the order-64 archive-only ratios beat the best native archive ratio on all five selected logs: Linux reaches 0.012, Zookeeper 0.018, Apache 0.020, Mac 0.022, and Proxifier 0.022. Higher orders can drive some archive-only ratios below 0.001, but the model bytes dominate if each archive has to carry its own model.

The format fits a specific class of systems: repeated, related text streams where the predictor can be installed once and amortized across many archives. Log families, telemetry exports, generated reports, protocol traces, or corpora distributed with a known sidecar model fit the design better than one-off files. For arbitrary standalone compression, a mature native codec is still the practical choice.

• Self-contained logs: native zstd/xz remains smaller and much faster in the current benchmark set.
• Shared-model logs: AIXC order-64 trained-byte archives reach roughly 1.2%-2.2% of original size on the selected LogHub files.
• Shakespeare max-context experiments show the upper bound of the idea: an order-64 shared model produced a 3,760-byte archive, about 0.0007 of the original text, with an 11.3 MB sidecar model.
• Best fit: many related archives decoded in an environment that can already provide the predictor manifest and sidecar model.