AIXC / docsProject documentation

Reference

Deterministic Decoding Contract

Look up syntax, contracts, layouts, algorithms, and exact behavior.

AIXC treats the predictor as part of the format, not as an informal dependency. A decoder needs to know exactly which model, tokenizer, runtime family, quantization setting, special-token policy, and tie-break rule were used. Otherwise two machines can agree on the file bytes but disagree on the next predicted unit.

F=SHA256⁡(canonical_json⁡(M))F = \operatorname{SHA256}(\operatorname{canonical\_json}(M))
Manifest fingerprint. A canonical predictor manifest can be hashed and recorded in the archive so decoders can reject mismatched runtimes.

For model-token compression, the tokenizer must also round-trip the source text exactly. If the tokenizer normalizes or drops information, the codec either has to reject the input or store a raw passthrough block. Byte-mode predictors avoid that tokenizer problem but usually give up the native advantage of off-the-shelf language models.

decode⁡(encode⁡(x))=x\operatorname{decode}(\operatorname{encode}(x)) = x
Round-trip condition. Model-token mode is only exact when tokenizer encode/decode preserves the original text.
  • Run predictors in evaluation mode with sampling disabled.
  • Select the next unit by argmax, with a fixed lowest-id tie break.
  • Pin tokenizer files, model weights, runtime versions, and quantization settings.
  • Feed actual decoded units back into predictor state after both hits and misses.

Continue exploring

Tutorials →How-to guides →Explanation →Source repository →