Nested enumeration to decrease file size

[AlexanderLife]

nested enumeration to decrease file size

users[2]{id,name,role(admin,user)}:
  1,Alice,0
  2,Bob,1

On finite values this enables not reprinting longer duplicates over and over.
1,Alice,admin
2,Bob,user
3,Bob1,user
4,Bob2,user
5,Bob3,user
6,Bob4,user
7,Bob5,user
...

[AlexanderLife]

if you wanted to save more space and add some complexity

you could make the enumeration indexing rather then using base 10 you could use base 62 (https://en.wikipedia.org/wiki/Base62)

that way after enumeration index 10 to 61 still only requires 1 space,

and enumeration 62 to 3843 only requires 2 spaces.

You could do this for all array indexing in this.

[johannschopplich]

Thanks for taking the time to think about compression tricks!

To make sure I understand your proposal:

1. You're suggesting a per‑column enumeration / dictionary syntax in the header, e.g.:

   users[2]{id,name,role(admin,user)}:
     1,Alice,0
     2,Bob,1
   

   where role(admin,user) declares a finite set of values, and the cells in that column store integer codes (0/1) instead of the full strings.

2. You then suggest switching those indices (and possibly the [N] lengths) to base‑62, so that larger counts still fit in 1–2 characters.

Conceptually this is a form of schema + dictionary encoding built into the format.

For TOON specifically, this runs into a couple of deliberate design constraints:

• The spec is intentionally not a schema or compression format. From the intro:

  TOON carries the JSON data model; it is a transport/authoring format with explicit structure, not an extended type system or schema language.

  The data model is "just JSON": objects, arrays, strings, numbers, booleans, null. Encoders/decoders don't interpret extra type information beyond that.

• The enumeration syntax would introduce new semantics that decoders have to understand:

  • role(admin,user) is no longer just a field name; it’s a mini‑schema.
  • Cell values in that column are no longer "the actual JSON values"... they’re references into a per‑column dictionary.
  • You now need rules for how this maps back to plain JSON, how it interacts with strict mode, how LLMs are supposed to reason over it, etc.

  That’s a different layer (schema/compression) than what TOON is trying to standardize.

• On the numeric side, the spec explicitly calls out alternative number systems as out of scope and requires canonical decimal form for numbers:

  Out of scope: alternative number systems or locale‑specific formatting.
  Encoders MUST emit numbers in canonical decimal form (no exponent notation, no leading zeros, etc.)

  Switching [N] to base‑62 would violate that core rule and make lengths harder to read/validate by humans and simple tooling.

• In practice, for the LLM use case, the win here is small:

  • Tokens like admin and user are already a single token in modern tokenizers.
  • You have to add extra syntax to define the dictionary and extra instructions in the prompt to explain it.
  • You also make the table less self‑describing for both humans and models (they now have to mentally map 0 → admin, 1 → user).

Where I absolutely agree with you is that dictionary‑style compression is useful, especially for very repetitive columns. I think the right place for that, though, is on top of TOON as an application convention, not inside the core syntax. For example, you can already do:

roles[2]: admin,user

users[3]{id,name,roleIndex}:
  1,Alice,0
  2,Bob,1
  3,Bob1,1

and describe in your prompt or application logic that roles is a dictionary for users[*].roleIndex. TOON just sees JSON primitives and arrays; your app (and/or the LLM instructions) give them higher‑level meaning.

Similarly, if you really want base‑62 codes, you can use them as strings in your own convention:

users[2]{id,code}:
  1,A
  2,B

TOON doesn't need to know that A/B are "base‑62 indices"... it just treats them as strings.

Given the spec's current goals and constraints (JSON data model only, decimal numbers only, no schema layer), I don't plan to add inline enumerations or base‑62 indices to the core format.

I'm going to close this as out of scope for the spec, but I do appreciate the idea very much! 🙌 If you end up building a "TOON + dictionary encoding" convention or pre‑processor and have benchmark numbers (tokens, accuracy) for that vs plain TOON, I'd be very interested in seeing those results – they'd be great to link from the docs as an advanced pattern.