Encoder quotes non-ASCII keys only when they occur at position 0 (§7.3)

[antrixy]

SPEC Version

v4.1

SPEC Section

Section 7.3: Key Encoding

SPEC Requirement

§7.3 states that object keys and tabular field names MAY be unquoted only if they match ^[A-Za-z_][A-Za-z0-9_.]*$, and otherwise MUST be quoted and escaped per §7.1.

The pattern is ASCII-only and applies to the whole key, not just its first character. A non-ASCII character anywhere in the key takes it outside the pattern, so the key must be quoted.

This wording is unchanged between v3.0 and v4.1, so the finding does not depend on which version the package targets. v4.1's §16 spells the consequence out: conforming encoders quote every non-ASCII key.

Current Behavior

encode quotes a non-ASCII key when the non-ASCII character is FIRST, but leaves it unquoted when it occurs anywhere later.

Against toon_format 0.9.0b1:

input key	non-ASCII at	output
a-b	—	"a-b": 1 — correct
éabc	0	"éabc": 1 — correct
日abc	0	"日abc": 1 — correct
aébc	1	aébc: 1
a日bc	1	a日bc: 1
abcé	last	abcé: 1
_é	1	_é: 1
a_é_b	2	a_é_b: 1

Behaviour is identical for BMP (é) and CJK (日) at both positions, so it is positional rather than an encoding-width issue.

The head character appears to be validated against §7.3's head class correctly, while the remaining characters are checked with an ASCII-blind test. _é is the clearest case: _ is a legal head character, so the head check passes and the tail check never rejects the é.

Test Case

toon_format 0.9.0b1

from toon_format import encode

for k in ["a-b", "éabc", "日abc", "aébc", "a日bc", "abcé", "_é", "a_é_b"]:
print(repr(encode({k: 1})))

Expected (matches @toon-format/toon 2.3.0, which quotes all of these):

'"a-b": 1'

'"éabc": 1'

'"日abc": 1'

'"aébc": 1'

'"a日bc": 1'

'"abcé": 1'

'"_é": 1'

'"a_é_b": 1'

Actual:

'"a-b": 1' <- correct

'"éabc": 1' <- correct

'"日abc": 1' <- correct

'aébc: 1'

'a日bc: 1'

'abcé: 1'

'_é: 1'

'a_é_b: 1'

Additional Context

Why this survives round-trip tests: §7.4 requires decoders to accept any unquoted key token as a literal key, even one §7.3 forbids an encoder from emitting. So the package decodes its own output correctly and self-consistency tests pass. It surfaces only against another implementation — @toon-format/toon 2.3.0 rejects the output in strict mode.

The positional nature is likely why it hasn't been caught: a fixture with a leading non-ASCII key — the obvious one to write — passes.

Scope tested: object keys at positions 0, 1, 2 and last, across BMP and CJK. Not tested: tabular field names, entry keys in keyed tabular form (§9.5), or keys that would require quoting for an additional §7.2 reason.

A related encoder issue was filed against toon-rust as toon-format/toon-rust#77; the mechanism there differs (no position is quoted).

Found with toon-diff (https://github.com/antrixy/toon-diff), a differential conformance fuzzer that round-trips generated values through the TypeScript, Python and Rust implementations against a lossless oracle. It generates values from the JSON grammar rather than from fixtures, which is how a non-leading non-ASCII key came up.

[antrixy]

Follow-up: I think I've found the exact mechanism, and it isn't a general ASCII-blind check — it looks like the pre-1.4 spec wording.

Spec CHANGELOG [1.4] 2025-11-05 records: replaced \w with explicit [A-Za-z0-9_] in key regexes for cross-language clarity (§7.3).

In Python's re, \w is Unicode-aware by default, so é and 日 match it. A pattern of the pre-1.4 shape — an explicit head class followed by \w — therefore rejects a leading non-ASCII character but accepts one anywhere after it:

import re
old = re.compile(r'^[A-Za-z_]\w*$')            # pre-1.4 wording
new = re.compile(r'^[A-Za-z_][A-Za-z0-9_.]*$')  # 1.4+ explicit ASCII

for k in ['abc', 'éabc', '日abc', 'aébc', 'a日bc', 'abcé', '_é', 'a_é_b', 'a-b']:
    print(f'{k:8} old={bool(old.match(k))!s:5} new={bool(new.match(k))!s:5}')

old matches all nine of the observed cases exactly — including _é staying unquoted (an underscore is a legal head character, so the head class passes and \w accepts the é) and é/日 behaving identically.

If that's the pattern in use, the fix is to make the character class explicit rather than relying on \w. Python is arguably the one language where this bites hardest: in most other regex flavours \w is ASCII-only by default, which is likely why the ambiguity survived until the spec spelled it out.

Worth noting §7.3 also covers entry keys (§9.5) and field names in a header field list, so the same pattern may need fixing at each site it's used.