diff --git a/src/odr/internal/csv/PLAN.md b/src/odr/internal/csv/PLAN.md new file mode 100644 index 000000000..b84aeb1d4 --- /dev/null +++ b/src/odr/internal/csv/PLAN.md @@ -0,0 +1,284 @@ +# CSV plan + +Where the csv module is going, and in what order. Written before stage 1; keep +it honest as stages land. + +## Today + +`CsvFile` is detection only — 60 lines, `is_decodable() == false`, rendered by +`html::create_text_service` as a line list. `check_csv_file` reads the **whole** +file, hard-codes `,` and `"`, and demands a globally uniform field count > 1. +`text::TextFile`'s constructor runs `guess_charset` over the **whole** stream, +and `open_strategy` does both speculatively while listing file types. + +The detected charset is currently used nowhere: `html/text_file.cpp` pipes raw +bytes into a document declaring ``, so any non-UTF-8 text +file already renders as mojibake (`// TODO charset`, `html/text_file.cpp:71`). + +## Target + +A csv opens as a spreadsheet document — one sheet, cells reachable through +`SheetAdapter` — so the generic HTML renderer and every binding get table +rendering without format-specific code. Separator, charset, quoting and header +are autodetected from a bounded probe, overridable by the caller, and the +resolved values are readable back. + +## Decisions taken up front + +**Cell strings are UTF-8.** `Text::content()` returns `std::string` and every +binding treats it as UTF-8 (JNI `NewStringUTF`, embind, pybind11, `NSString`). +Passing legacy bytes through and letting a browser sort it out only works for +the html path, not the api path, so it is not an option. This is what makes an +encoding we cannot decode fatal to the *sheet* path specifically, while the +text path stays open to it. + +**No element per cell.** `ElementIdentifier` is a `std::uint64_t` +(`definitions.hpp:7`): encode `kind | row | column` into it and decode in the +adapter. A flat `ElementRegistry` over cells does not survive a large file, and +the virtual-id seam is what keeps every later scaling decision an +implementation detail. This is a deliberate deviation from the pattern in the +root `AGENTS.md`; it belongs in a `csv/AGENTS.md` once stage 2 lands. + +**Detection rejects; the parser does not.** These are two jobs and +`check_csv_file` currently conflates them — `open_strategy.cpp:281` even +constructs a `CsvFile` *as* the probe, which is why detection pays for a full +scan. Split into a probe returning resolved options plus a verdict, and a parser +taking resolved options and judging nothing. + +Detection heuristics (sniffing an unknown text file): score candidate separators +over the probe, require ≥ 2 columns, reject an unterminated quote at EOF. Both +rules are discriminators — one column is every line of prose ever written, and a +dangling quote is good evidence of not-csv — not statements about what is valid. + +The parser, given a separator, is **total**: any bytes plus a transcodable +charset yield some sheet. A one-column csv is a legitimate csv; a truncated +quoted field closes at EOF; an empty file is an empty sheet; ragged rows pad +short and widen long. Nothing here is unrepresentable, so nothing throws. + +What is left to fail is an incoherent options struct (separator equal to the +quote char, a separator of `"` or `\n`) — an argument error — and a charset we +cannot transcode. `NoCsvFile` therefore becomes purely a detection exception, +which is what its name always claimed. + +**Quoting is not type information.** RFC 4180 quoting is lexical — Excel quotes +any field holding a delimiter, quote or newline, and many writers quote +everything. Infer nothing from it. + +**Encoding is a public enum with a table behind it.** Encodings carry aliases +the way mime types do (`UTF-8`/`utf8`, `windows-1252`/`cp1252`, +`ISO-8859-1`/`latin1`) and uchardet hands back a name that has to map home, so +unlike `FileType` this needs both directions. Mirror `file_type_table.cpp`: +canonical name first, aliases after, one row per encoding, and a test that +fails when an alias is claimed twice. + +```cpp +enum class TextEncoding { unknown, utf8, utf16le, utf16be, utf32le, utf32be, + windows_1252, iso_8859_1, iso_8859_15, shift_jis, … }; + +std::vector all_text_encodings(); +std::string text_encoding_to_string(TextEncoding); +TextEncoding text_encoding_by_name(std::string_view) noexcept; // unknown if none +std::span text_encoding_names(TextEncoding) noexcept; +bool text_encoding_is_decodable(TextEncoding) noexcept; +``` + +Named `TextEncoding`/`encoding()` rather than `Charset`/`charset()` so it does +not collide with the accessor it replaces. `TextFile::charset()` stays, +implemented as `text_encoding_to_string(encoding())` and marked both +`/// @deprecated` (the existing convention, `html.hpp:49`) and `[[deprecated]]`. +There is no `-Werror` (`CMakeLists.txt:38`), so the attribute is warnings only; +the three binding call sites (`jni_file.cpp:288`, `bind_file.cpp:218`, +`ODRFile.mm:516`) move to `text_encoding_to_string(encoding())` and keep their +own public surface unchanged. + +**Reuse from `pdf` means extraction first.** Nothing reaches across into +`internal/pdf`. Anything shared moves into `internal/encoding` with pdf as one +of its users. Stage 1 as designed needs no runtime reuse at all, so this binds +only on the multibyte tier below. + +**Encoding tiers**, expressed as that last predicate. + +- **UTF-8/16/32 (+BOM)** — `utfcpp`, already a dep. +- **Single-byte legacy** (windows-1252, ISO-8859-1/-15) — a generated 256-entry + byte→codepoint table per encoding. Generate from Python's built-in codecs + (`bytes([i]).decode('cp1252')`), which are the standard mappings; only the + *shape* comes from `tools/pdf/generate_encoding_data.py` → `pdf_encoding_data.cpp` + (auto-generated header, `clang-format off`, licence note). Do **not** reuse + WinAnsiEncoding as cp1252: they are close, not equal, and ISO-8859-x is not + in the pdf tables at all — the header on `pdf_doc_encoding_to_unicode` + documents exactly this class of near-miss. +- **Multibyte legacy** (Shift-JIS/cp932, GBK, Big5, EUC-KR) — *deferred, not + declined*, with two viable routes and no need to pick until someone wants CJK + csv. Measured mapping counts: cp932 22 736, gbk 21 791, shift_jis 18 912, + big5 13 710, euc_kr 8 225. + - *Generate our own*, again from Python's codecs: exact, no refactor, but + ~85 k pairs ≈ 340 KB of static data as sorted `(uint16, char16)`, less with + the run encoding `pdf_cid_data` uses. + - *Hoist and share*: `pdf_cid.hpp:19` documents `translate_predefined_cmap` + as decoding the legacy CJK CMaps, "whose codes are the legacy encoding's + bytes", to UTF-8, and `pdf_cid_data.cpp` carries 17 RKSJ CMaps — + `90ms-RKSJ-H` being the Microsoft variant, i.e. cp932. That data is already + compiled into every binary, so sharing it costs no size. The price is the + extraction into `internal/encoding` and the lossiness of routing through a + character collection's UCS2 map. + +Until then a non-decodable encoding is *known but not decoded*: reported by +name, rendered as text with `text_encoding_to_string(...)` in the html header so +the browser decodes it. Only the spreadsheet path, which must hand UTF-8 to the +bindings, is closed to it. + +--- + +## Stage 1 — encoding, in `internal/encoding` + +No csv in this stage at all. It is the substrate everything else stands on, it +ships alone, and it closes a live bug: the detected charset is used nowhere +today, so every non-UTF-8 text file renders as mojibake. A package of its own +rather than a corner of `internal/text`, because it is what `pdf` would later +share. + +- public `TextEncoding` enum + `text_encoding_table.cpp` (aliases, both + directions, decodable predicate) as above; `TextFile::encoding()` added, + `charset()` deprecated, bindings moved off it. +- `to_utf8(std::string_view, TextEncoding)`. BOM sniffing is deterministic and + runs before uchardet. Single-byte tables generated from Python's codecs, in + the `tools/pdf` output shape. Invalid bytes become U+FFFD, never a throw + mid-render — the override is the answer to a bad guess. +- `guess_charset(std::istream &, std::size_t max_bytes)` — honour the existing + TODO in `text_util.hpp`. 64 KiB is enough for uchardet. Accept the + consequence: a file that is ASCII for 64 KiB with a `é` at 2 MB detects as + UTF-8 and meets an invalid byte later. That is what U+FFFD and the override + are for. +- a binary test (NUL byte / share of C0 controls) as the actual "is this text" + oracle. uchardet names a charset, it does not gate. +- `html/text_file.cpp` transcodes, or declares the charset for a non- + transcodable one, and drops its `// TODO charset`. + +## Stage 2 — the csv probe + +Splits detection from parsing and stops detection paying for a full scan. + +- `probe` reads one bounded prefix, shared by charset detection and separator + scoring, and returns resolved options plus a verdict. Drop the trailing + partial record from the sample. +- separator scoring over `{',', ';', '\t', '|'}`: quote-aware split, modal field + count per candidate, pick by share-of-lines-matching then field count; a + winner yielding one column loses. Honour Excel's `sep=;` first line. +- `open_strategy` keeps constructing a `CsvFile` to probe, because that + constructor *is* the probe now and is bounded. Detection reads its own 64 KiB + rather than sharing `text::TextFile`'s: caching the probe would make every + text file carry 64 KiB for the life of the object, and re-reading it costs + nothing. +- the existing tests in `csv_file_test.cpp` move to the probe: ragged rows and + dangling quotes stay rejections *there*, and stop being rejections in the + parser. The comment on `a_quoted_field_must_be_terminated` records a real + misclassification — the rule survives, on the detection side only. + +## Stage 3 — options, shaped like decryption + +`nullopt` means autodetect; the resolved values are readable back so a caller +can show "detected `;`, UTF-16LE" and offer an override. The realistic flow is +open → wrong → adjust → reopen. + +```cpp +struct CsvOptions { + std::optional charset; + std::optional separator; + std::optional quote; + std::optional header_row; + std::size_t probe_bytes{1 << 16}; +}; +``` + +Follow `decrypt`: an immutable handle deriving another handle, narrowed on the +concrete type (`file.cpp:332`, `file.cpp:345`). `TextFile::charset()` is already +half of this pattern — stage 3 finishes it. + +```cpp +CsvOptions options() const; +CsvFile with_options(const CsvOptions &) const; +``` + +This keeps `odr::open` untouched — no new overload across six `open` signatures +and six `DecodedFile` constructors — and the bindings already have the call +pattern from decrypt (`wasm/src/wasm_file.cpp:86`, `jni/src/jni_file.cpp:162`). +Do **not** copy `decrypt`'s state machine: `EncryptionState` gates rendering +because an encrypted file cannot be rendered at all (`file.cpp:167`), whereas a +csv without options always has a guess. An `options_required()` state would +invent a gate the format does not have. Nor does the failure map: a wrong +password leaves nothing to hand back, whereas the parser is total, so +`with_options` fails only on an incoherent struct or an untranscodable charset. + +Options are per-format, so they need a concrete handle, but stage 2 routes csv +through `as_document_file()` which has nowhere to put them. Add a public +`CsvFile` handle alongside `TextFile`/`PdfFile` — `as_csv_file()`, `options()`, +`with_options()`, `document()`. `PdfFile` is the precedent for a format-specific +handle narrowing an inherited operation. The same shape gives `TextFile` a +`with_charset(std::string)` next to its existing `charset()`. + +`header_row` is not in the struct: it would be inert until a sheet exists to +mark a header on. It arrives with stage 4, or not at all. + +**Bindings come after stage 4, not here.** `CsvFile` is the handle stage 4 hangs +`document()` off, and stage 4 also decides the file category — binding it now +means binding it twice. One binding pass, once the shape has settled. + +## Stage 4 — the sheet document + +The visible win. Whole file in memory; the renderer caps at +`spreadsheet_limit{10000, 500}` (`html.hpp:124`) so that is sufficient for +output at any file size worth caring about today. + +- `CsvDocument` : `internal::Document`, `DocumentType::spreadsheet`, one sheet. +- virtual element ids as decided above; `cell`/`dimensions` are the only way in + to the data, so stage 6 can change what is behind them. +- the sheet is rectangular even where the file is not — a short row pads, a + long one widens. The counterpart to detection no longer rejecting ragged + files. +- csv **stays** `FileCategory::text` (`file_type_table.cpp:416`), carrying + `DocumentType::spreadsheet`. Moving it to `document` was the plan and is not + what landed: a csv is still text, `TextFile::text()` still reads it, and + `document()` is the second view rather than a replacement. So `is_text_file()` + keeps answering true and `html::translate` orders its csv branch first. The + capabilities test enforces whatever the row claims. +- `html_output_test` skips csv (`// TODO enable zip, csv, json`). Csv now + produces real output, so enabling it needs reference output committed to the + output repo and the pointer advanced — a separate repo, a separate change. + +## Stage 5 — value types + +`ValueType` is only `{unknown, string, float_number}` +(`document_element.hpp:135`) and drives one CSS class — right alignment — so it +stays small. Decided per *column*: a reader compares down a column, so one +number in a column of prose is not a quantity. The first row is left out of the +sample, because a header names its column rather than holding a value. + +The grammar is strict on purpose. A leading zero is what tells `007` from a +quantity, and a thousands separator does not say which side of the Atlantic +wrote it — `1,234` is a thousand or one point two three four depending. Dates +are left alone entirely: `03/04/2026` is two different days and guessing gets +it wrong confidently. The raw string is always what is displayed. + +## Stage 6 — large files + +Deferred: with the 10 000-row render cap this buys nothing for html output. It +matters for open cost and for api consumers walking the sheet. Build it when +something needs it; `CsvDocument::cell`/`dimensions` are the seam. + +- Row-start offsets are the only index needed — a row boundary is the one place + where "outside quotes" is unambiguous. +- Checkpoint every ~1024 rows (offset + row index), binary search then scan + forward. 10 M rows at 8 bytes each is 80 MB and unshippable on a phone; + checkpoints are ~80 KB per GB. +- `sheet_dimensions()` needs an exact count, i.e. one full quote-aware scan. + Make it lazy — first dimension query, not open — and build the checkpoints in + that pass. +- Window staging: unescape **into** the staged buffer. `""` means a cell's value + is not a contiguous range of the original bytes, but unescaping only shrinks, + so each field compacts within its own slot and a `string_view` into the + rewritten window is valid. The window also holds the transcoded UTF-8, so one + buffer solves both. LRU 2–4 windows; the renderer's row-major walk hits them + sequentially. +- `abstract::File` offers `memory_data()` (whole file as a `string_view`, no + staging needed) and `disk_path()` (mmap-able). A csv inside a zip has neither + — that path stays full in-memory, by decision not by accident. diff --git a/src/odr/internal/iwork/PLAN.md b/src/odr/internal/iwork/PLAN.md new file mode 100644 index 000000000..f57831fc3 --- /dev/null +++ b/src/odr/internal/iwork/PLAN.md @@ -0,0 +1,296 @@ +# iWork plan + +Where an iwork module would go, and in what order. Written before stage 1; keep +it honest as stages land. + +## Today + +Nothing decodes, and unlike rtf there is not even a `FileType` yet. A `.pages` +is a zip, so `magic.cpp:95` reports `FileType::zip`, `list_file_types` probes +odf then ooxml (`open_strategy.cpp:213-238`), both fail, and the caller gets +`[zip]`. `odr::open` hands back a `zip::ZipFile` — an archive, not a document. + +Two fixtures are already committed: +`test/data/input/odr-public/pages/{empty.pages,style-various-1.pages}`, both +written by iWork 13.2 (`Metadata/BuildVersionHistory.plist`). Neither is listed +in `index.csv` and neither has reference output, so nothing exercises them. +`style-various-1.pages` carries `Index/Tables/` and nine files under `Data/`, +which is most of the surface below. + +New `FileType` entries append at the end of the enum — `file.hpp:98` says so — +and **the bindings do need updating** here, unlike rtf: `python/src/bind_file.cpp`, +`jni/java/app/opendocument/core/FileType.java`, +`apple/include/OdrCoreObjC/ODRFile.h` + `apple/src/ODRFile.mm`. Wasm does not: +it derives its enums from `odr::all_file_types()` at runtime +(`wasm/src/wasm_core.cpp:35`, `:69`). + +## Spec + +There is none, and none is coming. Nothing is vendored under +`offline/documentation/`, and Apple has never published the `.proto` schemas. + +The evidence available is, in order of trust: + +1. **The fixtures.** Byte layout verified against a file in the repo is the only + claim this plan treats as fact; everything else below is marked as needing + confirmation. +2. **MIT-licensed reverse engineering** — `numbers-parser`, `keynote-parser`, + `obriensp/iWorkFileFormat`. +3. **`libetonyek`** (Document Liberation Project, MPL-2.0). + +Read all of them for facts; **copy code from none of them**. Where `oldms/` +cites `[MS-XLS] §2.4.1`, this module cites a fixture and an offset instead — +`empty.pages Index/Document.iwa +0` — because that is the strongest reference +that exists. + +## Target + +`.pages` opens as a text document, `.numbers` as a spreadsheet, `.key` as a +presentation, all through the abstract document model, so the generic HTML +renderer and every binding get them without format-specific code. +`is_editable()` false, `save` throws, exactly as `oldms/text`. + +**iWork '13 and later only.** The 2005–2009 XML era is a different format and is +deferred by decision — see below. + +## Decisions taken up front + +**The module is `iwork`, not `apple`.** `apple/` at the repo root is already the +Objective-C bindings and the Swift package; a second meaning would be a trap. +Namespace `odr::internal::iwork`. + +**Three file types, one engine.** `iwork_pages`, `iwork_numbers`, +`iwork_keynote` appended to `FileType`, each with its own +`file_type_table.cpp` row and `DocumentType`. One `IworkFile` switches on the +type, exactly as `odf::OpenDocumentFile` (`odf_file.hpp:21`) does for the four +opendocument types — same constructor over an +`abstract::ReadableFilesystem`, same `document()` dispatch. + +**No new dependencies.** Two pieces would normally be a conan line each, and +both are wrong here: + +- **Snappy.** The `.iwa` framing is Apple's own: a 4-byte header per block, + `0x00` then a little-endian 24-bit compressed length, repeated to EOF. Stock + Snappy *framing* (the `sNaPpY` stream identifier, per-chunk CRC-32C) is not + present, so a stream decoder does not apply — only the **block** decoder does, + and that is a varint length plus literal/copy tags in about 150 lines. + Verified: `empty.pages Index/Document.iwa` opens `00 fb 1d 00`, i.e. 7675 + bytes, and the file is 7679; the block then starts `82 6e`, a varint + uncompressed length of 14082. +- **Protobuf.** Only the wire format is needed — varint, 64-bit, + length-delimited, 32-bit, with groups skipped. There are no schemas to + generate from, so a code generator would have nothing to do, and linking + conan `protobuf` would drag one into the wasm, android and apple builds to + replace ~150 lines. Read fields by number against hand-written accessors. + +Both live inside `iwork/` for now. The csv plan's rule applies — anything +genuinely shared moves out to its own package *first* — and a wire reader with +one user has not earned a package. + +**An `.iwa` is an object graph, not a tree.** Each file is a sequence of +`(varint length, TSP.ArchiveInfo, payload bytes)`. `ArchiveInfo` is field 1 = +identifier, field 2 = repeated `MessageInfo{1: type, 2: version, 3: length, +4: field_infos}`. Verified on the same fixture: `08 01` (identifier 1), +`12 52` (an 82-byte `MessageInfo`), `08 90 4e` (type 10000), `12 03 01 00 05` +(version 1.0.5), `18 e0 0c` (payload length 1632). + +Objects reference each other by identifier, so the engine **indexes first and +resolves lazily**: build `identifier → (type, component, span)` across the +package, then walk the graph from the document archive. Reading files in +directory order and hoping the tree falls out is the mistake to avoid. + +**Components come from `Index/Metadata.iwa`, not from file names.** It carries +the package's component list (identifier → locator). The fixture's names carry +id suffixes — `CalculationEngine-1732585.iwa`, +`AnnotationAuthorStorage-1732584.iwa`, `Index/Tables/DataList-1732820-2.iwa` — +so globbing for `CalculationEngine.iwa` finds nothing in one fixture and finds +it in the other. (`style-various-1.pages` has the unsuffixed spelling.) + +**A pinned type table, and fail soft on what is not in it.** A `constexpr` +table maps message type IDs to the archive kinds we understand, annotated with +the app versions it was confirmed against, keyed off +`Metadata/BuildVersionHistory.plist`. An unknown type ID is **skipped**. + +This is not a hole in the root `AGENTS.md`'s fail-fast rule, for the same +reason the rtf plan's leniency is not: that rule says throw *where the spec +dictates what to expect*. Here there is no spec, and an unknown type ID means +Apple shipped a version we have not mapped — not that the input is corrupt. A +reader that throws on one cannot open next year's files. + +Do throw on: framing that overruns the file, a Snappy block that decompresses +past its declared length, a varint that does not terminate, a reference to an +identifier the index does not hold, and nesting past a depth bound. + +**`Data/` needs no decoder.** Media is stored as ordinary zip entries; open it +through the filesystem, hand it to `ImageAdapter::image_file` with +`image_is_internal() == true`. + +**Flat `ElementRegistry`, copied not shared.** An iWork document's element count +is bounded by its paragraph and drawable count, so the pattern in the root +`AGENTS.md` applies unchanged and `oldms/text/doc_element_registry.*` is the +template — not csv, whose id-is-the-coordinate scheme exists because a sheet has +an element per cell. + +**Pages has two document modes.** Word processing is a text flow; page layout is +a canvas of floating text boxes, structurally closer to `odg` than `odt`. One +`FileType` and `DocumentType::text` either way — word processing first, page +layout rendered as pages of frames once frames exist (stage 4). + +**A Numbers sheet holds many tables.** Our `Sheet` (`document_element.hpp:300`) +is one grid plus `shapes()`. Map **one odr sheet per Numbers table**, named +` – `; taking only the first table per sheet would drop data +silently. + +**No writer, no editing.** Out of scope in every stage. + +--- + +## Stage 1 — detection and the container + +Nothing renders yet. The point is that the bytes come apart correctly and the +type is reported, which is also the whole of what a file picker needs. + +- `iwork_snappy.{hpp,cpp}` — Apple framing plus block decompression, over the + `std::istream *` / `std::streambuf *` shape `pdf::ObjectParser` uses + (`pdf_object_parser.hpp`). +- `iwork_protobuf.{hpp,cpp}` — wire reader: varints, the four wire types, skip + for unknown fields, a `NestingGuard` for length-delimited recursion. +- `iwork_archive.{hpp,cpp}` — `.iwa` → objects; the package index built from + `Index/Metadata.iwa`; `identifier → object` lookup. +- `iwork_file.{hpp,cpp}` — `IworkFile : abstract::DocumentFile` over an + `abstract::ReadableFilesystem`, shaped like `odf::OpenDocumentFile`. + `password_encrypted()` reports true when `Index/Metadata.iwph` is present and + `decrypt` throws (see Deferred). +- detection: a third probe after odf and ooxml in `list_file_types` + (`open_strategy.cpp:213-238`), a branch in `open_file_as` beside odf's + (`:48-66`), and in `open_file` (`:328`) and `open_document_file` (`:498`). + `Index/Document.iwa` present ⇒ iWork; **which app comes from the root + archive's type ID**, read off a fixture per app and put in the pinned table — + not guessed, and not taken from the extension, which a caller may have lost. +- three rows in `file_type_table.cpp`: extensions `pages` / `numbers` / `key`, + mime types `application/vnd.apple.{pages,numbers,keynote}` plus the + `application/x-iwork-*-sff*` spellings, `FileCategory::document`, the matching + `DocumentType`, `{.detect_by_content = true, .open = true}`. `odr_test` fails + if declared capabilities exceed what the engine does, so `translate_html` + arrives per format in the stages below. +- `FileType` entries at the end of the enum; the three bindings above. +- `CMakeLists.txt` `ODR_SOURCE_FILES`. + +**Tests** are inline byte strings: a hand-built Snappy block (literal, copy with +1-byte and 2-byte offsets, a length that overruns), varint edge cases +(continuation past 10 bytes, a 64-bit field), an `ArchiveInfo` with an unknown +type ID that must be skipped rather than thrown on, and framing truncated +mid-block. Only the type-reporting test needs the fixtures — the data repos are +fetched and optional, so everything that can be inline is. + +## Stage 2 — Pages text + +- walk from the document archive to the body's text storage + (`TSWP.StorageArchive` in the reverse-engineering literature; confirm the type + ID against the fixture before pinning it). +- the storage holds the text as a small number of large strings plus **parallel + run tables** — index/value pairs for paragraph styles, character styles and + attachments. **Paragraph boundaries come from the paragraph run table**, not + from splitting on `\n`; `U+2028` is a line break inside a paragraph and maps + to `line_break`. The text is already UTF-8, so `internal/encoding` is not + involved. +- `root → paragraph → span → text` in a flat `ElementRegistry`. +- `empty.pages` is the regression that matters here: an empty document must + produce an empty body and not an exception. +- table row: `iwork_pages` gains `.translate_html = true`. + +## Stage 3 — Pages styles + +- `Index/DocumentStylesheet.iwa`. Style archives are sparse property sets with a + parent reference, so resolution is an inheritance walk terminating at the + theme's default — cache resolved styles by identifier, and let equal property + sets share one resolved style the way `.doc` does. +- character and paragraph properties → `TextStyle` / `ParagraphStyle`; adjacent + equal-style runs merge into one span. +- font names interned in the style registry and never mutated afterwards, so + `TextStyle::font_name` (`const char *`) stays valid — the `.doc`/`.xls` rule. +- page geometry → `TextRootAdapter::text_root_page_layout`, which stage 2 + returns empty from. + +## Stage 4 — drawables, images, frames + +- drawable archives carry a geometry (position, size, transform) and a content + reference → `Frame` plus `Image`, `Rect`, `Line`, `CustomShape` as they map. +- images resolve to a `Data/` entry by name; hand the zip entry through + unchanged. +- Pages **page-layout mode** falls out here: it is drawables on pages with no + body flow, so once frames exist it is a different root assembly, not new + parsing. + +## Stage 5 — Keynote + +Cheaper than it looks, and therefore before Numbers: slides are a container +above the *same* text storage stage 2 and 3 already read. + +- slide archives → `Slide`, the slide's master → `MasterPage`, text boxes and + drawables reuse stages 2–4 unchanged. +- notes, builds and transitions are out (see Deferred). +- `iwork_keynote` gains `.translate_html = true`; needs a `.key` fixture in the + public data repo first. + +## Stage 6 — the tile reader + +Tables in iWork are stored as **tiles** — row ranges holding packed cell +records — with strings, formats and formulas kept in side "data lists" that +cells reference by index. Several cell-storage layouts exist across app +versions; this is where the pinned table earns itself, and where a version we +have not mapped must degrade to an empty cell rather than a wrong one. + +Do this for **Pages tables first** (`Table`, `TableRow`, `TableCell`), because +`style-various-1.pages` already carries `Index/Tables/` and exercises the reader +without a Numbers fixture existing. + +## Stage 7 — Numbers + +- sheets → one odr `Sheet` per table (see decisions), on top of stage 6. +- cached values only. `CalculationEngine.iwa` holds the formula graph and is not + read; a cell shows what Numbers last computed, which is the position `.xls` + takes. +- `iwork_numbers` gains `.translate_html = true`. + +--- + +## Deferred, by decision + +- **iWork '05–'09** (Pages 1–4, Keynote 1–5, Numbers 1–2) — gzipped XML, + `index.xml.gz` for Pages/Numbers and `index.apxl(.gz)` for Keynote. A + different format needing a different engine and its own stage 1, for files no + application has written since 2013. If it ever happens it lives at + `iwork/legacy/` behind the same three `FileType`s, which is why those are + named for the app and not the era. +- **Password-protected files** — `Index/Metadata.iwph` and encrypted `.iwa`s. + `password_encrypted()` reports it from stage 1; `decrypt` throws. +- **The package form.** On macOS a `.pages` can be a directory rather than a + zip. The engine would not notice — `common::SystemFilesystem` + (`common/filesystem.hpp:16`) roots at a directory and satisfies the same + interface — but nothing in the public API opens a directory today, so this is + a seam, not a stage. +- **Templates and iBooks Author** (`.template`, `.kth`, `.nmbtemplate`, `.iba`) + — same container; add the extensions to the existing rows once documents + work. +- **Charts, comments, footnotes, change tracking, formulas, `ViewState.iwa`, + Keynote builds and presenter notes.** +- **Writing and editing.** +- **A preview fallback.** Every iWork file embeds a rendered preview + (`preview.jpg`, `preview-web.jpg`, `preview-micro.jpg` in the fixtures; the + '09 era wrote `QuickLook/Preview.pdf`). Deliberately not an iWork feature: + odf and ooxml carry thumbnails too, so rendering a preview in place of a + decode is one cross-cutting mechanism with one honest capability story, to be + decided library-wide — not invented here, where it would make + `translate_html = true` claim a fidelity this engine does not have. + +## Test data + +`empty.pages` and `style-various-1.pages` are already in +`test/data/input/odr-public/pages/` but absent from `index.csv` and from +reference output. Add them to the index in stage 1, and regenerate reference +output when stage 2 flips `translate_html` on. + +Stages 5 and 7 each need a fixture that does not exist yet — one `.key` and one +`.numbers` in the public repo. Everything at container level stays inline, per +stage 1. diff --git a/src/odr/internal/json/PLAN.md b/src/odr/internal/json/PLAN.md new file mode 100644 index 000000000..14e20b250 --- /dev/null +++ b/src/odr/internal/json/PLAN.md @@ -0,0 +1,185 @@ +# JSON plan + +Where the json module is going, and in what order. Written before stage 1; keep +it honest as stages land. Sibling of [`csv/PLAN.md`](../csv/PLAN.md), and it +leans on what stage 1 there already landed in `internal/encoding`. + +## Today + +`JsonFile` is detection only — 35 lines, `is_decodable() == false`, +`FileCategory::text` — so `html::translate` routes it through +`translate(file.as_text_file())` to `html::create_text_service` and a json +renders exactly like a `.txt`: numbered lines, no folding, nothing the raw bytes +did not already say. + +`check_json_file` runs `nlohmann::json::parse` over the **whole** stream and +throws the result away (`// TODO limit check size`, `// TODO check if that even +works`, `json_util.cpp:8`). `open_strategy.cpp:288` constructs a `JsonFile` +purely to *name* the type while listing candidates, so a 200 MB json is fully +parsed just to be called a json, and the dom is destroyed immediately. + +The parse reads the **undecoded** stream — `check_json_file(*m_file->file()-> +stream())`, `json_file.cpp:10` — even though `JsonFile` holds a +`text::TextFile` whose `text()` has decoded to UTF-8 since the csv plan's stage +1. nlohmann accepts UTF-8 and skips a UTF-8 BOM, so a UTF-16 json — blessed by +RFC 4627, still emitted by Windows tooling — fails detection today and falls +back to plain text, where `html/text_file.cpp` then renders it correctly. The +rendering path is the one that already got fixed; the parse is what is left. + +json is on the skip list in `html_output_test.cpp:115`, and there is no +`test/src/internal/json/` at all. + +`nlohmann_json` is already a direct link dependency (`CMakeLists.txt:272`). + +## Target + +A json opens on its value tree: indented, foldable, keys visually distinct from +values, and a collapsed subtree saying how much it hides. Read-only, no +round-trip, no schema. The text view stays reachable and stays the floor. + +## Decisions taken up front + +**Render the parsed tree, not the bytes.** Colouring the existing text service +with a lexer is what "syntax highlighting" would mean, and it is the wrong axis: +what makes a large json readable is *folding and structure*, not colour, and +anyone who wants the literal bytes already has the text view. We parse the file +anyway. + +That said, a little colour is nearly free here and worth taking. Walking a typed +dom, every scalar already knows whether it is a string, a number, a boolean or +null, so four CSS classes plus one for keys cost an `if` in the writer — no +lexer, no grammar, nothing that can disagree with the parser. That is the whole +of the "highlighting" this needs. + +**Not a document.** `ElementType` (`document_element.hpp:87`) has no key/value +pair. Mapping objects onto `list`/`list_item` either drops the keys or smuggles +them into `text`, and then every binding walking the tree sees prose where the +file had structure — worse than the honest answer, which is that the element +model does not describe json. So `is_decodable()` stays `false`, json stays +`FileCategory::text`, and the html service is hand-written like +`html/text_file.cpp` and `html/pdf_file.cpp` rather than derived from the +generic renderer. That is a deliberate deviation from step 3 of "Adding / +extending a document format" in the root `AGENTS.md`; it belongs in a +`json/AGENTS.md` once stage 3 lands. + +If a json-as-data api is ever wanted, the honest shape is a small value api in +the public headers, not an `ElementAdapter`. Nothing here forecloses it. + +**The service parses; the file detects.** The tempting move is to keep the dom +on `JsonFile` so it is parsed once. Resist it. It puts `nlohmann/json.hpp` — the +single heaviest header in the build — into `json_file.hpp` and from there into +`open_strategy.cpp`; the escape (`nlohmann/json_fwd.hpp` plus a +`std::unique_ptr` member and an out-of-line destructor) is a real technique but +it buys nothing, because after stage 2 detection no longer builds a dom to +share. So: detection reads a bounded prefix and keeps nothing, the html service +parses `TextFile::text()` once in its constructor and owns the dom for its own +lifetime, and nlohmann stays confined to `json_util.cpp` and +`html/json_file.cpp`. + +**The text view is the floor.** A bounded probe accepts a prefix; the full parse +can still fail on byte 900 000. So `html::translate(const TextFile &)` builds +the json service inside a `try` and falls back to `create_text_service` when the +parse throws — a broken json renders as text with its syntax error visible, +which is the useful outcome. This is the one place the fallback lives; no other +call site needs to know. + +**Dispatch by narrowing, not by a new handle.** `translate(const DecodedFile &)` +switches on category (`html.cpp:213`) and json is `text`, so json arrives at +`translate(const TextFile &)`. Branch there on +`file_type() == FileType::javascript_object_notation`. The alternative — a +public `JsonFile` handle with `as_json_file()`, the way csv just got one — costs +the handle plus its surface in four bindings, and csv needed it only because it +had `options()` and `document()` to hang there. json has neither. Revisit if +json ever grows per-file options; view-shaping knobs belong in `HtmlConfig` +anyway. + +**Detection is a discriminator, not a validity check.** RFC 8259 allows a bare +scalar at the top level, so `42` and `"hello"` are valid json documents — and +also valid txt, and valid one-column csv. Requiring the first token to be `{` or +`[` is the same kind of rule as csv's "at least two columns": it is about +telling formats apart, not about what json is. The by-type path +(`open_strategy.cpp:167`, reached when the caller says "this is json" or the +extension does) parses in full and accepts what RFC 8259 accepts. + +**Number spelling is not preserved.** nlohmann keeps `std::int64_t` / +`std::uint64_t` / `double`, not the source token, so re-serialising a scalar +gives the shortest round-trip form: `1.50` renders as `1.5`, `1e3` as `1000.0`. +The value is faithful, the spelling is not. Accepted — the text view is one +click away and shows the file. Emitting the raw token instead would mean a +second, token-preserving parse, which is a large cost for a cosmetic gain. + +**Strings out of the dom are valid UTF-8.** nlohmann rejects invalid UTF-8 at +parse time, so the writer never has to sanitise — it takes +`get_ref()` straight through `html::escape_text` +(`html/common.hpp:48`). Only the *probe* can hold a `U+FFFD`, from a multi-byte +sequence cut at the boundary, and a replacement character inside a string +literal is syntactically fine. + +**Folding is `
`/``.** Native, scriptless, static html, and it +survives the fragment-writing `HtmlService` shape without a frontend object. The +summary line carries what the fold hides — `"users": [ … 128 items ]` — which +the dom knows for free. Expand-all/collapse-all needs script; add it when +someone asks. + +**Truncation is loud.** The renderer emits one row per scalar, so the browser's +node budget is the real limit and it arrives long before ours does. Collapsing +does not help: a `
`-hidden subtree is still in the DOM. So cap the +number of *emitted* values and close the output with a visible +`… 4 213 891 more values, not rendered`, rather than silently stopping. +`HtmlConfig::spreadsheet_limit` (`html.hpp:124`) is the precedent for where the +knob goes. Streaming a json too large for that is out of scope, exactly as csv +stage 5 is. + +--- + +## Stage 1 — parse decoded text + +A bug fix that ships alone, and the smallest possible diff. + +- `check_json_file` takes a `std::string_view` of decoded UTF-8 instead of an + `std::istream` of raw bytes; `JsonFile` passes `m_file->text()`. +- a non-decodable encoding is not a json: RFC 8259 §8.1 makes json Unicode by + definition, so there is nothing to fall back to. `text()` already throws + `UnsupportedTextEncoding`, and `open_strategy`'s `catch (...)` already turns + that into "not a json". +- first `test/src/internal/json/json_util_test.cpp`, inline string literals: an + object, an array, a bare scalar, a truncated document, a UTF-16LE json with a + BOM. + +## Stage 2 — bounded detection + +Stops type listing paying for a full dom, and splits detection from parsing the +way the csv plan did — same reason, same shape. + +- probe with `encoding::read_probe` + `encoding::to_utf8` + (`encoding/detect.hpp`, `encoding/transcode.hpp`), 64 KiB, the same default + the rest of the codebase uses. +- verdict from `nlohmann::json::sax_parse` with a callback that keeps no dom and + stops at the cut: first non-whitespace byte is `{` or `[`, and no syntax error + before the end of the probe. A prefix is *supposed* to end mid-document, so + running out of input is not a rejection — only a real syntax error is. +- `open_strategy` keeps constructing a `JsonFile` to probe, because that + constructor *is* the probe now and is bounded — as with csv. +- the by-type path stays a full parse and keeps accepting top-level scalars. + +## Stage 3 — the tree view + +- `internal/html/json_file.{hpp,cpp}` with `create_json_service`, alongside + `create_text_service`; one view, `json.html`. +- `html::translate(const TextFile &)` narrows on file type and falls back to the + text service when the full parse throws. +- the writer: nested `
`s with a CSS indent (not `
` — we control every
+  byte of the output), `
`/`` per object and array with the + child count in the summary, `odr-json-key` / `-string` / `-number` / + `-literal` classes, `escape_text` on everything, node budget with the loud + marker. +- `write_json_style` in `html/frontend.cpp` next to `write_text_style`. +- drop `javascript_object_notation` from the skip list at + `html_output_test.cpp:115`. + +## Stage 4 — only if wanted + +Auto-expand depth and node budget as `HtmlConfig` fields; expand/collapse-all +and search (both need script); copy-a-json-pointer per row; NDJSON +(`application/x-ndjson`) as a separate file type rendering as a list of trees. +None of it is needed for the view to be worth having. diff --git a/src/odr/internal/markdown/PLAN.md b/src/odr/internal/markdown/PLAN.md new file mode 100644 index 000000000..0de11438b --- /dev/null +++ b/src/odr/internal/markdown/PLAN.md @@ -0,0 +1,243 @@ +# Markdown plan + +Where markdown support is going, and in what order. Written before stage 1; +keep it honest as stages land. + +## Today + +`FileType::markdown` exists (`file.hpp:73`) and has a table row with extensions +and mime types (`file_type_table.cpp:431`), declared classification-only: +capabilities `{}`, `FileCategory::text`, `DocumentType::unknown`. Nothing +constructs it. `magic` does not know it, `open_strategy` never names it, and +`open_file(file, FileType::markdown)` falls through to +`UnsupportedFileType` (`open_strategy.cpp:198`). A `.md` opened today comes back +as `text_file` and renders as a line list through +`html::create_text_service`. + +So there is no code to undo. The work is a decoder, plus flipping the row. + +## Target + +A markdown file opens as a **text document** — `TextRoot` with paragraphs, +spans, lists, tables, links and images — so the generic HTML renderer and every +binding get it without format-specific code. That is the whole argument for +doing it this way rather than writing a markdown→HTML renderer next to +`html/text_file.cpp`: the latter would produce html output only, with no +element api, nothing for JNI/embind/pybind/ObjC, and no path to +`back_translate`. + +CommonMark is the base; GFM's tables, strikethrough, task lists and permissive +autolinks are the extensions worth having. Everything beyond that is somebody's +dialect and stays out. + +## Decisions taken up front + +**A library, not a hand-rolled parser.** This is the opposite call from +`d8c8715` (dropping vincentlaucsb-csv-parser to scan csv in-tree), and +deliberately so. Csv is a hundred lines of quote-aware scanning. CommonMark has +~650 conformance cases, and the places hand-written parsers rot — lazy +continuation, link reference definitions, the emphasis left/right-flanking +rules, list-item indent arithmetic — are exactly the ones that look easy for a +weekend and are then wrong forever. + +**md4c** (`md4c/0.5.2` on conancenter, MIT, ~5k LOC C). Chosen over cmark-gfm: +cmark-gfm has no conan recipe at all, builds an allocating AST we would +immediately walk once and throw away, and is a GitHub fork with a lax release +cadence. md4c is a SAX parser — `enter_block` / `leave_block` / `enter_span` / +`leave_span` / `text` — which maps onto `create_element` / `append_child` with a +single id stack and no intermediate tree. Its `MD_DIALECT_GITHUB` flag is +exactly the extension set above. Add it to `conanfile.py:51` next to +`nlohmann_json` / `uchardet`, and `find_package` + link in `CMakeLists.txt:59` +/ `:271`. + +Two things to check when wiring it up, because they decide small amounts of +work in stage 2: +- whether the conan package exports the `md4c-html` component, whose + `entity.h`/`entity_lookup` resolves the ~2000 named html entities. If not, we + handle `&`-class and numeric references ourselves and pass the rest + through literally. +- md4c parses bytes and assumes UTF-8 (`MD4C_USE_UTF8`), so decoding happens + before it, not inside it. + +**Markdown is a `DocumentFile`, not a `TextFile`.** `abstract::TextFile` fixes +`file_category()` to `text` (`abstract/file.hpp`); a document has to be +`FileCategory::document` with `DocumentType::text`. The table row changes +category with it. This is api-visible for `FileType::markdown` — and free, +because the row declares no capabilities today, so nothing can be relying on +it. + +**Input is UTF-8, produced by `internal/encoding`.** `MarkdownFile` takes a +`std::shared_ptr` exactly as `CsvFile` and `JsonFile` do +(`csv_file.hpp`), and calls `text()` for the decoded bytes. An encoding we +cannot decode throws `UnsupportedTextEncoding` (`exceptions.hpp:42`) — the same +rule the csv plan sets for the sheet path, and for the same reason: +`Text::content()` is UTF-8 to every binding, so passing legacy bytes through +and letting a browser sort it out is not available to us. + +**Detection is by caller, not by content.** `open_strategy` is entirely +content-driven — magic plus speculative probes — and has no extension path +anywhere. Markdown has no signature, and a content probe for it is a probe for +"prose with occasional punctuation", which is every plain text file with a `#` +comment or an `*` bullet in it. Sniffing would steal `text_file` matches and be +confidently wrong. So: `detect_by_content` stays **false**, markdown never joins +the speculative chain in `list_file_types` (`open_strategy.cpp:272`), and the +only way in is `DecodedFile(file, FileType::markdown)` (`file.hpp:341`) via a +new branch in `open_file` next to the text/csv/json ones +(`open_strategy.cpp:146`). Callers route on the filename, which is what they +already have. + +The consequence to accept: `.md` still opens as `text_file` by default. That is +correct behaviour for a format that is, by construction, valid plain text. + +**There is no `NoMarkdownFile`.** Every other format's exception exists because +detection rejects. Nothing rejects here: md4c is total — any UTF-8 byte +sequence is some markdown document — and there is no detection to fail. The +only failure mode is the undecodable encoding above. + +**Headings ride on paragraphs, as ODF already does.** `ElementType` has no +heading (`document_element.hpp:87`), and `odf_parser.cpp:304` maps `text:h` to +`ElementType::paragraph` with the level carried in the style. Markdown does the +same: `MD_BLOCK_H` with `detail->level` becomes a paragraph plus a `TextStyle` +holding `font_size` and `font_weight` from a small `StyleRegistry`. The cost is +honest and shared with odt — the html output is `

` with inline styles, not +`

`, so heading semantics are lost to screen readers and to anything walking +the api for an outline. Stage 5 is where that gets fixed properly, once, for +both engines. + +**Raw html is dropped.** CommonMark passes html blocks and inline html through +verbatim. There is no passthrough element in the model, and inventing one means +deciding what `Text::content()` returns for it in four bindings — a real +question, not a markdown question. `MD_BLOCK_HTML` and `MD_SPAN`-level raw html +are skipped in stage 1. Inline `` in a markdown file therefore renders as +nothing, which is worse than the status quo for exactly one use case, and is +the price of not opening that seam early. + +**Images stay external.** `ImageAdapter` (`abstract/document.hpp:495`) offers +`image_is_internal()` plus `image_href()`, so `![](diagram.svg)` becomes a +`Frame` + `Image` with `is_internal() == false` and the href passed through +untouched. No filesystem, no relative-path resolution, no fetching. A viewer +resolves it against wherever it got the document, which is the only party that +knows. + +## Graphs and drawings + +Worth separating two things that get said in one breath. + +*Inline SVG* is the raw-html question above, and blocked behind the same +decision. + +*Mermaid / graphviz / plantuml fences* are not a markdown feature at all — they +are a fenced code block with an info string, which a downstream renderer picks +up. Rendering them in C++ means embedding a graph layout engine, which is a +larger project than markdown support entire, and one that duplicates what every +frontend already ships. The right move is to keep the fence as a code block +that *carries its language*, and let the frontend render it. There is nowhere +to put that language today — which, with the heading level, makes two entries +for the same stage 5. + +## Element mapping + +| md4c | model | +|---|---| +| `MD_BLOCK_DOC` | `root` (`TextRoot`), default `PageLayout` as `doc_document.cpp:144` | +| `MD_BLOCK_H` (level 1–6) | `paragraph` + heading `TextStyle` | +| `MD_BLOCK_P` | `paragraph` | +| `MD_BLOCK_UL` / `OL` | `list` | +| `MD_BLOCK_LI` | `list_item` | +| `MD_BLOCK_QUOTE` | `group` + left `margin` on the paragraphs inside | +| `MD_BLOCK_CODE` | `paragraph` + monospace `TextStyle`; info string dropped | +| `MD_BLOCK_HR` | — (nothing in the model) | +| `MD_BLOCK_TABLE` / `THEAD` / `TBODY` / `TR` / `TH` / `TD` | `table` / `table_row` / `table_cell` | +| `MD_SPAN_EM` / `STRONG` / `DEL` | `span` + `font_style` / `font_weight` / `font_line_through` | +| `MD_SPAN_CODE` | `span` + monospace `font_name` | +| `MD_SPAN_A` | `link` | +| `MD_SPAN_IMG` | `frame` + `image` | +| `MD_TEXT_NORMAL` / `ENTITY` / `CODE` | `text` | +| `MD_TEXT_BR` | `line_break` | +| `MD_TEXT_SOFTBR` | a space appended to the current text | +| `MD_TEXT_NULLCHAR` | U+FFFD | + +Two renderer gaps this exposes, both pre-existing: +`html::translate_list` hardcodes `
    ` (`html/document_element.cpp:342`), so an +ordered list renders as bullets; and `ElementType::group` renders as its +children with no wrapper (`:72`), so a blockquote's structure survives only in +the margins its paragraphs carry. + +## Module layout + +Mirrors `oldms/text`, which is the reference the root `AGENTS.md` points at. + +``` +src/odr/internal/markdown/ + markdown_file.hpp/.cpp abstract::DocumentFile over a text::TextFile + markdown_document.hpp/.cpp internal::Document + the ElementAdapter + markdown_element_registry.hpp/.cpp flat vector, side maps for text/link/image + markdown_parser.hpp/.cpp md4c callbacks → registry, one id stack + markdown_style.hpp/.cpp StyleRegistry: heading sizes, code font, quote margin +``` + +Every `.cpp` goes into `ODR_SOURCE_FILES` (`CMakeLists.txt:86`). + +--- + +## Stage 1 — blocks + +The skeleton, end to end, with the least that is worth rendering. + +- md4c as a conan dependency; `MarkdownFile` over `text::TextFile`; the + `open_file` branch; the table row flipped to `FileCategory::document`, + `DocumentType::text`, `{.open = true, .translate_html = true}`. +- registry, adapter and `Document` per the pattern; `TextRootAdapter`, + `ParagraphAdapter`, `TextAdapter`, `LineBreakAdapter`. +- the block half of the table above: doc, headings, paragraphs, code blocks. + Lists and quotes land here too — they are blocks and cost a stack push each. +- tests inline, string literal in, element tree out; no fixture files + (`text_file_test.cpp` is the shape). + +`FileTypeCapabilities.declaration_matches_the_engines` +(`odr_test.cpp:163`) opens two files per type from the test data and asserts the +engines do not exceed the row, so a handful of `.md` samples go into the +test-data repo alongside this stage. + +## Stage 2 — inlines and styles + +- `SpanAdapter`, `LinkAdapter`; emphasis, strong, inline code, links. +- `markdown_style`: one `StyleRegistry` handing out the heading scale, the + monospace face and the quote margin, indexed from the registry the way + `doc_style` is. +- entity and soft-break handling per the table; the `entity_lookup` question + above resolves here. + +## Stage 3 — GFM + +`MD_DIALECT_GITHUB`: tables, strikethrough, task lists, permissive autolinks. +Tables are the substantial one — `TableAdapter` plus row/column/cell — and the +reason to do it before images: it is what people actually put in readmes. + +Task list items have no checkbox in the model. Render the box as text (`☐`/`☑`) +in the item's first text element rather than inventing an element type for it. + +## Stage 4 — images and frontmatter + +- `FrameAdapter` + `ImageAdapter`, external hrefs per the decision above. +- YAML/TOML frontmatter: md4c does not know it, so strip a leading `---` fence + before parsing and expose what it holds through `FileMeta`. Parse only the + flat scalars we have somewhere to put (`title`, `author`, `date`); do not + take on a YAML dependency for the rest. + +## Stage 5 — the two model gaps + +Both are api changes shared with the other engines, which is why they come +last and together rather than being smuggled in with the parser. + +- **heading level** — the odf mapping loses it too. Either an + `ElementType::heading` with a level, or a level on `ParagraphStyle`. The + latter is smaller and lets `html::translate_paragraph` emit `

    `…`

    ` for + odt and markdown at once; the former is more honest about it being a + different kind of thing. Decide with odf in the room. +- **code-block language** — a `std::string` on the paragraph, or a dedicated + code element. This is what makes the mermaid story work without odr rendering + anything, and it is the only reason a frontend can syntax-highlight. + +Deferred beyond this: raw html passthrough, footnotes, definition lists, +`back_translate` to markdown, and any dialect that is not GFM. diff --git a/src/odr/internal/rtf/PLAN.md b/src/odr/internal/rtf/PLAN.md new file mode 100644 index 000000000..b82a859dd --- /dev/null +++ b/src/odr/internal/rtf/PLAN.md @@ -0,0 +1,286 @@ +# RTF plan + +Where an rtf module would go, and in what order. Written before stage 1; keep +it honest as stages land. + +## Today + +Nothing decodes. `FileType::rich_text_format` exists (`file.hpp:62`, under the +"Detection only" comment), `magic.cpp:123` matches `7B 5C 72 74 66 31` +(`{\rtf1`), and `file_type_table.cpp:390` carries a row with `rtf` extensions, +three mime types, `FileCategory::document`, `DocumentType::unknown` and +`{.detect_by_content = true}`. `open_strategy::open_file` has no branch for it, +so `odr::open` on an rtf reaches the fallthrough and throws `UnknownFileType`. + +The public enum is already mirrored by the bindings (`bind_file.cpp:39`, +`ODRFile.mm:43`), so **no binding work is needed for any stage below** — the +ordinal is spent, and everything else arrives through the abstract document +model the generic renderer already walks. + +## Spec + +`offline/documentation/MSFT-RTF/MSFT-RTF-080320/` — the RTF Specification +version 1.9.1 (March 2008, the final one). Cite section names in code, the way +`oldms/` cites `[MS-DOC] §2.4.1`; RTF's spec is not numbered, so cite the +heading (e.g. *Conventions of an RTF Reader*, *Table Definitions*, *Pictures*) +and the control word. + +Not the same document as `[MS-OXRTFEX]` (Exchange's HTML-in-RTF encapsulation) +or `[MS-OXRTFCP]` (RTF compression) — neither is in scope. + +## Target + +An rtf opens as a text document — `root → paragraph → span → text`, tables and +frames where the file has them — so the generic HTML renderer and every binding +get it without format-specific code. `DocumentType::text`, `open` and +`translate_html` in the table row; `is_editable()` false, `save` throws, exactly +as `oldms/text`. + +## Decisions taken up front + +**Mirror `pdf`, not `oldms`.** RTF is a *text* format: `{`, `}`, +`\controlword`, `\'hh`, and literal bytes. None of `oldms/text`'s +machinery applies — no CFB container, no FIB, no piece table, no +`PlcBteChpx`→`ChpxFkp` walk, no packed structs, no host-endianness caveat. What +does apply is `internal/pdf`'s parser shape, and it applies unusually well: + +- **`RtfTokenizer` follows `pdf::ObjectParser`** (`pdf_object_parser.hpp`): hold + `std::istream *` + `std::streambuf *`, prepare the stream once with an + `std::istream::sentry` in the constructor, and expose + `getc`/`bumpc`/`bumpnc(n)`/`ungetc` that throw `std::runtime_error` on + unexpected exhaust. This is not decoration: `\binN` needs a raw n-byte read + mid-stream (`bumpnc`), and `\'hh` needs the same `hex_char_to_int` / + `two_hex_to_char` statics. Those two helpers are six lines — **duplicate + them**, do not reach into `internal/pdf`. (The csv plan's rule: anything + genuinely shared moves out to its own package first, and two hex helpers do + not earn a package.) +- **`read_token()` returns a variant, like `GraphicsOperatorParser::read_operator()`** + — `GroupOpen`, `GroupClose`, `ControlWord{name, optional parameter}`, + `ControlSymbol{char}`, `Text{bytes}`, `Binary{bytes}`, `End`. The spec's + Appendix A sample reader is a callback design over globals; a pull-based token + stream is what the rest of this codebase looks like and is what makes the + tokenizer testable on inline strings. +- **`RtfState` follows `pdf::GraphicsState`** (`pdf_graphics_state.hpp`): + `std::vector stack` that is never empty, `save()` on `{`, `restore()` + on `}`, and a `ContentScope`-style RAII guard for destination groups. The + spec's *Conventions of an RTF Reader* prescribes precisely this ("stores its + current state on the stack … retrieves the current state from the stack"), and + `pdf` already carries the leniency precedent — "an unmatched `Q` is ignored: + the state the running content stream started from always remains". + +**Leniency is the spec, and is therefore not a violation of fail-fast.** The +root `AGENTS.md` says throw where the spec dictates what to expect. Here the +spec dictates the opposite: an unknown control word "should be ignored", and +`{\*` opens an ignorable destination the reader "should discard all text up to +and including the closing brace" of. A reader that throws on an unknown control +word cannot read any file written by a newer writer, which is the stated design +goal of `\*`. So: + +- **Ignore** unknown control words and control symbols; **skip** `{\*` groups + whose destination we do not implement; ignore an unmatched `}`. +- **Throw** where the spec does dictate: a group left open at EOF, an invalid + hex digit after `\'`, `\binN` running past EOF, nesting past a depth bound + (borrow `ObjectParser::NestingGuard` — the group stack is heap, but the + destination skip is recursive). +- The one place the spec *invites* rejection is `\cellxN` outside a table + ("probably created maliciously"). Not worth a throw at our fidelity; note and + ignore. + +**Text is bytes until a run ends.** `\'hh` yields one *byte*, not one character: +in a Shift-JIS run two consecutive `\'hh` escapes are one character. So the +accumulator is a byte buffer plus the run's `TextEncoding`, flushed through +`encoding::to_utf8` when the encoding changes, the formatting changes, or the +paragraph ends. Decoding per escape would corrupt every multibyte run and is the +single easiest mistake to make here. + +`\uN` is different — it bypasses the byte buffer (flushing it first, to keep +order). Three traps: + +- N is a **UTF-16 code unit, not a code point**, and anything above the BMP + arrives as a *surrogate pair*: two consecutive `\uN`, each with its own `\ucN` + fallback in between. So a high surrogate is held pending and combined with the + low one that follows; only the combined code point goes through + `util::string::append_c32`. Appending each half on its own turns every emoji + into two replacement characters. An unpaired surrogate still standing at a + flush is U+FFFD. +- N is *signed 16-bit*, so `U+F020` arrives as `\u-4064` and must be folded back + with `+ 65536` — before the surrogate test, which is what the folded value is + for. +- `\ucN` gives the number of following characters to skip as the ANSI fallback, + is scoped like a character property (so it lives in `RtfState`, restored by + `}`), defaults to 1, and counts *any* control word or symbol as one + character — with a `\binN` and its payload counting as one. + +**Encoding comes from `internal/encoding`.** `\ansi`/`\mac`/`\pc`/`\pca` set a +document default, `\ansicpgN` overrides it, the font table's `\fcharsetN` (or +`\cpgN`, which supersedes it) sets it per font, and `\fN` selects the font. That +resolution chain lands on a `TextEncoding` and the transcode is +`encoding::to_utf8`. The single-byte tier covers what real-world rtf uses; +`\fcharset128`/`129`/`134`/`136` (Shift-JIS, EUC-KR, GB2312, Big5) hit the +"named but not decoded" tier, so their `\'hh` runs degrade to U+FFFD — acceptable, +because a writer emitting CJK generally emits `\uN` alongside, which is decoded +regardless of the run's encoding. + +**Flat `ElementRegistry`, copied not shared.** Unlike csv, an rtf's element +count is bounded by its paragraph count, so the pattern from the root +`AGENTS.md` applies unchanged and `oldms/text/doc_element_registry.*` is the +template. Every engine has its own registry (`doc`, `ppt`, `xls` all do); copy +it rather than inventing a shared one. + +**No writer.** RTF is the one format here that would be pleasant to *emit*, and +`back_translate` exists. Out of scope, in every stage. + +--- + +## Stage 1 — plumbing, tokenizer, plain text + +The narrowest thing that renders. No formatting at all beyond paragraph +structure, so the tokenizer and the group machinery can be proven before +anything is layered on them. + +- `rtf_tokenizer.{hpp,cpp}` as above, with the delimiter rules from *Control + Word* exactly: a control word is `\` + ASCII letters, terminated by a space + (consumed), by a digit or `-` (a parameter of up to 10 digits follows), or by + any other character (not consumed). **The parameter's terminator is a + delimiter under the same rule** — a space there is consumed, anything else is + left unread. So `\fs24 Text` starts its text at `T`, not at a space, and + `\bin4 ` puts the four raw bytes immediately after the consumed space; + swallowing that space is the difference between correct text and a payload + read shifted by one. A control *symbol* is `\` + one non-letter and takes no + delimiter — a space after it is text. +- `rtf_state.{hpp,cpp}`: the group stack, the current destination, and the + character/paragraph property structs (empty but for `\uc` in this stage). +- destination handling: a table of known destinations; `{\*\unknown` skips to the + matching `}`. The skip must **still tokenize** — a `\binN` inside a skipped + group carries raw bytes that can contain braces, so a brace-counting scan over + raw bytes desyncs. Same reason `\'7b` must not be mistaken for a group open. +- text: `\par` ends a paragraph, `\line` a line break, `\tab` a tab, `\page` a + page break; `\\`, `\{`, `\}` are literal; a bare CR/LF is *not* text and is + dropped; `\~` non-breaking space, `\_` non-breaking hyphen, `\-` optional + hyphen dropped. (`oldms/text`'s `TextCleaner` is the same job with different + spellings.) +- fields need no code: `\fldinst` is `\*`-marked, so the ignorable-destination + rule hides the instruction and leaves `\fldrslt`'s text flowing — the same + "cached result only" position `.doc` takes. +- `rtf_file.{hpp,cpp}` — `RtfFile : abstract::DocumentFile` over a plain + `abstract::File`, shaped like `pdf::PdfFile` (`pdf_file.hpp:16`); a branch in + both `open_strategy::open_file` and `open_document_file`; `types_by_content` + needs nothing, magic already reports the type. +- `file_type_table.cpp:390` → `DocumentType::text`, `.open = true`, + `.translate_html = true`. `odr_test` fails if declared capabilities exceed + what the engine does, so this row moves stage by stage. +- `CMakeLists.txt` `ODR_SOURCE_FILES`. + +**Tests** are inline string literals against the tokenizer and against a small +whole document — the delimiter rules (space eaten after a control word *and* +after its parameter, digit kept, `\b0` vs `\b`), `\'hh` in a windows-1252 run, +`\uN` negative folding, a non-BMP character as a surrogate pair across two +`\uN`, an unpaired surrogate, `\ucN` skipping across a control word, an +unmatched `}`, an unterminated group, and a `\bin` payload containing `}`. + +## Stage 2 — character formatting + +- `{\fonttbl{\fN\fcharsetN Name;}…}` and `{\colortbl;\redN\greenN\blueN;…}` — + both are destinations, both parsed into a `StyleRegistry` alongside the + element registry, mirroring `oldms/text/doc_style.*`. Note the colour table's + leading `;`: entry 0 is the "auto" colour and is empty. +- `\b \i \ul \ulnone \strike \scaps \caps \sub \super \nosupersub`, `\fN`, + `\fsN` (half-points), `\cfN`/`\cbN`, `\highlightN`, `\plain` (reset) → + `TextStyle`, resolved to spans exactly as `.doc` does: equal property sets + share one resolved style, adjacent equal-style runs merge. +- font names interned in the `StyleRegistry` and never mutated afterwards, so + `TextStyle::font_name` (`const char *`) stays valid — the `.doc`/`.xls` rule. + +## Stage 3 — paragraph, section, page + +- `\pard` reset, `\ql \qc \qr \qj`, `\liN \riN \fiN`, `\sbN \saN \slN` → + `ParagraphStyle`. Twips throughout; `Measure` already carries units. +- `\paperwN \paperhN \marglN \margrN \margtN \margbN \lndscpsxn` → + `PageLayout` off `TextRootAdapter::text_root_page_layout`, which stage 1 + returns empty from. +- `\sect`/`\sectd` — with one page layout there is nowhere to put a second + section, so treat a section break as a page break and take the first section's + geometry. Honest and enough; revisit only if a reference document needs it. +- lists: `{\listtext …}` and `{\pntext …}` carry the rendered bullet or number. + Emitting them as literal text is right at this fidelity and costs nothing; + real `ListItem` elements need `{\*\listtable}` + `{\*\listoverridetable}` + + `\lsN\ilvlN` resolution and belong in a later stage, not here. + +## Stage 4 — tables + +The largest single piece, and the one place RTF is genuinely awkward. + +*Table Definitions*: there is no table group. A row is a run of paragraphs each +carrying `\intbl`, cells terminated by `\cell`, the row by `\row`, and the +geometry given by a `` — `\trowd` followed by one `\cellxN` per cell, +where N is that cell's cumulative **right edge** in twips. + +- **Buffer the row; do not stream it.** The spec is explicit that Word 97 wrote + `` before the cells, Word 2002–2007 write it *after* (and repeat it + before), and the grammar admits all three orders. So accumulate cells into a + pending row and apply whichever `` was seen when `\row` fires — and + make a second, identical `\trowd` idempotent rather than a second row. +- column widths are differences of successive `\cellxN`; `\trleftN` is the row's + left edge and the first difference is measured from it. +- `\clmgf`/`\clmrg` (horizontal) and `\clvmgf`/`\clvmrg` (vertical) merges map + onto `TableCellAdapter::table_cell_span` plus `table_cell_is_covered` — the + `f` variants start a merged region, the bare ones continue it, so the span is + only known once the run ends. Another reason the row is buffered. +- consecutive rows sharing a `` are one table; a row whose cell edges + differ starts a new one. `\itapN` gives the nesting depth, with + `\nestcell`/`\nestrow` and `{\*\nesttableprops}` for the inner levels — + implement depth 0 first and treat deeper rows as their own table, then nest. +- borders and shading (`\clbrdr*`, `\trbrdr*`, `\clcbpat`) into `TableStyle` / + `TableCellStyle` last; they are independent of the reconstruction. + +## Stage 5 — images + +- `{\pict …}` is a destination whose payload is hex `#SDATA` by default or + `\binN` + raw `#BDATA`. Decode to a `std::string`, wrap in + `common::MemoryFile` (`common/file.hpp:34`), hand back through + `ImageAdapter::image_file` with `image_is_internal() == true`. +- `\pngblip` and `\jpegblip` render. `\emfblip`, `\wmetafileN`, `\macpict`, + `\pmmetafileN`, `\dibitmapN`, `\wbitmapN` have no decoder here + (`windows_metafile` / `enhanced_metafile` are classification-only file types), + so **drop the frame** rather than emit a broken `` — "pass through what we + don't model". Word-97-era files lean on `\wmetafile8` heavily; anything modern + is png/jpeg. +- **`\nonshppict` must be skipped explicitly.** The pair is + `{\*\shppict{\pict …}}{\nonshppict{\pict …}}` — the second group is the same + image again, for readers that cannot do `\shppict`, and it is *not* `\*`-marked, + so the ignorable-destination rule will not hide it. Miss this and every image + appears twice. +- `\picwgoalN`/`\pichgoalN` are the display size in twips → `Frame` measures; + `\picwN`/`\pichN` are the intrinsic pixel (or metafile extent) size and are the + fallback. `\picscalexN`/`\picscaleyN` are percentages applied on top. +- shapes (`{\shp{\*\shpinst{\sp{\sn pib}{\sv …}}}}`) hold pictures too, and + `{\object\objemb}` holds embedded OLE. Both out of scope; both are skipped + cleanly by the destination rule if left unimplemented. + +--- + +## Deferred, by decision + +- **Headers, footers, footnotes, endnotes, comments** — each its own destination + (`\header*`, `\footer*`, `\footnote`, `\annotation`). Cheap to reach once the + destination table exists, but there is nowhere in the element model to put + them today; `.doc` drops them for the same reason. +- **Real list items** — see stage 3. +- **Style sheet** (`{\stylesheet}`) and `\sN`/`\csN` inheritance. Direct + formatting first; this is the same layer `.doc` still misses (§2.4.6.5 there). + Worth more here than in `.doc`, because rtf writers lean on styles for heading + fonts. +- **Math** (`{\mmath …}`), **drawing objects**, **bidi** (`\rtlch`/`\ltrch` and + the associated-font `\af*` chain), **revision marks**, **East Asian composite + fonts**. +- **Multibyte `\'hh` runs** — blocked on the multibyte tier in the encoding + package, and largely masked by `\uN` in practice. + +## Test data + +There is no `.rtf` anywhere under `test/data`, and that tree is fetched from the +pinned repos rather than vendored. Stage 1–4 parser tests are inline string +literals (which is the better test anyway — an rtf fragment is readable in a +`R"(...)"`), but a render test needs a real fixture committed to +`test/data/input` plus the reference-output regen. Stage 5 needs one either way, +since a picture cannot reasonably be written inline. diff --git a/src/odr/internal/xml/PLAN.md b/src/odr/internal/xml/PLAN.md new file mode 100644 index 000000000..2e9e3b408 --- /dev/null +++ b/src/odr/internal/xml/PLAN.md @@ -0,0 +1,298 @@ +# XML plan + +Where an xml renderer would go, and in what order. Written before stage 1; keep +it honest as stages land. + +## Today + +`FileType::xml` exists (`file.hpp:150`, under the "Classification only" +comment) and carries a table row — `xml` extension, `application/xml` and +`text/xml`, `FileCategory::text`, `DocumentType::unknown`, +`{.detect_by_content = true}` (`file_type_table.cpp:664`). + +Detection already works. `list_file_types` parses the file with +`util::xml::check_xml_file` and reports `[text_file, xml]`, plus +`scalable_vector_graphics` on top when the root element says so +(`open_strategy.cpp:296`). What is missing is everything after that: +`open_file_as` has no `FileType::xml` branch, and `open_file`'s unknown-type +path tries csv, json and svg before falling through to `text::TextFile` +(`open_strategy.cpp:432`). So a `.xml` decodes as `text_file` and renders +through `html::create_text_service` as a numbered line list. + +Which is the whole problem. The xml files anyone opens on purpose — +`content.xml`, `document.xml`, anything a writer emitted rather than a human — +have no newlines in them, so the line list is one line several megabytes wide. + +Already in place and reusable: `NoXmlFile` (`exceptions.hpp:146`), thrown by +`util::xml::parse` (`util/xml_util.cpp:18`); pugixml 1.15 as a dependency +(`conanfile.py:51`); and `internal/encoding` for transcoding. + +## Target + +An xml file opens as `XmlFile` and renders as a **source view**: indented, +syntax-highlighted, foldable, in one self-contained html document with no +JavaScript and no external resources. It stays a `TextFile`. The work is a +decoder shell plus one html service — no `Document`, no element adapters. + +## Why not leave it to the browser + +Every current browser ships an xml tree viewer, and none of them is reachable +from here. They fire on a *response* served as `text/xml`; odr serves nothing — +`HtmlService` hands the host an html document that a WebView displays, and +inside an html document the viewer never engages. + +It would be the wrong lever even if it could be pulled. An +`` PI makes the browser run the XSLT instead of showing the +tree — silently rendering something else entirely. The viewers differ from each +other in folding, attribute display and error reporting. And the two engines +that matter most for this library, Android's WebView and WKWebView, are not the +browsers whose behaviour anyone checked. + +## Decisions taken up front + +**A file-level html service, not a document.** Xml has no document semantics. +Routing it through `ElementAdapter` would mean picking a `DocumentType` that is +a lie, and the generic renderer would have nothing to contribute — there is no +paragraph, no page, no sheet, only nesting. This is the opposite call from the +csv plan, and for a reason that transfers: a csv *is* a sheet, so the model +earns its keep and every binding gets a table for free. An xml file is not a +document that happens to be serialised as xml; it is the serialisation. So it +renders the way an image or a media file does — one `HtmlService`, html output +only, no element api. See also *the archive seam* below, which is where the +api-free choice does eventually cost something. + +**`XmlFile` mirrors `JsonFile`.** `abstract::TextFile` over a +`std::shared_ptr`, constructed with the same probe-in-the- +constructor shape (`json/json_file.cpp:10`), `is_decodable()` false. The table +row flips to `{.detect_by_content = true, .open = true, .translate_html = +true}`. + +The api-visible consequence: a `.xml` that reports `text_file` today will +report `xml`. No binding work — the enumerator already exists and the bindings +mirror the enum by ordinal — but a caller switching on `file_type()` sees the +change. That is the point of the change, and it is the same step csv and json +already took. + +**The tree is not the bytes.** "We can already read it" is true only of the +parsed tree, and a pugixml tree is a normalisation of the file, not a view of +it. Lost, unavoidably: the original indentation, attribute quote style, whether +a character came in as `A` or `A`, and — with `parse_eol` and +`parse_wconv_attribute`, both on by default — line-ending and in-attribute +newline spelling. + +Accept that, deliberately. The alternative is a byte-faithful lexer over the +raw text, which keeps the author's formatting but does nothing for the minified +file that motivated the whole feature, and still has to reconstruct nesting +before it can fold anything. Pretty-printing is the feature; fidelity is the +price. If someone later wants a true "view source", it is a second mode over +the same css, not a redesign — noted under *Deferred*. + +**Parse with `parse_full`, and keep whitespace-only text.** `util::xml::parse` +uses pugixml's defaults, which drop comments, processing instructions, the +declaration and the doctype — invisible in a viewer whose job is to show what +is in the file. `parse_full` is exactly those four added to `parse_default`. + +`parse_ws_pcdata` is a separate flag and a separate question: keeping every +whitespace-only text node preserves fidelity but fills the tree with nodes we +are about to reindent anyway. Take `parse_ws_pcdata_single`, which keeps +whitespace-only text only where it is an element's sole child — so ` ` +survives as content while the newline-and-tab between two sibling elements does +not. Note that this is the flag that makes the mixed-content rule below +decidable at all. + +Do **not** reuse `util::xml::parse` for this; it hard-codes the default flags +and every existing caller wants them. Add the options at the xml module's own +call site. + +**Encoding is declared in band, and pugixml will not honour it.** An xml file +names its own encoding in the declaration, which is better information than +`encoding::detect`'s uchardet guess over a 64 KiB probe. pugixml's +`encoding_auto` resolves UTF-8/16/32 from a BOM or the `` document is read as UTF-8 +and yields invalid UTF-8 in the node strings, silently. (Verify against 1.15 +before relying on the negative — but the design below is right either way.) + +So: read the declaration's `encoding` pseudo-attribute from the head of the +file, map it through `text_encoding_by_name`, transcode with +`encoding::to_utf8`, and hand pugixml UTF-8. Precedence is declaration, then +BOM, then `text::TextFile::encoding()`'s guess. `XmlFile::encoding()` returns +the resolved value, so a caller can show it. An encoding we can name but not +decode throws `UnsupportedTextEncoding`, as the csv sheet path does and for the +same reason: the tree path has to produce UTF-8, and there is no "let the +browser sort it out" once the bytes are inside a parser. + +**Mixed content is not reindented.** `

    a x b

    ` carries significant +whitespace, and nothing short of a schema can tell it from the insignificant +kind. The rule: an element with any non-whitespace text child renders its +children inline, on one line, untouched; an element whose children are all +elements is indented and foldable. This is what every xml viewer does, it is +the one non-trivial rule in the writer, and it is the first thing a test should +pin. + +**Highlighting is server-side spans.** One `` per +token, emitted by the writer. Not a JavaScript highlighter: the output has been +self-contained with no external resources since the css and js moved into the +document (`HtmlResource::is_shipped`, `html.hpp:51`), and shipping a +highlighter would reverse that for a job the writer is already doing as it +walks the tree. + +Classes, following the `odr-text-*` naming in `frontend.cpp`: `odr-xml` on the +root, then `-tag`, `-name`, `-attr`, `-value`, `-text`, `-cdata`, `-comment`, +`-pi`, `-decl`, `-doctype`. Light palette only, as `text_css` is — a dark mode +is a question for every view at once, not for this one. + +**Folding is `
    `/``, with no script.** The disclosure element +gets keyboard access, screen-reader semantics and — the reason it wins — +find-in-page that expands a collapsed section natively, which a `display:none` +toggle does not. Verify the Safari behaviour before promising it; Chrome and +Firefox have done it for years. + +The layout objection is answerable: `details`/`summary` both `display:block`, +`summary::marker` removed via `list-style:none`, indentation carried inside the +summary so the `white-space:pre` flow stays intact. The start tag goes in the +``, the children in the body, the end tag on a line after it. + +The cost, recorded honestly: bulk expand-all/collapse-all needs JavaScript, so +stage 2 ships without it. + +**No line numbers.** The text view has a numbered gutter (`html/text_file.cpp:101`). +Reproducing it here would number *our* lines, not the file's, which for a +reindented minified document is actively misleading. The gutter column goes to +the fold handles instead. + +**Xml is the last resort in detection.** Insert the branch in `open_file`'s +unknown-type path *after* svg and before the text fallthrough +(`open_strategy.cpp:423`), so anything with a more specific reading keeps it. + +Two behaviour changes fall out, both worth naming before they surprise someone. +A flat-xml ODF (`.fods` and friends) has no detection today — the flat mimetypes +are only aliases on the zip-backed rows (`file_type_table.cpp:26`) — so it +currently decodes as text and would now decode as xml. That is an improvement +(a source tree beats a single line) but it is not what a flat ODF should +eventually do, and it must not be mistaken for support. Likewise `.xhtml`, +`.rels`, `.plist` and every rss feed become source views rather than line +lists — correct for a source viewer, and correct that we do not try to *render* +xhtml. + +**No DTD processing, and that is a feature.** pugixml does not resolve external +entities and does not expand internal entity declarations; it handles the five +predefined entities and numeric character references, and leaves `&foo;` as +literal text. For a viewer that opens files from the internet this closes XXE +and entity-expansion attacks by construction rather than by policy. The +fidelity note is the same sentence read the other way: an undefined entity is +shown as written, which for a source view is the right answer anyway. + +**A parse failure falls back to text.** `XmlFile`'s constructor throws +`NoXmlFile`, `open_file` catches it and reaches `text::TextFile`, and a +malformed file renders as the line list it renders as today. Automatic and +correct — with one thing conceded up front: "show me the broken xml" is exactly +when a viewer is most wanted, and the tree path structurally cannot serve it. +That is the strongest argument for the byte-faithful second mode, and it is not +strong enough to build both now. + +## Module layout + +``` +src/odr/internal/xml/ + xml_file.hpp/.cpp abstract::TextFile over a text::TextFile; parse probe, encoding resolution +src/odr/internal/html/ + xml_file.hpp/.cpp create_xml_service — the HtmlService and the writer +``` + +Both `.cpp` go into `ODR_SOURCE_FILES`: the html one next to +`html/text_file.cpp` (`CMakeLists.txt:148`), the module one after `util/` and +before `zip/` (`CMakeLists.txt:252`). + +No `xml_util.cpp` — `internal/util/xml_util` is the shared xml helper and stays +where it is. Anything this module needs that is genuinely general (the +declaration sniff) belongs there, not in a second utility with the same name. + +--- + +## Stage 1 — it opens, and it renders + +The skeleton end to end, flat: highlighted and indented, not yet foldable. + +- `XmlFile` per the `JsonFile` shape; declaration sniff, transcode, + `parse_full | parse_ws_pcdata_single`; `encoding()` resolved as above. +- `open_file_as` gains a `FileType::xml` branch throwing `NoXmlFile`; + `open_file` gains one after svg. Table row flipped to `{.open = true, + .translate_html = true}`. +- `html/xml_file.cpp` with `create_xml_service`, and a `file_type()` branch in + `html::translate(const TextFile &)` (`html.cpp:256`) so xml gets the tree and + everything else keeps the line list. One view, `xml.html`, mirroring + `html/text_file.cpp:26`. +- the writer: declaration, doctype, PI, comment, element, attribute, text and + CDATA, escaped through `html::escape_text` / `escape_attribute`, indented, + each token in its span. The mixed-content rule lands here, not later. +- `xml_css` in `frontend.cpp`, `write_xml_style` in `frontend.hpp`. +- `test/src/internal/xml/xml_file_test.cpp`, inline string literals in, html + out (`text_file_test.cpp` is the shape). Minimum set: minified input + reindents; mixed content does not; comments/PI/doctype/CDATA all survive; + a declared non-UTF-8 encoding decodes; malformed input throws `NoXmlFile`. + +`FileTypeCapabilities.declaration_matches_the_engines` (`odr_test.cpp`) opens +files per type from the test data against the row, so a handful of `.xml` +samples go into the test-data repo with this stage — see *Test data*. + +## Stage 2 — folding + +- `
    `/`` per element with element children, per the + markup and css above. Elements with no children stay a plain line. +- everything open by default. Collapsing by default hurts find-in-page and + hides the thing the user opened the file to see; the only case for it is + size, which stage 3 handles with a threshold rather than a habit. +- the fold handle in the gutter column the line numbers do not occupy. + +## Stage 3 — size + +A `content.xml` is routinely tens of megabytes, and this path multiplies it: +pugixml's dom is roughly 1.5–2× the file plus the in-memory buffer, and a span +per token can be 5–10× the input in emitted html. Both land in a WebView on a +phone. + +- a node budget in `HtmlConfig`, following `spreadsheet_limit` + (`html.hpp:124`) — `std::optional xml_node_limit`, `nullopt` + for unlimited — and, past it, stop and emit a visible truncation notice + rather than a silently short document. +- past a lower threshold, default the fold state to closed below some depth. + This is the only case where collapsed-by-default is right, and it is a + response to a measurement, not a preference. +- measure before choosing the numbers, on a real `content.xml`. + +## Stage 4 — the archive seam + +The filesystem view links every entry as an `application/octet-stream` data url +(`html/filesystem.cpp:110`), so browsing into a zip and looking at +`word/document.xml` downloads it. Routing entries through `html::translate` +instead is a separate feature with its own questions (which types, resource +paths, how deep), but it is the one that turns this from an xml-file viewer +into a way to inspect any office document's parts. Named here so the +dependency is on record; not scoped here. + +## Deferred, by decision + +- **Byte-faithful mode.** A lexer over the raw text, sharing the css, keeping + the author's formatting and — the real payoff — able to render a malformed + file up to the point where it breaks. Wanted; not wanted enough to build two + renderers before one exists. +- **XSLT.** An `` PI is shown as the processing instruction it + is. Applying it means an XSLT engine, which is larger than every format in + this repository put together. +- **Rendering xhtml as html.** Same class of decision, and the answer is no for + the same reason: this is a source viewer. +- **Namespace resolution.** pugixml does not process namespaces + (`svg/svg_util.cpp:15` works around exactly this), and a source view should + show the prefixes the file actually uses. Nothing to do. +- **Expand-all / collapse-all**, and **search within the tree** — both need + JavaScript, and neither is worth being the reason this view starts shipping a + script. +- **Dark mode**, which is a question for `frontend.cpp` as a whole. + +## Test data + +Content, not fixtures, for everything a string literal can express — the parser +and writer tests are inline. The test-data repo needs only what +`declaration_matches_the_engines` opens: a minified `content.xml` lifted from an +odt, a hand-formatted document with comments and a doctype, one non-UTF-8 +declared encoding, and one file that is xml-shaped but malformed.