Skip to content

AsciiDoc → Markdown converter for elastic.co/guide books - #3802

Draft
Mpdreamz wants to merge 25 commits into
mainfrom
feature/ascii-to-md-converter
Draft

AsciiDoc → Markdown converter for elastic.co/guide books#3802
Mpdreamz wants to merge 25 commits into
mainfrom
feature/ascii-to-md-converter

Conversation

@Mpdreamz

@Mpdreamz Mpdreamz commented Aug 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Adds docs-migrate, a standalone CLI that converts the legacy elastic.co/guide AsciiDoc corpus into docs-builder Markdown docsets. Each converted book becomes a self-contained docset that can be served immediately with docs-builder serve.

  • Bare-clone strategy: repos are cloned once, versions checked out via worktrees
  • shared/attributes.asciidoc loaded before each book so product-name substitutions resolve
  • Block anchors emitted as $$$anchor$$$ inline anchors so docs-builder cross-reference validation passes
  • Include files become separate pages regardless of section nesting depth
  • Xrefs rewritten to internal links; guide URLs rewritten to elastic.co/docs equivalents
  • include:: and include-tagged:: directives inside verbatim blocks are resolved (covers ESQL spec pages)
  • ifeval/ifdef/endif markers inside verbatim content are stripped

Getting started

Prerequisites

dotnet build

Full corpus (all books, latest major + one minor)

# 1. Pull conf.yaml from the legacy docs repo
dotnet run --project src/tooling/docs-migrate -- init

# 2. Clone source repos (latest major, 1 minor per major)
dotnet run --project src/tooling/docs-migrate -- clone --majors 1 --minors 1

# 3. Convert
dotnet run --project src/tooling/docs-migrate -- convert --majors 1 --minors 1

# 4. Serve
dotnet run --project src/tooling/docs-migrate -- serve
# → http://localhost:3001

Single book — Elasticsearch reference 8.19 only

dotnet run --project src/tooling/docs-migrate -- init
dotnet run --project src/tooling/docs-migrate -- clone  --book en/elasticsearch/reference --majors 1 --minors 1
dotnet run --project src/tooling/docs-migrate -- convert --book en/elasticsearch/reference --majors 1 --minors 1
dotnet run --project src/tooling/docs-migrate -- serve

Use list to browse available book prefixes:

dotnet run --project src/tooling/docs-migrate -- list

Output lands in .artifacts/migrated/ relative to the working directory (defaults to the OS app-data folder; override with --work-dir).

What's still open

Area Detail
Cross-version links 19 links to versions not in the converted set — reported as warnings, not expected to be zero until the full corpus is converted
ESQL result tables include:: inside |=== table blocks is not expanded; ESQL function pages show the query example but not the result table

Mpdreamz and others added 24 commits August 5, 2026 15:26
@Mpdreamz
Add legacy /guide AsciiDoc-to-Markdown migration tool
8a93a8c 
Implements a native C# converter to migrate elastic.co/guide content
(AsciiDoc served by the legacy elastic/docs system) into Markdown
docsets that docs-builder can render.

New project: Elastic.LegacyDocs.Migration with:
- conf.yaml parser for the elastic/docs book definitions
- Source repo manager for resolving branches and paths
- AsciiDoc lexer, parser, and AST (sections, code blocks, tables,
  admonitions, lists, includes, conditionals, inline formatting)
- Markdown emitter targeting MyST syntax
- Page chunker to split documents by section level
- Archive and Latest docset generators producing docset.yml/toc.yml
- CLI command: `docs-builder guide migrate`

Made-with: Cursor
@Mpdreamz
Fix migration output to pass docs-builder validation
c66dc6a 
Fixes found by running the converter end-to-end and building with
docs-builder:

- Fix section level off-by-one in lexer (= is level 0, == is level 1)
- Fix Repos model to handle nested YAML objects (Dict<string, LegacyRepo>)
- Fix basePath in generators to use index file directory for includes
- Fix chunker to walk into level-0 sections and promote chunked
  sections to H1 for standalone pages
- Use docs-builder anchor syntax (# Title [anchor]) instead of MyST
  (anchor)= targets
- Map unsupported directives: caution→warning, sidebar→admonition
- Add cross-page anchor-to-slug map so refs emit file links
- Add bare URL[text] pattern to inline parser
- Prevent double image path prefix

Made-with: Cursor
@Mpdreamz
Extract migration to isolated CLI tool and add missing parser features
44565f6 
- Add DSV/CSV/TSV table format support (384 occurrences in ES|QL docs)
- Add pass:[] and +inline+ passthrough inline patterns
- Add unconstrained bold (**text**) support
- Add [role]#text# inline macro support
- Extract migration command into standalone docs-migrate CLI tool
  (removable by deleting src/tooling/docs-migrate/ and
  src/authoring/Elastic.LegacyDocs.Migration/)
- Remove migration coupling from docs-builder

Made-with: Cursor
@Mpdreamz @claude
Refactor docs-migrate into subcommands and switch to bare-clone + wor…
68b5ac3 
…ktree strategy

- Split monolithic MigrateCommand into init/list/clone/convert/serve subcommands
- SourceRepoManager now uses bare clones + git worktrees + sparse checkout (cone mode)
  to avoid redundant full clones per branch and only fetch needed directory trees
- Add Proc package for process execution in place of raw System.Diagnostics.Process
- Add guide-nav feature flag to configuration layer (FeatureFlags, DocumentationSetFile,
  ConfigurationFile) so converted docsets can signal the guide-archive nav mode

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Fix conf.yaml parser missing nested section groups
18cbfaf 
The conf.yaml structure has up to 3 levels of nesting: a category's
sections can contain sub-groups (with base_dir + sections) rather than
books directly. LegacyBook had no Sections property so IgnoreUnmatchedProperties
silently dropped all nested books, causing entire families to disappear
(Elasticsearch clients, all APM agents, ECS logging, etc.).

Add BaseDir and Sections to LegacyBook, then recursively flatten after
deserialization, accumulating the base_dir prefix so leaf book prefixes
are fully qualified (e.g. en/elasticsearch/client/net-api).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Normalize docs-migrate default work dir to AppData
067f6c2 
Use LocalApplicationData/elastic/docs-migrate as the default work
directory, matching the pattern from Paths.ApplicationData in the main
codebase. Includes the same Docker/CI fallback to GetTempPath() when
LocalApplicationData returns an empty string.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Write conversion output to cwd/.artifacts/migrated
668b721 
Output from convert and serve now lands in .artifacts/migrated relative
to the current working directory, separate from the AppData work dir
that holds conf.yaml and repo clones.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz
Add --no-hud flag to serve and fix branch resolution for repos withou…
16a0316 
…t versioned branches

- Add --no-hud to docs-builder serve: disables diagnostics HUD and
  skips background in-memory validation builds
- docs-migrate serve passes --no-hud automatically
- Fix SourceRepoManager branch resolution: check local branches before
  fetching, use --detach for worktrees so repos like elastic/docs
  (only has master) can serve multiple version worktrees
- Use Proc.Start for branch probing to suppress expected git errors

Made-with: Cursor
@Mpdreamz
Rewrite guide URLs to internal links and emit docs-builder substitutions
2ccf3bb 
- Rewrite https://www.elastic.co/guide/... URLs to absolute internal
  markdown links (/prefix/version/page.md) so docs-builder validates them
- Emit {{name}} (double braces) for unresolved AsciiDoc attribute
  references so docs-builder recognizes them as substitution variables
- Add SharedAttributes with ~130 well-known product-name attributes
  from shared/attributes.asciidoc (es, kib, agent, fleet, etc.)
- Write subs: section in generated docset.yml with all product names

Made-with: Cursor
@Mpdreamz
Preserve block anchors as docs-builder inline anchors
87db152 
AsciiDoc [[id]] anchors on non-section blocks (paragraphs, tables,
description lists, etc.) were silently discarded. Now they are wrapped
in an AnchoredBlock node and emitted as $$$id$$$ inline anchors that
docs-builder renders as <a id="..."></a> link targets.

Made-with: Cursor
@Mpdreamz @claude
Fix conditionals inside verbatim blocks, anchor map coverage, and sou…
22e296c 
…rce attributes

- Fix lexer: conditionals inside verbatim/table blocks were being parsed
  as conditional tokens instead of raw text; now only recognized outside
  those contexts
- Fix table delimiter regex to accept |=== with 3+ = signs
- Fix parser: attribute substitution in section titles and image alt/title;
  preserve pending id/title/blockAttr across list-continuation blocks;
  track pendingStart to reset pos on section-level break
- Fix page chunker: include AnchoredBlock IDs in anchor-to-slug map
- Fix ConvertCommand: pass source_branch and per-repo root attributes to
  parser; guard against missing Current version when building overview

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Migrate docs-migrate CLI to Nullean.Argh
cbf569e 
docs-migrate was the last CLI still on ConsoleAppFramework, which had no
PackageVersion entry and so failed restore under Central Package Management.
Align it with docs-builder and essc: Host.CreateApplicationBuilder +
AddArgh, app.Map<T>() registration, and CancellationToken in place of Cancel.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Fix converter defects blocking elastic.co/guide book migration
9499a2c 
Fixes eight root-cause bugs that produced garbled output across 21,962+
files in the converted corpus. All changes are covered by a new
tests/Elastic.LegacyDocs.Migration.Tests/ project (19 tests).

Step 0: Add test project, diagnostics channel (OnDiagnostic on
  AsciidocParserOptions), and AGENTS.md test-project table row.

Step 1: Fix `--` open blocks — lexer dispatched on delimChar before
  checking length so `--` fell into the verbatim path. Now length < 4
  routes to BlockDelimiter; IsMatchingDelimiter/IsMatchingClose require
  exact length so `--------` can no longer close a `----` block.
  PageChunker now recurses into OpenBlockNode for anchor collection and
  page extraction.

Step 2: Add IncludeDirective case to ParseBlock's switch — includes
  were silently skipped (SkipToken) instead of being processed.

Step 3: Attribute resolution — eager expansion via SetAttribute helper,
  SetBasePath keeps docdir in sync, Path.GetFullPath normalization in
  ProcessInclude fixes 863 `../../shared/attributes.asciidoc` paths,
  docs-root/asciidoc-dir seeded in ConvertCommand, docdir saved and
  restored across include file boundaries.

Step 4: Emitter substitution — undefined attributes emit `{name}` not
  `{{name}}`; ProductNames keys pass through as `{{name}}` for
  docs-builder subs. SetAttribute skips ProductNames so shared/
  attributes.asciidoc cannot shadow them.

Step 5: Callout annotations — populate CodeBlockNode.Callouts from
  trailing `<n> text` lines after a code block closing delimiter,
  making the emitter's ordered-list output live.

Step 6: include-tagged:: — new ResolveVerbatimIncludes handles
  `include-tagged::` directives inside listing blocks; tag/end regexes
  widened to allow hyphens and drop the //prefix requirement.

Step 7: Xref regex uses non-greedy `.+?` to allow `>` in link text;
  multi-line admonition paragraphs collect continuation Text tokens
  instead of truncating at the first line.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Fix 4 conversion quality bugs in MarkdownEmitter
f053db4 
- Headings: apply attribute substitution to document/section titles so
  {es} → {{es}} rather than leaking as raw text
- Same-page xrefs: compare resolved anchor slug against current page
  slug; emit #anchor for same-page and slug.md#anchor for cross-page
- AnchoredBlock: emit a blank line after $$$anchor$$$ so docs-builder
  treats it as a standalone inline anchor block
- titleabbrev: PassthroughNode wrapping <titleabbrev>…</titleabbrev>
  was missed by the ParagraphNode filter; add the check to the
  PassthroughNode emit branch

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
docs-migrate: --minors flag, default port 3001, version URL routing fix
8589ff6 
- Add --minors N to clone and convert commands to cap minor versions
  per major; stored in FilterOptions and persisted to .clone-options.json
- Change docs-migrate serve default port from 3000 to 3001
- Fix /8.19 URL routing: Path.GetExtension treats version segments like
  8.19 as having extension .19; restrict hasKnownExtension check to
  actual document file types so version-number slugs resolve correctly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Fix xref display text, link attr substitution, and chunk level
aaf25f9 
- Build AnchorToTitleMap alongside AnchorToSlugMap in PageChunker;
  when a section has <titleabbrev>, use that as the display title
  (matches the abbreviated nav title on elastic.co, e.g. "Indices and
  documents" vs full "Indices, documents, and fields")
- EmitCrossRef: resolve <<anchor>> with no text to the section title
  from AnchorToTitleMap, not the raw anchor ID
- EmitLink: apply SubstituteTitleAttrs to link text so {es}//{kib}
  references in URL link text become {{es}}/{{kib}} instead of leaking
  as single-brace literals
- ConvertCommand: conf.yaml chunk:N maps to chunkLevel N+1 in our AST
  (chunk:1 → chunkLevel 2 creates separate pages for === sections,
  matching elastic.co's page-per-section structure; 423 → 3376 pages)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Fix xref conversion: inline regex ordering, bold/italic spanning, DL …
5fb5c2d 
…multi-line collection, title xrefs

- Prevent double-backtick AsciiDoc curly-quotes from opening a code span by adding
  (?<!`) lookbehind on the mono pattern (group 16)
- Restrict bold (groups 13,14) and italic (group 15) from matching across < so they
  cannot consume <<anchor>> xref openers in cross-mark content (e.g. *word*<<xref>>)
- Fix ParseDescriptionList to collect consecutive Text tokens into one paragraph
  (matching ParseParagraph behaviour) so xrefs that wrap across continuation lines
  within a DL item are kept in a single ParseInlines call
- Add SubstituteTitleXrefs helper to MarkdownEmitter and apply it to section headings,
  document titles, and callout descriptions, which previously used SubstituteTitleAttrs
  only and left <<anchor,text>> patterns unconverted
- Add ifeval:: lexer fix: relax middle group from .+ to .* to allow empty attribute name
- Add GetBlockAttributeRegex trailing-whitespace trim to avoid false negatives
- Add test coverage: curly-quote context, ordered-list items, bold-asterisk interference

Result: raw xref leftovers in en/elasticsearch/reference/8.19 drop from 273 to 9 files.
The remaining 9 are source-level bugs (malformed single >), code-block placeholders,
or xrefs genuinely inside code spans -- all correct not-converted behaviour.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Inline include:: directives inside pipe tables
e9204b8 
When a pipe table contains an include:: on its own line (typical
for shared common-options snippets), the lexer emits it as a Text
token (not IncludeDirective) because the inTable state flag is not
set during include resolution. ParseTable now detects the include::
pattern in Text tokens, reads the referenced file, and appends its
| ... | rows directly to the current table instead of leaking the
directive text into the last cell.

Also handles the IncludeDirective token type for future-proofing.

Result: include:: leftovers drop from 58 to 24 (remaining 24 are
{esql-specs} references to external test fixture files that are not
part of the repository and cannot be resolved).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Fix callout collection: skip comments/blank after code blocks, add Pa…
dc25c2c 
…rseCalloutList for orphan markers

- After a listing block, skip any trailing comment tokens (// TEST[...]) and
  one optional blank line before attempting callout collection; restore position
  if no callout markers follow, preventing skipped blank lines from orphaning later content
- Add ParseCalloutList routed from ParseBlock for standalone <n> callout lines
  that appear after continuation text or in other contexts where the code-block
  collector already broke; collects multi-line descriptions via continuation join
- Lexer: relax ifeval:: middle group .+ → .* to allow empty attribute name
- Lexer: allow trailing whitespace on block attribute lines [attr]\s*

Result: raw <n> callout markers in en/elasticsearch/reference/8.19 drop from ~396 to ~90 files.
Remaining cases are multi-callout blocks where <1> description wraps across lines,
breaking the code-block collector before reaching <2>+.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Fix multi-line callout descriptions and DL inline xref spanning
1855311 
- ParseParagraph: stop collecting Text tokens at callout markers (<n>)
  so ParseCalloutList gets to handle the run instead of having them swallowed
  into paragraph content
- ParseDescriptionList: same guard in the inline Text-continuation loop
- Code-block callout collector: collect continuation lines for each <n>
  description so multi-line callout text is fully captured rather than
  truncated at the first line
- ParseDescriptionList: when inline description text (after ::) contains
  an unclosed <<, join continuation Text tokens until the xref closes,
  enabling multi-line xref spanning within DL items

Result: callout markers drop from 90 to 0; residual raw xrefs drop from 23
to 22 all of which are either inside code spans/fences (correct behaviour)
or source-level malformed >> (not fixable).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Load shared/attributes.asciidoc before converting each book
0fc5fe6 
Pre-parse docs-repo/shared/attributes.asciidoc using the existing AsciidocParser
with branch/version seed attributes already set, so feature and product-name
abbreviations like {transform}, {ilm-init}, {anomaly-detect} and cross-guide
URL attributes like {ref}, {logstash-ref} are resolved at conversion time.

ProductNames keys (es, kib, etc.) are still excluded from eager expansion so they
continue to emit as {{name}} docs-builder substitution placeholders that get
resolved from the generated docset.yml subs: block.

Adds AsciidocParser.LoadAttributeFile static helper and ResolvedAttributes
property to support this pre-parse pattern.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Convert AsciiDoc typographic double quotes to straight quotes
11b8be3 
Add group 23 to InlineCombinedRegex to match ``text'' (AsciiDoc opening
double-backtick + closing double-apostrophe) and emit as "text" (straight
double-quote pair). The existing lookbehind on the mono group already
prevents `` from opening a code span, so group 23 matches the remaining
cases that fall through to raw text. Eliminates 21 occurrences of visible
AsciiDoc typographic quote syntax in en/elasticsearch/reference/8.19.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Fix chunking: each included file becomes a separate page regardless o…
338fb33 
…f section level

- Add IsIncludeRoot flag to SectionNode to mark sections produced at the
  top level of a ProcessInclude call
- Set IsIncludeRoot=true in ParseSection's IncludeDirective handler and in
  ProcessInclude's main loop
- Fix chunkLevel calculation: conf.yaml chunk:N maps directly to AST level N,
  so pass book.Chunk directly instead of book.Chunk+1
- Fix ExtractPages to recurse into inline sections (level > chunkLevel) when
  looking for nested IsIncludeRoot sections — this handles the case where an
  include appears after a discrete section, which nests the included content
  under the discrete section in the AST
- Fix CollectAnchors to also recurse into inline sections for the same reason
- Add IncludeChain_EachIncludedFile_BecomesASeparatePage test that mirrors
  the elastic.co search-your-data structure with 3-level include nesting and
  discrete sections

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Mpdreamz @claude
Fix: emit $$$anchor$$$ before H1 headings so docs-builder registers them
791e1e4 
docs-builder's anchor registry includes H2+ heading slugs and $$$...$$$ inline
anchors but NOT H1 heading slugs. Cross-references like
[text](page.md#page-anchor-id) were triggering 96 'does not exist' errors
because the page-level anchor (from the original [[block-anchor]] before the
first section) was emitted as '# Title [#anchor-id]' — a format docs-builder
parses for H2+ but does not register in the anchor collection for H1.

Fix: emit $$$anchor-id$$$ on a standalone line before H1 headings (both from
document.Title and from Level=0 SectionNode). The $$$...$$$ syntax is the
docs-builder InlineAnchor format, which is collected via
document.Descendants<InlineAnchor>() and included in markdown.Anchors.

With this fix: 0 errors and 0 warnings from docs-builder serve.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown

Label error. Requires exactly 1 of: automation, breaking, bug, changelog:skip, chore, ci, dependencies, documentation, enhancement, feature, fix, redesign. Found:

@Mpdreamz @claude
Fix verbatim include resolution, ifeval stripping, and +..+ passthrou…
8008879 
…gh bug

- Extend ResolveVerbatimIncludes to handle standard include::[tag=] and
  full-file include::[] directives inside verbatim blocks (fixes 164 ESQL pages)
- Strip ifeval::/ifdef::/endif:: conditional markers from verbatim block
  content where they can't be evaluated (fixes 2 files)
- Add lookbehind to constrained +..+ passthrough regex so '8.0+' and similar
  version indicators don't start a passthrough span (fixes xref leaking past
  the passthrough into prose)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LF2iCfVK3ecJgPeFCP6p6S
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Mpdreamz