diff --git a/content/ai-agent-resources.md b/content/ai-agent-resources.md index 74765c041f..36c5973aa6 100644 --- a/content/ai-agent-resources.md +++ b/content/ai-agent-resources.md @@ -37,6 +37,34 @@ Each documentation page has a corresponding JSON file at the same URL with `/ind - Page: `https://redis.io/docs/latest/commands/set/` - JSON: `https://redis.io/docs/latest/commands/set/index.json` +### What the feeds cover + +The feeds contain one record for every documentation page. They deliberately do **not** +contain the taxonomy pages that appear in +[sitemap.xml](https://redis.io/docs/latest/sitemap.xml), so a straight comparison of the two +shows the sitemap with more URLs. + +The excluded pages are: + +- the `categories/` listing and each `categories/` page +- the `tags/` listing and each `tags/` page +- the documentation home page + +These are generated index pages that list other pages. They carry no documentation prose of +their own, so a record for them would add navigation noise without adding content. The +exclusion is a consequence of how the output formats are configured rather than a filter +applied afterwards: JSON and Markdown are produced for Hugo's `section` and `page` kinds +only, and taxonomy, term and home pages are none of those. + +Two consequences worth knowing if you diff the feed against the sitemap: + +- Pages excluded from Hugo's page lists with `_build.list: never` are **absent from the + sitemap but present in the feeds**, because they are still built and rendered. They are + real documentation pages, usually reference material reached by direct link rather than by + browsing. +- Both figures move. The feed is rebuilt at least daily and the corpus grows, so treat any + page count as a snapshot. + ### JSON schema Each document contains: @@ -54,9 +82,11 @@ Each document contains: | `children` | array | Child pages (index pages only) | Each **section** contains: -- `id`: Slugified heading +- `id`: Slugified heading, matching the heading's anchor on the rendered page, so + `#
` links to that section - `title`: Original heading text -- `role`: Semantic role (`overview`, `syntax`, `example`, `parameters`, `returns`, etc.) +- `role`: Semantic role, assigned from the heading text. See + [Section roles](#section-roles) for the current values. - `text`: Section content (code blocks replaced with `[code example]` placeholder) Each **example** contains: @@ -65,6 +95,40 @@ Each **example** contains: - `code`: The code content - `section_id`: Which section this example came from +### Section roles + +Each section carries a `role`, derived from its heading text. These are the values currently +in use: + +| Role | Assigned when the heading begins with | +|------|----------------------------------------| +| `overview` | `overview`, `introduction`, `about`, `description` | +| `syntax` | `syntax`, `usage`, `command`, `signature` | +| `example` | `example`, `demo`, `sample`, `code example` | +| `parameters` | `option`, `parameter`, `argument`, `flag` | +| `returns` | `return`, `response`, `output`, `result` | +| `errors` | `error`, `exception`, `troubleshoot` | +| `performance` | `performance`, `complexity`, `benchmark` | +| `limits` | `limit`, `constraint`, `restriction` | +| `related` | `see also`, `related`, `learn more`, `reference` | +| `setup` | `install`, `setup`, `getting started`, `quickstart` | +| `configuration` | `configur`, `setting` | +| `security` | `security`, `auth`, `permission`, `acl` | +| `history` | `history`, `changelog`, `version history` | +| `compatibility` | `compatib`, `support`, `version` | +| `content` | none of the above | + +The table is in priority order and the first match wins, which matters where the patterns +overlap: a heading of "Version history" is `history` rather than `compatibility`, because +`history` is tested first. + +A page's introductory text, before its first heading, is also given the `overview` role. + +{{< note >}}This vocabulary is descriptive, not a contract. It reflects the values produced +today and may gain entries, or change how a heading maps to a role, without notice. If you +filter or rank on `role`, treat an unrecognized value as `content` rather than discarding the +section, and do not assume a value you rely on will keep its current name.{{< /note >}} + ### Verifying content_hash The `content_hash` can be verified by computing: