DOC-6939 Document feed coverage and the section role vocabulary

content/ai-agent-resources.md

@@ -37,6 +37,34 @@ Each documentation page has a corresponding JSON file at the same URL with `/ind
 - Page: `https://redis.io/docs/latest/commands/set/`
 - JSON: `https://redis.io/docs/latest/commands/set/index.json`
 
+### What the feeds cover
+
+The feeds contain one record for every documentation page. They deliberately do **not**
+contain the taxonomy pages that appear in
+[sitemap.xml](https://redis.io/docs/latest/sitemap.xml), so a straight comparison of the two
+shows the sitemap with more URLs.
+
+The excluded pages are:
+
+- the `categories/` listing and each `categories/<name>` page
+- the `tags/` listing and each `tags/<name>` page
+- the documentation home page
+
+These are generated index pages that list other pages. They carry no documentation prose of
+their own, so a record for them would add navigation noise without adding content. The
+exclusion is a consequence of how the output formats are configured rather than a filter
+applied afterwards: JSON and Markdown are produced for Hugo's `section` and `page` kinds
+only, and taxonomy, term and home pages are none of those.
+
+Two consequences worth knowing if you diff the feed against the sitemap:
+
+- Pages excluded from Hugo's page lists with `_build.list: never` are **absent from the
+  sitemap but present in the feeds**, because they are still built and rendered. They are
+  real documentation pages, usually reference material reached by direct link rather than by
+  browsing.
+- Both figures move. The feed is rebuilt at least daily and the corpus grows, so treat any
+  page count as a snapshot.
+
 ### JSON schema
 
 Each document contains:
@@ -54,9 +82,11 @@ Each document contains:
 | `children` | array | Child pages (index pages only) |
 
 Each **section** contains:
-- `id`: Slugified heading
+- `id`: Slugified heading, matching the heading's anchor on the rendered page, so
+  `<url>#<section id>` links to that section
 - `title`: Original heading text
-- `role`: Semantic role (`overview`, `syntax`, `example`, `parameters`, `returns`, etc.)
+- `role`: Semantic role, assigned from the heading text. See
+  [Section roles](#section-roles) for the current values.
 - `text`: Section content (code blocks replaced with `[code example]` placeholder)
 
 Each **example** contains:
@@ -65,6 +95,40 @@ Each **example** contains:
 - `code`: The code content
 - `section_id`: Which section this example came from
 
+### Section roles
+
+Each section carries a `role`, derived from its heading text. These are the values currently
+in use:
+
+| Role | Assigned when the heading begins with |
+|------|----------------------------------------|
+| `overview` | `overview`, `introduction`, `about`, `description` |
+| `syntax` | `syntax`, `usage`, `command`, `signature` |
+| `example` | `example`, `demo`, `sample`, `code example` |
+| `parameters` | `option`, `parameter`, `argument`, `flag` |
+| `returns` | `return`, `response`, `output`, `result` |
+| `errors` | `error`, `exception`, `troubleshoot` |
+| `performance` | `performance`, `complexity`, `benchmark` |
+| `limits` | `limit`, `constraint`, `restriction` |
+| `related` | `see also`, `related`, `learn more`, `reference` |
+| `setup` | `install`, `setup`, `getting started`, `quickstart` |
+| `configuration` | `configur`, `setting` |
+| `security` | `security`, `auth`, `permission`, `acl` |
+| `history` | `history`, `changelog`, `version history` |
+| `compatibility` | `compatib`, `support`, `version` |
+| `content` | none of the above |
+
+The table is in priority order and the first match wins, which matters where the patterns
+overlap: a heading of "Version history" is `history` rather than `compatibility`, because
+`history` is tested first.
+
+A page's introductory text, before its first heading, is also given the `overview` role.
+
+{{< note >}}This vocabulary is descriptive, not a contract. It reflects the values produced
+today and may gain entries, or change how a heading maps to a role, without notice. If you
+filter or rank on `role`, treat an unrecognized value as `content` rather than discarding the
+section, and do not assume a value you rely on will keep its current name.{{< /note >}}
+
 ### Verifying content_hash
 
 The `content_hash` can be verified by computing: