From 602d2d61c0d5608f534facbb720dfbbff830f92a Mon Sep 17 00:00:00 2001 From: Justin Dunham Date: Thu, 30 Jul 2026 09:05:29 -0700 Subject: [PATCH 1/2] Add learn page on triaging reader feedback with AI. --- .../ai-triage-act-on-reader-feedback-docs.md | 70 +++++++++++++++++++ learn/ai-for-docs/sidebars.yaml | 2 + 2 files changed, 72 insertions(+) create mode 100644 learn/ai-for-docs/ai-triage-act-on-reader-feedback-docs.md diff --git a/learn/ai-for-docs/ai-triage-act-on-reader-feedback-docs.md b/learn/ai-for-docs/ai-triage-act-on-reader-feedback-docs.md new file mode 100644 index 000000000..0cd789544 --- /dev/null +++ b/learn/ai-for-docs/ai-triage-act-on-reader-feedback-docs.md @@ -0,0 +1,70 @@ +--- +seo: + title: Use AI to triage and act on reader feedback on your docs + description: Turn scattered page ratings, code-snippet reports, and support tickets into a prioritized fix list with AI, then ship the fix through a reviewed pull request. +--- + +# Use AI to triage and act on reader feedback on your docs + +Most docs teams collect reader feedback long before they have a plan to use it. A star rating lands in one dashboard, a comment lands in a support ticket, and a note about a broken code sample shows up in Slack. Each signal is real, but none of them says what to fix first. + +AI is good at this kind of sorting. It can read raw comments, group them by cause, and draft a plan a writer can act on in minutes instead of hours. This article shows how to turn reader feedback into a fix list with AI, then turn that list into shipped changes with a lint check and a reviewed pull request. + +## Why reader feedback piles up faster than teams can act on it + +Reader feedback keeps growing because most teams built a channel to collect it, then never built a step to act on it. A page-level "sentiment" widget invites a thumbs up or down on nearly every guide by default, and a code snippet carries its own report icon, separate from that page score. On top of both, a support ticket or a stray comment from a sales call adds a third stream that never touches the same dashboard as the other two. + +None of these signals is wrong on its own. The real problem is priority: forty comments might point at the same outdated example, while one is just noise from a reader on the wrong page. Without a step that sorts before anyone fixes anything, the list a writer works from is whatever comment they read most recently, not the one that would help the most readers. + +## Where reader feedback comes from + +Redocly ships [page feedback forms](https://redocly.com/docs/realm/config/feedback) by default, in sentiment, rating, comment, mood, and scale types. Readers can [leave that feedback](https://redocly.com/docs/end-user/interact-with-pages) on a Markdown page or on a single API operation. Code snippets carry a separate report icon next to the copy button, for a narrower complaint: this example does not run as written. + +Reunite collects both streams in one [Feedback dashboard](https://redocly.com/docs/realm/reunite/project/feedback). Each item starts with a status of "new," and a writer can mark it archived or spam once it has been checked. Redocly Realm even names a project role after this work: the enterprise [`triage` role](https://redocly.com/docs/realm/access/roles) lets someone manage issues, discussions, and pull requests without full write access. That role is a good reminder that sorting feedback is its own job, separate from fixing it. + +## Use AI to sort feedback into a fix list + +Feeding raw comments to an AI model with no instructions produces the same vague summary a person would write after skimming too fast. Give the model a clear task instead: read every comment since the last release, tag each one by cause, and flag duplicates. That way, a writer sees one fix instead of twelve near-identical complaints. + +```markdown +You are triaging reader feedback for API documentation. + +Input: raw feedback rows (page path, rating or sentiment, comment text, date). + +For each comment, output: +1. category: broken example, missing content, confusing wording, + outdated screenshot, or not-docs-related +2. page path affected +3. a one-line paraphrase of the complaint +4. duplicate_of: the row number of an earlier comment with the + same cause, or "none" + +Group the output by category. Do not invent a page path that is +not in the input. +``` + +Categories matter more than raw counts. A page with ten "confusing wording" comments needs a rewrite. A page with one comment tagged "not-docs-related" needs nothing. Once the model groups rows by cause, a writer can open the Feedback dashboard, filter to one category, and treat the duplicates as a single decision instead of ten separate ones. + +## Before and after: a vague comment becomes a shippable fix + +Before: a reader leaves a two-word comment, "doesn't work," on the webhook signing guide. On its own, that comment tells a writer nothing. + +After: the AI triage step pulls three other comments tagged to the same page. Two of them mention a header name that no longer matches the current API version, so the model drafts a one-line summary: the signature header changed, but the example still shows the old name. That is a specific, assignable fix instead of a mystery. + +The model did not write the fix. It read four scattered comments and produced the one sentence a writer needed to open the file and know what to change. That is what triage is for: turning noise into a task with a clear owner. + +## Let deterministic checks catch the pattern next time + +AI triage is good at spotting a pattern in prose, but it will not stop the same mistake from shipping again on its own. That is where [Redocly CLI's built-in rules](https://redocly.com/docs/cli/rules/built-in-rules) take over. A rule that checks header names against the spec can catch the same drift that caused the webhook complaint, before a reader ever sees it. + +Promote a recurring AI finding into a [lint](https://redocly.com/docs/cli/commands/lint) rule once you have seen the same complaint twice. The first time, a human fixes the page by hand. The second time, a rule fails the build in CI, so the mistake cannot come back without someone noticing it in a pull request, not in a reader comment. + +## Build a triage habit, not a backlog + +At Redocly, we treat page feedback as a metric, not a mailbox. We track an engagement rate (feedback divided by unique page views) and a positive-feedback rate, then read the negative comments closely enough to change the page. That practice is described in [our own SEO write-up](https://redocly.com/blog/seo-api-docs). Our [migration guidance](https://redocly.com/blog/documentation-migration-tips) tells teams to build a process for quickly triaging and fixing user-reported issues, instead of leaving a feedback form nobody reviews. + +A short weekly triage pass keeps the Feedback dashboard from turning into a permanent backlog. Run the AI grouping step once a week, clear the duplicates, and ship the top category. Let lint rules hold the fix in place after that. This rhythm matters more than which tool runs it, because a backlog nobody opens is worse than no feedback form at all. + +## How Redocly can help + +Once feedback is grouped by cause, the fix still has to land as a real change, reviewed and shipped like any other edit. [Reunite](https://redocly.com/reunite) is built for that. Its Feedback dashboard already holds the page ratings and code-snippet reports this triage step sorts through, and its [pull request review](https://redocly.com/docs/realm/reunite/project/pull-request/review-pull-request) shows a reviewer the before-and-after page instead of a raw diff, so a triaged fix gets a real check before it merges. Pair Reunite with [Redocly CLI](https://redocly.com/docs/cli/), so the rule you wrote after the second complaint keeps the same mistake from shipping a third time. diff --git a/learn/ai-for-docs/sidebars.yaml b/learn/ai-for-docs/sidebars.yaml index 7dd3c8899..2b2448871 100644 --- a/learn/ai-for-docs/sidebars.yaml +++ b/learn/ai-for-docs/sidebars.yaml @@ -16,6 +16,8 @@ label: Use AI to review code examples for completeness and accuracy - page: ai-automate-documentation-reviews-pr-workflow.md label: Use AI to automate documentation reviews in your PR workflow +- page: ai-triage-act-on-reader-feedback-docs.md + label: Use AI to triage and act on reader feedback on your docs - page: ai-review-docs-navigation-structure.md label: Use AI to review your docs navigation structure - page: ai-enforce-api-style-guide-at-scale.md From 52c8a934929450d73f135d982e790fd0e87ba213 Mon Sep 17 00:00:00 2001 From: Justin Dunham Date: Thu, 30 Jul 2026 11:31:07 -0700 Subject: [PATCH 2/2] Refresh learn/ai-for-docs draft from latest Sonnet 5 run. --- .../ai-triage-act-on-reader-feedback-docs.md | 65 +++++++++---------- 1 file changed, 29 insertions(+), 36 deletions(-) diff --git a/learn/ai-for-docs/ai-triage-act-on-reader-feedback-docs.md b/learn/ai-for-docs/ai-triage-act-on-reader-feedback-docs.md index 0cd789544..0f99849e9 100644 --- a/learn/ai-for-docs/ai-triage-act-on-reader-feedback-docs.md +++ b/learn/ai-for-docs/ai-triage-act-on-reader-feedback-docs.md @@ -1,70 +1,63 @@ --- seo: title: Use AI to triage and act on reader feedback on your docs - description: Turn scattered page ratings, code-snippet reports, and support tickets into a prioritized fix list with AI, then ship the fix through a reviewed pull request. + description: How to sort, prioritize, and draft fixes for reader feedback on your docs with AI, then route the change through a normal pull request review. --- # Use AI to triage and act on reader feedback on your docs -Most docs teams collect reader feedback long before they have a plan to use it. A star rating lands in one dashboard, a comment lands in a support ticket, and a note about a broken code sample shows up in Slack. Each signal is real, but none of them says what to fix first. +Most doc teams collect reader feedback: a thumbs up or down, a star rating, a code snippet flagged as broken, sometimes a full comment explaining what went wrong. Few teams read all of it the same week it arrives, because a handful of writers cannot triage forty comments and still ship new content. -AI is good at this kind of sorting. It can read raw comments, group them by cause, and draft a plan a writer can act on in minutes instead of hours. This article shows how to turn reader feedback into a fix list with AI, then turn that list into shipped changes with a lint check and a reviewed pull request. +That backlog is not a sign the feedback is unimportant. It is a sign nobody has a fast way to sort the useful reports from the noise. -## Why reader feedback piles up faster than teams can act on it +This article covers how to use AI to group, prioritize, and draft a first-pass fix for reader feedback, then hand the result to the same pull request review your docs already use for spec changes. -Reader feedback keeps growing because most teams built a channel to collect it, then never built a step to act on it. A page-level "sentiment" widget invites a thumbs up or down on nearly every guide by default, and a code snippet carries its own report icon, separate from that page score. On top of both, a support ticket or a stray comment from a sales call adds a third stream that never touches the same dashboard as the other two. +## Where reader feedback comes from -None of these signals is wrong on its own. The real problem is priority: forty comments might point at the same outdated example, while one is just noise from a reader on the wrong page. Without a step that sorts before anyone fixes anything, the list a writer works from is whatever comment they read most recently, not the one that would help the most readers. +Reader feedback rarely arrives through one channel. A project can collect page-level "sentiment," rating, mood, scale, and comment [feedback forms](https://redocly.com/docs/realm/config/feedback) at the bottom of Markdown and API reference pages, plus a separate report button next to code snippets for readers who hit a broken example. Readers can also [leave a comment on the page](https://redocly.com/docs/end-user/interact-with-pages) after most form types, and logged-in readers' feedback carries an email automatically so a writer can follow up. -## Where reader feedback comes from +All of it lands in one place: a project's [Feedback tab](https://redocly.com/docs/realm/reunite/project/feedback), split into Page feedback and Code reports, where each entry starts with a status of New and can be moved to Archived or Spam once someone has looked at it. That status field is built for triage, but somebody still has to read every row and decide what "New" should become. + +## Why raw feedback piles up untouched -Redocly ships [page feedback forms](https://redocly.com/docs/realm/config/feedback) by default, in sentiment, rating, comment, mood, and scale types. Readers can [leave that feedback](https://redocly.com/docs/end-user/interact-with-pages) on a Markdown page or on a single API operation. Code snippets carry a separate report icon next to the copy button, for a narrower complaint: this example does not run as written. +A rating without a comment tells you little on its own: a two-star page could mean confusing wording, a missing prerequisite, or a reader who wanted a feature the product does not have. A comment field mixes real bugs with one-off requests and the occasional typo report, so a writer opening the tab cold has to read every entry before knowing which ones are worth ten minutes of their day. -Reunite collects both streams in one [Feedback dashboard](https://redocly.com/docs/realm/reunite/project/feedback). Each item starts with a status of "new," and a writer can mark it archived or spam once it has been checked. Redocly Realm even names a project role after this work: the enterprise [`triage` role](https://redocly.com/docs/realm/access/roles) lets someone manage issues, discussions, and pull requests without full write access. That role is a good reminder that sorting feedback is its own job, separate from fixing it. +Volume compounds the problem. A handful of comments a week is manageable by hand, but once a docs site gets meaningful traffic, the backlog grows faster than a small team can read it, and the reports that matter (a broken auth example, a missing endpoint) sit next to reports that do not (a reader who disagreed with a design choice). -## Use AI to sort feedback into a fix list +## Use AI to triage incoming feedback -Feeding raw comments to an AI model with no instructions produces the same vague summary a person would write after skimming too fast. Give the model a clear task instead: read every comment since the last release, tag each one by cause, and flag duplicates. That way, a writer sees one fix instead of twelve near-identical complaints. +Export the current batch of feedback and code reports as CSV or JSON, then hand it to AI with a short prompt instead of asking someone to read every row cold. -```markdown -You are triaging reader feedback for API documentation. +### Group similar comments before you read them one by one -Input: raw feedback rows (page path, rating or sentiment, comment text, date). +Ask AI to cluster the export by page and by theme: broken code samples, missing steps, terminology confusion, and everything else. Five readers describing the same broken `curl` example in five different ways should collapse into one item, not five separate tickets a writer re-reads independently. -For each comment, output: -1. category: broken example, missing content, confusing wording, - outdated screenshot, or not-docs-related -2. page path affected -3. a one-line paraphrase of the complaint -4. duplicate_of: the row number of an earlier comment with the - same cause, or "none" +### Ask AI to separate the fixable from the unclear -Group the output by category. Do not invent a page path that is -not in the input. -``` +Within each cluster, ask AI to flag which comments point at something specific enough to act on and which ones need a follow-up question before anyone can fix anything. -Categories matter more than raw counts. A page with ten "confusing wording" comments needs a rewrite. A page with one comment tagged "not-docs-related" needs nothing. Once the model groups rows by cause, a writer can open the Feedback dashboard, filter to one category, and treat the duplicates as a single decision instead of ten separate ones. +A vague entry like "this page is confusing" gives a writer nothing to act on. A triaged version, "the auth example on the OAuth page is missing the token refresh step a reader hit at the two-star rating," names the page, the missing step, and the reader's blocker. That second version is what a human should see first. -## Before and after: a vague comment becomes a shippable fix +## Turn a triaged item into a drafted fix -Before: a reader leaves a two-word comment, "doesn't work," on the webhook signing guide. On its own, that comment tells a writer nothing. +Once AI has surfaced the items worth acting on, ask it to draft the fix instead of only describing the problem. Point AI at the page in question and the triaged comment, and ask for a proposed edit: an added prerequisite, a corrected code sample, a reworded step. [Use AI to accelerate and improve reviews](https://redocly.com/learn/ai-for-docs/ai-reviews) covers a related pattern for briefing AI with a short checklist rather than a long style guide; the same discipline applies here, since a focused prompt about one reader's blocker produces a more usable draft than an open-ended "fix this page" request. -After: the AI triage step pulls three other comments tagged to the same page. Two of them mention a header name that no longer matches the current API version, so the model drafts a one-line summary: the signature header changed, but the example still shows the old name. That is a specific, assignable fix instead of a mystery. +A useful prompt names the page, quotes the triaged comment, and states what "fixed" looks like, for example: "A reader on the OAuth page reported that the token refresh step is missing after the initial auth example. Draft a short paragraph and code sample that adds that step in the same style as the rest of the page." That level of detail gives AI one job instead of an open invitation to rewrite the page. -The model did not write the fix. It read four scattered comments and produced the one sentence a writer needed to open the file and know what to change. That is what triage is for: turning noise into a task with a clear owner. +Treat the draft as a starting point, not a merge-ready change. AI can misread which step a reader meant, or propose a fix that contradicts a decision the team already made on purpose, so a human still reads the draft against the original comment before it goes anywhere near a pull request. -## Let deterministic checks catch the pattern next time +## Route the fix through review, not straight to production -AI triage is good at spotting a pattern in prose, but it will not stop the same mistake from shipping again on its own. That is where [Redocly CLI's built-in rules](https://redocly.com/docs/cli/rules/built-in-rules) take over. A rule that checks header names against the spec can catch the same drift that caused the webhook complaint, before a reader ever sees it. +A drafted fix still needs the same checks any other documentation change gets. [Use AI to automate documentation reviews in your PR workflow](https://redocly.com/learn/ai-for-docs/ai-automate-documentation-reviews-pr-workflow) describes a three-layer pattern that fits here directly: an AI checklist pass on the diff, [Redocly CLI lint](https://redocly.com/docs/cli/commands/lint) against the spec when the fix touches an OpenAPI file, and a human reviewer who decides whether the change is right before it merges. -Promote a recurring AI finding into a [lint](https://redocly.com/docs/cli/commands/lint) rule once you have seen the same complaint twice. The first time, a human fixes the page by hand. The second time, a rule fails the build in CI, so the mistake cannot come back without someone noticing it in a pull request, not in a reader comment. +Open the AI-drafted fix as a normal pull request and link back to the source feedback entry so the reviewer can see the original reader comment next to the proposed change. That link matters because a reviewer approving a fix in isolation cannot tell whether it actually addresses what the reader reported. -## Build a triage habit, not a backlog +## Close the loop with the reader -At Redocly, we treat page feedback as a metric, not a mailbox. We track an engagement rate (feedback divided by unique page views) and a positive-feedback rate, then read the negative comments closely enough to change the page. That practice is described in [our own SEO write-up](https://redocly.com/blog/seo-api-docs). Our [migration guidance](https://redocly.com/blog/documentation-migration-tips) tells teams to build a process for quickly triaging and fixing user-reported issues, instead of leaving a feedback form nobody reviews. +Once the fix merges, update the feedback entry's status and, when the reader left an email, send a short note that the page changed because of their report. That last step is easy to skip and it is the one that turns a feedback form from a suggestion box nobody checks into a channel readers trust enough to keep using. -A short weekly triage pass keeps the Feedback dashboard from turning into a permanent backlog. Run the AI grouping step once a week, clear the duplicates, and ship the top category. Let lint rules hold the fix in place after that. This rhythm matters more than which tool runs it, because a backlog nobody opens is worse than no feedback form at all. +Reader feedback and [AI usability testing](https://redocly.com/learn/ai-for-docs/ai-usability-testing) surface different problems: usability testing shows what a simulated reader hits on a task you chose, while page feedback shows whatever a real reader happened to notice. Running both means less goes unnoticed between the two. ## How Redocly can help -Once feedback is grouped by cause, the fix still has to land as a real change, reviewed and shipped like any other edit. [Reunite](https://redocly.com/reunite) is built for that. Its Feedback dashboard already holds the page ratings and code-snippet reports this triage step sorts through, and its [pull request review](https://redocly.com/docs/realm/reunite/project/pull-request/review-pull-request) shows a reviewer the before-and-after page instead of a raw diff, so a triaged fix gets a real check before it merges. Pair Reunite with [Redocly CLI](https://redocly.com/docs/cli/), so the rule you wrote after the second complaint keeps the same mistake from shipping a third time. +[Reunite](https://redocly.com/reunite) collects reader feedback directly on the page, through sentiment, rating, comment, and code-snippet report forms, and stores it in a Feedback tab you can triage without leaving the platform. Because Reunite is also where documentation changes move through Git branches, commits, and pull requests, the same surface that captures a reader's comment can carry the resulting fix through review, so triaging feedback and shipping the fix do not require switching tools.