Skip to content

fix: preserve all PRD sections during revision#64

Closed
ekuris-redhat wants to merge 1 commit into
mainfrom
fix/prd-executive-summary-preserved-on-revision
Closed

fix: preserve all PRD sections during revision#64
ekuris-redhat wants to merge 1 commit into
mainfrom
fix/prd-executive-summary-preserved-on-revision

Conversation

@ekuris-redhat
Copy link
Copy Markdown
Collaborator

@ekuris-redhat ekuris-redhat commented Jun 1, 2026

Summary

  • Root cause: _strip_preamble checked text.find("\n#") before text.startswith("#"). When the LLM revision output started with ## 1. Executive Summary (no document title), the method found \n## 2. Problem Statement and stripped everything before it — including the entire Executive Summary.
  • Fix: Swap the check order — startswith("#") first so content that already begins with a heading is never stripped.
  • Also: Strengthened the shared regenerate.md prompt with explicit preservation rules for all content types (PRD, spec, epic, task).

Changes (3 files)

File Change
src/forge/integrations/agents/agent.py _strip_preamble: check startswith("#") before find("\n#")
src/forge/prompts/v1/regenerate.md Added explicit "Preserve what was not objected to" instructions
tests/unit/prompts/test_prompt_templates.py Regression test for preservation instructions

Test plan

  • All 21 prompt template unit tests pass
  • E2E on AISOS-742 (no new skill, _strip_preamble fix only):
    • Generated PRD → rejected with "change threshold to 500K"
    • Executive Summary: preserved
    • All 11 sections: intact
    • Only requested change applied: 50,000 → 500,000 (3 references, 0 old values remain)

🤖 Generated with Claude Code

@ekuris-redhat ekuris-redhat requested a review from eshulman2 June 1, 2026 11:04
@ekuris-redhat ekuris-redhat force-pushed the fix/prd-executive-summary-preserved-on-revision branch from 4cbc65d to 314b9e7 Compare June 2, 2026 12:01
_strip_preamble checked text.find("\n#") before text.startswith("#").
When the LLM revision output started with "## 1. Executive Summary"
(no document title), the method found "\n## 2. Problem Statement" and
stripped everything before it — including the entire Executive Summary.

Fix: check startswith("#") first so content that already begins with a
heading is never stripped.

Also strengthened the shared regenerate.md prompt with explicit
preservation rules ("Preserve what was not objected to") for all
content types (PRD, spec, epic, task).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ekuris-redhat ekuris-redhat force-pushed the fix/prd-executive-summary-preserved-on-revision branch from 314b9e7 to 376700f Compare June 2, 2026 12:48
@eshulman2
Copy link
Copy Markdown
Collaborator

fixed in #75

@eshulman2 eshulman2 closed this Jun 8, 2026
@ekuris-redhat
Copy link
Copy Markdown
Collaborator Author

Closing this PR — we were unable to consistently reproduce the Executive Summary deletion during PRD revision in local end-to-end testing.

While we did observe _strip_preamble stripping the Executive Summary once (when the LLM output started directly with ## 1. Executive Summary without a document title), the issue is non-deterministic and depends on the LLM's output formatting choices. Across ~15+ revision attempts, the bug only triggered once.

The root cause hypothesis (check order in _strip_preamble + weak regeneration prompt) remains plausible but needs more evidence before merging a fix. Will revisit if the issue is reported again with reproducible steps.

@ekuris-redhat ekuris-redhat deleted the fix/prd-executive-summary-preserved-on-revision branch June 8, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants