feat: 7 - living doc copilot#9
Draft
miroslavpojer wants to merge 60 commits into
Draft
Conversation
…DME and documentation for clarity
…nes and skill overview
…lete glossary file
…mpact tracing, and entity validation - Implemented `next_id.py` for auto-assigning IDs to user stories, features, functionalities, and acceptance criteria. - Created `compute_gaps.py` to analyze a catalog snapshot and identify gaps in documentation and testing coverage. - Developed `trace_impact.py` to trace the impact of code changes on features, functionalities, and user stories based on a catalog. - Added `validate_entity.py` to validate living doc entities against a canonical schema, ensuring required fields and referential integrity.
- Implemented `scan_ac_links.py` to check for missing or malformed AC link headers in Gherkin scenarios. - Validates AC ID format and checks for duplicates within the same feature file. docs: create gherkin-scenario skill documentation - Added `SKILL.md` for `gherkin-scenario` detailing standards for writing BDD scenarios in Gherkin. - Covers traceability requirements, language use, and anti-pattern avoidance. docs: create gherkin-step skill documentation - Added `SKILL.md` for `gherkin-step` outlining best practices for implementing Gherkin step definitions. - Emphasizes keeping steps thin, encapsulating selectors, and sharing state correctly. docs: create living-doc-pageobject-scan skill documentation - Added `SKILL.md` for `living-doc-pageobject-scan` detailing how to generate and maintain PageObject classes. - Describes modes for creating and maintaining PageObjects, including selector preferences and output artifacts. feat: add manifest diff script for PageObject validation - Implemented `manifest_diff.py` to compare the manifest against PageObject files on disk. - Identifies stale manifest entries and undocumented PageObjects. docs: create living-doc-scenario-creator skill documentation - Added `SKILL.md` for `living-doc-scenario-creator` detailing the process of generating BDD scenarios from User Stories. - Includes workflow steps for mapping acceptance criteria to scenarios and identifying missing steps. feat: add coverage report script for AC tracking - Implemented `coverage_report.py` to generate a report on AC coverage by Gherkin scenarios. - Scans feature files for AC links and compares them against User Stories to identify gaps.
- Updated Gherkin step definitions to remove Java and Scala references, focusing on Python behave and TypeScript Cucumber. - Enhanced Living Doc PageObject scan to include TypeScript examples and clarified output artifact locations. - Revised Living Doc Scenario Creator to improve missing step handling and stub generation, ensuring better integration with PageObjects. - Created a comprehensive implementation roadmap for the Agentic Engineering Toolkit, detailing progress, file layout, and validation checklists.
…n skills with new trigger phrases and improved descriptions
- Introduced trigger evaluations for Gherkin living doc sync to identify sync-related queries. - Added evaluation scenarios for Gherkin scenario creation, focusing on writing and reviewing BDD scenarios. - Implemented evaluations for Gherkin step definitions, emphasizing the distinction between Gherkin text and step binding code. - Created evaluations for living doc page object scanning, including bootstrap and maintain modes for PageObject generation. - Established evaluations for living doc scenario creation, generating BDD scenarios from user stories and handling coverage reports.
…s and new trigger phrases
…nality - Updated SKILL.md for living-doc-create-functionality to clarify Functionality naming and acceptance criteria elicitation. - Enhanced living-doc-create-user-story to streamline narrative elicitation and improve AC generation process. - Revised living-doc-gap-finder to normalize script output and report gaps more effectively. - Improved living-doc-impact-analysis to flag missing coverage and provide a re-test checklist. - Enhanced living-doc-pageobject-scan to better handle fragile selectors and update PageObjects. - Updated living-doc-scenario-creator to ensure accurate scenario generation and coverage reporting. - Refined living-doc-update to maintain AC ID stability and improve documentation practices. - Added gap-report.json to track documentation coverage and identify gaps in User Stories and Functionalities.
…-bdd-copilot agents
…y and completeness
…ntext and function naming conventions
…for living documentation analysis
…ent testing; refine trigger phrases in living documentation
…umeric IDs and improve normalization logic
…nsistency and improve documentation clarity
- Updated `.github/copilot-instructions.md` to clarify that the token-saving skill is always active and applies concise formatting rules. - Revised `docs/guides/token-saving.md` to reflect the always-active status of the token-saving skill and provided instructions for enabling it in other repositories. - Changed references from `FUNC-promo-validate` to `FUNC-001` in various files to standardize functionality IDs. - Updated evaluation prompts and expected outputs in `skills/data-cy-instrument` and `skills/living-doc-create-functionality` to reflect the new functionality ID format. - Adjusted gap-finding logic in `skills/living-doc-gap-finder` to ensure consistency with the new functionality IDs. - Improved error handling in `skills/shared/lib/living_doc_id.py` to raise appropriate exceptions for invalid catalog formats. - Refined the description of the token-saving skill to emphasize response discipline and formatting rules.
- Updated living-doc-create-feature and living-doc-create-functionality to use numeric Feature IDs exclusively. - Modified evals and documentation to reflect changes in ID handling and naming conventions. - Enhanced error handling for slug-based Feature IDs in living_doc_id.py. - Adjusted related tests to ensure compliance with new ID standards. - Updated manifest schema and related documentation to use normalized test_id keys.
…he project - Updated compute_gaps.py to normalize AC status checks to lowercase. - Modified SKILL.md in living-doc-scenario-creator to reflect lowercase AC states in documentation. - Adjusted evals.json to use lowercase for AC states in prompts and expected outputs. - Revised coverage_report.py to include lowercase AC states in coverage checks and documentation. - Changed SKILL.md in living-doc-update to use lowercase for active ACs in versioning guidelines. - Updated validate_entity.py to validate AC states as lowercase. - Altered living-doc-bdd-schemas.md to specify lowercase with underscores for AC states. - Modified living-doc-glossary.md to reflect lowercase AC states in definitions. - Updated project-profile.schema.json to describe AC state vocabulary in lowercase with underscores.
…SKILL files; adjust script path for find_unused_po_methods.py
| | [Responsible Agent Use](./responsible-agent-use.md) | Not burning your Copilot token budget — context, models, agent mode, MCP, plugins, skills, and a must-do checklist | | ||
| | [Agent Design Best Practices](./guides/agent-design.md) | Core principles, file structure, context management, tool guidance, examples, and stopping conditions for `.agent.md` files | | ||
| | [Skill Testing](./testing/skill-testing.md) | Eval creation, fixtures, regression loops, trigger and description optimization | | ||
| | [Agent Testing](./testing/agent-testing.md) | Eval creation, trigger accuracy tuning, and body quality testing for `.agent.md` files | |
| | 6 | negative | *(no file — create US redirect to living-doc-create-user-story)* | | ||
| | 7 | negative | *(no file — gap-finding redirect to living-doc-gap-finder)* | | ||
| | 8 | paraphrase | *(no file — add AC phrased as "update the story")* | | ||
| | 9 | edge-case | *(no file — descope AC mid-sprint: status=descoped, do not delete)* | |
Comment on lines
+24
to
+42
| def test_slug_based_feat_ids_raise_error(): | ||
| """Slug-based FEAT IDs should raise ValueError.""" | ||
| catalog = { | ||
| "features": [ | ||
| {"id": "FEAT-checkout", "name": "Checkout Page"}, | ||
| {"id": "FEAT-orders-api", "name": "Orders API"}, | ||
| ] | ||
| } | ||
| try: | ||
| next_entity_id(catalog, "FEAT") | ||
| print("✗ Expected ValueError for slug-based FEAT IDs, but none was raised") | ||
| return False | ||
| except ValueError as e: | ||
| error_msg = str(e) | ||
| assert "slug-based IDs" in error_msg, f"Error message missing slug reference: {error_msg}" | ||
| assert "FEAT-checkout" in error_msg, f"Error should mention FEAT-checkout: {error_msg}" | ||
| print(f"✓ Slug-based FEAT IDs raise clear error (as expected)") | ||
| return True | ||
|
|
Comment on lines
+44
to
+45
| "description": "AC state vocabulary, lowercase with underscores inside '# AC:' blocks and entity files, e.g. [planned, in_review, active, deprecated]." | ||
| }, |
Comment on lines
+36
to
+39
| "feature_id": { | ||
| "type": "string", | ||
| "pattern": "^FEAT-(\\d{3,}|[a-z0-9]+(?:-[a-z0-9]+)*|UNKNOWN)$" | ||
| }, |
| "url": { "type": "string", "minLength": 1 }, | ||
| "title": { "type": "string" }, | ||
| "pageobject_path": { "type": "string", "minLength": 1 }, | ||
| "feature_id": { "type": "string", "pattern": "^FEAT-(\\d{3,}|[a-z0-9]+(?:-[a-z0-9]+)*|UNKNOWN)$" }, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR: Living Documentation & BDD Automation Framework
Overview
This PR introduces the Living Documentation & BDD Automation framework to the Agentic Toolkit — a comprehensive system for creating, maintaining, and synchronizing business-level acceptance criteria with executable Gherkin scenarios and PageObject automation.
The framework unifies requirements management (User Stories, Features, Functionalities) with BDD automation (Gherkin scenarios, step definitions, PageObjects) through a single agent (
living-doc-bdd-copilot) and 11 complementary skills. This enables traceability between business requirements and test automation while reducing maintenance overhead.Total Changes: 93 files changed, 14,419 insertions(+), 28 deletions(-)
Major Components Added
1. Living Documentation Agent
.github/agents/living-doc-bdd-copilot.agent.md.copilot/bdd/.project-profile.yaml) for config-driven conventions2. Living Documentation Skills (7 skills)
living-doc-create-user-storyliving-doc-create-featureliving-doc-create-functionalityliving-doc-updateliving-doc-gap-finderliving-doc-impact-analysisliving-doc-pageobject-scan3. BDD Automation Skills (4 skills)
living-doc-scenario-creatorgherkin-stepgherkin-living-doc-syncbdd-maintain4. Supporting Infrastructure
data-cy-instrumentskill to resolve missingdata-cyattributes in Angular templates and sync PageObjects to usegetByTestId()skills/shared/lib/module with utility functions (living_doc_id.pyfor ID parsing/validation)living-doc-bdd-schemas.md— templates for Project Profile, feature file headers, seed.yaml, manifest.jsonliving-doc-glossary.md— entity definitions, status vocabularies, AC state model, relationship diagramproject-profile.schema.json,seed.schema.json,manifest.schema.jsonbdd-maintain:find_unused_steps.py,find_unused_po_methods.py,find_unused_po_components.pyfor dead code auditsgherkin-living-doc-sync:scan_ac_links.pyto audit AC link healthliving-doc-*:next_id.pyfor entity ID generation across skillsliving-doc-update:validate_entity.pyfor entity validationliving-doc-scenario-creator:coverage_report.pyfor scenario coverage reportsWhat's New
Skills Additions
Documentation Enhancements
docs/guides/living-doc-bdd-copilot.md— end-to-end walkthrough of catalog and automation workflowsdocs/guides/agent-design.md— how to build orchestrating agents with skills and tool orchestrationdocs/testing/agent-testing.md— testing strategies for multi-step agent workflowsdocs/testing/skill-testing.md(updated) — skill evaluation best practicesdocs/responsible-agent-use.md— principles for using agents safely and effectivelydocs/getting-started.md— new section on agents, project-scoped installs, Copilot CLIdocs/troubleshooting.mdwith agent-specific issuesShared References
Configuration & Conventions
test_id_attribute(default:data-cy)Key Files Changed
Agent & Skills
.github/agents/living-doc-bdd-copilot.agent.md— New orchestrating agentskills/living-doc-create-user-story/SKILL.md— New skill + evalsskills/living-doc-create-feature/SKILL.md— New skill + evals + scriptsskills/living-doc-create-functionality/SKILL.md— New skill + evalsskills/living-doc-update/SKILL.md— New skill + evals + scriptsskills/living-doc-impact-analysis/SKILL.md— New skill + evalsskills/living-doc-gap-finder/SKILL.md— New skill + evals + scriptsskills/living-doc-pageobject-scan/SKILL.md— New skill + evalsskills/living-doc-scenario-creator/SKILL.md— New skill + evals + scriptsskills/gherkin-step/SKILL.md— New skill + evalsskills/gherkin-living-doc-sync/SKILL.md— New skill + evals + scriptsskills/bdd-maintain/SKILL.md— New skill + evals + scriptsskills/data-cy-instrument/SKILL.md— New skill + evalsDocumentation
docs/guides/living-doc-bdd-copilot.md— New end-to-end agent walkthroughdocs/guides/agent-design.md— New agent architecture guidedocs/testing/agent-testing.md— New agent testing guidedocs/responsible-agent-use.md— New principles guidedocs/getting-started.md— Updated with agents, Copilot CLI, project-scoped installsREADME.md— Updated skill catalog and agent roster sectionShared Resources
skills/shared/lib/living_doc_id.py— Utility module for ID parsing/validationskills/shared/references/living-doc-bdd-schemas.md— BDD schema templates and referenceskills/shared/references/living-doc-glossary.md— Entity definitions and vocabularyskills/shared/references/schemas/project-profile.schema.json— Project Profile validationskills/shared/references/schemas/seed.schema.json— Business Seed validationskills/shared/references/schemas/manifest.schema.json— Exploration Manifest validationConfiguration
.gitignore— Updated to exclude.DS_Storeand BDD artifact directoriesHighlights & Improvements
Configuration-Driven Toolkit
.copilot/bdd/.project-profile.yaml, making the toolkit portable across projects with different directory layouts, naming conventions, or test-id attributes.Traceability & Impact Analysis
scan_ac_links.pyvalidates that every Gherkin scenario is correctly linked to a living-doc AC.Lifecycle Management
Skill Coordination
Enhanced Documentation
Changes Summary
Acceptance Criteria State Standardization
[Planned, In Review, Active, Deprecated](Title-case)Skill Descriptions & Trigger Phrases
ID Format & Coverage Reporting
AC_001) for consistencyliving_doc_id.pyTesting
Evaluation Coverage
Each skill includes:
Example test counts:
living-doc-create-user-story: 20+ triggers, 247 assertionsgherkin-step: 20+ triggers, 236 assertionsliving-doc-pageobject-scan: 20+ triggers, 265 assertionsAgent Evaluation
living-doc-bdd-copilotagent includes 20+ multi-step workflow testsBreaking Changes
None
This PR is backward compatible. Existing skills remain unchanged. The new living documentation framework is additive — projects can opt-in by:
living-doc-bdd-copilotagent.copilot/bdd/.project-profile.yamlfileInstallation & Getting Started
Quick Install
npx skills add https://github.com/AbsaOSS/agentic-toolkit -g # Or install just the living doc agent: npx skills add https://github.com/AbsaOSS/agentic-toolkit -g --agent living-doc-bdd-copilotFirst Steps
.copilot/bdd/.project-profile.yamlRelated Issues & PRs
Review Notes
For Reviewers
skills/shared/references/schemas/for correctnessliving_doc_id.pyfind_unused_*) for correctnessTesting Recommendation
Run the full skill evaluation suite:
npx skills test --allAuthor Notes
skills/shared/lib/) allows future skills to reuse ID validation, schema parsing, and common utilitiesChecklist
Closes #7