diff --git a/.github/ISSUE_TEMPLATE/docs-reproduction-question.md b/.github/ISSUE_TEMPLATE/docs-reproduction-question.md new file mode 100644 index 0000000..4d62e4e --- /dev/null +++ b/.github/ISSUE_TEMPLATE/docs-reproduction-question.md @@ -0,0 +1,32 @@ +--- +name: Docs reproduction question +about: Ask for clarification when local reproduction docs are unclear or incomplete +title: "[Docs reproduction]: " +labels: ["docs", "reproduction", "reviewer-contract"] +--- + +## Reproduction Doc Or Command + +Which doc section or command was unclear? + +## What You Tried + +```bash +# paste the exact command sequence +``` + +## Where You Got Stuck + +Describe the first unclear step, missing prerequisite, unexpected output, or +documentation mismatch. + +## Expected Operator Path + +What did you expect the docs to tell you next? + +## Boundary Check + +- [ ] This is about local reproduction docs for committed synthetic artifacts. +- [ ] This is not a request for a new demo. +- [ ] No live ingestion, production SIEM, dashboard, alert routing, or case-management scope is involved. +- [ ] No real account IDs, credentials, or private data are included. diff --git a/README.md b/README.md index b521439..b8ee404 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,8 @@ Latest tagged release: [v1.0 — reviewer contract release](https://github.com/s - [`docs/reviewer-brief.md`](docs/reviewer-brief.md): scope, value, evidence, and boundaries - [`docs/reviewer-path.md`](docs/reviewer-path.md): choose the right demo by review question - [`docs/reviewer-pack.md`](docs/reviewer-pack.md): demo matrix, artifact contract, and v1 readiness gate -- [`docs/operator-reproduction.md`](docs/operator-reproduction.md): shortest local path from clone to artifact regeneration +- [`docs/operator-reproduction.md`](docs/operator-reproduction.md): shortest local path from clone to verifying all five demos +- [`docs/release-v1.1.md`](docs/release-v1.1.md): draft v1.1 operator reproduction and issue triage release notes - [`docs/v1-contract-freeze.md`](docs/v1-contract-freeze.md): v1.0 five-demo contract freeze, release status, and contract scope - [`docs/v1-readiness-gate.md`](docs/v1-readiness-gate.md): fixed inputs, fixed outputs, schema validation, artifact regeneration, and test pass requirements - [`docs/release-v1.0.md`](docs/release-v1.0.md): v1.0 reviewer-contract release notes and explicit non-SIEM boundary @@ -73,6 +74,8 @@ python -m telemetry_window_demo.cli run --config configs/default.yaml Use the same Python interpreter for install, tests, and demo commands. On machines with multiple Python installs, replace `python` with the intended interpreter path. To run the test suite in a fresh environment, install the dev extra with `python -m pip install -e ".[dev]"`. +## Verify Locally In 3 Commands + If you want to verify v1.0 locally, run these three commands. ```bash @@ -185,7 +188,8 @@ Cooldown behavior: - [`demos/config-change-investigation-demo/README.md`](demos/config-change-investigation-demo/README.md) explains the config-change investigation demo and its committed artifacts - [`demos/cloud-iam-change-investigation-demo/README.md`](demos/cloud-iam-change-investigation-demo/README.md) explains the synthetic CloudTrail-like IAM investigation demo and its committed artifacts - [`docs/README.md`](docs/README.md) indexes current reviewer docs, supporting design notes, and historical release evidence -- [`docs/operator-reproduction.md`](docs/operator-reproduction.md) gives the shortest local path from clone to artifact regeneration and the release contract gate +- [`docs/operator-reproduction.md`](docs/operator-reproduction.md) gives the shortest local path from clone to running the five demos, artifact regeneration, schema tests, and full tests +- [`docs/release-v1.1.md`](docs/release-v1.1.md) drafts the v1.1 operator reproduction and issue triage release notes - [`docs/reviewer-pack.md`](docs/reviewer-pack.md) is the top-level no-guessing reviewer pack and artifact naming contract - [`docs/v1-contract-freeze.md`](docs/v1-contract-freeze.md) defines the v1.0 five-demo contract freeze gate - [`docs/v1-readiness-gate.md`](docs/v1-readiness-gate.md) defines the fixed-input, fixed-output, schema-validation, artifact-regeneration, and test-pass readiness gate diff --git a/docs/README.md b/docs/README.md index 33c6d3f..7d29f90 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,12 +5,13 @@ This directory separates the current reviewer route from supporting design notes ## Current reviewer route - [`reviewer-pack.md`](reviewer-pack.md): top-level reviewer pack, demo matrix, artifact naming contract, and v1 readiness gate -- [`operator-reproduction.md`](operator-reproduction.md): shortest local path from clone to artifact regeneration +- [`operator-reproduction.md`](operator-reproduction.md): shortest local path from clone to verifying all five demos - [`reviewer-path.md`](reviewer-path.md): choose a demo by review question - [`reviewer-brief.md`](reviewer-brief.md): short problem, value, evidence, and boundary summary - [`v1-contract-freeze.md`](v1-contract-freeze.md): v1.0 five-demo contract freeze, release status, and contract scope - [`v1-readiness-gate.md`](v1-readiness-gate.md): fixed inputs, fixed outputs, schema validation, artifact regeneration, and test pass requirements - [`release-v1.0.md`](release-v1.0.md): v1.0 reviewer-contract release notes with the explicit non-SIEM boundary +- [`release-v1.1.md`](release-v1.1.md): draft v1.1 operator reproduction and issue triage release notes - [`v0.6-to-v1-artifact-diff.md`](v0.6-to-v1-artifact-diff.md): additive artifact contract and compatibility diff from the fourth demo to the fifth - [`evidence-pipeline-contract.md`](evidence-pipeline-contract.md): JSON/JSONL schema contracts for reviewer-facing evidence artifacts - [`reviewer-artifact-diff.md`](reviewer-artifact-diff.md): release diff contract for reviewer-facing artifact changes diff --git a/docs/operator-reproduction.md b/docs/operator-reproduction.md index 652a10c..78086dc 100644 --- a/docs/operator-reproduction.md +++ b/docs/operator-reproduction.md @@ -5,7 +5,7 @@ fresh clone. It does not add a new demo and does not require any live service, cloud account, SIEM, dashboard, alert routing, case-management system, or production telemetry source. -## Fresh Clone +## Install ```bash git clone https://github.com/stacknil/telemetry-lab.git @@ -13,10 +13,33 @@ cd telemetry-lab python -m pip install -e ".[dev]" ``` -Use the same Python interpreter for install, artifact regeneration, schema -validation, and tests. +Use the same Python interpreter for install, demo runs, artifact regeneration, +schema validation, and tests. -## Reproduce v1.0 Artifacts +## Run The Five Demos + +```bash +python -m telemetry_window_demo.cli run --config configs/default.yaml +python -m telemetry_window_demo.cli run-ai-demo +python -m telemetry_window_demo.cli run-rule-dedup-demo +python -m telemetry_window_demo.cli run-config-change-demo +python -m telemetry_window_demo.cli run-cloud-iam-change-demo +``` + +Expected operator checkpoints: + +- `data/processed/summary.json` reports `41` events, `24` windows, and `12` + alerts for `telemetry-window-demo`. +- `demos/ai-assisted-detection-demo/artifacts/case_report.md` reports `3` + deterministic cases. +- `demos/rule-evaluation-and-dedup-demo/artifacts/dedup_report.md` reports + `10` raw hits reduced to `6` retained alerts. +- `demos/config-change-investigation-demo/artifacts/investigation_report.md` + reports `4` normalized changes and `3` investigations. +- `demos/cloud-iam-change-investigation-demo/artifacts/investigation_report.md` + reports `14` CloudTrail-like events and `5` investigation signals. + +## Regenerate Artifacts ```bash python scripts/regenerate_artifacts.py --check @@ -32,7 +55,22 @@ If this command fails, open an artifact regeneration issue with the command, exit code, failing artifact path, and whether the failure is a missing file or content difference. -## Full Reviewer Contract Gate +## Schema Tests + +```bash +python -m pytest tests/test_evidence_pipeline_schemas.py +``` + +This validates reviewer-facing JSON and JSONL artifacts against the schemas in +`schemas/`. + +## Full Tests + +```bash +python -m pytest +``` + +## Release Contract Gate The v1.0 reviewer contract is the exact three-command gate: diff --git a/docs/release-v1.1.md b/docs/release-v1.1.md new file mode 100644 index 0000000..74eb0fd --- /dev/null +++ b/docs/release-v1.1.md @@ -0,0 +1,74 @@ +# v1.1 Operator Reproduction Release Notes (Draft) + +Theme: operator reproduction and issue triage, no demo expansion. + +Release status: draft until a `v1.1` tag or GitHub release is explicitly +published. + +## Scope + +v1.1 keeps the existing five-demo matrix unchanged: + +1. `telemetry-window-demo` +2. `ai-assisted-detection-demo` +3. `rule-evaluation-and-dedup-demo` +4. `config-change-investigation-demo` +5. `cloud-iam-change-investigation-demo` + +This release is about helping an outside operator reproduce the frozen v1.0 +reviewer contract from a clean local checkout. It does not add a sixth demo, a +live data source, a dashboard, alert routing, case management, autonomous +response, or final incident verdicts. + +## Operator Reproduction + +The maintained operator path is [`docs/operator-reproduction.md`](operator-reproduction.md): + +1. install the repo with the dev extra +2. run all five demos locally +3. regenerate committed artifacts with `python scripts/regenerate_artifacts.py --check` +4. validate schemas with `python -m pytest tests/test_evidence_pipeline_schemas.py` +5. run the full test suite with `python -m pytest` + +For a labeled one-command wrapper, run: + +```bash +python scripts/check_release_contract.py +``` + +## Issue Triage + +v1.1 adds focused issue templates for operator-facing contract questions: + +- schema drift reports +- artifact regeneration failures +- demo boundary questions +- documentation reproduction questions + +These templates keep reports scoped to local, synthetic reviewer artifacts and +ask reporters not to include credentials, real account IDs, production telemetry, +or private data. + +## Validation Snapshot + +Refresh this snapshot from the final release commit before publishing v1.1: + +```bash +python scripts/check_release_contract.py +``` + +Expected gate sequence: + +- artifact regeneration +- schema validation +- full test suite + +## Boundaries + +- No demo expansion. +- No live ingestion. +- No production SIEM or dashboard. +- No alert routing or case-management service. +- No autonomous response. +- No final incident verdict. + diff --git a/docs/roadmap.md b/docs/roadmap.md index 17c83bc..abf0340 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -22,7 +22,8 @@ Recently added: - [`docs/vocabulary.md`](vocabulary.md) defines cross-demo evidence workflow terms. - [`docs/reviewer-artifact-diff.md`](reviewer-artifact-diff.md) defines the release diff contract for reviewer-facing artifact changes. - [`docs/v0.6-to-v1-artifact-diff.md`](v0.6-to-v1-artifact-diff.md) records the additive fourth-to-fifth-demo artifact contract. -- [`docs/operator-reproduction.md`](operator-reproduction.md) records the shortest clone-to-regeneration path. +- [`docs/operator-reproduction.md`](operator-reproduction.md) records the shortest clone-to-five-demo-verification path. +- [`docs/release-v1.1.md`](release-v1.1.md) drafts the operator reproduction and issue triage release notes. - `scripts/check_release_contract.py` wraps artifact regeneration, schema validation, and the full test suite. ## v1.1 Operator Reproduction Release @@ -33,7 +34,7 @@ Deliverable focus: 1. Keep the five-demo matrix unchanged. 2. Make clone-to-artifact-regeneration instructions short and explicit. -3. Provide issue templates for schema drift, artifact regeneration failures, and demo boundary questions. +3. Provide issue templates for schema drift, artifact regeneration failures, demo boundary questions, and documentation reproduction questions. 4. Provide one reviewer-friendly release contract gate command. 5. Keep README verification commands copy/paste runnable. diff --git a/tests/test_reviewer_docs.py b/tests/test_reviewer_docs.py index 31ace21..42700e8 100644 --- a/tests/test_reviewer_docs.py +++ b/tests/test_reviewer_docs.py @@ -164,6 +164,7 @@ def test_docs_index_separates_current_route_from_history() -> None: "v1-contract-freeze.md", "v1-readiness-gate.md", "release-v1.0.md", + "release-v1.1.md", "v0.6-to-v1-artifact-diff.md", "evidence-pipeline-contract.md", "reviewer-artifact-diff.md", @@ -535,6 +536,15 @@ def test_operator_reproduction_doc_and_readme_define_short_gate() -> None: assert "# Operator Reproduction" in operator_doc assert "git clone https://github.com/stacknil/telemetry-lab.git" in operator_doc assert "python -m pip install -e \".[dev]\"" in operator_doc + assert "## Run The Five Demos" in operator_doc + for demo_command in [ + "python -m telemetry_window_demo.cli run --config configs/default.yaml", + "python -m telemetry_window_demo.cli run-ai-demo", + "python -m telemetry_window_demo.cli run-rule-dedup-demo", + "python -m telemetry_window_demo.cli run-config-change-demo", + "python -m telemetry_window_demo.cli run-cloud-iam-change-demo", + ]: + assert demo_command in operator_doc assert "python scripts/regenerate_artifacts.py --check" in operator_doc assert "python -m pytest tests/test_evidence_pipeline_schemas.py" in operator_doc assert "python -m pytest" in operator_doc @@ -542,6 +552,7 @@ def test_operator_reproduction_doc_and_readme_define_short_gate() -> None: assert "does not add a new demo" in operator_doc assert "does not claim production readiness" in operator_doc + assert "## Verify Locally In 3 Commands" in readme assert "If you want to verify v1.0 locally, run these three commands." in readme assert "docs/operator-reproduction.md" in readme assert "operator-reproduction.md" in docs_index @@ -556,6 +567,9 @@ def test_operator_issue_templates_keep_reviewer_contract_scope() -> None: "artifact-regeneration-failure.md" ), "demo-boundary-question.md": _read_issue_template("demo-boundary-question.md"), + "docs-reproduction-question.md": _read_issue_template( + "docs-reproduction-question.md" + ), } feature_template = _read_issue_template("feature_request.yml") @@ -574,4 +588,43 @@ def test_operator_issue_templates_keep_reviewer_contract_scope() -> None: "artifact-regeneration-failure.md" ] assert "No new demo expansion for v1.1." in templates["demo-boundary-question.md"] + assert "documentation mismatch" in templates["docs-reproduction-question.md"] + assert "This is not a request for a new demo." in templates[ + "docs-reproduction-question.md" + ] assert "next demo" not in feature_template.lower() + + +def test_v11_release_note_keeps_operator_reproduction_scope() -> None: + release_note = _read_repo_file("docs/release-v1.1.md") + docs_index = _read_repo_file("docs/README.md") + readme = _read_repo_file("README.md") + roadmap = _read_repo_file("docs/roadmap.md") + + assert "# v1.1 Operator Reproduction Release Notes (Draft)" in release_note + assert "Theme: operator reproduction and issue triage, no demo expansion." in release_note + assert "Release status: draft" in release_note + assert "python scripts/check_release_contract.py" in release_note + assert "documentation reproduction questions" in release_note + + for demo_name in [ + "telemetry-window-demo", + "ai-assisted-detection-demo", + "rule-evaluation-and-dedup-demo", + "config-change-investigation-demo", + "cloud-iam-change-investigation-demo", + ]: + assert f"`{demo_name}`" in release_note + + for forbidden_scope in [ + "No demo expansion.", + "No live ingestion.", + "No production SIEM or dashboard.", + "No alert routing or case-management service.", + "No autonomous response.", + "No final incident verdict.", + ]: + assert forbidden_scope in release_note + + for text in [docs_index, readme, roadmap]: + assert "release-v1.1.md" in text