Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .github/ISSUE_TEMPLATE/docs-reproduction-question.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
name: Docs reproduction question
about: Ask for clarification when local reproduction docs are unclear or incomplete
title: "[Docs reproduction]: "
labels: ["docs", "reproduction", "reviewer-contract"]
---

## Reproduction Doc Or Command

Which doc section or command was unclear?

## What You Tried

```bash
# paste the exact command sequence
```

## Where You Got Stuck

Describe the first unclear step, missing prerequisite, unexpected output, or
documentation mismatch.

## Expected Operator Path

What did you expect the docs to tell you next?

## Boundary Check

- [ ] This is about local reproduction docs for committed synthetic artifacts.
- [ ] This is not a request for a new demo.
- [ ] No live ingestion, production SIEM, dashboard, alert routing, or case-management scope is involved.
- [ ] No real account IDs, credentials, or private data are included.
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ Latest tagged release: [v1.0 — reviewer contract release](https://github.com/s
- [`docs/reviewer-brief.md`](docs/reviewer-brief.md): scope, value, evidence, and boundaries
- [`docs/reviewer-path.md`](docs/reviewer-path.md): choose the right demo by review question
- [`docs/reviewer-pack.md`](docs/reviewer-pack.md): demo matrix, artifact contract, and v1 readiness gate
- [`docs/operator-reproduction.md`](docs/operator-reproduction.md): shortest local path from clone to artifact regeneration
- [`docs/operator-reproduction.md`](docs/operator-reproduction.md): shortest local path from clone to verifying all five demos
- [`docs/release-v1.1.md`](docs/release-v1.1.md): draft v1.1 operator reproduction and issue triage release notes
- [`docs/v1-contract-freeze.md`](docs/v1-contract-freeze.md): v1.0 five-demo contract freeze, release status, and contract scope
- [`docs/v1-readiness-gate.md`](docs/v1-readiness-gate.md): fixed inputs, fixed outputs, schema validation, artifact regeneration, and test pass requirements
- [`docs/release-v1.0.md`](docs/release-v1.0.md): v1.0 reviewer-contract release notes and explicit non-SIEM boundary
Expand Down Expand Up @@ -73,6 +74,8 @@ python -m telemetry_window_demo.cli run --config configs/default.yaml
Use the same Python interpreter for install, tests, and demo commands. On machines with multiple Python installs, replace `python` with the intended interpreter path.
To run the test suite in a fresh environment, install the dev extra with `python -m pip install -e ".[dev]"`.

## Verify Locally In 3 Commands

If you want to verify v1.0 locally, run these three commands.

```bash
Expand Down Expand Up @@ -185,7 +188,8 @@ Cooldown behavior:
- [`demos/config-change-investigation-demo/README.md`](demos/config-change-investigation-demo/README.md) explains the config-change investigation demo and its committed artifacts
- [`demos/cloud-iam-change-investigation-demo/README.md`](demos/cloud-iam-change-investigation-demo/README.md) explains the synthetic CloudTrail-like IAM investigation demo and its committed artifacts
- [`docs/README.md`](docs/README.md) indexes current reviewer docs, supporting design notes, and historical release evidence
- [`docs/operator-reproduction.md`](docs/operator-reproduction.md) gives the shortest local path from clone to artifact regeneration and the release contract gate
- [`docs/operator-reproduction.md`](docs/operator-reproduction.md) gives the shortest local path from clone to running the five demos, artifact regeneration, schema tests, and full tests
- [`docs/release-v1.1.md`](docs/release-v1.1.md) drafts the v1.1 operator reproduction and issue triage release notes
- [`docs/reviewer-pack.md`](docs/reviewer-pack.md) is the top-level no-guessing reviewer pack and artifact naming contract
- [`docs/v1-contract-freeze.md`](docs/v1-contract-freeze.md) defines the v1.0 five-demo contract freeze gate
- [`docs/v1-readiness-gate.md`](docs/v1-readiness-gate.md) defines the fixed-input, fixed-output, schema-validation, artifact-regeneration, and test-pass readiness gate
Expand Down
3 changes: 2 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,13 @@ This directory separates the current reviewer route from supporting design notes
## Current reviewer route

- [`reviewer-pack.md`](reviewer-pack.md): top-level reviewer pack, demo matrix, artifact naming contract, and v1 readiness gate
- [`operator-reproduction.md`](operator-reproduction.md): shortest local path from clone to artifact regeneration
- [`operator-reproduction.md`](operator-reproduction.md): shortest local path from clone to verifying all five demos
- [`reviewer-path.md`](reviewer-path.md): choose a demo by review question
- [`reviewer-brief.md`](reviewer-brief.md): short problem, value, evidence, and boundary summary
- [`v1-contract-freeze.md`](v1-contract-freeze.md): v1.0 five-demo contract freeze, release status, and contract scope
- [`v1-readiness-gate.md`](v1-readiness-gate.md): fixed inputs, fixed outputs, schema validation, artifact regeneration, and test pass requirements
- [`release-v1.0.md`](release-v1.0.md): v1.0 reviewer-contract release notes with the explicit non-SIEM boundary
- [`release-v1.1.md`](release-v1.1.md): draft v1.1 operator reproduction and issue triage release notes
- [`v0.6-to-v1-artifact-diff.md`](v0.6-to-v1-artifact-diff.md): additive artifact contract and compatibility diff from the fourth demo to the fifth
- [`evidence-pipeline-contract.md`](evidence-pipeline-contract.md): JSON/JSONL schema contracts for reviewer-facing evidence artifacts
- [`reviewer-artifact-diff.md`](reviewer-artifact-diff.md): release diff contract for reviewer-facing artifact changes
Expand Down
48 changes: 43 additions & 5 deletions docs/operator-reproduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,41 @@ fresh clone. It does not add a new demo and does not require any live service,
cloud account, SIEM, dashboard, alert routing, case-management system, or
production telemetry source.

## Fresh Clone
## Install

```bash
git clone https://github.com/stacknil/telemetry-lab.git
cd telemetry-lab
python -m pip install -e ".[dev]"
```

Use the same Python interpreter for install, artifact regeneration, schema
validation, and tests.
Use the same Python interpreter for install, demo runs, artifact regeneration,
schema validation, and tests.

## Reproduce v1.0 Artifacts
## Run The Five Demos

```bash
python -m telemetry_window_demo.cli run --config configs/default.yaml
python -m telemetry_window_demo.cli run-ai-demo
python -m telemetry_window_demo.cli run-rule-dedup-demo
python -m telemetry_window_demo.cli run-config-change-demo
python -m telemetry_window_demo.cli run-cloud-iam-change-demo
```

Expected operator checkpoints:

- `data/processed/summary.json` reports `41` events, `24` windows, and `12`
alerts for `telemetry-window-demo`.
- `demos/ai-assisted-detection-demo/artifacts/case_report.md` reports `3`
deterministic cases.
- `demos/rule-evaluation-and-dedup-demo/artifacts/dedup_report.md` reports
`10` raw hits reduced to `6` retained alerts.
- `demos/config-change-investigation-demo/artifacts/investigation_report.md`
reports `4` normalized changes and `3` investigations.
- `demos/cloud-iam-change-investigation-demo/artifacts/investigation_report.md`
reports `14` CloudTrail-like events and `5` investigation signals.

## Regenerate Artifacts

```bash
python scripts/regenerate_artifacts.py --check
Expand All @@ -32,7 +55,22 @@ If this command fails, open an artifact regeneration issue with the command,
exit code, failing artifact path, and whether the failure is a missing file or
content difference.

## Full Reviewer Contract Gate
## Schema Tests

```bash
python -m pytest tests/test_evidence_pipeline_schemas.py
```

This validates reviewer-facing JSON and JSONL artifacts against the schemas in
`schemas/`.

## Full Tests

```bash
python -m pytest
```

## Release Contract Gate

The v1.0 reviewer contract is the exact three-command gate:

Expand Down
74 changes: 74 additions & 0 deletions docs/release-v1.1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# v1.1 Operator Reproduction Release Notes (Draft)

Theme: operator reproduction and issue triage, no demo expansion.

Release status: draft until a `v1.1` tag or GitHub release is explicitly
published.

## Scope

v1.1 keeps the existing five-demo matrix unchanged:

1. `telemetry-window-demo`
2. `ai-assisted-detection-demo`
3. `rule-evaluation-and-dedup-demo`
4. `config-change-investigation-demo`
5. `cloud-iam-change-investigation-demo`

This release is about helping an outside operator reproduce the frozen v1.0
reviewer contract from a clean local checkout. It does not add a sixth demo, a
live data source, a dashboard, alert routing, case management, autonomous
response, or final incident verdicts.

## Operator Reproduction

The maintained operator path is [`docs/operator-reproduction.md`](operator-reproduction.md):

1. install the repo with the dev extra
2. run all five demos locally
3. regenerate committed artifacts with `python scripts/regenerate_artifacts.py --check`
4. validate schemas with `python -m pytest tests/test_evidence_pipeline_schemas.py`
5. run the full test suite with `python -m pytest`

For a labeled one-command wrapper, run:

```bash
python scripts/check_release_contract.py
```

## Issue Triage

v1.1 adds focused issue templates for operator-facing contract questions:

- schema drift reports
- artifact regeneration failures
- demo boundary questions
- documentation reproduction questions

These templates keep reports scoped to local, synthetic reviewer artifacts and
ask reporters not to include credentials, real account IDs, production telemetry,
or private data.

## Validation Snapshot

Refresh this snapshot from the final release commit before publishing v1.1:

```bash
python scripts/check_release_contract.py
```

Expected gate sequence:

- artifact regeneration
- schema validation
- full test suite

## Boundaries

- No demo expansion.
- No live ingestion.
- No production SIEM or dashboard.
- No alert routing or case-management service.
- No autonomous response.
- No final incident verdict.

5 changes: 3 additions & 2 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ Recently added:
- [`docs/vocabulary.md`](vocabulary.md) defines cross-demo evidence workflow terms.
- [`docs/reviewer-artifact-diff.md`](reviewer-artifact-diff.md) defines the release diff contract for reviewer-facing artifact changes.
- [`docs/v0.6-to-v1-artifact-diff.md`](v0.6-to-v1-artifact-diff.md) records the additive fourth-to-fifth-demo artifact contract.
- [`docs/operator-reproduction.md`](operator-reproduction.md) records the shortest clone-to-regeneration path.
- [`docs/operator-reproduction.md`](operator-reproduction.md) records the shortest clone-to-five-demo-verification path.
- [`docs/release-v1.1.md`](release-v1.1.md) drafts the operator reproduction and issue triage release notes.
- `scripts/check_release_contract.py` wraps artifact regeneration, schema validation, and the full test suite.

## v1.1 Operator Reproduction Release
Expand All @@ -33,7 +34,7 @@ Deliverable focus:

1. Keep the five-demo matrix unchanged.
2. Make clone-to-artifact-regeneration instructions short and explicit.
3. Provide issue templates for schema drift, artifact regeneration failures, and demo boundary questions.
3. Provide issue templates for schema drift, artifact regeneration failures, demo boundary questions, and documentation reproduction questions.
4. Provide one reviewer-friendly release contract gate command.
5. Keep README verification commands copy/paste runnable.

Expand Down
53 changes: 53 additions & 0 deletions tests/test_reviewer_docs.py
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ def test_docs_index_separates_current_route_from_history() -> None:
"v1-contract-freeze.md",
"v1-readiness-gate.md",
"release-v1.0.md",
"release-v1.1.md",
"v0.6-to-v1-artifact-diff.md",
"evidence-pipeline-contract.md",
"reviewer-artifact-diff.md",
Expand Down Expand Up @@ -535,13 +536,23 @@ def test_operator_reproduction_doc_and_readme_define_short_gate() -> None:
assert "# Operator Reproduction" in operator_doc
assert "git clone https://github.com/stacknil/telemetry-lab.git" in operator_doc
assert "python -m pip install -e \".[dev]\"" in operator_doc
assert "## Run The Five Demos" in operator_doc
for demo_command in [
"python -m telemetry_window_demo.cli run --config configs/default.yaml",
"python -m telemetry_window_demo.cli run-ai-demo",
"python -m telemetry_window_demo.cli run-rule-dedup-demo",
"python -m telemetry_window_demo.cli run-config-change-demo",
"python -m telemetry_window_demo.cli run-cloud-iam-change-demo",
]:
assert demo_command in operator_doc
assert "python scripts/regenerate_artifacts.py --check" in operator_doc
assert "python -m pytest tests/test_evidence_pipeline_schemas.py" in operator_doc
assert "python -m pytest" in operator_doc
assert "python scripts/check_release_contract.py" in operator_doc
assert "does not add a new demo" in operator_doc
assert "does not claim production readiness" in operator_doc

assert "## Verify Locally In 3 Commands" in readme
assert "If you want to verify v1.0 locally, run these three commands." in readme
assert "docs/operator-reproduction.md" in readme
assert "operator-reproduction.md" in docs_index
Expand All @@ -556,6 +567,9 @@ def test_operator_issue_templates_keep_reviewer_contract_scope() -> None:
"artifact-regeneration-failure.md"
),
"demo-boundary-question.md": _read_issue_template("demo-boundary-question.md"),
"docs-reproduction-question.md": _read_issue_template(
"docs-reproduction-question.md"
),
}
feature_template = _read_issue_template("feature_request.yml")

Expand All @@ -574,4 +588,43 @@ def test_operator_issue_templates_keep_reviewer_contract_scope() -> None:
"artifact-regeneration-failure.md"
]
assert "No new demo expansion for v1.1." in templates["demo-boundary-question.md"]
assert "documentation mismatch" in templates["docs-reproduction-question.md"]
assert "This is not a request for a new demo." in templates[
"docs-reproduction-question.md"
]
assert "next demo" not in feature_template.lower()


def test_v11_release_note_keeps_operator_reproduction_scope() -> None:
release_note = _read_repo_file("docs/release-v1.1.md")
docs_index = _read_repo_file("docs/README.md")
readme = _read_repo_file("README.md")
roadmap = _read_repo_file("docs/roadmap.md")

assert "# v1.1 Operator Reproduction Release Notes (Draft)" in release_note
assert "Theme: operator reproduction and issue triage, no demo expansion." in release_note
assert "Release status: draft" in release_note
assert "python scripts/check_release_contract.py" in release_note
assert "documentation reproduction questions" in release_note

for demo_name in [
"telemetry-window-demo",
"ai-assisted-detection-demo",
"rule-evaluation-and-dedup-demo",
"config-change-investigation-demo",
"cloud-iam-change-investigation-demo",
]:
assert f"`{demo_name}`" in release_note

for forbidden_scope in [
"No demo expansion.",
"No live ingestion.",
"No production SIEM or dashboard.",
"No alert routing or case-management service.",
"No autonomous response.",
"No final incident verdict.",
]:
assert forbidden_scope in release_note

for text in [docs_index, readme, roadmap]:
assert "release-v1.1.md" in text
Loading