Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .epic-loop/epics/ai-lint/decision-log.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Decision Log

## Active Decisions

- Keep one command named `review:skills:ai`; do not split broad review and focused lint into separate package scripts.
- Convert the command behavior toward fixed semantic lint checks instead of open-ended model review.
- Do not rely on installed `skill-creator` or other local skills at runtime. Use their principles as design input, then encode the relevant rules in repo-owned prompts/check definitions.
- Treat deterministic checks as the CI-style validation path. AI review remains a maintainer workflow command outside `pnpm run validate`.

## Historical Decisions

- None recorded yet.
48 changes: 48 additions & 0 deletions .epic-loop/epics/ai-lint/docs/problem-framing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Epic Problem Framing

## Problem

Replace the broad AI skill review with focused semantic lint checks for epic-loop skill drift and degradation, based on the maintainer discussion about unstable model-backed review findings.

## Desired Outcome

- `pnpm run review:skills:ai` remains the single maintainer-facing AI review command.
- The command becomes a focused semantic lint boundary, not a broad code-review-like reviewer.
- The model evaluates a fixed repository-owned check catalog with stable check ids, target files, pass/fail criteria, and severity policy.
- The output is stable enough for maintainer workflow: repeated runs may vary in wording, but not in check identity, severity ownership, or scope.
- AI checks focus on semantic instruction contracts that deterministic tools cannot reliably verify.

## Scope

- Refactor the AI review prompt/schema to use fixed checks instead of free-form findings.
- Define a small initial semantic check catalog for the maintained `epic-loop` skill package.
- Preserve the existing command name: `review:skills:ai`.
- Keep deterministic validation separate from AI-backed validation; do not add AI review to `pnpm run validate`.
- Use skill-building principles from `skill-creator` as design input, but make the command self-contained and repo-owned.
- Add mocked tests for report validation, check formatting, blocking policy, and command-level behavior.
- Verify live `codex exec` behavior after the fixed-check boundary is implemented.

## Non-Scope

- Do not split the command into separate broad-review and lint commands.
- Do not depend on installed user/system skills being available inside `codex exec`.
- Do not make AI review a CI gate or deterministic validation replacement.
- Do not ask the model to perform general script security review, broad code review, or free-form skill critique.
- Do not fix unrelated semantic warnings unless they are part of the fixed check catalog and fail the new criteria.
- Do not change runtime hook behavior except where a fixed check reveals a confirmed narrow bug that gets scheduled as a separate implementation task.

## Constraints

- Generated artifacts must stay under ignored `.validation-output/skill-review/`.
- The runner must continue to validate model output deterministically before deciding pass/fail.
- Check ids and severity defaults are owned by repository code, not invented by the model.
- The model should return evidence for predefined checks, not create arbitrary finding categories.
- `status: fail` should be reserved for fixed blocking checks with confirmed semantic contract failure.
- Warning-level findings should support maintainer triage without blocking by default.
- Any live Codex review variability must be managed by schema design, check catalog boundaries, and tests.

## Open Questions

- Should the first implementation preserve the top-level `findings` array for backward compatibility, or migrate to a `checks` array with derived findings?
- Should blocking require a single failed `error` check, or should repeated live-run consensus be introduced later?
- Should broad advisory review remain available only through a separate manual prompt outside repository scripts?
182 changes: 182 additions & 0 deletions .epic-loop/epics/ai-lint/docs/semantic-lint-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
# Focused AI Semantic Lint Contract

## Context

The first AI-backed `review:skills:ai` runner behaved like a broad semantic reviewer. Three live runs produced useful signals, but the results were unstable:

- The same trigger-boundary concern appeared as both `error` and `warning`.
- A real session-id path-boundary bug appeared in one run and was missed in later runs.
- Warning-level concerns were phrased under different codes across runs.
- The command was validating JSON shape deterministically, but the model owned too much of the review scope, code taxonomy, and severity policy.

The maintainer decision is to keep a single `review:skills:ai` command and narrow it into a focused semantic lint boundary.

## Design Goal

The command should check fixed semantic contracts that ordinary deterministic tooling cannot verify well. It should not behave like a general code review.

The model should answer predefined questions with evidence. The repository should own:

- check ids
- target files
- pass/fail criteria
- severity defaults
- blocking policy
- output schema
- generated-output location

## Non-Goals

- Do not split the package scripts into separate broad-review and lint commands.
- Do not ask the model to inspect every possible script safety concern.
- Do not rely on local installed skills such as `skill-creator` being active inside `codex exec`.
- Do not include AI review in `pnpm run validate`.
- Do not treat a single live AI run as deterministic proof of package quality.

## Skill-Creation Principles To Encode

Use these principles as repo-owned check guidance, not as a runtime dependency on any installed skill:

- `SKILL.md` frontmatter controls trigger decisions and must be precise.
- `SKILL.md` should stay concise and route detailed mode behavior into references.
- References should be loaded conditionally and task-locally.
- Fragile operations need exact scripts and guardrails.
- Judgment-heavy work should preserve appropriate degrees of freedom.
- Validation integrity matters: tests should not leak the expected answer into the model context.

## Initial Fixed Checks

### `skill.description.trigger-boundary`

Target files:

- `plugins/epic-loop/skills/epic-loop/SKILL.md`

Question:

Does the skill description allow activation for useful `.epic-loop/` artifact understanding/editing while avoiding implied permission to run lifecycle commands, bind sessions, mutate runtime state, or start implementation unless the user intent or hook context asks for that?

Expected pass:

- Mentions active epic-loop workspace or artifact work clearly enough to trigger the skill for real epic artifacts.
- Does not imply ordinary package/source review of the plugin grants permission to run lifecycle/runtime actions.
- Preserves the contract that implementation starts only after explicit confirmation.

Default severity on fail: `warning`, unless the wording directly permits implementation start or runtime mutation without user intent.

### `skill.reentry.mode-conditional-orientation`

Target files:

- `plugins/epic-loop/skills/epic-loop/SKILL.md`
- `plugins/epic-loop/skills/epic-loop/references/implementation-techlead-role.md`
- `plugins/epic-loop/skills/epic-loop/references/implementation-manager-role.md`
- `plugins/epic-loop/skills/epic-loop/references/implementation-engineer-role.md`

Question:

Does the entrypoint avoid forcing broad artifact reads for every non-trivial turn, and does implementation mode orient from `role-summary.mjs` plus selective reads?

Expected pass:

- Normal orientation is mode-conditional.
- Implementation mode uses `role-summary.mjs` as the default entrypoint.
- Logs, decision registers, risk registers, and runtime/debug artifacts are read only when the mode decision needs them.

Default severity on fail: `warning`.

### `skill.lifecycle.explicit-implementation-start`

Target files:

- `plugins/epic-loop/skills/epic-loop/SKILL.md`
- `plugins/epic-loop/skills/epic-loop/references/implementation-cycle.md`

Question:

Does the skill consistently require explicit current-session user confirmation before binding a session and starting implementation mode?

Expected pass:

- Slug-only resume is orientation, not implementation permission.
- `bind-session --current --mode implementation` appears only after explicit confirmation.
- After binding, the agent stops and lets the trusted hook continue the loop.

Default severity on fail: `error`.

### `skill.parallel-sessions.mode-consistency`

Target files:

- `plugins/epic-loop/skills/epic-loop/SKILL.md`
- `plugins/epic-loop/skills/epic-loop/references/parallel-sessions.md`
- `plugins/epic-loop/skills/epic-loop/references/hooks-and-session-routing.md`

Question:

Do the entrypoint and references describe same-epic parallel sessions consistently?

Expected pass:

- Many sessions may be members of the same epic.
- A session has only one active mode at a time.
- Implementation has one exclusive driver while other members remain observers.
- Same-epic session behavior does not conflict between `SKILL.md` and references.

Default severity on fail: `error` for contradictions that can cause conflicting writes; otherwise `warning`.

### `skill.runtime-artifacts.normal-flow-boundary`

Target files:

- `plugins/epic-loop/skills/epic-loop/SKILL.md`
- `plugins/epic-loop/skills/epic-loop/references/implementation-techlead-role.md`
- `plugins/epic-loop/skills/epic-loop/references/implementation-manager-role.md`
- `plugins/epic-loop/skills/epic-loop/references/hooks-and-session-routing.md`

Question:

Do normal operating instructions keep agents away from `.runtime/**` debug artifacts unless a specific runtime/debug task requires them?

Expected pass:

- Human-facing artifacts are distinguished from runtime/debug artifacts.
- Normal implementation roles avoid prompt/progress logs, hook events, session files, and raw runtime traces.
- Runtime writes are routed through scripts/hooks rather than manual file edits.

Default severity on fail: `warning`, or `error` if instructions direct agents to inspect sensitive/raw runtime payloads by default.

## Output Shape Direction

Prefer a report shape centered on checks:

```json
{
"schemaVersion": 2,
"status": "pass",
"summary": "Focused semantic lint completed.",
"checks": [
{
"id": "skill.lifecycle.explicit-implementation-start",
"status": "pass",
"severity": "error",
"evidence": [],
"message": "Implementation start requires explicit confirmation.",
"recommendation": null
}
]
}
```

The runner may derive path-oriented findings from failed checks for display, but the model should not invent arbitrary finding codes.

## Verification Expectations

- Mocked valid report exits `0`.
- Mocked warning report exits `0` with stable diagnostics.
- Mocked failed `error` check exits non-zero.
- Unknown check ids fail schema validation.
- Missing checks fail schema validation.
- Malformed JSON and missing output remain non-zero.
- `pnpm run validate` remains green and non-AI-backed.
- Live repeated runs are compared by check id and status, not by free-form finding text.
12 changes: 12 additions & 0 deletions .epic-loop/epics/ai-lint/implementation-log.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Implementation Log

## 2026-07-08T11:25:43+00:00 - Epic Workspace Initialized

- Created epic workspace for `ai-lint`.
- Initial mode: shaping.

## 2026-07-08T11:25:43+00:00 - Initial Shaping Context Captured

- Captured maintainer decision to replace broad AI skill review behavior with focused fixed semantic checks.
- Added `docs/semantic-lint-contract.md` with initial check catalog, non-goals, output-shape direction, and verification expectations.
- Marked the first shaping documentation task done; next action is review of the proposed check catalog before implementation starts.
8 changes: 8 additions & 0 deletions .epic-loop/epics/ai-lint/risk-register.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Risk Register

| Risk | Impact | Mitigation | Status |
| --- | --- | --- | --- |
| Model output remains unstable even with fixed checks. | The command stays noisy and cannot support reliable maintainer triage. | Restrict check ids, severities, target files, and output schema; verify with repeated live runs. | open |
| Check catalog becomes too broad and recreates a general review. | The command drifts back into subjective review findings. | Keep the initial catalog small and reject free-form model-created codes. | open |
| Fixed checks miss real script safety bugs. | AI review may appear more reliable than it is. | Keep script safety in deterministic tests where possible; use AI only for semantic instruction contracts. | open |
| Installed skill availability differs across maintainer environments. | Review behavior becomes environment-dependent. | Do not require `skill-creator` or other installed skills inside `codex exec`; bake needed guidance into repo-owned prompt/checks. | open |
22 changes: 22 additions & 0 deletions .epic-loop/epics/ai-lint/state-of-epic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# State Of Epic

Epic: Focused AI Skill Review Checks
Slug: `ai-lint`
Created: 2026-07-08T11:25:43+00:00
Active phase: Phase 1 - Shape The Epic
Active task: Phase 1 Task 2 - Review the fixed semantic check catalog

## Current State

- This epic follows the completed `set-up` epic and owns the next iteration of `pnpm run review:skills:ai`.
- Maintainer discussion established that the current AI review is too broad and unstable for lint-like drift detection.
- The desired direction is to keep one AI review command, but convert it from an open-ended semantic review into a focused semantic lint with fixed checks.
- The fixed checks should cover instruction drift and degradation that ordinary deterministic linters cannot catch.

## Blockers

- None recorded.

## Next Action

- Review `docs/problem-framing.md` and `docs/semantic-lint-contract.md`, then refine or approve the roadmap before implementation starts.
73 changes: 73 additions & 0 deletions .epic-loop/epics/ai-lint/tracker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Tracker

Epic: Focused AI Skill Review Checks

## Task Statuses

- todo
- doing
- need-review
- blocked
- partially-satisfied
- deferred
- reset-required
- done

## Task Kinds

- implementation
- verification
- review
- follow-up
- architecture-reset
- documentation-only

## Active Roadmap

### Phase 1: Fixed Semantic Check Contract

- Phase status: todo

- [x] Kind: documentation-only | Status: done | Define the fixed semantic AI lint contract.
- Outcome: The AI review command has an explicit check catalog and severity policy before implementation changes begin.
- Surface: `.epic-loop/epics/ai-lint/docs/semantic-lint-contract.md`, `.epic-loop/epics/ai-lint/decision-log.md`.
- Acceptance: The doc names the initial check ids, target files, pass/fail criteria, evidence requirements, and non-goals.
- Docs: `docs/semantic-lint-contract.md`, `docs/problem-framing.md`.

- [ ] Kind: verification | Status: todo | Review the check catalog against the maintainer discussion.
- Outcome: The first implementation slice is grounded in the desired lint-like behavior, not broad review behavior.
- Surface: `docs/semantic-lint-contract.md`.
- Acceptance: The catalog explicitly excludes general code review, model-created finding codes, and installed-skill runtime dependency.
- Docs: `docs/semantic-lint-contract.md`.

### Phase 2: Runner And Schema Refactor

- Phase status: todo

- [ ] Kind: implementation | Status: todo | Refactor `review:skills:ai` to evaluate fixed semantic checks.
- Outcome: The runner prompt and report schema use repository-owned check ids instead of free-form model findings.
- Surface: `scripts/review-skills-ai.mjs`, `tests/unit/skill-review-ai.test.mjs`.
- Acceptance: Mocked valid, failing, malformed, and missing-output paths still behave deterministically; the model cannot invent blocking codes outside the fixed catalog.
- Docs: `docs/semantic-lint-contract.md`.

- [ ] Kind: implementation | Status: todo | Add focused tests for fixed-check parsing, severity policy, and stable diagnostics.
- Outcome: The AI boundary is protected by deterministic unit tests without invoking live Codex.
- Surface: `tests/unit/skill-review-ai.test.mjs`, optional small helper modules under `scripts/`.
- Acceptance: Tests prove pass, warning, blocking, unknown-check, and malformed-report behavior.
- Docs: `docs/semantic-lint-contract.md`.

- [ ] Kind: verification | Status: todo | Verify deterministic validation remains separate from AI review.
- Outcome: The aggregate validation path remains stable and non-AI-backed.
- Surface: `package.json`, `scripts/review-skills-ai.mjs`, `.validation-output/skill-review/`.
- Acceptance: `pnpm run validate` passes and does not invoke `review:skills:ai`; generated AI reports remain ignored.
- Docs: `docs/problem-framing.md`.

### Phase 3: Live Stability Verification

- Phase status: todo

- [ ] Kind: verification | Status: todo | Run repeated live AI review checks and compare normalized results.
- Outcome: Maintainers understand whether the fixed-check boundary reduces model jitter enough for the intended workflow.
- Surface: `pnpm run review:skills:ai`, `.validation-output/skill-review/latest.json`.
- Acceptance: At least three sequential live runs are compared by check id, status, severity, and evidence; any remaining jitter is recorded as a risk or follow-up.
- Docs: `docs/semantic-lint-contract.md`, `risk-register.md`.
6 changes: 4 additions & 2 deletions .epic-loop/epics/set-up/decision-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,12 @@
## Active Decisions

- 2026-07-07: Split skill repository checks into two implementation phases. Mechanical skill package invariants should be enforced by deterministic scripts, while semantic skill quality review should run through a headless `codex exec --ephemeral` workflow wrapped by a repository script that requires structured JSON output in an ignored directory and validates that schema before reporting findings.
- 2026-07-07: Accept current oxlint `max-lines` failures in `plugins/epic-loop/skills/epic-loop/scripts/lib/loop.mjs` and `plugins/epic-loop/skills/epic-loop/scripts/lib/hooks.mjs` as known baseline debt for now. Do not relax the targeted limits; continue implementation and track a Phase 1 refactor task to split those modules before final phase verification.
- 2026-07-07: Scope Oxfmt formatting to JS/MJS and JSON config/source surfaces. Exclude markdown from the formatter command because `oxfmt@0.57.0` rewrote inline-code spacing and template placeholder emphasis unsafely in repository docs/templates during verification.
- 2026-07-07: Remove the repository English-only lexical validation phase. A real read-only audit found no Cyrillic, suspicious non-Latin scripts, or obvious non-English prose in the repository; remaining non-ASCII findings are punctuation and contract strings, so strict ASCII cleanup is not part of this epic.
- 2026-07-07: Add targeted oxlint maintainability limits: source files may contain up to 600 lines, test files may contain up to 900 lines, and individual functions may contain up to 150 lines. Configure these limits to skip blank lines and comment-only lines.
- 2026-07-06: Use oxlint for JavaScript/ESM linting, Oxfmt (`oxfmt`) for formatting, and a small repository-owned Node.js script for the English-only lexical policy. The custom script should cover repo-specific language policy with explicit include/ignore patterns and allowlists.
- 2026-07-06: Treat `pnpm run validate` as the durable aggregate validation entry point. New checks should be integrated there unless implementation finds a stronger existing convention.

## Historical Decisions

- None recorded yet.
- Superseded 2026-07-07: Use oxlint for JavaScript/ESM linting, Oxfmt (`oxfmt`) for formatting, and a small repository-owned Node.js script for the English-only lexical policy. The custom script should cover repo-specific language policy with explicit include/ignore patterns and allowlists.
Loading
Loading