Add contract hardening, oracle regression, and repair audit trail.

Ship P1 JSON repair, P2 semantic policies, versioned oracle snapshots with CI semantic changelog, repair_audit/unstable_repair gates, and live-baseline tooling so release decisions stay auditable in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: CHDev2116

.ci/live_quality_kpi_baseline.json

@@ -0,0 +1,8 @@
+{
+  "profile": "live_baseline",
+  "recorded_at": null,
+  "batch_report": null,
+  "inference_backend": null,
+  "observed": {},
+  "notes": "Run: bash scripts/run_live_kpi_baseline.sh — then inspect or commit after review."
+}

.ci/live_quality_kpi_thresholds.json

@@ -0,0 +1,16 @@
+{
+  "profile": "live_baseline",
+  "description": "Loose ceilings for local live runs. Tighten after record_live_kpi_baseline.py --propose-thresholds.",
+  "max_fallback_ratio": 0.35,
+  "max_review_rate": 1.0,
+  "max_pass_rate_variance": 0.15,
+  "max_image_score_variance": 0.15,
+  "max_json_repair_exhausted_count": 10,
+  "max_json_repair_exhausted_ratio": 0.25,
+  "max_semantic_assert_fail_count": 50,
+  "max_semantic_code_mismatch_count": 50,
+  "max_strict_contract_violation_count": 5,
+  "max_llm_judge_calls": 20,
+  "max_llm_judge_overrides": 20,
+  "max_unstable_repair_count": 10
+}

.ci/quality_kpi_thresholds.json

@@ -2,5 +2,11 @@
   "max_fallback_ratio": 0.01,
   "max_review_rate": 1.0,
   "max_pass_rate_variance": 0.05,
-  "max_image_score_variance": 0.05
+  "max_image_score_variance": 0.05,
+  "max_json_repair_exhausted_count": 0,
+  "max_json_repair_exhausted_ratio": 0.0,
+  "max_semantic_assert_fail_count": 50,
+  "max_semantic_code_mismatch_count": 50,
+  "max_strict_contract_violation_count": 0,
+  "max_unstable_repair_count": 0
 }

.ci/replay_quality_kpi_thresholds.json

@@ -0,0 +1,9 @@
+{
+  "max_fallback_ratio": 0.01,
+  "max_review_rate": 1.0,
+  "max_json_repair_exhausted_count": 0,
+  "max_json_repair_exhausted_ratio": 0.0,
+  "max_semantic_assert_fail_count": 0,
+  "max_semantic_code_mismatch_count": 0,
+  "max_strict_contract_violation_count": 0
+}

.github/workflows/ci.yml

@@ -50,6 +50,17 @@ jobs:
         run: |
           PYTHONPATH=src pytest
 
+      - name: Oracle historical regression
+        id: oracle_regression
+        run: |
+          PYTHONPATH=src pytest tests/test_oracle_regression.py -q
+
+      - name: Oracle semantic changelog (job summary)
+        if: always()
+        env:
+          CI_ORACLE_SNAPSHOT_ENFORCE: ${{ github.event_name == 'pull_request' && '1' || '0' }}
+        run: bash scripts/ci_oracle_semantic_summary.sh
+
       - name: Coverage baseline gate (PR only)
         if: github.event_name == 'pull_request'
         run: |
@@ -65,8 +76,7 @@ jobs:
             --loopback-planner simulated \
             --planner-skip-health-check
           python scripts/check_quality_kpis.py --enforce \
-            --max-fallback-ratio 0.01 \
-            --max-review-rate 1.0
+            --thresholds-file .ci/quality_kpi_thresholds.json
 
       - name: Repeatability KPI smoke
         run: |
@@ -76,8 +86,7 @@ jobs:
             --loopback-planner simulated \
             --planner-skip-health-check
           python scripts/check_quality_kpis.py --enforce \
-            --max-pass-rate-variance 0.05 \
-            --max-image-score-variance 0.05
+            --thresholds-file .ci/quality_kpi_thresholds.json
 
   generate-reports:
     runs-on: ubuntu-latest

.gitignore

@@ -40,5 +40,6 @@ docs/InterviewNarratives.md
 docs/InterviewLanguagePrep.md
 docs/linkedin-self-healing-vision-qa.md
 docs/CollaborationPlan.md
+docs/ContractHardening.md
 docs/*Collaboration*.md
 docs/*Plan*.md

README.md

@@ -202,6 +202,65 @@ python3 src/ai_quality_agent.py --profile dev --replay-mode record --replay-file
 python3 src/ai_quality_agent.py --profile dev --replay-mode replay --replay-file results/replay_trace.jsonl
 ```
 
+Inference JSON contract (self-repair + strict mode):
+
+Default is **off** (`max_json_repair_attempts: 0`) so CI and simulated batches stay deterministic. Enable in config under `model_settings.inference.contract`:
+
+```json
+{
+  "model_settings": {
+    "inference": {
+      "backend": "ollama_vision",
+      "fallback_to_simulated": true,
+      "contract": {
+        "max_json_repair_attempts": 2,
+        "strict_contract": false,
+        "repair_on_empty_dict": true,
+        "repair_prompt_suffix": "Return ONLY a single JSON object with keys decision, code, msg."
+      }
+    }
+  }
+}
+```
+
+Behavior:
+
+- Parse/validate failures trigger up to N extra LLM calls with validation feedback (`contract_validator.py`).
+- Success attaches `contract_meta.repair_attempts` on the inference dict.
+- After exhausted repair: `ERR_MODEL_RESPONSE_422` + `repair_exhausted` in `msg`; may fall back to `simulated` unless `strict_contract` is true.
+- `runtime.replay_mode=replay` disables repair (same as CI replay smoke).
+
+Semantic asserts (P2) run by default via `eval_settings.semantic_asserts_enabled` (default true). Row-level issues appear in `contract.semantic_errors`; batch summary includes `semantic_assert_fail_count` and `review_breakdown.SEMANTIC_ASSERT_MISMATCH` when arbitration input is overridden.
+
+Oracle historical regression (frozen release semantics):
+
+```bash
+PYTHONPATH=src pytest tests/test_oracle_regression.py -q
+```
+
+Import a row from a past batch report into the oracle corpus:
+
+```bash
+PYTHONPATH=src python scripts/append_oracle_case_from_batch.py \
+  --batch-report results/dev/batch_report_YYYYMMDD_HHMMSS.json \
+  --file your_image.jpg \
+  --id hist-NNN-short-name \
+  --description "What this incident was"
+```
+
+See `tests/regression/README.md` and `docs/RegressionVersioning.md`. Rule-change drift vs committed snapshots:
+
+```bash
+python scripts/diff_oracle_semantics.py
+python scripts/refresh_oracle_snapshot.py   # after intentional policy change
+```
+
+CI posts a **semantic changelog** to the GitHub job summary on every run; PRs also enforce snapshot parity (`scripts/ci_oracle_semantic_summary.sh`).
+
+**JSON repair audit:** when `contract.max_json_repair_attempts > 0`, each row gets `contract_meta.repair_audit` and `unstable_repair` if decision flips across repair rounds. Default `contract.unstable_repair_release: REVIEW` downgrades release for audit. See `docs/RepairAudit.md`, `docs/FailureTaxonomy.md` (triage **IN**).
+
+Replay CI uses stricter KPI thresholds (`.ci/replay_quality_kpi_thresholds.json`: `max_semantic_assert_fail_count=0`).
+
 </details>
 
 <details>
@@ -226,12 +285,29 @@ docker run --rm \
 
 </details>
 
+<details>
+<summary><strong>Live KPI baseline (optional, not CI)</strong></summary>
+
+Profile `live_baseline` runs real inference + contract repair + adaptive backoff + LLM judge (Ollama). Requires a local server (`llama.cpp` on `:8080` or Ollama on `:11434`).
+
+```bash
+bash scripts/run_live_kpi_baseline.sh
+INFERENCE_BACKEND=ollama_vision bash scripts/run_live_kpi_baseline.sh
+PROPOSE_THRESHOLDS=1 bash scripts/run_live_kpi_baseline.sh   # tighten .ci/live_quality_kpi_thresholds.json
+```
+
+Writes `.ci/live_quality_kpi_baseline.json` and checks loose ceilings with `--warn-only`. See `docs/LiveKPIBaseline.md`.
+
+</details>
+
 <details>
 <summary><strong>CI / local tests</strong></summary>
 
 ```bash
 python -m pip install -U pip
 pip install -e ".[dev]"
+PYTHONPATH=src pytest tests/test_oracle_regression.py -q
+python scripts/check_quality_kpis.py --enforce --thresholds-file .ci/quality_kpi_thresholds.json
 bash scripts/dev_prepush_check.sh
 ```
 
@@ -265,6 +341,9 @@ Workflow reference: `.github/workflows/ci.yml`
 - [x] Batch ranking + release arbitration
 - [x] Repeatability / performance / overhead analysis
 - [x] Automated JSON error reporting with retention
+- [x] Deterministic replay (JSONL planner trace + CI smoke)
+- [x] Contract hardening (JSON repair, semantic asserts, oracle regression)
+- [x] Adaptive backoff module for async HTTP (config-gated, see `docs/AdaptiveBackoff.md`)
 
 **Backlog (intentionally deferred)**
 
@@ -273,14 +352,8 @@ Workflow reference: `.github/workflows/ci.yml`
 
 **Future roadmap (agentic testing hardening)**
 
-- **Deterministic replay mode (VCR-style)**  
-  Priority: **P0** | Effort: **M** | Impact: **High**  
-  Record planner prompts/responses and decision-state transitions on a known-good run, then support playback-only regression mode to eliminate flaky LLM variance and reduce token spend.
-
-- **Adaptive backoff + dynamic concurrency**  
-  Priority: **P1** | Effort: **M-L** | Impact: **High**  
-  Evolve from fixed semaphore limits to runtime-aware rate control (429/503 detection, exponential backoff with jitter, and temporary permit reduction) so test loops remain stable under service pressure.
-
+- **LLM judge on REVIEW rows (P2.1)** — shipped (simulated, cost-capped); enable via `eval_settings.llm_judge.enabled`.
+- **Critique Agent** — outline in `docs/CritiqueAgent.md` (assertion strength / schema coverage recommendations, not a CI gate).
 - **Schema-driven auto assertion generation**  
   Priority: **P2** | Effort: **L** | Impact: **High**  
   Use observed response samples and Pydantic contracts to infer boundary/type assertions and scaffold `tests/test_generated_*.py` candidates, reducing manual test-authoring overhead for newly explored paths.

configs/base.json

@@ -52,6 +52,10 @@
         "loopback_planner": {
             "mode": "simulated",
             "require_healthy_on_startup": true,
+            "contract": {
+                "max_json_repair_attempts": 0,
+                "unstable_repair_release": "REVIEW"
+            },
             "llm": {
                 "host": "http://127.0.0.1:8080",
                 "endpoint": "/v1/chat/completions",
@@ -60,6 +64,14 @@
                 "temperature": 0.0,
                 "max_tokens": 200
             }
+        },
+        "adaptive_backoff": {
+            "enabled": false,
+            "max_retries": 2,
+            "base_delay_s": 0.25,
+            "max_delay_s": 8.0,
+            "jitter_ratio": 0.2,
+            "concurrency_floor": 1
         }
     },
     "quality_gate": {
@@ -71,6 +83,18 @@
             "engine_weight": 0.6,
             "model_weight": 0.4
         },
-        "auto_tag_conflicts": true
+        "auto_tag_conflicts": true,
+        "semantic_asserts_enabled": true,
+        "semantic_policy": {
+            "invalid_label_release": "NO_GO",
+            "confidence_violation_policy": "review",
+            "inference_error_release": "NO_GO"
+        },
+        "llm_judge": {
+            "enabled": false,
+            "max_calls_per_batch": 5,
+            "mode": "simulated",
+            "tie_break": "conservative"
+        }
     }
 }

configs/live_baseline.json

@@ -0,0 +1,87 @@
+{
+    "version": "1.2.0",
+    "environment": "live_baseline",
+    "metadata": {
+        "purpose": "One-off / periodic live inference KPI baseline (not used in CI)",
+        "notes": "Requires llama.cpp server :8080 or Ollama :11434. See docs/LiveKPIBaseline.md"
+    },
+    "model_settings": {
+        "name": "Agentic Testing Framework - Live KPI",
+        "inference": {
+            "backend": "llama_cpp",
+            "fallback_to_simulated": true,
+            "contract": {
+                "max_json_repair_attempts": 2,
+                "strict_contract": false,
+                "repair_on_empty_dict": true
+            },
+            "ollama": {
+                "host": "http://localhost:11434",
+                "model": "llava:7b",
+                "timeout_s": 60
+            },
+            "llama_cpp": {
+                "host": "http://127.0.0.1:8080",
+                "endpoint": "/v1/chat/completions",
+                "model": "local-model",
+                "timeout_s": 60,
+                "temperature": 0.0,
+                "max_tokens": 256,
+                "use_response_format": true
+            }
+        }
+    },
+    "thresholds": {
+        "min_sharpness": 20.0,
+        "min_brightness": 40.0,
+        "max_brightness": 220.0,
+        "timeout_ms": 8000
+    },
+    "folders": {
+        "input": "test_images",
+        "output": "results/live_baseline",
+        "logs": "logs/live_baseline"
+    },
+    "runtime": {
+        "oom_probability": 0.0,
+        "async_per_image_timeout_s": 45,
+        "async_backend_health_check": true,
+        "async_backend_health_timeout_s": 3.0,
+        "adaptive_backoff": {
+            "enabled": true,
+            "max_retries": 3,
+            "base_delay_s": 0.5,
+            "max_delay_s": 12.0,
+            "jitter_ratio": 0.2,
+            "concurrency_floor": 1
+        },
+        "loopback_planner": {
+            "mode": "simulated",
+            "require_healthy_on_startup": false,
+            "contract": {
+                "max_json_repair_attempts": 1
+            }
+        }
+    },
+    "quality_gate": {
+        "target_pass_rate": 75.0
+    },
+    "eval_settings": {
+        "conflict_strategy": "conservative",
+        "semantic_asserts_enabled": true,
+        "semantic_policy": {
+            "invalid_label_release": "NO_GO",
+            "confidence_violation_policy": "review",
+            "inference_error_release": "NO_GO"
+        },
+        "llm_judge": {
+            "enabled": true,
+            "max_calls_per_batch": 8,
+            "mode": "ollama",
+            "tie_break": "conservative",
+            "host": "http://localhost:11434",
+            "model": "llama3.2",
+            "timeout_s": 45
+        }
+    }
+}

docs/AdaptiveBackoff.md

@@ -0,0 +1,37 @@
+# Adaptive Backoff (async inference)
+
+Implemented in `src/util/adaptive_backoff.py` and wired into `src/models/async_inference.py`.
+
+## Purpose
+
+When live inference backends return **429/5xx**, async batch runs can retry with exponential backoff + jitter instead of immediately falling back to simulated results.
+
+## Config (`runtime.adaptive_backoff`)
+
+| Key | Default | Meaning |
+|-----|---------|---------|
+| `enabled` | `false` | Master switch |
+| `max_retries` | `2` | Extra attempts after pressure response |
+| `base_delay_s` | `0.25` | Initial backoff |
+| `max_delay_s` | `8.0` | Cap per sleep |
+| `jitter_ratio` | `0.2` | Randomized delay fraction |
+| `concurrency_floor` | `1` | Future hook for lowering async concurrency under pressure |
+
+## Enable (example)
+
+```json
+"runtime": {
+  "adaptive_backoff": {
+    "enabled": true,
+    "max_retries": 3,
+    "base_delay_s": 0.5,
+    "max_delay_s": 10.0
+  }
+}
+```
+
+## Notes
+
+- Retries are **HTTP-layer only**; they do not re-run vision metrics.
+- Simulated backend is unaffected (no HTTP).
+- Pair with `--async-concurrency` tuning on live profiles after measuring 429 rates.