feat(medic): CI Medic mode for post-CI triage, retry, and auto-fix

[factory-nizar]

Adds CI Medic, a drop-in workflow that runs after CI completes on a pull request, diagnoses the failures, retries what looks flaky or environmental, and either commits a focused fix or posts inline suggestions.

Tracked by AUT-1522 / AUT-1523 (Dell ask for a post-CI watchdog action).

How it works

Triggered on workflow_run, so it runs in base-repo context with access to the Actions API. It waits for every check on the head commit to finish before acting, which keeps a pull request with several failing workflows to one aggregated comment rather than one per workflow.

# .github/workflows/ci-medic.yml
on:
  workflow_run:
    workflows: ["CI"]
    types: [completed]
permissions:
  actions: write        # read failed job logs, rerun jobs
  contents: write
  pull-requests: write
  id-token: write       # Droid App token

Configuration comes from action inputs merged over an optional .github/droid-ci.yml read from the base branch, so a pull request cannot reconfigure the bot. Repository config is untrusted input, so every field is coerced back to its declared type and a malformed value degrades to the default.

Budgets

Three separate limits, each with a distinct scope:

Setting	Scope	Resets
max_retries	reruns of one job for one commit	each new commit
max_fix_attempts	consecutive fix commits	any human commit
max_runs_per_pr	all invocations over the PR lifetime	never

max_runs_per_pr is enforced at the gate, before Droid is invoked, so an exhausted budget costs nothing.

Two auth details that matter

github_token is deliberately not passed, so the action authenticates as the Factory Droid App. GitHub does not start a new workflow run for a push made with secrets.GITHUB_TOKEN, so an auto-fix commit would otherwise land under the stale failing checks that triggered it and never be verified. Reading job logs and rerunning jobs still use the workflow token, which is why actions: write is also required.

The template checks out the head branch so fixes have a real branch to commit to.

CI Medic does not run on fork pull requests. workflow_run executes in the base repository with write-scoped tokens, the Factory API key, and OIDC, so checking out a fork's commit and running commands against it would hand the pull request author those credentials. A job-level condition enforces this and the gate re-checks it. An earlier revision of this branch claimed forks "downgrade to comment-only"; no such downgrade existed in code, and the claim has been removed rather than implemented.

Validation

scripts/ci-medic-sandbox.ts provisions a sandbox repository and a fleet of pull requests covering each path, then asserts the outcomes. workflow_run cannot be simulated locally: the triggering definition is read from the default branch and reruns go through the Actions API.

All 12 scenarios pass against a real repository: real failure with fix off and on, flaky retry, infra retry, retry disabled, two failing workflows, config failure, protected path, excluded workflow, draft, label opt-out, and budget exhaustion. Flakes are made deterministic via GITHUB_RUN_ATTEMPT so retry assertions are not coin flips. The end-to-end loop is proven: diagnosed an off-by-one, committed the fix, CI retriggered and went green.

507 unit tests pass, typecheck and format clean.

Bugs the fleet caught, fixed here

• The run budget erased itself. It counted comments carrying the ci-medic:run marker, but Droid rewrites that comment's body and dropped the marker, so each successful run destroyed its own record. Measured: three runs, one surviving marker. max_runs_per_pr would only ever have counted runs that crashed before Droid started. The count now lives in a marker on one reused tracking comment, which the comment server carries across rewrites. This also stops a new top-level comment appearing per run.
• Every run failed at startup. github_inline_comment___create_inline_comment was allowed, but the server behind it was gated on isEntityContext, which a workflow_run payload never satisfies, so the CLI rejected the run with Unknown tool identifier(s). That tool is how the auto_fix: false path delivers suggestions.
• The config parser crashed on valid YAML. Flow style workflows: { exclude: ["Deploy *"] } failed JSON.parse because YAML leaves keys unquoted, fell through as a string, and left config.workflows.exclude undefined for a .some() call.
• The template never checked out code, so auto-fix had no working tree.

Known caveats

• Fix commits can land in action_required when the app is a first-time contributor to the repo, which stalls the verify step until someone approves. Worth surfacing in the medic comment.
• The fork block is verified by unit test and by the workflow condition, not by the fleet: exercising it needs a pull request from a second account.
• The fleet ran before the hardening commit. Its config used flow-style lists, which the old parser handled, so the block-sequence bug never fired there; max_runs_per_pr in that file was silently ignored, and matched the default, so the outcome was identical. The budget scenario exercised the action input, not the file.
• Separately filed AUT-1531: when the app is not installed, token exchange returns an opaque 500 instead of app_not_installed, and the error copy links to apps/factory-ai, a private app users cannot install.

[factory-droid]

Droid finished @factory-nizar's task —— View job

---

CI Medic is a solid addition, but there are a few high-impact correctness and security gaps: the config parser currently mis-parses common YAML layouts, numeric inputs cannot be set to zero, and the workflow_run path needs stronger trust boundaries (both for config ref selection and for running at all on untrusted PRs).

[factory-nizar]

Evidence

CI Medic was validated against a real repository, Factory-AI/ci-medic-sandbox, not mocks. 12 of 12 scenarios pass, 636 tests pass, and this branch adds CI to droid-action.

Two things are worth a reviewer's attention: the fix scope is enforced by withholding tools rather than by asking, and two behaviors remain unproven (listed at the bottom).

scenario	comments	outcome
#44 real failure, auto-fix off	1	diagnosis and inline suggestion, nothing committed
#45 real failure, auto-fix on	1	fix(ci): divide average by the value count
#46 flaky	1	rerun to attempt=2 success
#47 infrastructure	1	rerun to attempt=2 success
#48 retry_mode: off	1	no rerun, stayed attempt=1 failure
#49 two failing workflows	1	one analysis, not two
#50 configuration failure	1	diagnosis only
#51 protected path	1	reported, .github/workflows/** untouched
#52 excluded workflow	0	silent skip
#53 draft PR	0	silent skip
#54 opted out by label	0	silent skip
#55 budget	2	budget note, skipped before invoking Droid

Every run charged exactly one budget unit and left exactly one tracking comment.

---

The fix scope is enforced by withholding tools

fix.scope describes what failed, not the diff, so it is decided before Droid starts: the failing jobs for the commit are fetched, classified, and if none fall inside the scope the editing tools are never granted.

Job names are matched by category, so unit, tsc and eslint count as tests, types and lint. Only steps GitHub named itself are read, which it does by prefixing the command with Run: Run bun test counts as a test, while a deployment step a human named Push preview bundle does not count as a build. An unrecognized job stays out of scope and says so in the log.

Both directions proven live:

• In scope — #42 (run). Job unit failed, classified tests, tools granted, fix committed.
• Out of scope — #43 (run). Only deploy-preview failed, auto_fix on:

  CI Medic is diagnosing only: no failing check is within the configured
  fix scope (lint, types, tests, build). Failing checks: CI / deploy-preview
  

  No Execute, Edit, Create or ApplyPatch in the tool list. It still diagnosed the deployment failure correctly.

One commit is charged once, even when several workflows fail

Every watched workflow that fails raises its own event, and the first run already waits for all checks on the commit, so the marker records the commit it analyzed and later events for that commit are skipped.

Rather than wait for a race to demonstrate it, the CI fixture is dispatchable, so a duplicate attempt-1 run can be raised on an unchanged commit deliberately. On #38:

first run   -> <!-- ci-medic:run=31048954594 count=1 sha=395c5a9a... -->
dispatched CI at the same sha, attempt 1, fails
medic run 31049379199 -> "CI Medic skipped: commit_already_processed"
marker afterwards -> unchanged, count=1

The skip line is in that run's log. A failed rerun carries a higher attempt and is still analyzed, because that is a new outcome rather than a duplicate.

Protected paths: two links proven, one not

1. The guard runs in production — from this run's log:

   ##[start-action display=CI Medic post-run]
   Run bun run ${GITHUB_ACTION_PATH}/src/medic/postrun.ts
   MEDIC_BASE_SHA: fba39426bf5a7ec94ad08d16c7d1a1389f1b8bde
   MEDIC_PROTECTED_PATHS: src/**
   

2. The logic works on real git — a test builds a repository, commits edits to a protected and an allowed path, then asserts the protected file is restored, the unrelated fix survives, and the restore is committed.
3. A model-driven violation is not proven. #39 put maximum pressure on it: auto_fix on, write tools granted, and protected_paths: ["src/**"] so the only possible repair was protected. The model declined and diagnosed instead. Forcing a violation would mean deceiving the model, which would say nothing about production.

---

Not covered

• Fork runs. Blocked by a job-level condition and re-checked in gate.ts, unit-tested, but exercising it needs a pull request from a second account.
• A model-driven protected-path violation. See above; the model declined every time, including under maximum pressure.
• action_required. A fix commit can land in a run awaiting approval, visible on #45. Unchanged by this work, but worth an opinion on whether CI Medic should say so in its own comment.
• run-droid.ts argv quoting. It interpolates MCP env values into an unquoted shell string, so any value containing a space or metacharacter breaks registration. CI Medic routes around it by passing only digit and hex values; the underlying weakness is pre-existing and untouched here.

[factory-droid]

[P1] Fix YAML parser dropping top-level keys after sections

parseConfig keeps the last section active until another section header is seen, so a later top-level scalar like max_runs_per_pr: 5 gets written under the previous section object (often skip) and is then ignored by mergeConfig, which makes common configs silently not apply (including the pattern of putting max_runs_per_pr at the end of the file).

Suggested change

	const match = /^\s*([A-Za-z_]+):\s*(.+)$/.exec(line);
	if (!match) continue;
	const key = match[1]!;
	const value = match[2]!;
	const parsed = value.replace(/^['"]|['"]$/g, "");
	const target = section
	? (result[section] as Record<string, unknown>)
	: result;
	target[key] = parseScalar(parsed);
	const match = /^\s*([A-Za-z_]+):\s*(.+)$/.exec(line);
	if (!match) continue;
	const key = match[1]!;
	const value = match[2]!;
	const parsed = value.replace(/^['\"]|['\"]$/g, "");
	const isNested = /^\s+/.test(line);
	const target =
	section && isNested
	? (result[section] as Record<string, unknown>)
	: result;
	if (!isNested) section = "";
	target[key] = parseScalar(parsed);

[factory-droid]

[P1] Allow numeric inputs to be set to "0"

The parseInt(...) || <default> pattern means inputs like MAX_RETRIES=0 (and similarly MAX_FIX_ATTEMPTS=0) are treated as “unset” and replaced with the default, so callers cannot express zero even though the surrounding Math.max(...) suggests 0 is a valid value.

Suggested change

	maxRetries: Math.max(
	0,
	parseInt(process.env.MAX_RETRIES ?? "1", 10) || 1,
	),
	maxFixAttempts: Math.max(
	0,
	parseInt(process.env.MAX_FIX_ATTEMPTS ?? "2", 10) || 2,
	),
	maxRunsPerPr: Math.max(
	1,
	parseInt(process.env.MAX_RUNS_PER_PR ?? "10", 10) || 10,
	),
	maxRetries: (() => {
	const parsed = parseInt(process.env.MAX_RETRIES ?? "1", 10);
	return Math.max(0, Number.isNaN(parsed) ? 1 : parsed);
	})(),
	maxFixAttempts: (() => {
	const parsed = parseInt(process.env.MAX_FIX_ATTEMPTS ?? "2", 10);
	return Math.max(0, Number.isNaN(parsed) ? 2 : parsed);
	})(),
	maxRunsPerPr: (() => {
	const parsed = parseInt(process.env.MAX_RUNS_PER_PR ?? "10", 10);
	return Math.max(1, Number.isNaN(parsed) ? 10 : parsed);
	})(),

[factory-droid]

[P1] Treat maxRetries=0 as an explicit override

The retry override gate uses (context.inputs.maxRetries && context.inputs.maxRetries !== 1), which will incorrectly treat 0 as “not provided” and skip installing the retry override; once MAX_RETRIES=0 parses correctly, this becomes a concrete behavior bug.

Suggested change

	...((context.inputs.retryMode && context.inputs.retryMode !== "smart") ||
	(context.inputs.maxRetries && context.inputs.maxRetries !== 1)
	? {
	retry: {
	mode: context.inputs.retryMode ?? "smart",
	max_per_job: context.inputs.maxRetries ?? 1,
	eligible: [],
	exclude: [],
	},
	}
	: {}),
	...(((context.inputs.retryMode ?? "smart") !== "smart") ||
	((context.inputs.maxRetries ?? 1) !== 1)
	? {
	retry: {
	mode: context.inputs.retryMode ?? "smart",
	max_per_job: context.inputs.maxRetries ?? 1,
	eligible: [],
	exclude: [],
	},
	}
	: {}),

[factory-droid]

[P0] [security] Gate CI Medic workflow_run execution for untrusted PRs

The workflow_run automation path goes straight to prepareMedicMode and bypasses the existing human-actor and write-permission gates (those only run for entity contexts), which means a fork PR can trigger a secrets-bearing, tool-enabled CI Medic run in base-repo context; add an explicit gate that resolves the PR and skips unless the head repository is the base repo (or the PR author is trusted, for example has write/admin) before invoking Droid/reading logs.

[factory-droid]

[P1] [security] Load CI Medic config from a trusted ref (not head_branch)

loadMedicConfig(..., ref: run.head_branch, ...) reads .github/droid-ci.yml from a PR-controlled ref for same-repo PRs, which breaks the documented safety boundary that PRs cannot reconfigure CI Medic (and can be used to flip fix.enabled, tweak protected paths, or inject prompt instructions) before the PR is merged; resolve the PR first and load config from pr.baseRef (or the repo default branch), and consider treating workflow inputs like auto_fix as the hard gate regardless of config-file values.

[factory-droid]

[P1] [security] Avoid checking out head_branch from base repo for fork PRs

Checking out ${{ github.event.workflow_run.head_branch }} without also setting repository: can silently check out the base repo’s branch when a fork PR’s branch name collides (commonly main), so CI Medic runs against the wrong code while still having access to secrets and privileged tokens; check out the exact head_sha first, and only switch to a branch checkout when the head repository matches the base repo.

Suggested change

	# Check out the head branch (not the merge commit) so CI Medic can
	# reproduce failures and, when auto_fix is on, commit to a real branch.
	- name: Checkout pull request branch
	id: checkout_branch
	continue-on-error: true
	uses: actions/checkout@v4
	with:
	ref: ${{ github.event.workflow_run.head_branch }}
	fetch-depth: 0
	# Fork branches do not exist in the base repository. Fall back to a
	# detached checkout so diagnosis still works; CI Medic downgrades to
	# comment-only because it cannot push to a fork.
	- name: Checkout pull request commit
	if: steps.checkout_branch.outcome == 'failure'
	uses: actions/checkout@v4
	with:
	ref: ${{ github.event.workflow_run.head_sha }}
	fetch-depth: 0
	# Always check out the exact head SHA so CI Medic operates on PR code.
	- name: Checkout pull request commit
	uses: actions/checkout@v4
	with:
	ref: ${{ github.event.workflow_run.head_sha }}
	fetch-depth: 0
	# For same-repo PRs, also check out the branch name so CI Medic can push fixes.
	- name: Checkout pull request branch (same repo only)
	if: github.event.workflow_run.head_repository.full_name == github.repository
	uses: actions/checkout@v4
	with:
	ref: ${{ github.event.workflow_run.head_branch }}
	fetch-depth: 0