PFR: expose pluggable agent runners via an `--agent` flag (so `functional`/`trigger`/`report` can use non-Claude CLIs)

[benlec]

Summary

The functional, trigger, report, and compare commands currently drive the
claude CLI exclusively. The codebase already contains the abstraction needed to
support other agent CLIs — AgentRunner (ABC), a register_runner() / get_runner()
registry, and per-runner skill-injection hooks — but only ClaudeRunner is registered,
and there is no user-facing way to select a different runner. As a result, anyone
without the Anthropic claude CLI installed cannot run the agent-backed evaluations
(functional / trigger / report), even though audit / init / snapshot / regression
work fine without an agent.

This request is to finish wiring up the abstraction that already exists: add a way
to select a registered runner (e.g. an --agent NAME flag), and document the
AgentRunner contract so third parties can register their own runners.

Why this matters

• The agent-backed commands are gated on one specific vendor CLI (claude). Teams
  standardized on other coding agents (Amazon Q / Kiro CLI, aider, Codex, Cursor CLI,
  etc.) currently can't produce a functional/trigger/unified grade at all.
• The design clearly anticipated this — agent_runner.py documents the interface as
  "enables support for any agent CLI by implementing the AgentRunner interface and
  registering it with register_runner()". Today that extensibility isn't reachable from
  the CLI; a user would have to monkey-patch the registry in a wrapper script.
• CI adoption: the GitHub Actions reusable workflow (skill-eval.yml) is similarly
  Claude-bound. A runner selector would let orgs run the same gate with whatever agent
  they already have credentials for.

Current behavior (as of main)

• skill_eval/agent_runner.py defines:
  • AgentRunner (ABC) with check_available(), run_prompt(), parse_output(),
    and a default total_tokens().
  • register_runner(name, cls) and get_runner(name="claude") — a working registry.
  • ClaudeRunner, registered as "claude", which shells out to claude -p ...
    with --append-system-prompt <SKILL.md> for the with-skill arm and
    --output-format stream-json parsing.
• get_runner() defaults to "claude" and nothing passes a different name, so the
  functional/trigger/report paths always resolve to ClaudeRunner.
• There is no --agent (or equivalent) CLI option, and no environment variable, to
  select a different registered runner.

Proposed behavior

1. Runner selector on the agent-backed commands. Add --agent NAME (default
   claude) to functional, trigger, report, and compare, threading NAME
   through to get_runner(NAME). Optionally honor a SKILL_EVAL_AGENT env var so CI
   can set it once.
2. Clear error when a runner isn't registered/available. If --agent foo is
   unknown, list the registered runners (the registry already produces this message);
   if the selected runner's check_available() fails, surface its install hint (as
   ClaudeRunner already does).
3. Document the AgentRunner contract in the README/AGENTS.md: the four methods to
   implement, the expected parse_output() shape (events, tool_calls, text,
   token_counts), the skill-injection convention, and a minimal example of a custom
   runner registered via register_runner(). Include what CLI shape a runner is
   expected to wrap (non-interactive "prompt in → machine-readable transcript out",
   ideally a streaming-JSON mode, plus a way to inject skill/system context).
4. (Optional) Entry-point discovery so runners can be contributed as separate
   packages without editing this repo (e.g. a skill_eval.runners entry-point group).

Motivating example

We built and validated a skill against a real agent and want to produce the unified
grade the project gates on, but the claude CLI isn't available in our environment; we
do have another agent CLI. With --agent <name> plus a small AgentRunner subclass, we
could run skill-eval report ./skills/<name> --agent <name> and get audit + functional

• trigger in one grade. (Happy to contribute a reference runner as a follow-up PR — see
  below.)

Backwards compatibility

Fully backwards compatible: --agent defaults to claude, so existing invocations and
the CI workflow behave exactly as today.

Willing to contribute

Yes — if the maintainers are open to it, we're happy to open a PR that (a) adds the
--agent flag threaded through the four commands, (b) adds the unknown-runner and
not-available error handling, and (c) documents the AgentRunner contract with a small
example runner. We'd like to align on the flag name and the runner-discovery approach
(explicit register_runner() vs. entry points) in this issue before sending code, per
CONTRIBUTING.md.

Environment

• sample-agent-skill-eval @ main
• Observed in skill_eval/agent_runner.py, skill_eval/_claude.py, skill_eval/cli.py