From 57f9920d8f4b5579c754a723983b4dcb5a0d84f8 Mon Sep 17 00:00:00 2001 From: dalongbao Date: Thu, 30 Jul 2026 23:17:52 +0200 Subject: [PATCH 1/9] feat: agl-skill --- skills/README.md | 54 +++++++++++++++++++++ skills/agl-optimizer/SKILL.md | 90 +++++++++++++++++++++++++++++++++++ 2 files changed, 144 insertions(+) create mode 100644 skills/README.md create mode 100644 skills/agl-optimizer/SKILL.md diff --git a/skills/README.md b/skills/README.md new file mode 100644 index 000000000..d083dbd1b --- /dev/null +++ b/skills/README.md @@ -0,0 +1,54 @@ +# Agent Skills + +Skills in the [Agent Skills](https://agentskills.io) format (`/SKILL.md`), installable into any compatible agent. + +## agl-optimizer + +Turns your coding agent into an **agent optimizer**: given an editable agent and a benchmark to hillclimb on, it improves the agent's accuracy, cost, and latency through focused, individually-measured edits — keeping only what moves the frontier. Developed in the [agl-skill harness](https://github.com/agent-lightning/agl-skill) and measured there against a no-skill control under a fair, leakage-free protocol. + +You provide the environment; the skill does the optimizing. Before invoking it, have ready: a working copy of the agent (keep the original pristine), labeled examples, a frozen eval command, and an objective + budget. See [here](https://github.com/agent-lightning/agl-skill/tree/main/targets) for examples. + +### Installation + +Claude Code users can alternatively install the packaged plugin, published from [agent-lightning/agl-skill](https://github.com/agent-lightning/agl-skill) to the community marketplace: `/plugin install agl-skill@claude-community` (the skill then appears as `/agl-skill:agl-optimizer`). + +Alternatively, install via GitHub: +```bash +gh skill install microsoft/agent-lightning agl-optimizer --agent claude-code +gh skill install microsoft/agent-lightning agl-optimizer --agent codex +gh skill install microsoft/agent-lightning agl-optimizer --agent github-copilot +``` + +or copy `agl-optimizer/` into your agent's skills folder (`~/.claude/skills/` for Claude Code, `~/.agents/skills/` for Codex, `~/.copilot/skills/` for Copilot). + +### Results + +**Main finding:** Coding-agent harnesses are already strong optimizers. The clearest opportunity is improving consistency while preserving their high average performance, rather than expecting large score gains. + +For context, these are the published GPT-5.4-mini direct-chat results from [SkillOpt](https://github.com/microsoft/SkillOpt) (Table 1): + +| Method | Spreadsheet (%) | OfficeQA (%) | ALFWorld (%) | +| :--- | ---: | ---: | ---: | +| No skill | 36.1 | 22.1 | 73.1 | +| Human skill | 42.9 | 45.9 | 56.7 | +| LLM skill | 36.8 | 36.6 | 65.7 | +| Trace2Skill | 40.7 | 20.9 | 82.8 | +| TextGrad | 38.2 | 30.0 | 70.9 | +| GEPA | 42.5 | 45.3 | 81.3 | +| **SkillOpt** | **47.5** | **48.8** | **85.8** | + +Our evaluation uses a different protocol: Claude Code, Codex, and Copilot act as agentic optimizers that can edit the whole agent. Results are held-out test means pooled across \$5, \$10, and \$25 budgets (`n = 9`) and are shown as mean ± standard deviation. Higher is better; bold marks the better mean within each row. + +| Benchmark (train/test) | Optimizer | agl-skill v9.7 (%) | No-skill control (%) | Change (pp) | +| :--- | :--- | ---: | ---: | ---: | +| SpreadsheetBench (120/280) | Claude Code | 67.8 ± 5.7 | **68.9** ± 2.5 | −1.1 | +| SpreadsheetBench (120/280) | Codex | **64.4** ± 2.9 | 58.4 ± 17.7 | +5.9 | +| SpreadsheetBench (120/280) | Copilot | **68.1** ± 3.2 | 61.5 ± 13.1 | +6.6 | +| OfficeQA (50/172) | Claude Code | **59.9** ± 4.4 | 58.7 ± 3.2 | +1.2 | +| OfficeQA (50/172) | Codex | **51.6** ± 2.9 | 50.6 ± 1.3 | +1.0 | +| OfficeQA (50/172) | Copilot | 52.1 ± 2.4 | **53.0** ± 2.7 | −0.8 | +| ALFWorld (3553/134) | Claude Code | **89.1** ± 26.4 | 88.2 ± 33.1 | +0.9 | +| ALFWorld (3553/134) | Codex | 97.5 ± 4.7 | **97.8** ± 5.9 | −0.3 | +| ALFWorld (3553/134) | Copilot | 97.5 ± 7.2 | **99.9** ± 0.3 | −2.4 | + +Because the two tables use different protocols and data splits, compare results within a table, not across tables. Full per-budget scores, cost breakdowns, and caveats are available in [RESULTS_V9_7.md](https://github.com/agent-lightning/agl-skill/blob/verifier-v8/RESULTS_V9_7.md). diff --git a/skills/agl-optimizer/SKILL.md b/skills/agl-optimizer/SKILL.md new file mode 100644 index 000000000..5e39d2914 --- /dev/null +++ b/skills/agl-optimizer/SKILL.md @@ -0,0 +1,90 @@ +--- +name: agl-optimizer +description: >- + Provides the action space, tradeoffs, and evaluation context for improving an + editable AI agent against a benchmark while preserving its deployment contract. + Use when optimizing agent accuracy, cost, latency, or reliability. +--- + +# Agent Optimization + +Agent optimization is a search over interacting choices. The useful question is +not which architecture is most sophisticated, but which change moves the requested +accuracy, cost, latency, and reliability frontier for this agent. + +The optimizer's development budget and the resulting agent's per-run cost are +different quantities. More development budget creates room to learn; it does not +imply that the deployed agent should spend more on every task. + +Your remaining development budget is reported at `/artifacts/cost_budget.json` +(`{spent, budget, remaining}`), refreshed as you work. Read that file to see how +much is left, and keep iterating — measure, edit, re-score — while meaningful +budget remains; do not stop at the first plausible result. A run is finished not +because one change worked, but because further measured changes no longer improve +the frontier within the budget you still have. If `remaining` is large, there is +more search to do: ground more cases, probe a lever you have not tested, or add +reps to resolve a noisy comparison. Check `remaining` again after each expensive +step so the decision to stop is evidence-based, not a default. + +## Action space + +| Lever | What it changes | Useful signal | Main tradeoff | +| ------------------------ | -------------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------ | +| Input grounding | Information and state visible to the model | Relevant deployment-visible context is missing | Longer context can distract or cost more | +| Output contract | Representation, types, schema, files, and terminal state | Work looks reasonable but is rejected or unreadable | Can overfit evaluator quirks | +| Prompt | Interpretation, priorities, and constraints | Instructions are misunderstood or important details are ignored | Prompt gains can be brittle | +| Tools | Deterministic inspection, computation, and execution | The model is approximating work a tool can do reliably | More code and new failure modes | +| Model | Base capability and knowledge | The primary cannot solve grounded cases | Cost, latency, and availability | +| Reasoning effort | Computation used by the primary call | Grounded hard cases remain | Cost and latency can grow nonlinearly | +| Failure isolation | Whether one failure damages other work | Individual tasks crash, time out, or corrupt shared state | Isolation does not recover the failed task | +| Conditional repair | A second attempt informed by failure evidence | Deployment-visible checks expose a recoverable failure | Extra calls and possible regressions | +| Routing | Different handling for different task classes | Difficulty or failure risk varies predictably | Router mistakes and operational complexity | +| Planning and interaction | State, ordering, and tool use across multiple steps | Long tasks lose goals or ignore observations | More state and control-flow overhead | +| Retrieval | Facts supplied from an available corpus | Correct answers depend on external knowledge | Retrieval errors and added latency | +| Critique or selection | Additional views or candidates | Independent attempts expose different useful information | Multiplied calls, cost, and latency | + +## Reading the evidence + +Different failures expose different amounts of information: + +- Exceptions, missing artifacts, invalid schemas, and timeouts are objective + signals. They can support deterministic checks or focused recovery. +- A valid-looking but wrong answer may expose no label-free repair signal. More + calls with the same information can repeat the same mistake. +- Repeated failures across different attempts suggest shared blindness, a contract + mismatch, or missing capability. Diverse failures make routing, critique, or + selection more plausible. +- A development gain that disappears under validation may come from randomness, + memorized examples, training-only fields, or a different deployment path. +- An unavailable measurement is unknown evidence, not proof that a candidate + improved or regressed. + +Levers interact. More reasoning cannot recover information the model never sees. +Repair cannot fix a systematic contract error when the retry receives no new +evidence. Failure isolation preserves the batch but does not repair an item. A +global model or effort increase and conditional escalation occupy different points +on the frontier. + +## Evaluation context + +Agent evaluations are often stochastic. A score can improve while many individual +cases regress, and a single strong result can be a lucky draw. Fixed cases, +validation splits, repeated runs, frozen primary outputs, and small probes are +different ways to reduce uncertainty; their value depends on the decision and the +available budget. + +Comparisons are easiest to interpret when the checkpoint, cases, model, effort, +concurrency, scorer, and execution path are held constant except for the variable +being studied. Accuracy is only one result: completion rate, downside, cost, latency, +and variance can change the decision. + +Development may expose labels, metadata, or tools that do not exist at deployment. +An improvement that depends on them is not a deployed improvement. Likewise, +measurement plumbing can fail independently of the agent being tested. + +## Boundaries + +- Preserve the target's external interface and deployment environment. +- Do not expose held-out labels or training-only answer fields to the deployed path. +- Treat the scorer and evaluation contract as immutable measurement surfaces. +- Leave a coherent measured checkpoint, not an unfinished or partially tested edit. From a3233c93e8fb62f527f29c4217a87dc14f54494a Mon Sep 17 00:00:00 2001 From: dalongbao Date: Fri, 31 Jul 2026 14:08:23 +0200 Subject: [PATCH 2/9] Publish AGL skill --- .github/workflows/skills.yml | 33 + skills/README.md | 83 +- skills/agl/.claude-plugin/plugin.json | 11 + skills/{agl-optimizer => agl}/SKILL.md | 2 +- skills/agl_results.csv | 55 + skills/assets/agl-alf-world-accuracy-bars.svg | 2460 ++++++++++++++++ skills/assets/agl-officeqa-accuracy-bars.svg | 2502 ++++++++++++++++ .../agl-spreadsheetbench-accuracy-bars.svg | 2526 +++++++++++++++++ 8 files changed, 7644 insertions(+), 28 deletions(-) create mode 100644 .github/workflows/skills.yml create mode 100644 skills/agl/.claude-plugin/plugin.json rename skills/{agl-optimizer => agl}/SKILL.md (99%) create mode 100644 skills/agl_results.csv create mode 100644 skills/assets/agl-alf-world-accuracy-bars.svg create mode 100644 skills/assets/agl-officeqa-accuracy-bars.svg create mode 100644 skills/assets/agl-spreadsheetbench-accuracy-bars.svg diff --git a/.github/workflows/skills.yml b/.github/workflows/skills.yml new file mode 100644 index 000000000..356713d14 --- /dev/null +++ b/.github/workflows/skills.yml @@ -0,0 +1,33 @@ +name: Validate AGL Skill + +permissions: + contents: read + +on: + push: + branches: [main] + paths: + - 'skills/**' + - '.github/workflows/skills.yml' + pull_request: + branches: [main] + paths: + - 'skills/**' + - '.github/workflows/skills.yml' + workflow_dispatch: + +jobs: + validate: + name: Validate Agent Skills and Claude plugin formats + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v6 + - uses: astral-sh/setup-uv@v7 + - name: Validate Agent Skills format + run: uvx --from 'skills-ref==0.1.1' agentskills validate skills/agl + - uses: actions/setup-node@v6 + with: + node-version: '22' + - name: Validate Claude Code plugin + run: npx --yes @anthropic-ai/claude-code@2.1.218 plugin validate skills/agl diff --git a/skills/README.md b/skills/README.md index d083dbd1b..c5f796809 100644 --- a/skills/README.md +++ b/skills/README.md @@ -2,30 +2,36 @@ Skills in the [Agent Skills](https://agentskills.io) format (`/SKILL.md`), installable into any compatible agent. -## agl-optimizer +## agl -Turns your coding agent into an **agent optimizer**: given an editable agent and a benchmark to hillclimb on, it improves the agent's accuracy, cost, and latency through focused, individually-measured edits — keeping only what moves the frontier. Developed in the [agl-skill harness](https://github.com/agent-lightning/agl-skill) and measured there against a no-skill control under a fair, leakage-free protocol. +Turns your coding agent into an **agent optimizer**: given an editable agent and a benchmark to hillclimb on, it improves the agent's accuracy, cost, and latency through focused, individually-measured edits — keeping only what moves the frontier. It was measured against a no-skill control under a fair, leakage-free protocol. -You provide the environment; the skill does the optimizing. Before invoking it, have ready: a working copy of the agent (keep the original pristine), labeled examples, a frozen eval command, and an objective + budget. See [here](https://github.com/agent-lightning/agl-skill/tree/main/targets) for examples. +You provide the environment; the skill does the optimizing. Before invoking it, have ready: a working copy of the agent (keep the original pristine), labeled examples, a frozen eval command, and an objective + budget. ### Installation -Claude Code users can alternatively install the packaged plugin, published from [agent-lightning/agl-skill](https://github.com/agent-lightning/agl-skill) to the community marketplace: `/plugin install agl-skill@claude-community` (the skill then appears as `/agl-skill:agl-optimizer`). +Install the skill from this repository for Claude Code, Codex, or GitHub Copilot: -Alternatively, install via GitHub: ```bash -gh skill install microsoft/agent-lightning agl-optimizer --agent claude-code -gh skill install microsoft/agent-lightning agl-optimizer --agent codex -gh skill install microsoft/agent-lightning agl-optimizer --agent github-copilot +gh skill install microsoft/agent-lightning agl --agent claude-code +gh skill install microsoft/agent-lightning agl --agent codex +gh skill install microsoft/agent-lightning agl --agent github-copilot ``` -or copy `agl-optimizer/` into your agent's skills folder (`~/.claude/skills/` for Claude Code, `~/.agents/skills/` for Codex, `~/.copilot/skills/` for Copilot). +Claude Code users can alternatively install the packaged plugin from the community marketplace: + +```text +/plugin marketplace add anthropics/claude-plugins-community +/plugin install agl@claude-community +``` + +The `skills/agl/` directory is both the canonical Agent Skills package and the Claude Code plugin root, so both publication paths use the same `SKILL.md` without a copied or symlinked wrapper. ### Results **Main finding:** Coding-agent harnesses are already strong optimizers. The clearest opportunity is improving consistency while preserving their high average performance, rather than expecting large score gains. -For context, these are the published GPT-5.4-mini direct-chat results from [SkillOpt](https://github.com/microsoft/SkillOpt) (Table 1): +SkillOpt and the other non-agentic results are taken from the [SkillOpt paper](https://github.com/microsoft/SkillOpt) (Table 1); our agentic rows use the same splits and average all optimizers, budgets, and replicates. | Method | Spreadsheet (%) | OfficeQA (%) | ALFWorld (%) | | :--- | ---: | ---: | ---: | @@ -35,20 +41,43 @@ For context, these are the published GPT-5.4-mini direct-chat results from [Skil | Trace2Skill | 40.7 | 20.9 | 82.8 | | TextGrad | 38.2 | 30.0 | 70.9 | | GEPA | 42.5 | 45.3 | 81.3 | -| **SkillOpt** | **47.5** | **48.8** | **85.8** | - -Our evaluation uses a different protocol: Claude Code, Codex, and Copilot act as agentic optimizers that can edit the whole agent. Results are held-out test means pooled across \$5, \$10, and \$25 budgets (`n = 9`) and are shown as mean ± standard deviation. Higher is better; bold marks the better mean within each row. - -| Benchmark (train/test) | Optimizer | agl-skill v9.7 (%) | No-skill control (%) | Change (pp) | -| :--- | :--- | ---: | ---: | ---: | -| SpreadsheetBench (120/280) | Claude Code | 67.8 ± 5.7 | **68.9** ± 2.5 | −1.1 | -| SpreadsheetBench (120/280) | Codex | **64.4** ± 2.9 | 58.4 ± 17.7 | +5.9 | -| SpreadsheetBench (120/280) | Copilot | **68.1** ± 3.2 | 61.5 ± 13.1 | +6.6 | -| OfficeQA (50/172) | Claude Code | **59.9** ± 4.4 | 58.7 ± 3.2 | +1.2 | -| OfficeQA (50/172) | Codex | **51.6** ± 2.9 | 50.6 ± 1.3 | +1.0 | -| OfficeQA (50/172) | Copilot | 52.1 ± 2.4 | **53.0** ± 2.7 | −0.8 | -| ALFWorld (3553/134) | Claude Code | **89.1** ± 26.4 | 88.2 ± 33.1 | +0.9 | -| ALFWorld (3553/134) | Codex | 97.5 ± 4.7 | **97.8** ± 5.9 | −0.3 | -| ALFWorld (3553/134) | Copilot | 97.5 ± 7.2 | **99.9** ± 0.3 | −2.4 | - -Because the two tables use different protocols and data splits, compare results within a table, not across tables. Full per-budget scores, cost breakdowns, and caveats are available in [RESULTS_V9_7.md](https://github.com/agent-lightning/agl-skill/blob/verifier-v8/RESULTS_V9_7.md). +| SkillOpt | 47.5 | 48.8 | 85.8 | +| Agentic optimizer average, no skill | 62.9 | 54.1 | **95.3** | +| **Agentic optimizer average, AGL** | **66.7** | **54.5** | 94.7 | + +#### Accuracy by budget and optimizer + +Each benchmark is split into the \$5, \$10, and \$25 nominal budgets. Within each budget, every optimizer's with-skill and no-skill held-out finale accuracies are shown side by side. Solid bars use the skill; hatched bars are no-skill controls. + +![SpreadsheetBench accuracy by budget and optimizer](assets/agl-spreadsheetbench-accuracy-bars.svg) + +![OfficeQA accuracy by budget and optimizer](assets/agl-officeqa-accuracy-bars.svg) + +![ALFWorld accuracy by budget and optimizer](assets/agl-alf-world-accuracy-bars.svg) + + +#### \$5 budget snapshot + +| Benchmark (train/test) | Result | Accuracy (%) | Actual total cost | +| :--- | :--- | ---: | ---: | +| SpreadsheetBench (120/280) | Before optimizer | 25.66 ± 2.65 | \$1.51 ± 0.04 | +| | Claude Code with skill | 63.79 ± 5.24 | **\$7.58 ± 0.90** | +| | Claude Code without skill | **68.23 ± 0.55** | \$7.59 ± 0.95 | +| | Codex with skill | **65.47 ± 4.59** | \$5.41 ± 0.17 | +| | Codex without skill | 41.49 ± 24.28 | **\$3.44 ± 1.78** | +| | Copilot with skill | **66.31 ± 2.05** | \$5.55 ± 0.69 | +| | Copilot without skill | 51.68 ± 20.82 | **\$3.40 ± 1.06** | +| OfficeQA (50/172) | Before optimizer | 31.78 ± 1.21 | \$2.78 ± 0.06 | +| | Claude Code with skill | 56.78 ± 3.87 | \$10.42 ± 1.59 | +| | Claude Code without skill | **59.69 ± 4.88** | **\$9.82 ± 1.72** | +| | Codex with skill | **49.81 ± 2.98** | **\$7.69 ± 0.21** | +| | Codex without skill | 49.61 ± 0.67 | \$8.03 ± 0.34 | +| | Copilot with skill | 51.55 ± 3.74 | **\$7.79 ± 0.91** | +| | Copilot without skill | **54.65 ± 2.01** | \$8.09 ± 1.28 | +| ALFWorld (3553/134) | Before optimizer | 58.71 ± 1.56 | \$5.78 ± 0.17 | +| | Claude Code with skill | **94.53 ± 4.56** | \$7.37 ± 1.81 | +| | Claude Code without skill | 64.68 ± 56.09 | **\$6.30 ± 2.03** | +| | Codex with skill | 94.78 ± 7.12 | \$3.82 ± 1.21 | +| | Codex without skill | **100.00 ± 0.00** | **\$1.31 ± 0.56** | +| | Copilot with skill | 92.79 ± 12.49 | **\$1.94 ± 0.80** | +| | Copilot without skill | **99.75 ± 0.43** | \$2.37 ± 0.63 | diff --git a/skills/agl/.claude-plugin/plugin.json b/skills/agl/.claude-plugin/plugin.json new file mode 100644 index 000000000..0dd374140 --- /dev/null +++ b/skills/agl/.claude-plugin/plugin.json @@ -0,0 +1,11 @@ +{ + "name": "agl", + "displayName": "AGL", + "description": "Turns coding agents into agent optimizers that improve accuracy, cost, latency, and reliability against a benchmark.", + "author": { + "name": "Agent Lightning Team" + }, + "repository": "https://github.com/microsoft/agent-lightning", + "homepage": "https://microsoft.github.io/agent-lightning/", + "license": "MIT" +} diff --git a/skills/agl-optimizer/SKILL.md b/skills/agl/SKILL.md similarity index 99% rename from skills/agl-optimizer/SKILL.md rename to skills/agl/SKILL.md index 5e39d2914..8810264ca 100644 --- a/skills/agl-optimizer/SKILL.md +++ b/skills/agl/SKILL.md @@ -1,5 +1,5 @@ --- -name: agl-optimizer +name: agl description: >- Provides the action space, tradeoffs, and evaluation context for improving an editable AI agent against a benchmark while preserving its deployment contract. diff --git a/skills/agl_results.csv b/skills/agl_results.csv new file mode 100644 index 000000000..fc1b739f2 --- /dev/null +++ b/skills/agl_results.csv @@ -0,0 +1,55 @@ +benchmark,optimizer,condition,nominal_budget,accuracy_mean,accuracy_sd,total_cost_mean,total_cost_sd +SpreadsheetBench,Claude Code,skill,5,0.6379,0.0524,7.58,0.90 +SpreadsheetBench,Claude Code,skill,10,0.6775,0.0240,11.86,0.33 +SpreadsheetBench,Claude Code,skill,25,0.7170,0.0711,32.46,5.46 +SpreadsheetBench,Claude Code,no-skill,5,0.6823,0.0055,7.59,0.95 +SpreadsheetBench,Claude Code,no-skill,10,0.6942,0.0432,12.26,0.25 +SpreadsheetBench,Claude Code,no-skill,25,0.6894,0.0198,30.60,5.64 +SpreadsheetBench,Codex,skill,5,0.6547,0.0459,5.41,0.17 +SpreadsheetBench,Codex,skill,10,0.6463,0.0075,5.58,0.86 +SpreadsheetBench,Codex,skill,25,0.6295,0.0252,14.19,2.29 +SpreadsheetBench,Codex,no-skill,5,0.4149,0.2428,3.44,1.78 +SpreadsheetBench,Codex,no-skill,10,0.6859,0.0116,4.98,0.47 +SpreadsheetBench,Codex,no-skill,25,0.6523,0.0240,4.86,0.60 +SpreadsheetBench,Copilot,skill,5,0.6631,0.0205,5.55,0.69 +SpreadsheetBench,Copilot,skill,10,0.6930,0.0432,10.29,1.49 +SpreadsheetBench,Copilot,skill,25,0.6871,0.0312,10.61,6.64 +SpreadsheetBench,Copilot,no-skill,5,0.5168,0.2082,3.40,1.06 +SpreadsheetBench,Copilot,no-skill,10,0.6439,0.0288,3.87,0.34 +SpreadsheetBench,Copilot,no-skill,25,0.6847,0.0360,5.54,1.06 +OfficeQA,Claude Code,skill,5,0.5678,0.0387,10.42,1.59 +OfficeQA,Claude Code,skill,10,0.6260,0.0374,16.16,1.02 +OfficeQA,Claude Code,skill,25,0.6027,0.0470,27.05,1.62 +OfficeQA,Claude Code,no-skill,5,0.5969,0.0488,9.82,1.72 +OfficeQA,Claude Code,no-skill,10,0.5930,0.0174,14.55,2.43 +OfficeQA,Claude Code,no-skill,25,0.5717,0.0298,32.64,12.01 +OfficeQA,Codex,skill,5,0.4981,0.0298,7.69,0.21 +OfficeQA,Codex,skill,10,0.5407,0.0116,9.67,1.69 +OfficeQA,Codex,skill,25,0.5078,0.0287,14.09,1.10 +OfficeQA,Codex,no-skill,5,0.4961,0.0067,8.03,0.34 +OfficeQA,Codex,no-skill,10,0.5000,0.0058,7.64,0.58 +OfficeQA,Codex,no-skill,25,0.5213,0.0034,8.81,1.67 +OfficeQA,Copilot,skill,5,0.5155,0.0374,7.79,0.91 +OfficeQA,Copilot,skill,10,0.5368,0.0089,11.01,1.14 +OfficeQA,Copilot,skill,25,0.5116,0.0174,19.29,6.23 +OfficeQA,Copilot,no-skill,5,0.5465,0.0201,8.09,1.28 +OfficeQA,Copilot,no-skill,10,0.5116,0.0407,8.05,2.98 +OfficeQA,Copilot,no-skill,25,0.5310,0.0034,8.94,0.33 +ALFWorld,Claude Code,skill,5,0.9453,0.0456,7.37,1.81 +ALFWorld,Claude Code,skill,10,0.7313,0.4653,10.55,1.64 +ALFWorld,Claude Code,skill,25,0.9975,0.0043,20.14,8.47 +ALFWorld,Claude Code,no-skill,5,0.6468,0.5609,6.30,2.03 +ALFWorld,Claude Code,no-skill,10,1.0000,0.0000,11.08,1.55 +ALFWorld,Claude Code,no-skill,25,1.0000,0.0000,10.44,1.61 +ALFWorld,Codex,skill,5,0.9478,0.0712,3.82,1.21 +ALFWorld,Codex,skill,10,1.0000,0.0000,6.38,3.14 +ALFWorld,Codex,skill,25,0.9776,0.0388,8.13,4.42 +ALFWorld,Codex,no-skill,5,1.0000,0.0000,1.31,0.56 +ALFWorld,Codex,no-skill,10,1.0000,0.0000,1.98,1.41 +ALFWorld,Codex,no-skill,25,0.9328,0.0976,3.64,0.30 +ALFWorld,Copilot,skill,5,0.9279,0.1249,1.94,0.80 +ALFWorld,Copilot,skill,10,1.0000,0.0000,1.52,1.05 +ALFWorld,Copilot,skill,25,0.9975,0.0043,2.79,1.78 +ALFWorld,Copilot,no-skill,5,0.9975,0.0043,2.37,0.63 +ALFWorld,Copilot,no-skill,10,1.0000,0.0000,2.39,2.23 +ALFWorld,Copilot,no-skill,25,1.0000,0.0000,1.47,0.43 diff --git a/skills/assets/agl-alf-world-accuracy-bars.svg b/skills/assets/agl-alf-world-accuracy-bars.svg new file mode 100644 index 000000000..643aebf6c --- /dev/null +++ b/skills/assets/agl-alf-world-accuracy-bars.svg @@ -0,0 +1,2460 @@ + + + + + + + + image/svg+xml + + + Matplotlib v3.11.1, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/skills/assets/agl-officeqa-accuracy-bars.svg b/skills/assets/agl-officeqa-accuracy-bars.svg new file mode 100644 index 000000000..767740e0b --- /dev/null +++ b/skills/assets/agl-officeqa-accuracy-bars.svg @@ -0,0 +1,2502 @@ + + + + + + + + image/svg+xml + + + Matplotlib v3.11.1, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/skills/assets/agl-spreadsheetbench-accuracy-bars.svg b/skills/assets/agl-spreadsheetbench-accuracy-bars.svg new file mode 100644 index 000000000..0d93e2456 --- /dev/null +++ b/skills/assets/agl-spreadsheetbench-accuracy-bars.svg @@ -0,0 +1,2526 @@ + + + + + + + + image/svg+xml + + + Matplotlib v3.11.1, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + From be5eda3ef9bd59bfbb9951ab38f87cc7f32ffe42 Mon Sep 17 00:00:00 2001 From: dalongbao Date: Tue, 4 Aug 2026 08:01:53 +0200 Subject: [PATCH 3/9] docs: graphs and results at other budgets --- skills/README.md | 61 + skills/agl_results.csv | 55 - .../agl-alf-world-accuracy-log-cost.svg | 2278 ++++++++++++++++ .../assets/agl-officeqa-accuracy-log-cost.svg | 2277 ++++++++++++++++ ...agl-spreadsheetbench-accuracy-log-cost.svg | 2322 +++++++++++++++++ 5 files changed, 6938 insertions(+), 55 deletions(-) delete mode 100644 skills/agl_results.csv create mode 100644 skills/assets/agl-alf-world-accuracy-log-cost.svg create mode 100644 skills/assets/agl-officeqa-accuracy-log-cost.svg create mode 100644 skills/assets/agl-spreadsheetbench-accuracy-log-cost.svg diff --git a/skills/README.md b/skills/README.md index c5f796809..502841d9e 100644 --- a/skills/README.md +++ b/skills/README.md @@ -55,6 +55,15 @@ Each benchmark is split into the \$5, \$10, and \$25 nominal budgets. Within eac ![ALFWorld accuracy by budget and optimizer](assets/agl-alf-world-accuracy-bars.svg) +#### Accuracy versus actual cost + +Each line connects the \$5, \$10, and \$25 nominal budgets in that order. The x-axis is average actual total cost on a log scale, including optimizer calls, train/self-evaluation, and held-out finale generation. + +![SpreadsheetBench accuracy versus actual cost](assets/agl-spreadsheetbench-accuracy-log-cost.svg) + +![OfficeQA accuracy versus actual cost](assets/agl-officeqa-accuracy-log-cost.svg) + +![ALFWorld accuracy versus actual cost](assets/agl-alf-world-accuracy-log-cost.svg) #### \$5 budget snapshot @@ -81,3 +90,55 @@ Each benchmark is split into the \$5, \$10, and \$25 nominal budgets. Within eac | | Codex without skill | **100.00 ± 0.00** | **\$1.31 ± 0.56** | | | Copilot with skill | 92.79 ± 12.49 | **\$1.94 ± 0.80** | | | Copilot without skill | **99.75 ± 0.43** | \$2.37 ± 0.63 | + +#### \$10 budget snapshot + +| Benchmark (train/test) | Result | Accuracy (%) | Actual total cost | +| :--- | :--- | ---: | ---: | +| SpreadsheetBench (120/280) | Before optimizer | 25.66 ± 2.65 | \$1.51 ± 0.04 | +| | Claude Code with skill | 67.75 ± 2.40 | **\$11.86 ± 0.33** | +| | Claude Code without skill | **69.42 ± 4.32** | \$12.26 ± 0.25 | +| | Codex with skill | 64.63 ± 0.75 | \$5.58 ± 0.86 | +| | Codex without skill | **68.59 ± 1.16** | **\$4.98 ± 0.47** | +| | Copilot with skill | **69.30 ± 4.32** | \$10.29 ± 1.49 | +| | Copilot without skill | 64.39 ± 2.88 | **\$3.87 ± 0.34** | +| OfficeQA (50/172) | Before optimizer | 31.78 ± 1.21 | \$2.78 ± 0.06 | +| | Claude Code with skill | **62.60 ± 3.74** | \$16.16 ± 1.02 | +| | Claude Code without skill | 59.30 ± 1.74 | **\$14.55 ± 2.43** | +| | Codex with skill | **54.07 ± 1.16** | \$9.67 ± 1.69 | +| | Codex without skill | 50.00 ± 0.58 | **\$7.64 ± 0.58** | +| | Copilot with skill | **53.68 ± 0.89** | \$11.01 ± 1.14 | +| | Copilot without skill | 51.16 ± 4.07 | **\$8.05 ± 2.98** | +| ALFWorld (3553/134) | Before optimizer | 58.71 ± 1.56 | \$5.78 ± 0.17 | +| | Claude Code with skill | 73.13 ± 46.53 | **\$10.55 ± 1.64** | +| | Claude Code without skill | **100.00 ± 0.00** | \$11.08 ± 1.55 | +| | Codex with skill | 100.00 ± 0.00 | \$6.38 ± 3.14 | +| | Codex without skill | 100.00 ± 0.00 | **\$1.98 ± 1.41** | +| | Copilot with skill | 100.00 ± 0.00 | **\$1.52 ± 1.05** | +| | Copilot without skill | 100.00 ± 0.00 | \$2.39 ± 2.23 | + +#### \$25 budget snapshot + +| Benchmark (train/test) | Result | Accuracy (%) | Actual total cost | +| :--- | :--- | ---: | ---: | +| SpreadsheetBench (120/280) | Before optimizer | 25.66 ± 2.65 | \$1.51 ± 0.04 | +| | Claude Code with skill | **71.70 ± 7.11** | \$32.46 ± 5.46 | +| | Claude Code without skill | 68.94 ± 1.98 | **\$30.60 ± 5.64** | +| | Codex with skill | 62.95 ± 2.52 | \$14.19 ± 2.29 | +| | Codex without skill | **65.23 ± 2.40** | **\$4.86 ± 0.60** | +| | Copilot with skill | **68.71 ± 3.12** | \$10.61 ± 6.64 | +| | Copilot without skill | 68.47 ± 3.60 | **\$5.54 ± 1.06** | +| OfficeQA (50/172) | Before optimizer | 31.78 ± 1.21 | \$2.78 ± 0.06 | +| | Claude Code with skill | **60.27 ± 4.70** | **\$27.05 ± 1.62** | +| | Claude Code without skill | 57.17 ± 2.98 | \$32.64 ± 12.01 | +| | Codex with skill | 50.78 ± 2.87 | \$14.09 ± 1.10 | +| | Codex without skill | **52.13 ± 0.34** | **\$8.81 ± 1.67** | +| | Copilot with skill | 51.16 ± 1.74 | \$19.29 ± 6.23 | +| | Copilot without skill | **53.10 ± 0.34** | **\$8.94 ± 0.33** | +| ALFWorld (3553/134) | Before optimizer | 58.71 ± 1.56 | \$5.78 ± 0.17 | +| | Claude Code with skill | 99.75 ± 0.43 | \$20.14 ± 8.47 | +| | Claude Code without skill | **100.00 ± 0.00** | **\$10.44 ± 1.61** | +| | Codex with skill | **97.76 ± 3.88** | \$8.13 ± 4.42 | +| | Codex without skill | 93.28 ± 9.76 | **\$3.64 ± 0.30** | +| | Copilot with skill | 99.75 ± 0.43 | \$2.79 ± 1.78 | +| | Copilot without skill | **100.00 ± 0.00** | **\$1.47 ± 0.43** | diff --git a/skills/agl_results.csv b/skills/agl_results.csv deleted file mode 100644 index fc1b739f2..000000000 --- a/skills/agl_results.csv +++ /dev/null @@ -1,55 +0,0 @@ -benchmark,optimizer,condition,nominal_budget,accuracy_mean,accuracy_sd,total_cost_mean,total_cost_sd -SpreadsheetBench,Claude Code,skill,5,0.6379,0.0524,7.58,0.90 -SpreadsheetBench,Claude Code,skill,10,0.6775,0.0240,11.86,0.33 -SpreadsheetBench,Claude Code,skill,25,0.7170,0.0711,32.46,5.46 -SpreadsheetBench,Claude Code,no-skill,5,0.6823,0.0055,7.59,0.95 -SpreadsheetBench,Claude Code,no-skill,10,0.6942,0.0432,12.26,0.25 -SpreadsheetBench,Claude Code,no-skill,25,0.6894,0.0198,30.60,5.64 -SpreadsheetBench,Codex,skill,5,0.6547,0.0459,5.41,0.17 -SpreadsheetBench,Codex,skill,10,0.6463,0.0075,5.58,0.86 -SpreadsheetBench,Codex,skill,25,0.6295,0.0252,14.19,2.29 -SpreadsheetBench,Codex,no-skill,5,0.4149,0.2428,3.44,1.78 -SpreadsheetBench,Codex,no-skill,10,0.6859,0.0116,4.98,0.47 -SpreadsheetBench,Codex,no-skill,25,0.6523,0.0240,4.86,0.60 -SpreadsheetBench,Copilot,skill,5,0.6631,0.0205,5.55,0.69 -SpreadsheetBench,Copilot,skill,10,0.6930,0.0432,10.29,1.49 -SpreadsheetBench,Copilot,skill,25,0.6871,0.0312,10.61,6.64 -SpreadsheetBench,Copilot,no-skill,5,0.5168,0.2082,3.40,1.06 -SpreadsheetBench,Copilot,no-skill,10,0.6439,0.0288,3.87,0.34 -SpreadsheetBench,Copilot,no-skill,25,0.6847,0.0360,5.54,1.06 -OfficeQA,Claude Code,skill,5,0.5678,0.0387,10.42,1.59 -OfficeQA,Claude Code,skill,10,0.6260,0.0374,16.16,1.02 -OfficeQA,Claude Code,skill,25,0.6027,0.0470,27.05,1.62 -OfficeQA,Claude Code,no-skill,5,0.5969,0.0488,9.82,1.72 -OfficeQA,Claude Code,no-skill,10,0.5930,0.0174,14.55,2.43 -OfficeQA,Claude Code,no-skill,25,0.5717,0.0298,32.64,12.01 -OfficeQA,Codex,skill,5,0.4981,0.0298,7.69,0.21 -OfficeQA,Codex,skill,10,0.5407,0.0116,9.67,1.69 -OfficeQA,Codex,skill,25,0.5078,0.0287,14.09,1.10 -OfficeQA,Codex,no-skill,5,0.4961,0.0067,8.03,0.34 -OfficeQA,Codex,no-skill,10,0.5000,0.0058,7.64,0.58 -OfficeQA,Codex,no-skill,25,0.5213,0.0034,8.81,1.67 -OfficeQA,Copilot,skill,5,0.5155,0.0374,7.79,0.91 -OfficeQA,Copilot,skill,10,0.5368,0.0089,11.01,1.14 -OfficeQA,Copilot,skill,25,0.5116,0.0174,19.29,6.23 -OfficeQA,Copilot,no-skill,5,0.5465,0.0201,8.09,1.28 -OfficeQA,Copilot,no-skill,10,0.5116,0.0407,8.05,2.98 -OfficeQA,Copilot,no-skill,25,0.5310,0.0034,8.94,0.33 -ALFWorld,Claude Code,skill,5,0.9453,0.0456,7.37,1.81 -ALFWorld,Claude Code,skill,10,0.7313,0.4653,10.55,1.64 -ALFWorld,Claude Code,skill,25,0.9975,0.0043,20.14,8.47 -ALFWorld,Claude Code,no-skill,5,0.6468,0.5609,6.30,2.03 -ALFWorld,Claude Code,no-skill,10,1.0000,0.0000,11.08,1.55 -ALFWorld,Claude Code,no-skill,25,1.0000,0.0000,10.44,1.61 -ALFWorld,Codex,skill,5,0.9478,0.0712,3.82,1.21 -ALFWorld,Codex,skill,10,1.0000,0.0000,6.38,3.14 -ALFWorld,Codex,skill,25,0.9776,0.0388,8.13,4.42 -ALFWorld,Codex,no-skill,5,1.0000,0.0000,1.31,0.56 -ALFWorld,Codex,no-skill,10,1.0000,0.0000,1.98,1.41 -ALFWorld,Codex,no-skill,25,0.9328,0.0976,3.64,0.30 -ALFWorld,Copilot,skill,5,0.9279,0.1249,1.94,0.80 -ALFWorld,Copilot,skill,10,1.0000,0.0000,1.52,1.05 -ALFWorld,Copilot,skill,25,0.9975,0.0043,2.79,1.78 -ALFWorld,Copilot,no-skill,5,0.9975,0.0043,2.37,0.63 -ALFWorld,Copilot,no-skill,10,1.0000,0.0000,2.39,2.23 -ALFWorld,Copilot,no-skill,25,1.0000,0.0000,1.47,0.43 diff --git a/skills/assets/agl-alf-world-accuracy-log-cost.svg b/skills/assets/agl-alf-world-accuracy-log-cost.svg new file mode 100644 index 000000000..bc74e4129 --- /dev/null +++ b/skills/assets/agl-alf-world-accuracy-log-cost.svg @@ -0,0 +1,2278 @@ + + + + + + + + image/svg+xml + + + Matplotlib v3.11.1, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/skills/assets/agl-officeqa-accuracy-log-cost.svg b/skills/assets/agl-officeqa-accuracy-log-cost.svg new file mode 100644 index 000000000..f90f029a0 --- /dev/null +++ b/skills/assets/agl-officeqa-accuracy-log-cost.svg @@ -0,0 +1,2277 @@ + + + + + + + + image/svg+xml + + + Matplotlib v3.11.1, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/skills/assets/agl-spreadsheetbench-accuracy-log-cost.svg b/skills/assets/agl-spreadsheetbench-accuracy-log-cost.svg new file mode 100644 index 000000000..b534899bb --- /dev/null +++ b/skills/assets/agl-spreadsheetbench-accuracy-log-cost.svg @@ -0,0 +1,2322 @@ + + + + + + + + image/svg+xml + + + Matplotlib v3.11.1, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + From 3d48f4d6baa61124fd2063ac2c86112dfa8904a1 Mon Sep 17 00:00:00 2001 From: dalongbao Date: Tue, 4 Aug 2026 08:27:06 +0200 Subject: [PATCH 4/9] style: optimize cost only, crop y axis on line graph --- skills/README.md | 10 +- .../agl-alf-world-accuracy-log-cost.svg | 1277 ++++++++-------- .../assets/agl-officeqa-accuracy-log-cost.svg | 1303 ++++++++--------- ...agl-spreadsheetbench-accuracy-log-cost.svg | 1255 ++++++++-------- 4 files changed, 1903 insertions(+), 1942 deletions(-) diff --git a/skills/README.md b/skills/README.md index 502841d9e..8d72f6ed1 100644 --- a/skills/README.md +++ b/skills/README.md @@ -55,15 +55,15 @@ Each benchmark is split into the \$5, \$10, and \$25 nominal budgets. Within eac ![ALFWorld accuracy by budget and optimizer](assets/agl-alf-world-accuracy-bars.svg) -#### Accuracy versus actual cost +#### Accuracy versus optimize-stage cost -Each line connects the \$5, \$10, and \$25 nominal budgets in that order. The x-axis is average actual total cost on a log scale, including optimizer calls, train/self-evaluation, and held-out finale generation. +Each line connects the \$5, \$10, and \$25 nominal budgets in that order. The x-axis is average actual optimize-stage cost on a log scale, including optimizer calls and train/self-evaluation but excluding held-out finale generation. -![SpreadsheetBench accuracy versus actual cost](assets/agl-spreadsheetbench-accuracy-log-cost.svg) +![SpreadsheetBench accuracy versus optimize-stage cost](assets/agl-spreadsheetbench-accuracy-log-cost.svg) -![OfficeQA accuracy versus actual cost](assets/agl-officeqa-accuracy-log-cost.svg) +![OfficeQA accuracy versus optimize-stage cost](assets/agl-officeqa-accuracy-log-cost.svg) -![ALFWorld accuracy versus actual cost](assets/agl-alf-world-accuracy-log-cost.svg) +![ALFWorld accuracy versus optimize-stage cost](assets/agl-alf-world-accuracy-log-cost.svg) #### \$5 budget snapshot diff --git a/skills/assets/agl-alf-world-accuracy-log-cost.svg b/skills/assets/agl-alf-world-accuracy-log-cost.svg index bc74e4129..d73c138e7 100644 --- a/skills/assets/agl-alf-world-accuracy-log-cost.svg +++ b/skills/assets/agl-alf-world-accuracy-log-cost.svg @@ -1,7 +1,7 @@ - + @@ -9,7 +9,7 @@ image/svg+xml - Matplotlib v3.11.1, https://matplotlib.org/ + Matplotlib v3.11.0, https://matplotlib.org/ @@ -20,42 +20,42 @@ - - - + - - + - - + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + - + - + - + - + - - + + + - + - + - + - - - - + - - - + + + - + - + - + - + - - - + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - + - - + - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - @@ -1004,20 +953,20 @@ z - - - + + + - + - + - + - + - - + + + - + - + - + - + - - + + + - + - + - + - + - - + + + - + - + - + - + @@ -1170,9 +1119,9 @@ L 698.489844 51.429656 - + - + - - - - - - - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - - + + + + + + + + + + + + + + @@ -1914,32 +1974,40 @@ z - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + - - + + - + - + - + - - - + + - + - + - + @@ -2094,18 +2149,18 @@ L 200.272344 387.740713 - - + + - + - + - + - - + + - + - + - + @@ -2173,46 +2228,18 @@ L 354.705312 387.740713 - - + + - + - + - - - - + @@ -2235,18 +2262,18 @@ z - - + + - + - + - + @@ -2271,8 +2298,8 @@ L 480.192031 387.740713 - - + + diff --git a/skills/assets/agl-officeqa-accuracy-log-cost.svg b/skills/assets/agl-officeqa-accuracy-log-cost.svg index f90f029a0..2eeecdccd 100644 --- a/skills/assets/agl-officeqa-accuracy-log-cost.svg +++ b/skills/assets/agl-officeqa-accuracy-log-cost.svg @@ -1,7 +1,7 @@ - + @@ -9,7 +9,7 @@ image/svg+xml - Matplotlib v3.11.1, https://matplotlib.org/ + Matplotlib v3.11.0, https://matplotlib.org/ @@ -20,42 +20,42 @@ - - - + - - + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - + - - + - - - + + + - - - - - - - - - - - - - - - - - - - + + - - - + + + - + - + - - - + + + - - - + + + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - + - - - + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + - + - + - - - - - - - - + + + + + - - - - - - - - - - - - - - - - - - - - + - + - - - - - - - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - - + + + + + + + + + + + + + + @@ -1913,32 +1879,40 @@ z - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + - - + + - + - + - + - - - + + - + - + - + @@ -2093,18 +2054,18 @@ L 200.272344 387.740713 - - + + - + - + - + - - + + - + - + - + @@ -2172,46 +2133,18 @@ L 354.705312 387.740713 - - + + - + - + - - - - + @@ -2234,18 +2167,18 @@ z - - + + - + - + - + @@ -2270,8 +2203,8 @@ L 480.192031 387.740713 - - + + diff --git a/skills/assets/agl-spreadsheetbench-accuracy-log-cost.svg b/skills/assets/agl-spreadsheetbench-accuracy-log-cost.svg index b534899bb..4b69cde03 100644 --- a/skills/assets/agl-spreadsheetbench-accuracy-log-cost.svg +++ b/skills/assets/agl-spreadsheetbench-accuracy-log-cost.svg @@ -1,7 +1,7 @@ - + @@ -9,7 +9,7 @@ image/svg+xml - Matplotlib v3.11.1, https://matplotlib.org/ + Matplotlib v3.11.0, https://matplotlib.org/ @@ -20,42 +20,42 @@ - - - + - - + - - + + + + + + + + + + + + + + + + + + + + + + - - + + + - + - + - + - + - - + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - + - + - - + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + - - + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - + - - + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - + + + - + - + - + - + - - + + + - + - + - + + + + + + + + + + + + + + + + + + + - + - - - + + + - + - + - + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + - - + + - + - + - + - - + + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + - - - - - - - - - - - - - - - - - - - - + - + - - - - - - - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - + + - + - - - - + + + + - - - + + + + + + + + + + + + + @@ -1958,32 +1992,40 @@ z - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + - - + + - + - + - + - - - + + - + - + - + @@ -2138,18 +2167,18 @@ L 200.272344 387.740713 - - + + - + - + - + - - + + - + - + - + @@ -2217,46 +2246,18 @@ L 354.705312 387.740713 - - + + - + - + - - - - + @@ -2279,18 +2280,18 @@ z - - + + - + - + - + @@ -2315,8 +2316,8 @@ L 480.192031 387.740713 - - + + From c082015b656437afb1477d51c35e22d6df4e38b7 Mon Sep 17 00:00:00 2001 From: dalongbao Date: Tue, 4 Aug 2026 09:03:01 +0200 Subject: [PATCH 5/9] style: rename to agent-lightning --- .github/workflows/skills.yml | 6 ++--- skills/README.md | 26 +++++++++---------- .../.claude-plugin/plugin.json | 4 +-- skills/{agl => agent-lightning}/SKILL.md | 4 +-- ...ent-lightning-alf-world-accuracy-bars.svg} | 0 ...lightning-alf-world-accuracy-log-cost.svg} | 0 ...gent-lightning-officeqa-accuracy-bars.svg} | 0 ...-lightning-officeqa-accuracy-log-cost.svg} | 0 ...htning-spreadsheetbench-accuracy-bars.svg} | 0 ...ng-spreadsheetbench-accuracy-log-cost.svg} | 0 10 files changed, 20 insertions(+), 20 deletions(-) rename skills/{agl => agent-lightning}/.claude-plugin/plugin.json (83%) rename skills/{agl => agent-lightning}/SKILL.md (99%) rename skills/assets/{agl-alf-world-accuracy-bars.svg => agent-lightning-alf-world-accuracy-bars.svg} (100%) rename skills/assets/{agl-alf-world-accuracy-log-cost.svg => agent-lightning-alf-world-accuracy-log-cost.svg} (100%) rename skills/assets/{agl-officeqa-accuracy-bars.svg => agent-lightning-officeqa-accuracy-bars.svg} (100%) rename skills/assets/{agl-officeqa-accuracy-log-cost.svg => agent-lightning-officeqa-accuracy-log-cost.svg} (100%) rename skills/assets/{agl-spreadsheetbench-accuracy-bars.svg => agent-lightning-spreadsheetbench-accuracy-bars.svg} (100%) rename skills/assets/{agl-spreadsheetbench-accuracy-log-cost.svg => agent-lightning-spreadsheetbench-accuracy-log-cost.svg} (100%) diff --git a/.github/workflows/skills.yml b/.github/workflows/skills.yml index 356713d14..03a143a5d 100644 --- a/.github/workflows/skills.yml +++ b/.github/workflows/skills.yml @@ -1,4 +1,4 @@ -name: Validate AGL Skill +name: Validate Agent Lightning Skill permissions: contents: read @@ -25,9 +25,9 @@ jobs: - uses: actions/checkout@v6 - uses: astral-sh/setup-uv@v7 - name: Validate Agent Skills format - run: uvx --from 'skills-ref==0.1.1' agentskills validate skills/agl + run: uvx --from 'skills-ref==0.1.1' agentskills validate skills/agent-lightning - uses: actions/setup-node@v6 with: node-version: '22' - name: Validate Claude Code plugin - run: npx --yes @anthropic-ai/claude-code@2.1.218 plugin validate skills/agl + run: npx --yes @anthropic-ai/claude-code@2.1.218 plugin validate skills/agent-lightning diff --git a/skills/README.md b/skills/README.md index 8d72f6ed1..9f1e3cbff 100644 --- a/skills/README.md +++ b/skills/README.md @@ -2,7 +2,7 @@ Skills in the [Agent Skills](https://agentskills.io) format (`/SKILL.md`), installable into any compatible agent. -## agl +## Agent Lightning Turns your coding agent into an **agent optimizer**: given an editable agent and a benchmark to hillclimb on, it improves the agent's accuracy, cost, and latency through focused, individually-measured edits — keeping only what moves the frontier. It was measured against a no-skill control under a fair, leakage-free protocol. @@ -13,19 +13,19 @@ You provide the environment; the skill does the optimizing. Before invoking it, Install the skill from this repository for Claude Code, Codex, or GitHub Copilot: ```bash -gh skill install microsoft/agent-lightning agl --agent claude-code -gh skill install microsoft/agent-lightning agl --agent codex -gh skill install microsoft/agent-lightning agl --agent github-copilot +gh skill install microsoft/agent-lightning agent-lightning --agent claude-code +gh skill install microsoft/agent-lightning agent-lightning --agent codex +gh skill install microsoft/agent-lightning agent-lightning --agent github-copilot ``` Claude Code users can alternatively install the packaged plugin from the community marketplace: ```text /plugin marketplace add anthropics/claude-plugins-community -/plugin install agl@claude-community +/plugin install agent-lightning@claude-community ``` -The `skills/agl/` directory is both the canonical Agent Skills package and the Claude Code plugin root, so both publication paths use the same `SKILL.md` without a copied or symlinked wrapper. +The `skills/agent-lightning/` directory is both the canonical Agent Skills package and the Claude Code plugin root, so both publication paths use the same `SKILL.md` without a copied or symlinked wrapper. ### Results @@ -43,27 +43,27 @@ SkillOpt and the other non-agentic results are taken from the [SkillOpt paper](h | GEPA | 42.5 | 45.3 | 81.3 | | SkillOpt | 47.5 | 48.8 | 85.8 | | Agentic optimizer average, no skill | 62.9 | 54.1 | **95.3** | -| **Agentic optimizer average, AGL** | **66.7** | **54.5** | 94.7 | +| **Agentic optimizer average, Agent Lightning** | **66.7** | **54.5** | 94.7 | #### Accuracy by budget and optimizer Each benchmark is split into the \$5, \$10, and \$25 nominal budgets. Within each budget, every optimizer's with-skill and no-skill held-out finale accuracies are shown side by side. Solid bars use the skill; hatched bars are no-skill controls. -![SpreadsheetBench accuracy by budget and optimizer](assets/agl-spreadsheetbench-accuracy-bars.svg) +![SpreadsheetBench accuracy by budget and optimizer](assets/agent-lightning-spreadsheetbench-accuracy-bars.svg) -![OfficeQA accuracy by budget and optimizer](assets/agl-officeqa-accuracy-bars.svg) +![OfficeQA accuracy by budget and optimizer](assets/agent-lightning-officeqa-accuracy-bars.svg) -![ALFWorld accuracy by budget and optimizer](assets/agl-alf-world-accuracy-bars.svg) +![ALFWorld accuracy by budget and optimizer](assets/agent-lightning-alf-world-accuracy-bars.svg) #### Accuracy versus optimize-stage cost Each line connects the \$5, \$10, and \$25 nominal budgets in that order. The x-axis is average actual optimize-stage cost on a log scale, including optimizer calls and train/self-evaluation but excluding held-out finale generation. -![SpreadsheetBench accuracy versus optimize-stage cost](assets/agl-spreadsheetbench-accuracy-log-cost.svg) +![SpreadsheetBench accuracy versus optimize-stage cost](assets/agent-lightning-spreadsheetbench-accuracy-log-cost.svg) -![OfficeQA accuracy versus optimize-stage cost](assets/agl-officeqa-accuracy-log-cost.svg) +![OfficeQA accuracy versus optimize-stage cost](assets/agent-lightning-officeqa-accuracy-log-cost.svg) -![ALFWorld accuracy versus optimize-stage cost](assets/agl-alf-world-accuracy-log-cost.svg) +![ALFWorld accuracy versus optimize-stage cost](assets/agent-lightning-alf-world-accuracy-log-cost.svg) #### \$5 budget snapshot diff --git a/skills/agl/.claude-plugin/plugin.json b/skills/agent-lightning/.claude-plugin/plugin.json similarity index 83% rename from skills/agl/.claude-plugin/plugin.json rename to skills/agent-lightning/.claude-plugin/plugin.json index 0dd374140..a7e076663 100644 --- a/skills/agl/.claude-plugin/plugin.json +++ b/skills/agent-lightning/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { - "name": "agl", - "displayName": "AGL", + "name": "agent-lightning", + "displayName": "Agent Lightning", "description": "Turns coding agents into agent optimizers that improve accuracy, cost, latency, and reliability against a benchmark.", "author": { "name": "Agent Lightning Team" diff --git a/skills/agl/SKILL.md b/skills/agent-lightning/SKILL.md similarity index 99% rename from skills/agl/SKILL.md rename to skills/agent-lightning/SKILL.md index 8810264ca..128cc2155 100644 --- a/skills/agl/SKILL.md +++ b/skills/agent-lightning/SKILL.md @@ -1,12 +1,12 @@ --- -name: agl +name: agent-lightning description: >- Provides the action space, tradeoffs, and evaluation context for improving an editable AI agent against a benchmark while preserving its deployment contract. Use when optimizing agent accuracy, cost, latency, or reliability. --- -# Agent Optimization +# Agent Lightning Agent optimization is a search over interacting choices. The useful question is not which architecture is most sophisticated, but which change moves the requested diff --git a/skills/assets/agl-alf-world-accuracy-bars.svg b/skills/assets/agent-lightning-alf-world-accuracy-bars.svg similarity index 100% rename from skills/assets/agl-alf-world-accuracy-bars.svg rename to skills/assets/agent-lightning-alf-world-accuracy-bars.svg diff --git a/skills/assets/agl-alf-world-accuracy-log-cost.svg b/skills/assets/agent-lightning-alf-world-accuracy-log-cost.svg similarity index 100% rename from skills/assets/agl-alf-world-accuracy-log-cost.svg rename to skills/assets/agent-lightning-alf-world-accuracy-log-cost.svg diff --git a/skills/assets/agl-officeqa-accuracy-bars.svg b/skills/assets/agent-lightning-officeqa-accuracy-bars.svg similarity index 100% rename from skills/assets/agl-officeqa-accuracy-bars.svg rename to skills/assets/agent-lightning-officeqa-accuracy-bars.svg diff --git a/skills/assets/agl-officeqa-accuracy-log-cost.svg b/skills/assets/agent-lightning-officeqa-accuracy-log-cost.svg similarity index 100% rename from skills/assets/agl-officeqa-accuracy-log-cost.svg rename to skills/assets/agent-lightning-officeqa-accuracy-log-cost.svg diff --git a/skills/assets/agl-spreadsheetbench-accuracy-bars.svg b/skills/assets/agent-lightning-spreadsheetbench-accuracy-bars.svg similarity index 100% rename from skills/assets/agl-spreadsheetbench-accuracy-bars.svg rename to skills/assets/agent-lightning-spreadsheetbench-accuracy-bars.svg diff --git a/skills/assets/agl-spreadsheetbench-accuracy-log-cost.svg b/skills/assets/agent-lightning-spreadsheetbench-accuracy-log-cost.svg similarity index 100% rename from skills/assets/agl-spreadsheetbench-accuracy-log-cost.svg rename to skills/assets/agent-lightning-spreadsheetbench-accuracy-log-cost.svg From e61069da0c64e482a64841e32bd829206726ea02 Mon Sep 17 00:00:00 2001 From: dalongbao Date: Mon, 10 Aug 2026 10:36:48 +0200 Subject: [PATCH 6/9] docs: refresh skill results --- skills/README.md | 142 +- ...gent-lightning-alf-world-accuracy-bars.svg | 2460 ----------------- ...-lightning-alf-world-accuracy-log-cost.svg | 2305 --------------- ...agent-lightning-officeqa-accuracy-bars.svg | 1485 +++++----- ...ghtning-officeqa-accuracy-finale-cost.svg} | 1826 ++++++------ ...ghtning-spreadsheetbench-accuracy-bars.svg | 1485 +++++----- ...spreadsheetbench-accuracy-finale-cost.svg} | 1907 ++++++------- 7 files changed, 3355 insertions(+), 8255 deletions(-) delete mode 100644 skills/assets/agent-lightning-alf-world-accuracy-bars.svg delete mode 100644 skills/assets/agent-lightning-alf-world-accuracy-log-cost.svg rename skills/assets/{agent-lightning-officeqa-accuracy-log-cost.svg => agent-lightning-officeqa-accuracy-finale-cost.svg} (69%) rename skills/assets/{agent-lightning-spreadsheetbench-accuracy-log-cost.svg => agent-lightning-spreadsheetbench-accuracy-finale-cost.svg} (69%) diff --git a/skills/README.md b/skills/README.md index 9f1e3cbff..a56226a10 100644 --- a/skills/README.md +++ b/skills/README.md @@ -18,13 +18,6 @@ gh skill install microsoft/agent-lightning agent-lightning --agent codex gh skill install microsoft/agent-lightning agent-lightning --agent github-copilot ``` -Claude Code users can alternatively install the packaged plugin from the community marketplace: - -```text -/plugin marketplace add anthropics/claude-plugins-community -/plugin install agent-lightning@claude-community -``` - The `skills/agent-lightning/` directory is both the canonical Agent Skills package and the Claude Code plugin root, so both publication paths use the same `SKILL.md` without a copied or symlinked wrapper. ### Results @@ -33,112 +26,87 @@ The `skills/agent-lightning/` directory is both the canonical Agent Skills packa SkillOpt and the other non-agentic results are taken from the [SkillOpt paper](https://github.com/microsoft/SkillOpt) (Table 1); our agentic rows use the same splits and average all optimizers, budgets, and replicates. -| Method | Spreadsheet (%) | OfficeQA (%) | ALFWorld (%) | -| :--- | ---: | ---: | ---: | -| No skill | 36.1 | 22.1 | 73.1 | -| Human skill | 42.9 | 45.9 | 56.7 | -| LLM skill | 36.8 | 36.6 | 65.7 | -| Trace2Skill | 40.7 | 20.9 | 82.8 | -| TextGrad | 38.2 | 30.0 | 70.9 | -| GEPA | 42.5 | 45.3 | 81.3 | -| SkillOpt | 47.5 | 48.8 | 85.8 | -| Agentic optimizer average, no skill | 62.9 | 54.1 | **95.3** | -| **Agentic optimizer average, Agent Lightning** | **66.7** | **54.5** | 94.7 | +| Method | Spreadsheet (%) | OfficeQA (%) | +| :--- | ---: | ---: | +| No skill | 36.1 | 22.1 | +| Human skill | 42.9 | 45.9 | +| LLM skill | 36.8 | 36.6 | +| Trace2Skill | 40.7 | 20.9 | +| TextGrad | 38.2 | 30.0 | +| GEPA | 42.5 | 45.3 | +| SkillOpt | 47.5 | 48.8 | +| Agentic optimizer average, no skill | 62.9 | 54.1 | +| **Agentic optimizer average, Agent Lightning** | **66.7** | **54.5** | #### Accuracy by budget and optimizer -Each benchmark is split into the \$5, \$10, and \$25 nominal budgets. Within each budget, every optimizer's with-skill and no-skill held-out finale accuracies are shown side by side. Solid bars use the skill; hatched bars are no-skill controls. +Each benchmark is split into the \$5, \$10, and \$25 nominal budgets. Within each budget, every optimizer's with-skill and no-skill held-out finale accuracies are shown side by side. Solid bars use the skill; hatched bars are no-skill controls. The y-axis is cropped to 30–80%; labels report the exact values, including results above 80%. ![SpreadsheetBench accuracy by budget and optimizer](assets/agent-lightning-spreadsheetbench-accuracy-bars.svg) ![OfficeQA accuracy by budget and optimizer](assets/agent-lightning-officeqa-accuracy-bars.svg) -![ALFWorld accuracy by budget and optimizer](assets/agent-lightning-alf-world-accuracy-bars.svg) - -#### Accuracy versus optimize-stage cost - -Each line connects the \$5, \$10, and \$25 nominal budgets in that order. The x-axis is average actual optimize-stage cost on a log scale, including optimizer calls and train/self-evaluation but excluding held-out finale generation. +#### Accuracy versus finale cost -![SpreadsheetBench accuracy versus optimize-stage cost](assets/agent-lightning-spreadsheetbench-accuracy-log-cost.svg) +Each benchmark uses the \$5 nominal-budget group and shows seven points: the pristine baseline plus Claude Code, Codex, and Copilot with and without the skill. The x-axis is average finale cost; the y-axis is held-out finale accuracy. -![OfficeQA accuracy versus optimize-stage cost](assets/agent-lightning-officeqa-accuracy-log-cost.svg) +![SpreadsheetBench accuracy versus finale cost](assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg) -![ALFWorld accuracy versus optimize-stage cost](assets/agent-lightning-alf-world-accuracy-log-cost.svg) +![OfficeQA accuracy versus finale cost](assets/agent-lightning-officeqa-accuracy-finale-cost.svg) #### \$5 budget snapshot -| Benchmark (train/test) | Result | Accuracy (%) | Actual total cost | +| Benchmark (train/test) | Result | Accuracy (%) | Finale cost | | :--- | :--- | ---: | ---: | | SpreadsheetBench (120/280) | Before optimizer | 25.66 ± 2.65 | \$1.51 ± 0.04 | -| | Claude Code with skill | 63.79 ± 5.24 | **\$7.58 ± 0.90** | -| | Claude Code without skill | **68.23 ± 0.55** | \$7.59 ± 0.95 | -| | Codex with skill | **65.47 ± 4.59** | \$5.41 ± 0.17 | -| | Codex without skill | 41.49 ± 24.28 | **\$3.44 ± 1.78** | -| | Copilot with skill | **66.31 ± 2.05** | \$5.55 ± 0.69 | -| | Copilot without skill | 51.68 ± 20.82 | **\$3.40 ± 1.06** | +| | Claude Code with skill | 63.79 ± 5.24 | **\$2.45 ± 0.45** | +| | Claude Code without skill | **68.23 ± 0.55** | \$2.52 ± 0.93 | +| | Codex with skill | **65.47 ± 4.59** | **\$1.66 ± 0.19** | +| | Codex without skill | 41.49 ± 24.28 | \$1.73 ± 0.15 | +| | Copilot with skill | **66.31 ± 2.05** | **\$1.65 ± 0.13** | +| | Copilot without skill | 51.68 ± 20.82 | \$1.66 ± 0.09 | | OfficeQA (50/172) | Before optimizer | 31.78 ± 1.21 | \$2.78 ± 0.06 | -| | Claude Code with skill | 56.78 ± 3.87 | \$10.42 ± 1.59 | -| | Claude Code without skill | **59.69 ± 4.88** | **\$9.82 ± 1.72** | -| | Codex with skill | **49.81 ± 2.98** | **\$7.69 ± 0.21** | -| | Codex without skill | 49.61 ± 0.67 | \$8.03 ± 0.34 | -| | Copilot with skill | 51.55 ± 3.74 | **\$7.79 ± 0.91** | -| | Copilot without skill | **54.65 ± 2.01** | \$8.09 ± 1.28 | -| ALFWorld (3553/134) | Before optimizer | 58.71 ± 1.56 | \$5.78 ± 0.17 | -| | Claude Code with skill | **94.53 ± 4.56** | \$7.37 ± 1.81 | -| | Claude Code without skill | 64.68 ± 56.09 | **\$6.30 ± 2.03** | -| | Codex with skill | 94.78 ± 7.12 | \$3.82 ± 1.21 | -| | Codex without skill | **100.00 ± 0.00** | **\$1.31 ± 0.56** | -| | Copilot with skill | 92.79 ± 12.49 | **\$1.94 ± 0.80** | -| | Copilot without skill | **99.75 ± 0.43** | \$2.37 ± 0.63 | +| | Claude Code with skill | 56.78 ± 3.87 | \$5.35 ± 1.60 | +| | Claude Code without skill | **59.69 ± 4.88** | **\$4.69 ± 1.60** | +| | Codex with skill | **49.81 ± 2.98** | **\$3.38 ± 0.54** | +| | Codex without skill | 49.61 ± 0.67 | \$3.77 ± 0.22 | +| | Copilot with skill | 51.55 ± 3.74 | **\$3.69 ± 0.45** | +| | Copilot without skill | **54.65 ± 2.01** | \$4.20 ± 0.58 | #### \$10 budget snapshot -| Benchmark (train/test) | Result | Accuracy (%) | Actual total cost | +| Benchmark (train/test) | Result | Accuracy (%) | Finale cost | | :--- | :--- | ---: | ---: | | SpreadsheetBench (120/280) | Before optimizer | 25.66 ± 2.65 | \$1.51 ± 0.04 | -| | Claude Code with skill | 67.75 ± 2.40 | **\$11.86 ± 0.33** | -| | Claude Code without skill | **69.42 ± 4.32** | \$12.26 ± 0.25 | -| | Codex with skill | 64.63 ± 0.75 | \$5.58 ± 0.86 | -| | Codex without skill | **68.59 ± 1.16** | **\$4.98 ± 0.47** | -| | Copilot with skill | **69.30 ± 4.32** | \$10.29 ± 1.49 | -| | Copilot without skill | 64.39 ± 2.88 | **\$3.87 ± 0.34** | +| | Claude Code with skill | 67.75 ± 2.40 | **\$2.01 ± 0.26** | +| | Claude Code without skill | **69.42 ± 4.32** | \$2.11 ± 0.21 | +| | Codex with skill | 64.63 ± 0.75 | **\$1.59 ± 0.06** | +| | Codex without skill | **68.59 ± 1.16** | \$1.72 ± 0.08 | +| | Copilot with skill | **69.30 ± 4.32** | \$2.08 ± 0.74 | +| | Copilot without skill | 64.39 ± 2.88 | **\$1.63 ± 0.05** | | OfficeQA (50/172) | Before optimizer | 31.78 ± 1.21 | \$2.78 ± 0.06 | -| | Claude Code with skill | **62.60 ± 3.74** | \$16.16 ± 1.02 | -| | Claude Code without skill | 59.30 ± 1.74 | **\$14.55 ± 2.43** | -| | Codex with skill | **54.07 ± 1.16** | \$9.67 ± 1.69 | -| | Codex without skill | 50.00 ± 0.58 | **\$7.64 ± 0.58** | -| | Copilot with skill | **53.68 ± 0.89** | \$11.01 ± 1.14 | -| | Copilot without skill | 51.16 ± 4.07 | **\$8.05 ± 2.98** | -| ALFWorld (3553/134) | Before optimizer | 58.71 ± 1.56 | \$5.78 ± 0.17 | -| | Claude Code with skill | 73.13 ± 46.53 | **\$10.55 ± 1.64** | -| | Claude Code without skill | **100.00 ± 0.00** | \$11.08 ± 1.55 | -| | Codex with skill | 100.00 ± 0.00 | \$6.38 ± 3.14 | -| | Codex without skill | 100.00 ± 0.00 | **\$1.98 ± 1.41** | -| | Copilot with skill | 100.00 ± 0.00 | **\$1.52 ± 1.05** | -| | Copilot without skill | 100.00 ± 0.00 | \$2.39 ± 2.23 | +| | Claude Code with skill | **62.60 ± 3.74** | \$5.88 ± 0.82 | +| | Claude Code without skill | 59.30 ± 1.74 | **\$5.68 ± 0.74** | +| | Codex with skill | **54.07 ± 1.16** | \$3.95 ± 0.43 | +| | Codex without skill | 50.00 ± 0.58 | **\$3.77 ± 0.27** | +| | Copilot with skill | **53.68 ± 0.89** | \$3.46 ± 0.03 | +| | Copilot without skill | 51.16 ± 4.07 | **\$3.07 ± 1.46** | #### \$25 budget snapshot -| Benchmark (train/test) | Result | Accuracy (%) | Actual total cost | +| Benchmark (train/test) | Result | Accuracy (%) | Finale cost | | :--- | :--- | ---: | ---: | | SpreadsheetBench (120/280) | Before optimizer | 25.66 ± 2.65 | \$1.51 ± 0.04 | -| | Claude Code with skill | **71.70 ± 7.11** | \$32.46 ± 5.46 | -| | Claude Code without skill | 68.94 ± 1.98 | **\$30.60 ± 5.64** | -| | Codex with skill | 62.95 ± 2.52 | \$14.19 ± 2.29 | -| | Codex without skill | **65.23 ± 2.40** | **\$4.86 ± 0.60** | -| | Copilot with skill | **68.71 ± 3.12** | \$10.61 ± 6.64 | -| | Copilot without skill | 68.47 ± 3.60 | **\$5.54 ± 1.06** | +| | Claude Code with skill | **71.70 ± 7.11** | \$8.12 ± 5.26 | +| | Claude Code without skill | 68.94 ± 1.98 | **\$5.31 ± 5.71** | +| | Codex with skill | 62.95 ± 2.52 | **\$1.76 ± 0.24** | +| | Codex without skill | **65.23 ± 2.40** | \$1.84 ± 0.24 | +| | Copilot with skill | **68.71 ± 3.12** | \$3.48 ± 3.17 | +| | Copilot without skill | 68.47 ± 3.60 | **\$1.69 ± 0.01** | | OfficeQA (50/172) | Before optimizer | 31.78 ± 1.21 | \$2.78 ± 0.06 | -| | Claude Code with skill | **60.27 ± 4.70** | **\$27.05 ± 1.62** | -| | Claude Code without skill | 57.17 ± 2.98 | \$32.64 ± 12.01 | -| | Codex with skill | 50.78 ± 2.87 | \$14.09 ± 1.10 | -| | Codex without skill | **52.13 ± 0.34** | **\$8.81 ± 1.67** | -| | Copilot with skill | 51.16 ± 1.74 | \$19.29 ± 6.23 | -| | Copilot without skill | **53.10 ± 0.34** | **\$8.94 ± 0.33** | -| ALFWorld (3553/134) | Before optimizer | 58.71 ± 1.56 | \$5.78 ± 0.17 | -| | Claude Code with skill | 99.75 ± 0.43 | \$20.14 ± 8.47 | -| | Claude Code without skill | **100.00 ± 0.00** | **\$10.44 ± 1.61** | -| | Codex with skill | **97.76 ± 3.88** | \$8.13 ± 4.42 | -| | Codex without skill | 93.28 ± 9.76 | **\$3.64 ± 0.30** | -| | Copilot with skill | 99.75 ± 0.43 | \$2.79 ± 1.78 | -| | Copilot without skill | **100.00 ± 0.00** | **\$1.47 ± 0.43** | +| | Claude Code with skill | **60.27 ± 4.70** | **\$5.32 ± 0.62** | +| | Claude Code without skill | 57.17 ± 2.98 | \$11.81 ± 7.02 | +| | Codex with skill | 50.78 ± 2.87 | **\$3.28 ± 0.22** | +| | Codex without skill | **52.13 ± 0.34** | \$3.83 ± 0.41 | +| | Copilot with skill | 51.16 ± 1.74 | **\$4.13 ± 0.25** | +| | Copilot without skill | **53.10 ± 0.34** | \$4.38 ± 0.40 | diff --git a/skills/assets/agent-lightning-alf-world-accuracy-bars.svg b/skills/assets/agent-lightning-alf-world-accuracy-bars.svg deleted file mode 100644 index 643aebf6c..000000000 --- a/skills/assets/agent-lightning-alf-world-accuracy-bars.svg +++ /dev/null @@ -1,2460 +0,0 @@ - - - - - - - - image/svg+xml - - - Matplotlib v3.11.1, https://matplotlib.org/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/skills/assets/agent-lightning-alf-world-accuracy-log-cost.svg b/skills/assets/agent-lightning-alf-world-accuracy-log-cost.svg deleted file mode 100644 index d73c138e7..000000000 --- a/skills/assets/agent-lightning-alf-world-accuracy-log-cost.svg +++ /dev/null @@ -1,2305 +0,0 @@ - - - - - - - - image/svg+xml - - - Matplotlib v3.11.0, https://matplotlib.org/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/skills/assets/agent-lightning-officeqa-accuracy-bars.svg b/skills/assets/agent-lightning-officeqa-accuracy-bars.svg index 767740e0b..bdc17f1ed 100644 --- a/skills/assets/agent-lightning-officeqa-accuracy-bars.svg +++ b/skills/assets/agent-lightning-officeqa-accuracy-bars.svg @@ -1,7 +1,7 @@ - + @@ -9,7 +9,7 @@ image/svg+xml - Matplotlib v3.11.1, https://matplotlib.org/ + Matplotlib v3.11.0, https://matplotlib.org/ @@ -20,47 +20,28 @@ - - - - - - - - - - - - - - - - + - + - + - - - - - + - + - - - - - + - + - - - - - + - + @@ -368,7 +337,7 @@ z - + @@ -377,14 +346,10 @@ z - - - - - + - + @@ -394,14 +359,10 @@ z - - - - - + - + @@ -413,14 +374,10 @@ z - - - - - + - + @@ -429,7 +386,7 @@ z - + @@ -438,14 +395,10 @@ z - - - - - + - + @@ -455,14 +408,10 @@ z - - - - - + - + @@ -476,25 +425,57 @@ z - - + + - + - - + - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - @@ -705,20 +551,20 @@ z - - - + + + - + - + - + - + - - + + + - + - + - + - + - - + + + - + - + - + - + - - + + + - + - + - + - + - - + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + - - + + + + + + + + - - + + + + + + + + - - + + + + + + + + - - + + + + + + + + - - + + + + + + + + - + - + - - - - - - - - - - - - - - - - - - - + - + - - - - - - - - - - - - - - - - - - - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + - +" style="fill: url(#h851278c734); stroke: #64748b; stroke-linejoin: miter"/> - + - + - + + - + +" style="fill: #e76f00; stroke: #e76f00; stroke-width: 1.2; stroke-linecap: butt; stroke-linejoin: miter"/> - + +" style="fill: #059669; stroke: #059669; stroke-width: 1.2; stroke-linecap: butt; stroke-linejoin: miter"/> - + +" style="fill: #2563eb; stroke: #2563eb; stroke-width: 1.2; stroke-linecap: butt; stroke-linejoin: miter"/> - + +" style="fill: #64748b; stroke: #64748b; stroke-width: 1.2; stroke-linecap: butt; stroke-linejoin: miter"/> diff --git a/skills/assets/agent-lightning-officeqa-accuracy-log-cost.svg b/skills/assets/agent-lightning-officeqa-accuracy-finale-cost.svg similarity index 69% rename from skills/assets/agent-lightning-officeqa-accuracy-log-cost.svg rename to skills/assets/agent-lightning-officeqa-accuracy-finale-cost.svg index 2eeecdccd..70baf9f3e 100644 --- a/skills/assets/agent-lightning-officeqa-accuracy-log-cost.svg +++ b/skills/assets/agent-lightning-officeqa-accuracy-finale-cost.svg @@ -1,7 +1,7 @@ - + @@ -21,8 +21,8 @@ @@ -39,23 +39,23 @@ z - + - - + - - + + - + - - - - - - - - - - - - - - - - - - - - + + + - - - + + + - + - + - - - + + + - - - - + + + + + + + + + + + + + + + + + + + + - + - + - - + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - + + + + + + - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + - - + + - + - - + - - - + + + + - - + + - - + + - + - + - - - - + + + + - - + + - + - + - - - + + + - + - - + + - + - + - + - + - - - - - - - - - - - - - - - - - - + @@ -1019,53 +982,34 @@ Q 1469 3103 1208 2742 Q 947 2381 947 1747 z " transform="scale(0.015625)"/> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + - + + + - - - - - - - - - - - @@ -1848,71 +1638,337 @@ z - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - + - + - + - + @@ -2054,18 +2106,14 @@ L 136.352422 423.939431 - - + - + - + - + - - + - + - + - + @@ -2133,18 +2177,14 @@ L 313.944609 423.939431 - - + - + - + - + @@ -2167,18 +2207,14 @@ L 459.374297 408.93865 - - + - + - + - + @@ -2203,7 +2239,7 @@ L 459.374297 423.939431 - + diff --git a/skills/assets/agent-lightning-spreadsheetbench-accuracy-bars.svg b/skills/assets/agent-lightning-spreadsheetbench-accuracy-bars.svg index 0d93e2456..b4e4309ac 100644 --- a/skills/assets/agent-lightning-spreadsheetbench-accuracy-bars.svg +++ b/skills/assets/agent-lightning-spreadsheetbench-accuracy-bars.svg @@ -1,7 +1,7 @@ - + @@ -9,7 +9,7 @@ image/svg+xml - Matplotlib v3.11.1, https://matplotlib.org/ + Matplotlib v3.11.0, https://matplotlib.org/ @@ -20,47 +20,28 @@ - - - - - - - - - - - - - - - - + - + - + - - - - - + - + - - - - - + - + - - - - - + - + @@ -368,7 +337,7 @@ z - + @@ -377,14 +346,10 @@ z - - - - - + - + @@ -394,14 +359,10 @@ z - - - - - + - + @@ -413,14 +374,10 @@ z - - - - - + - + @@ -429,7 +386,7 @@ z - + @@ -438,14 +395,10 @@ z - - - - - + - + @@ -455,14 +408,10 @@ z - - - - - + - + @@ -476,25 +425,57 @@ z - - + + - + - - + - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - @@ -705,20 +551,20 @@ z - - - + + + - + - + - + - + - - + + + - + - + - + - + - - + + + - + - + - + - + - - + + + - + - + - + - + - - + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + - - + + + + + + + + - - + + + + + + + + - - + + + + + + + + - - + + + + + + + + - - + + + + + + + + - + - + - - - - - - - - - - - - - - - - - - - + - + - - - - - - - - - - - - - - - - - - - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + - +" style="fill: url(#h851278c734); stroke: #64748b; stroke-linejoin: miter"/> - + - + - + + - + +" style="fill: #e76f00; stroke: #e76f00; stroke-width: 1.2; stroke-linecap: butt; stroke-linejoin: miter"/> - + +" style="fill: #059669; stroke: #059669; stroke-width: 1.2; stroke-linecap: butt; stroke-linejoin: miter"/> - + +" style="fill: #2563eb; stroke: #2563eb; stroke-width: 1.2; stroke-linecap: butt; stroke-linejoin: miter"/> - + +" style="fill: #64748b; stroke: #64748b; stroke-width: 1.2; stroke-linecap: butt; stroke-linejoin: miter"/> diff --git a/skills/assets/agent-lightning-spreadsheetbench-accuracy-log-cost.svg b/skills/assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg similarity index 69% rename from skills/assets/agent-lightning-spreadsheetbench-accuracy-log-cost.svg rename to skills/assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg index 4b69cde03..052609e49 100644 --- a/skills/assets/agent-lightning-spreadsheetbench-accuracy-log-cost.svg +++ b/skills/assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg @@ -1,7 +1,7 @@ - + @@ -21,8 +21,8 @@ @@ -39,23 +39,23 @@ z - + - - + - - + + - + + - + + + - + - + - - + + - - + + + - + - + - - + + - - + + + - + - + - - + + - - - - + - + - + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + - - - - - - - - + + + - - - + + - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + - + +" clip-path="url(#p09c36dd3a0)" style="fill: none; stroke: #cbd5e1; stroke-opacity: 0.65; stroke-width: 0.7; stroke-linecap: square"/> - + - - + - - + + + - - + + - - + + - + - + - - - + + + - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + - - - - + + + + - + - + - + - - + + - - + - + - + - + - - + + - + - + - + - + - + - - - - - - - - - - - - - - - - - - + @@ -1165,52 +1095,33 @@ Q 1469 3103 1208 2742 Q 947 2381 947 1747 z " transform="scale(0.015625)"/> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + - + - - - - - - @@ -1961,71 +1766,337 @@ z - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - - + - + - + @@ -2167,18 +2234,14 @@ L 136.352422 423.939431 - - + - + - + - - + - + - + @@ -2246,18 +2305,14 @@ L 313.944609 423.939431 - - + - + - + @@ -2280,18 +2335,14 @@ L 459.374297 408.93865 - - + - + - + @@ -2316,7 +2367,7 @@ L 459.374297 423.939431 - + From 9824dabc7d060b7d3b112affe4a536f6ab7adb13 Mon Sep 17 00:00:00 2001 From: dalongbao Date: Mon, 10 Aug 2026 14:27:32 +0200 Subject: [PATCH 7/9] docs: revise skill result plots --- skills/README.md | 36 +- ...agent-lightning-officeqa-accuracy-bars.svg | 2407 ---------------- ...ightning-officeqa-accuracy-finale-cost.svg | 2246 --------------- ...tning-officeqa-correctness-finale-cost.svg | 1815 ++++++++++++ ...ning-officeqa-correctness-overall-cost.svg | 2067 ++++++++++++++ ...ghtning-spreadsheetbench-accuracy-bars.svg | 2431 ----------------- ...-spreadsheetbench-accuracy-finale-cost.svg | 2349 +++++++--------- ...spreadsheetbench-accuracy-overall-cost.svg | 2060 ++++++++++++++ 8 files changed, 6876 insertions(+), 8535 deletions(-) delete mode 100644 skills/assets/agent-lightning-officeqa-accuracy-bars.svg delete mode 100644 skills/assets/agent-lightning-officeqa-accuracy-finale-cost.svg create mode 100644 skills/assets/agent-lightning-officeqa-correctness-finale-cost.svg create mode 100644 skills/assets/agent-lightning-officeqa-correctness-overall-cost.svg delete mode 100644 skills/assets/agent-lightning-spreadsheetbench-accuracy-bars.svg create mode 100644 skills/assets/agent-lightning-spreadsheetbench-accuracy-overall-cost.svg diff --git a/skills/README.md b/skills/README.md index a56226a10..14125572b 100644 --- a/skills/README.md +++ b/skills/README.md @@ -26,7 +26,7 @@ The `skills/agent-lightning/` directory is both the canonical Agent Skills packa SkillOpt and the other non-agentic results are taken from the [SkillOpt paper](https://github.com/microsoft/SkillOpt) (Table 1); our agentic rows use the same splits and average all optimizers, budgets, and replicates. -| Method | Spreadsheet (%) | OfficeQA (%) | +| Method | SpreadsheetBench accuracy (%) | OfficeQA correctness (%) | | :--- | ---: | ---: | | No skill | 36.1 | 22.1 | | Human skill | 42.9 | 45.9 | @@ -38,34 +38,36 @@ SkillOpt and the other non-agentic results are taken from the [SkillOpt paper](h | Agentic optimizer average, no skill | 62.9 | 54.1 | | **Agentic optimizer average, Agent Lightning** | **66.7** | **54.5** | -#### Accuracy by budget and optimizer +#### Performance versus overall cost -Each benchmark is split into the \$5, \$10, and \$25 nominal budgets. Within each budget, every optimizer's with-skill and no-skill held-out finale accuracies are shown side by side. Solid bars use the skill; hatched bars are no-skill controls. The y-axis is cropped to 30–80%; labels report the exact values, including results above 80%. +Each benchmark includes the \$5, \$10, and \$25 nominal-budget groups. Every point is a held-out finale result: the x-axis is average overall cost per treatment cell on a log scale, and the y-axis is SpreadsheetBench accuracy or OfficeQA correctness. Color identifies the optimizer, filled markers use Agent Lightning, hollow markers are no-skill controls, and marker shape identifies the budget. Overall cost includes optimizer LLM calls, train/self-evaluation, and held-out finale deployment; it excludes the one shared pristine-baseline evaluation. -![SpreadsheetBench accuracy by budget and optimizer](assets/agent-lightning-spreadsheetbench-accuracy-bars.svg) +Claude Code uses Claude Opus 4.8; Codex and GitHub Copilot use GPT 5.6 Sol as their optimizer models. -![OfficeQA accuracy by budget and optimizer](assets/agent-lightning-officeqa-accuracy-bars.svg) +![SpreadsheetBench accuracy versus overall cost](assets/agent-lightning-spreadsheetbench-accuracy-overall-cost.svg) -#### Accuracy versus finale cost +![OfficeQA correctness versus overall cost](assets/agent-lightning-officeqa-correctness-overall-cost.svg) -Each benchmark uses the \$5 nominal-budget group and shows seven points: the pristine baseline plus Claude Code, Codex, and Copilot with and without the skill. The x-axis is average finale cost; the y-axis is held-out finale accuracy. +#### Performance versus finale cost + +The selected-budget views use the groups with the strongest aggregate skill-over-control lift: \$5 for SpreadsheetBench and \$10 for OfficeQA. Each shows seven points: the pristine baseline plus Claude Code, Codex, and GitHub Copilot with and without Agent Lightning. The x-axis is average finale cost; the y-axis is held-out SpreadsheetBench accuracy or OfficeQA correctness. ![SpreadsheetBench accuracy versus finale cost](assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg) -![OfficeQA accuracy versus finale cost](assets/agent-lightning-officeqa-accuracy-finale-cost.svg) +![OfficeQA correctness versus finale cost](assets/agent-lightning-officeqa-correctness-finale-cost.svg) #### \$5 budget snapshot -| Benchmark (train/test) | Result | Accuracy (%) | Finale cost | +| Benchmark metric (train/test) | Result | Score (%) | Finale cost | | :--- | :--- | ---: | ---: | -| SpreadsheetBench (120/280) | Before optimizer | 25.66 ± 2.65 | \$1.51 ± 0.04 | +| SpreadsheetBench accuracy (120/280) | Baseline | 25.66 ± 2.65 | \$1.51 ± 0.04 | | | Claude Code with skill | 63.79 ± 5.24 | **\$2.45 ± 0.45** | | | Claude Code without skill | **68.23 ± 0.55** | \$2.52 ± 0.93 | | | Codex with skill | **65.47 ± 4.59** | **\$1.66 ± 0.19** | | | Codex without skill | 41.49 ± 24.28 | \$1.73 ± 0.15 | | | Copilot with skill | **66.31 ± 2.05** | **\$1.65 ± 0.13** | | | Copilot without skill | 51.68 ± 20.82 | \$1.66 ± 0.09 | -| OfficeQA (50/172) | Before optimizer | 31.78 ± 1.21 | \$2.78 ± 0.06 | +| OfficeQA correctness (50/172) | Baseline | 31.78 ± 1.21 | \$2.78 ± 0.06 | | | Claude Code with skill | 56.78 ± 3.87 | \$5.35 ± 1.60 | | | Claude Code without skill | **59.69 ± 4.88** | **\$4.69 ± 1.60** | | | Codex with skill | **49.81 ± 2.98** | **\$3.38 ± 0.54** | @@ -75,16 +77,16 @@ Each benchmark uses the \$5 nominal-budget group and shows seven points: the pri #### \$10 budget snapshot -| Benchmark (train/test) | Result | Accuracy (%) | Finale cost | +| Benchmark metric (train/test) | Result | Score (%) | Finale cost | | :--- | :--- | ---: | ---: | -| SpreadsheetBench (120/280) | Before optimizer | 25.66 ± 2.65 | \$1.51 ± 0.04 | +| SpreadsheetBench accuracy (120/280) | Baseline | 25.66 ± 2.65 | \$1.51 ± 0.04 | | | Claude Code with skill | 67.75 ± 2.40 | **\$2.01 ± 0.26** | | | Claude Code without skill | **69.42 ± 4.32** | \$2.11 ± 0.21 | | | Codex with skill | 64.63 ± 0.75 | **\$1.59 ± 0.06** | | | Codex without skill | **68.59 ± 1.16** | \$1.72 ± 0.08 | | | Copilot with skill | **69.30 ± 4.32** | \$2.08 ± 0.74 | | | Copilot without skill | 64.39 ± 2.88 | **\$1.63 ± 0.05** | -| OfficeQA (50/172) | Before optimizer | 31.78 ± 1.21 | \$2.78 ± 0.06 | +| OfficeQA correctness (50/172) | Baseline | 31.78 ± 1.21 | \$2.78 ± 0.06 | | | Claude Code with skill | **62.60 ± 3.74** | \$5.88 ± 0.82 | | | Claude Code without skill | 59.30 ± 1.74 | **\$5.68 ± 0.74** | | | Codex with skill | **54.07 ± 1.16** | \$3.95 ± 0.43 | @@ -94,16 +96,16 @@ Each benchmark uses the \$5 nominal-budget group and shows seven points: the pri #### \$25 budget snapshot -| Benchmark (train/test) | Result | Accuracy (%) | Finale cost | +| Benchmark metric (train/test) | Result | Score (%) | Finale cost | | :--- | :--- | ---: | ---: | -| SpreadsheetBench (120/280) | Before optimizer | 25.66 ± 2.65 | \$1.51 ± 0.04 | +| SpreadsheetBench accuracy (120/280) | Baseline | 25.66 ± 2.65 | \$1.51 ± 0.04 | | | Claude Code with skill | **71.70 ± 7.11** | \$8.12 ± 5.26 | | | Claude Code without skill | 68.94 ± 1.98 | **\$5.31 ± 5.71** | | | Codex with skill | 62.95 ± 2.52 | **\$1.76 ± 0.24** | | | Codex without skill | **65.23 ± 2.40** | \$1.84 ± 0.24 | | | Copilot with skill | **68.71 ± 3.12** | \$3.48 ± 3.17 | | | Copilot without skill | 68.47 ± 3.60 | **\$1.69 ± 0.01** | -| OfficeQA (50/172) | Before optimizer | 31.78 ± 1.21 | \$2.78 ± 0.06 | +| OfficeQA correctness (50/172) | Baseline | 31.78 ± 1.21 | \$2.78 ± 0.06 | | | Claude Code with skill | **60.27 ± 4.70** | **\$5.32 ± 0.62** | | | Claude Code without skill | 57.17 ± 2.98 | \$11.81 ± 7.02 | | | Codex with skill | 50.78 ± 2.87 | **\$3.28 ± 0.22** | diff --git a/skills/assets/agent-lightning-officeqa-accuracy-bars.svg b/skills/assets/agent-lightning-officeqa-accuracy-bars.svg deleted file mode 100644 index bdc17f1ed..000000000 --- a/skills/assets/agent-lightning-officeqa-accuracy-bars.svg +++ /dev/null @@ -1,2407 +0,0 @@ - - - - - - - - image/svg+xml - - - Matplotlib v3.11.0, https://matplotlib.org/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/skills/assets/agent-lightning-officeqa-accuracy-finale-cost.svg b/skills/assets/agent-lightning-officeqa-accuracy-finale-cost.svg deleted file mode 100644 index 70baf9f3e..000000000 --- a/skills/assets/agent-lightning-officeqa-accuracy-finale-cost.svg +++ /dev/null @@ -1,2246 +0,0 @@ - - - - - - - - image/svg+xml - - - Matplotlib v3.11.0, https://matplotlib.org/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/skills/assets/agent-lightning-officeqa-correctness-finale-cost.svg b/skills/assets/agent-lightning-officeqa-correctness-finale-cost.svg new file mode 100644 index 000000000..00d649fac --- /dev/null +++ b/skills/assets/agent-lightning-officeqa-correctness-finale-cost.svg @@ -0,0 +1,1815 @@ + + + + + + + + 2026-08-10T14:26:35.502628 + image/svg+xml + + + Matplotlib v3.8.4, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/skills/assets/agent-lightning-officeqa-correctness-overall-cost.svg b/skills/assets/agent-lightning-officeqa-correctness-overall-cost.svg new file mode 100644 index 000000000..672e81120 --- /dev/null +++ b/skills/assets/agent-lightning-officeqa-correctness-overall-cost.svg @@ -0,0 +1,2067 @@ + + + + + + + + 2026-08-10T14:26:35.371972 + image/svg+xml + + + Matplotlib v3.8.4, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/skills/assets/agent-lightning-spreadsheetbench-accuracy-bars.svg b/skills/assets/agent-lightning-spreadsheetbench-accuracy-bars.svg deleted file mode 100644 index b4e4309ac..000000000 --- a/skills/assets/agent-lightning-spreadsheetbench-accuracy-bars.svg +++ /dev/null @@ -1,2431 +0,0 @@ - - - - - - - - image/svg+xml - - - Matplotlib v3.11.0, https://matplotlib.org/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/skills/assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg b/skills/assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg index 052609e49..338f8c7ef 100644 --- a/skills/assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg +++ b/skills/assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg @@ -1,15 +1,16 @@ - + + 2026-08-10T14:26:35.449119 image/svg+xml - Matplotlib v3.11.0, https://matplotlib.org/ + Matplotlib v3.8.4, https://matplotlib.org/ @@ -20,44 +21,44 @@ - - - + - - + - + - - - - - - - - + + + + - + - + - + - - - - - + + + + - + - + - + - - - - - + + + + - + - + - + - - - + + - + - + - - - - - + + + + + - + - + - - - - - + + + + + - + - + - - - - - + + + + + - + - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + - + - - + - - - - - - - - - - - - - - - - - - - - - - - - + + - + - - - + + - - - + + + - + - + - - - - - - + + + + + - - - + + + - + - + - - - + + + - - - - + + - - - + + + - + - + - - - - - - + + + + + - - - + + + - + - + - - - + + + - - - - + + - - - + + + - - - - - + - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + - + - + - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + - - + + - +" style="stroke: #059669; stroke-width: 1.9"/> - - + + - - - + + - +" style="stroke: #2563eb; stroke-width: 1.9"/> - - + + - - - + + - - - + + + + - - - - - - - - - - - - - - - - + + + - - + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - +" style="stroke: #ffffff; stroke-width: 1.2"/> + + + - - - - + + + + + + + + + - - - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + + + + - - - - - + + + + - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + + + + - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + - - - - + + + + + + + - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + diff --git a/skills/assets/agent-lightning-spreadsheetbench-accuracy-overall-cost.svg b/skills/assets/agent-lightning-spreadsheetbench-accuracy-overall-cost.svg new file mode 100644 index 000000000..9ab19b476 --- /dev/null +++ b/skills/assets/agent-lightning-spreadsheetbench-accuracy-overall-cost.svg @@ -0,0 +1,2060 @@ + + + + + + + + 2026-08-10T14:26:35.289991 + image/svg+xml + + + Matplotlib v3.8.4, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + From e3185801b7d28403f34b62491eb52c4e001a0c5c Mon Sep 17 00:00:00 2001 From: dalongbao Date: Tue, 11 Aug 2026 19:04:56 +0200 Subject: [PATCH 8/9] docs: refresh benchmark result plots --- skills/README.md | 51 +- ...lightning-alfworld-success-finale-cost.svg | 1725 ++++++++++++++++ ...ightning-alfworld-success-overall-cost.svg | 1675 +++++++++++++++ ...ning-officeqa-correctness-overall-cost.svg | 1826 ++++++---------- ...spreadsheetbench-accuracy-overall-cost.svg | 1829 +++++++---------- 5 files changed, 4865 insertions(+), 2241 deletions(-) create mode 100644 skills/assets/agent-lightning-alfworld-success-finale-cost.svg create mode 100644 skills/assets/agent-lightning-alfworld-success-overall-cost.svg diff --git a/skills/README.md b/skills/README.md index 14125572b..72b91254d 100644 --- a/skills/README.md +++ b/skills/README.md @@ -26,21 +26,21 @@ The `skills/agent-lightning/` directory is both the canonical Agent Skills packa SkillOpt and the other non-agentic results are taken from the [SkillOpt paper](https://github.com/microsoft/SkillOpt) (Table 1); our agentic rows use the same splits and average all optimizers, budgets, and replicates. -| Method | SpreadsheetBench accuracy (%) | OfficeQA correctness (%) | -| :--- | ---: | ---: | -| No skill | 36.1 | 22.1 | -| Human skill | 42.9 | 45.9 | -| LLM skill | 36.8 | 36.6 | -| Trace2Skill | 40.7 | 20.9 | -| TextGrad | 38.2 | 30.0 | -| GEPA | 42.5 | 45.3 | -| SkillOpt | 47.5 | 48.8 | -| Agentic optimizer average, no skill | 62.9 | 54.1 | -| **Agentic optimizer average, Agent Lightning** | **66.7** | **54.5** | +| Method | SpreadsheetBench accuracy (%) | OfficeQA correctness (%) | ALFWorld success (%) | +| :--- | ---: | ---: | ---: | +| No skill | 36.1 | 22.1 | 73.1 | +| Human skill | 42.9 | 45.9 | 56.7 | +| LLM skill | 36.8 | 36.6 | 65.7 | +| Trace2Skill | 40.7 | 20.9 | 82.8 | +| TextGrad | 38.2 | 30.0 | 70.9 | +| GEPA | 42.5 | 45.3 | 81.3 | +| SkillOpt | 47.5 | 48.8 | 85.8 | +| Agentic optimizer average, no skill | 62.9 | 54.1 | 88.6 | +| **Agentic optimizer average, Agent Lightning** | **66.7** | **54.5** | **94.9** | #### Performance versus overall cost -Each benchmark includes the \$5, \$10, and \$25 nominal-budget groups. Every point is a held-out finale result: the x-axis is average overall cost per treatment cell on a log scale, and the y-axis is SpreadsheetBench accuracy or OfficeQA correctness. Color identifies the optimizer, filled markers use Agent Lightning, hollow markers are no-skill controls, and marker shape identifies the budget. Overall cost includes optimizer LLM calls, train/self-evaluation, and held-out finale deployment; it excludes the one shared pristine-baseline evaluation. +Each benchmark includes the \$5, \$10, and \$25 nominal-budget groups with three runs per treatment cell. Every point is one held-out finale result: the x-axis is that run's overall cost on a log scale, and the y-axis is SpreadsheetBench accuracy, OfficeQA correctness, or ALFWorld success. Color and shape identify the optimizer; filled markers use Agent Lightning and hollow markers are no-skill controls. Budget is not encoded in the legend. Overall cost includes optimizer LLM calls, train/self-evaluation, and held-out finale deployment; it excludes the pristine-baseline evaluations. Claude Code uses Claude Opus 4.8; Codex and GitHub Copilot use GPT 5.6 Sol as their optimizer models. @@ -48,14 +48,18 @@ Claude Code uses Claude Opus 4.8; Codex and GitHub Copilot use GPT 5.6 Sol as th ![OfficeQA correctness versus overall cost](assets/agent-lightning-officeqa-correctness-overall-cost.svg) +![ALFWorld success versus overall cost](assets/agent-lightning-alfworld-success-overall-cost.svg) + #### Performance versus finale cost -The selected-budget views use the groups with the strongest aggregate skill-over-control lift: \$5 for SpreadsheetBench and \$10 for OfficeQA. Each shows seven points: the pristine baseline plus Claude Code, Codex, and GitHub Copilot with and without Agent Lightning. The x-axis is average finale cost; the y-axis is held-out SpreadsheetBench accuracy or OfficeQA correctness. +The selected-budget views use the groups with the strongest aggregate skill-over-control lift: \$5 for SpreadsheetBench and \$10 for OfficeQA and ALFWorld. Harness/treatment points average three runs. The x-axis is average finale cost; the y-axis is held-out SpreadsheetBench accuracy, OfficeQA correctness, or ALFWorld success. Finale cost measures LLM gateway spend, so an ALFWorld deterministic controller can have exactly \$0 finale cost while still executing and scoring real environment steps. SpreadsheetBench and OfficeQA show the pristine baseline as a point; the corrected ALFWorld records do not include baseline deployment cost, so its measured success is shown as a horizontal reference instead of assigning it an x-coordinate. ![SpreadsheetBench accuracy versus finale cost](assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg) ![OfficeQA correctness versus finale cost](assets/agent-lightning-officeqa-correctness-finale-cost.svg) +![ALFWorld success versus finale cost](assets/agent-lightning-alfworld-success-finale-cost.svg) + #### \$5 budget snapshot | Benchmark metric (train/test) | Result | Score (%) | Finale cost | @@ -74,6 +78,13 @@ The selected-budget views use the groups with the strongest aggregate skill-over | | Codex without skill | 49.61 ± 0.67 | \$3.77 ± 0.22 | | | Copilot with skill | 51.55 ± 3.74 | **\$3.69 ± 0.45** | | | Copilot without skill | **54.65 ± 2.01** | \$4.20 ± 0.58 | +| ALFWorld success (3553/134) | Baseline | 56.97 ± 0.43 | — | +| | Claude Code with skill | **95.02 ± 1.14** | \$3.83 ± 0.78 | +| | Claude Code without skill | 93.53 ± 4.11 | **\$3.21 ± 0.38** | +| | Codex with skill | 87.31 ± 21.97 | \$1.69 ± 2.92 | +| | Codex without skill | **96.52 ± 3.02** | **\$0.97 ± 1.68** | +| | Copilot with skill | **99.75 ± 0.43** | \$0.01 ± 0.02 | +| | Copilot without skill | 95.52 ± 7.12 | **\$0.00 ± 0.00** | #### \$10 budget snapshot @@ -93,6 +104,13 @@ The selected-budget views use the groups with the strongest aggregate skill-over | | Codex without skill | 50.00 ± 0.58 | **\$3.77 ± 0.27** | | | Copilot with skill | **53.68 ± 0.89** | \$3.46 ± 0.03 | | | Copilot without skill | 51.16 ± 4.07 | **\$3.07 ± 1.46** | +| ALFWorld success (3553/134) | Baseline | 56.97 ± 0.43 | — | +| | Claude Code with skill | 93.78 ± 0.43 | **\$3.62 ± 0.51** | +| | Claude Code without skill | **94.28 ± 3.02** | \$3.67 ± 0.90 | +| | Codex with skill | **99.00 ± 0.86** | \$0.75 ± 1.28 | +| | Codex without skill | 89.55 ± 18.10 | **\$0.00 ± 0.00** | +| | Copilot with skill | **100.00 ± 0.00** | \$0.00 ± 0.00 | +| | Copilot without skill | 66.92 ± 57.30 | \$0.00 ± 0.00 | #### \$25 budget snapshot @@ -112,3 +130,10 @@ The selected-budget views use the groups with the strongest aggregate skill-over | | Codex without skill | **52.13 ± 0.34** | \$3.83 ± 0.41 | | | Copilot with skill | 51.16 ± 1.74 | **\$4.13 ± 0.25** | | | Copilot without skill | **53.10 ± 0.34** | \$4.38 ± 0.40 | +| ALFWorld success (3553/134) | Baseline | 56.97 ± 0.43 | — | +| | Claude Code with skill | 82.59 ± 21.84 | \$4.28 ± 1.27 | +| | Claude Code without skill | **94.78 ± 4.48** | **\$3.02 ± 0.55** | +| | Codex with skill | **96.77 ± 5.60** | \$1.08 ± 1.87 | +| | Codex without skill | 66.67 ± 57.74 | **\$0.01 ± 0.02** | +| | Copilot with skill | 100.00 ± 0.00 | \$0.12 ± 0.20 | +| | Copilot without skill | 100.00 ± 0.00 | **\$0.00 ± 0.00** | diff --git a/skills/assets/agent-lightning-alfworld-success-finale-cost.svg b/skills/assets/agent-lightning-alfworld-success-finale-cost.svg new file mode 100644 index 000000000..2d2b5012f --- /dev/null +++ b/skills/assets/agent-lightning-alfworld-success-finale-cost.svg @@ -0,0 +1,1725 @@ + + + + + + + + 2026-08-11T18:08:59.980184 + image/svg+xml + + + Matplotlib v3.11.1, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/skills/assets/agent-lightning-alfworld-success-overall-cost.svg b/skills/assets/agent-lightning-alfworld-success-overall-cost.svg new file mode 100644 index 000000000..06b1a6242 --- /dev/null +++ b/skills/assets/agent-lightning-alfworld-success-overall-cost.svg @@ -0,0 +1,1675 @@ + + + + + + + + 2026-08-11T18:08:59.923766 + image/svg+xml + + + Matplotlib v3.11.1, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/skills/assets/agent-lightning-officeqa-correctness-overall-cost.svg b/skills/assets/agent-lightning-officeqa-correctness-overall-cost.svg index 672e81120..acc80f826 100644 --- a/skills/assets/agent-lightning-officeqa-correctness-overall-cost.svg +++ b/skills/assets/agent-lightning-officeqa-correctness-overall-cost.svg @@ -6,11 +6,11 @@ - 2026-08-10T14:26:35.371972 + 2026-08-11T18:08:59.868856 image/svg+xml - Matplotlib v3.8.4, https://matplotlib.org/ + Matplotlib v3.11.1, https://matplotlib.org/ @@ -30,109 +30,80 @@ z - - + - - + - - + + - - - - + - + - + - - + + - - - - - + + - + - + - - + + - - - - + + - + - + - - + + - - - - + + - - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - + - + - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - + - - + - - - - - - - - + + + + - - + + - + - + - - - - - - + + + - - + + - + - + - - - - + + + + - - + + - + - + - - - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - + + - + - + - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - + + + + + - + - + - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - - + - - + + + + + + + + + + - + - - + + + + + + + + + + - +" style="stroke: #2563eb; stroke-opacity: 0.95; stroke-width: 1.7"/> - - + + + + + + + + + + + + + + + + - + - - + + + + + + + + + + - +" style="stroke: #059669; stroke-opacity: 0.95; stroke-width: 1.7"/> - - + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +" style="stroke: #2563eb; stroke-opacity: 0.95; stroke-width: 1.7"/> - - + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + - - - + + + + - - - - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + - - - - + + - - - +" style="stroke: #059669; stroke-linejoin: miter"/> + + + + - - - - - - + + + + - + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - - - - - + + - - - + + + + - - - - - - + + + + - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - - - + + - - - + + + + - - - - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + - - + + diff --git a/skills/assets/agent-lightning-spreadsheetbench-accuracy-overall-cost.svg b/skills/assets/agent-lightning-spreadsheetbench-accuracy-overall-cost.svg index 9ab19b476..d5cb27dd3 100644 --- a/skills/assets/agent-lightning-spreadsheetbench-accuracy-overall-cost.svg +++ b/skills/assets/agent-lightning-spreadsheetbench-accuracy-overall-cost.svg @@ -6,11 +6,11 @@ - 2026-08-10T14:26:35.289991 + 2026-08-11T18:08:59.753083 image/svg+xml - Matplotlib v3.8.4, https://matplotlib.org/ + Matplotlib v3.11.1, https://matplotlib.org/ @@ -30,78 +30,79 @@ z - - + - - + - - + + - - + + + + + + + + + + + + + + + + + - - + - - - + + + - + - + - - - + + + - - - + - - - + + + - + - + - - - + + + - - - - - + + - - - + + + - + - + - - - + + + + + + + + + + + + + + + + + + + + - - - - + + - - + + - - + - - + + - + - - + + - + - - + + - + - - + + - + - - + + - + - - - + + + - - - - - - - - - - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +M 3481 434 +Q 3481 -459 3084 -895 +Q 2688 -1331 1869 -1331 +Q 1566 -1331 1297 -1286 +Q 1028 -1241 775 -1147 +L 775 -588 +Q 1028 -725 1275 -790 +Q 1522 -856 1778 -856 +Q 2344 -856 2625 -561 +Q 2906 -266 2906 331 +L 2906 616 +Q 2728 306 2450 153 +Q 2172 0 1784 0 +Q 1141 0 747 490 +Q 353 981 353 1791 +Q 353 2603 747 3093 +Q 1141 3584 1784 3584 +Q 2172 3584 2450 3431 +Q 2728 3278 2906 2969 +L 2906 3500 +L 3481 3500 +L 3481 434 +z +" transform="scale(0.015625)"/> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + - - + - - - - - - - - + + + + + - - + + - + - + - - - - - + + + + + - - + + - + - + - - - - - + + + + + - - + + - + - + - - - - - + + + + + - - + + - + - + - + - + - - - + + - - + + - + - + - - - - - - - - - - - - - - - - - - + - + - - - + + + + + + + + + + + + + + + + + + + + + + - + - + - + - - - - - - - - - - - - - - + + + + + + + + + + + + - - - - - - - + - - + + + + + + + + + + - + - - + + + + + + + + + + - +" style="stroke: #2563eb; stroke-opacity: 0.95; stroke-width: 1.7"/> - - + + + + + + + + + + + + + + + + - + - - + + + + + + + + + + - +" style="stroke: #059669; stroke-opacity: 0.95; stroke-width: 1.7"/> - - + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - +" style="stroke: #2563eb; stroke-opacity: 0.95; stroke-width: 1.7"/> - - + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + - - - + + + + - - - - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + - - - - + + - - - +" style="stroke: #059669; stroke-linejoin: miter"/> + + + + - - - - - - + + + + - + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + - - - - - - + + - - - + + + + - - - - - - + + + + - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - - - + + - - - + + + + - - - - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + - - + + From ae40840f40d4b7de947d0eaa093873c855da8c9e Mon Sep 17 00:00:00 2001 From: dalongbao Date: Tue, 11 Aug 2026 21:34:59 +0200 Subject: [PATCH 9/9] docs: show finale result replicates --- skills/README.md | 2 +- ...lightning-alfworld-success-finale-cost.svg | 858 ++++---- ...tning-officeqa-correctness-finale-cost.svg | 1705 ++++++++------- ...-spreadsheetbench-accuracy-finale-cost.svg | 1880 ++++++++--------- 4 files changed, 2078 insertions(+), 2367 deletions(-) diff --git a/skills/README.md b/skills/README.md index 72b91254d..f831ffe07 100644 --- a/skills/README.md +++ b/skills/README.md @@ -52,7 +52,7 @@ Claude Code uses Claude Opus 4.8; Codex and GitHub Copilot use GPT 5.6 Sol as th #### Performance versus finale cost -The selected-budget views use the groups with the strongest aggregate skill-over-control lift: \$5 for SpreadsheetBench and \$10 for OfficeQA and ALFWorld. Harness/treatment points average three runs. The x-axis is average finale cost; the y-axis is held-out SpreadsheetBench accuracy, OfficeQA correctness, or ALFWorld success. Finale cost measures LLM gateway spend, so an ALFWorld deterministic controller can have exactly \$0 finale cost while still executing and scoring real environment steps. SpreadsheetBench and OfficeQA show the pristine baseline as a point; the corrected ALFWorld records do not include baseline deployment cost, so its measured success is shown as a horizontal reference instead of assigning it an x-coordinate. +The selected-budget views use the groups with the strongest aggregate skill-over-control lift: \$5 for SpreadsheetBench and \$10 for OfficeQA and ALFWorld. Every harness/treatment point is one of three runs; the x-axis is that run's finale cost, and the y-axis is held-out SpreadsheetBench accuracy, OfficeQA correctness, or ALFWorld success. Finale cost measures LLM gateway spend, so an ALFWorld deterministic controller can have exactly \$0 finale cost while still executing and scoring real environment steps; coincident zero-cost ALFWorld results are offset slightly along the x-axis so each replicate remains visible. SpreadsheetBench and OfficeQA show their aggregate pristine-baseline results as single reference points. The corrected ALFWorld records do not include baseline deployment cost, so its aggregate measured success is shown as a horizontal reference instead of assigning it an x-coordinate. ![SpreadsheetBench accuracy versus finale cost](assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg) diff --git a/skills/assets/agent-lightning-alfworld-success-finale-cost.svg b/skills/assets/agent-lightning-alfworld-success-finale-cost.svg index 2d2b5012f..ae8f1efde 100644 --- a/skills/assets/agent-lightning-alfworld-success-finale-cost.svg +++ b/skills/assets/agent-lightning-alfworld-success-finale-cost.svg @@ -6,11 +6,11 @@ - 2026-08-11T18:08:59.980184 + 2026-08-11T20:51:34.507463 image/svg+xml - Matplotlib v3.11.1, https://matplotlib.org/ + Matplotlib v3.11.0, https://matplotlib.org/ @@ -30,8 +30,8 @@ z - - + - - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - + + + - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - + + - + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - - - - - - + + + - + - + - + - + - - - - - - - - - - - - - - - - - - - - + - + - + - + @@ -929,7 +771,7 @@ L 651.7548 66.684706 - + - - - - - - - + - + - - + + + + - + - - + + + + - +" style="stroke: #2563eb; stroke-opacity: 0.95; stroke-width: 1.7"/> - - + + + + + + + + + + - + - - + + + + - +" style="stroke: #059669; stroke-opacity: 0.95; stroke-width: 1.7"/> - - + + + + - +" style="stroke: #2563eb; stroke-opacity: 0.95; stroke-width: 1.7"/> - - + + + + - + - @@ -1180,6 +1021,22 @@ z + - + + + + @@ -1380,7 +1314,7 @@ z - - + - + - - + - + - - + - + - - - - + @@ -1597,7 +1529,7 @@ L 248.842162 464.263037 - - + - + - - + - + - + + diff --git a/skills/assets/agent-lightning-officeqa-correctness-finale-cost.svg b/skills/assets/agent-lightning-officeqa-correctness-finale-cost.svg index 00d649fac..e248fa869 100644 --- a/skills/assets/agent-lightning-officeqa-correctness-finale-cost.svg +++ b/skills/assets/agent-lightning-officeqa-correctness-finale-cost.svg @@ -6,11 +6,11 @@ - 2026-08-10T14:26:35.502628 + 2026-08-11T20:51:34.456714 image/svg+xml - Matplotlib v3.8.4, https://matplotlib.org/ + Matplotlib v3.11.0, https://matplotlib.org/ @@ -30,78 +30,79 @@ z - - + - - + - - + + - - + + + + + + + + + + + + + + + + + - - + - - - + + + - + - + - - - + + + - - - + - - - + + + - + - + - - - + + + - - - + - - - + + + - + - + - - - + + + - - - + - - - - - + + + + + + + + + + + + + - + + + + + + + + - - - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - + + - + - - + - + - + - - - + + - - + + - + - + - + - - - + + + - - + + - + - + - + - - - + + + - - + + - + - + - + - - - + + + - - + + - + - + - + - - - + + + - - + + - + - + - + - - - + + + - - + + - + - + - + - - - + + + - - + + - + - + - + - - - + + + - + - + - - + - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - - + - - + + + + - + - - + + + + - + - - + + + + + + + + + + - + - - + + + + - + - - + + + + - + + + + + + + + + + + - - + + - + - + - - + - + - + - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +M 3481 434 +Q 3481 -459 3084 -895 +Q 2688 -1331 1869 -1331 +Q 1566 -1331 1297 -1286 +Q 1028 -1241 775 -1147 +L 775 -588 +Q 1028 -725 1275 -790 +Q 1522 -856 1778 -856 +Q 2344 -856 2625 -561 +Q 2906 -266 2906 331 +L 2906 616 +Q 2728 306 2450 153 +Q 2172 0 1784 0 +Q 1141 0 747 490 +Q 353 981 353 1791 +Q 353 2603 747 3093 +Q 1141 3584 1784 3584 +Q 2172 3584 2450 3431 +Q 2728 3278 2906 2969 +L 2906 3500 +L 3481 3500 +L 3481 434 +z +" transform="scale(0.015625)"/> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + - - - + + + + - - - - + + + + + + + + + + + + + + + + + - - - +" style="stroke: #059669; stroke-linejoin: miter"/> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + - - - - - - + + + + - + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - - - - - + + - - - + + + + - - - - - - + + + + - - - - - - - - - + + + + + + + + + + - - - - + + - - - + + + + - - - - - - + + + + - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - - - + + - - - + + + + - - - - - - + + + + - - - - - - - - - - + + + + + + + + + + - - + + diff --git a/skills/assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg b/skills/assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg index 338f8c7ef..002fb5ae4 100644 --- a/skills/assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg +++ b/skills/assets/agent-lightning-spreadsheetbench-accuracy-finale-cost.svg @@ -6,11 +6,11 @@ - 2026-08-10T14:26:35.449119 + 2026-08-11T20:51:34.373170 image/svg+xml - Matplotlib v3.8.4, https://matplotlib.org/ + Matplotlib v3.11.0, https://matplotlib.org/ @@ -30,78 +30,35 @@ z - - + - - + - - + + - - - - - - - - + + + - + - + - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - + - - + + + - - - - - - - - - - - - - - - - - - - - - - + + + - + - + - - - - - - - + + + + + + - - - + + + - + - - - - - - - - - - + - - - - - - - - - - - + + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - + + - + - - + - + - - - - - - - + + + - - + + - + - + - + - - - + + + + + + - - + + - + - + - + - - - - - - + + + - - + + - + - + - + - - - + + + + + + - - + + - + - + - + - + - - - + + - + - + - + - + - - - - - - - - - - - - - + + + + + + + + + + + + - - - - - - - + - - + + + + - + - - + + + + - + - - + + + + + + + + + + - + - - + + + + - + - - + + + + - + + + + + + + + + + + - - + + - + - + - - - - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +M 3481 434 +Q 3481 -459 3084 -895 +Q 2688 -1331 1869 -1331 +Q 1566 -1331 1297 -1286 +Q 1028 -1241 775 -1147 +L 775 -588 +Q 1028 -725 1275 -790 +Q 1522 -856 1778 -856 +Q 2344 -856 2625 -561 +Q 2906 -266 2906 331 +L 2906 616 +Q 2728 306 2450 153 +Q 2172 0 1784 0 +Q 1141 0 747 490 +Q 353 981 353 1791 +Q 353 2603 747 3093 +Q 1141 3584 1784 3584 +Q 2172 3584 2450 3431 +Q 2728 3278 2906 2969 +L 2906 3500 +L 3481 3500 +L 3481 434 +z +" transform="scale(0.015625)"/> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + - - - + + + + - - - - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + - - - - + + - - - +" style="stroke: #059669; stroke-linejoin: miter"/> + + + + - - - - - - + + + + - + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + - - - - - - + + - - - + + + + - - - - - - - - - - - - + + + + + + + + + + + + - - - - + + - - - + + + + - - - - - - + + + + - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - - - + + - - - + + + + - - - - - - + + + + - - - - - - - - - - + + + + + + + + + + - - + +