feat: agl-skill - #534
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds an installable Agent Skills package (agl-optimizer) to the repository, along with a skills catalog README, so compatible agents can discover and load the skill via the Agent Skills format.
Changes:
- Added
skills/agl-optimizer/SKILL.mddefining theagl-optimizerskill (metadata + optimization guidance, action space, and boundaries). - Added
skills/README.mddocumenting the skill, installation options, and published evaluation context/results.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| skills/README.md | Introduces the skills catalog, installation instructions, and benchmark/result context for agl-optimizer. |
| skills/agl-optimizer/SKILL.md | Adds the Agent Skills spec file (frontmatter + content) for the agl-optimizer skill. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -0,0 +1,90 @@ | |||
| --- | |||
| name: agl-optimizer | |||
There was a problem hiding this comment.
should be agl
|
|
||
| Our evaluation uses a different protocol: Claude Code, Codex, and Copilot act as agentic optimizers that can edit the whole agent. Results are held-out test means pooled across \$5, \$10, and \$25 budgets (`n = 9`) and are shown as mean ± standard deviation. Higher is better; bold marks the better mean within each row. | ||
|
|
||
| | Benchmark (train/test) | Optimizer | agl-skill v9.7 (%) | No-skill control (%) | Change (pp) | |
There was a problem hiding this comment.
don't put 9.7 here
| Our evaluation uses a different protocol: Claude Code, Codex, and Copilot act as agentic optimizers that can edit the whole agent. Results are held-out test means pooled across \$5, \$10, and \$25 budgets (`n = 9`) and are shown as mean ± standard deviation. Higher is better; bold marks the better mean within each row. | ||
|
|
||
| | Benchmark (train/test) | Optimizer | agl-skill v9.7 (%) | No-skill control (%) | Change (pp) | | ||
| | :--- | :--- | ---: | ---: | ---: | |
| | ALFWorld (3553/134) | Claude Code | **89.1** ± 26.4 | 88.2 ± 33.1 | +0.9 | | ||
| | ALFWorld (3553/134) | Codex | 97.5 ± 4.7 | **97.8** ± 5.9 | −0.3 | | ||
| | ALFWorld (3553/134) | Copilot | 97.5 ± 7.2 | **99.9** ± 0.3 | −2.4 | | ||
|
|
There was a problem hiding this comment.
columns:
accuracy cost
rows:
before optimizer
claude code w/ skill
codex w/ skill
copilot w/ skill
| | ALFWorld (3553/134) | Codex | 97.5 ± 4.7 | **97.8** ± 5.9 | −0.3 | | ||
| | ALFWorld (3553/134) | Copilot | 97.5 ± 7.2 | **99.9** ± 0.3 | −2.4 | | ||
|
|
||
| Because the two tables use different protocols and data splits, compare results within a table, not across tables. Full per-budget scores, cost breakdowns, and caveats are available in [RESULTS_V9_7.md](https://github.com/agent-lightning/agl-skill/blob/verifier-v8/RESULTS_V9_7.md). |
There was a problem hiding this comment.
don't put links here. it won't be publicly accessible
|
@microsoft-github-policy-service agree company="Microsoft" |
| Claude Code users can alternatively install the packaged plugin from the community marketplace: | ||
|
|
||
| ```text | ||
| /plugin marketplace add anthropics/claude-plugins-community |
There was a problem hiding this comment.
have you actuallly published to this anthropics/claude-plugins-community?
There was a problem hiding this comment.
no, publishing requires the plugin to be on an open repo first, should i remove it for now then?
There was a problem hiding this comment.
we can hold until it's merged.
|
|
||
| #### \$5 budget snapshot | ||
|
|
||
| | Benchmark (train/test) | Result | Accuracy (%) | Actual total cost | |
There was a problem hiding this comment.
3 figures. 7 points in each figure. x-axis: target agent cost. y-axis: accuracy
There was a problem hiding this comment.
pick one group of 3 figures out of 3 groups of 3 (either $5, or $10 or $25)
No description provided.