Skip to content

feat: add Hermes Agent as a first-class backend and transcript source#114

Open
fgscaglioni wants to merge 6 commits into
microsoft:mainfrom
fgscaglioni:feat/hermes-agent-backend
Open

feat: add Hermes Agent as a first-class backend and transcript source#114
fgscaglioni wants to merge 6 commits into
microsoft:mainfrom
fgscaglioni:feat/hermes-agent-backend

Conversation

@fgscaglioni

Copy link
Copy Markdown

This adds full support for Nous Research's Hermes Agent across all three SkillOpt layers — core model backend, sleep-cycle backend, and transcript harvesting.

Core model layer (skillopt/model/):

  • Register the Hermes CLI as a chat backend (hermes_chat) alongside the existing Claude, Codex, Qwen, and MiniMax backends.
  • All chat entry points (optimizer, target, messages, deployment) route through the Hermes adapter.

Sleep-cycle backend (skillopt_sleep/backend.py):

  • Add HermesBackend extending CliBackend, driving hermes --profile chat -Q -q for attempt / judge / reflect calls.
  • CLI output is filtered to strip notices, warnings, and traceback debris so the pipeline sees only the model response.
  • Registered under get_backend("hermes") with aliases hermes_chat and hermes_cli.

Transcript harvesting (skillopt_sleep/harvest_hermes.py):

  • New harvest source reads Hermes session data from ~/.hermes/state.db (SQLite), building SessionDigest objects from sessions + messages.
  • Skips engine-internal sessions (temp dirs matching skillopt_sleep_*) so the mine step only sees real user sessions.
  • Supports the standard harvest contract: scope, since, limit, project.

Config additions (skillopt_sleep/config.py):

  • memory_filename — controls the project memory file name. Defaults to "CLAUDE.md" for backward compatibility. Set to "AGENTS.md" for Codex and Hermes.
  • hermes_home — path to Hermes state directory, defaults to ~/.hermes, overridable via $HERMES_HOME.

Tests:

  • 7 new tests in TestHermesBackendCli covering backend registration, command construction, error capture, output filtering, and env vars.

Backward compatible — defaults unchanged, all existing backends intact.

This adds full support for Nous Research's Hermes Agent across all three
SkillOpt layers — core model backend, sleep-cycle backend, and transcript
harvesting.

Core model layer (skillopt/model/):
- Register the Hermes CLI as a chat backend (hermes_chat) alongside the
  existing Claude, Codex, Qwen, and MiniMax backends.
- All chat entry points (optimizer, target, messages, deployment) route
  through the Hermes adapter.

Sleep-cycle backend (skillopt_sleep/backend.py):
- Add HermesBackend extending CliBackend, driving hermes --profile
  <name> chat -Q -q for attempt / judge / reflect calls.
- CLI output is filtered to strip notices, warnings, and traceback
  debris so the pipeline sees only the model response.
- Registered under get_backend("hermes") with aliases hermes_chat and
  hermes_cli.

Transcript harvesting (skillopt_sleep/harvest_hermes.py):
- New harvest source reads Hermes session data from ~/.hermes/state.db
  (SQLite), building SessionDigest objects from sessions + messages.
- Skips engine-internal sessions (temp dirs matching skillopt_sleep_*)
  so the mine step only sees real user sessions.
- Supports the standard harvest contract: scope, since, limit, project.

Config additions (skillopt_sleep/config.py):
- memory_filename — controls the project memory file name.
  Defaults to "CLAUDE.md" for backward compatibility.
  Set to "AGENTS.md" for Codex and Hermes.
- hermes_home — path to Hermes state directory, defaults to
  ~/.hermes, overridable via $HERMES_HOME.

Tests:
- 7 new tests in TestHermesBackendCli covering backend registration,
  command construction, error capture, output filtering, and env vars.

Backward compatible — defaults unchanged, all existing backends intact.
@fgscaglioni

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

@Yif-Yang

Copy link
Copy Markdown
Contributor

Thanks for the substantial Hermes integration. Supporting another agent/backend can improve SkillOpt, and we would like to preserve the work under your authorship, but the current PR is too broad and has several correctness issues that block merge.

Confirmed issues:

  • The branch conflicts with current backend registry files. Please rebase onto the latest main.
  • set_*_backend accepts hermes_chat, but is_optimizer_chat_backend() and is_target_chat_backend() still return false.
  • The public deployment setters do not dispatch to Hermes, and the Hermes setter changes a deployment variable while calls use profile constants.
  • Hermes and Claude share the same global token tracker, then both summaries are aggregated, so usage is double-counted.
  • Message APIs discard tools, tool_choice, retries, and return_message; return_message=True still returns a string. This violates the backend contract for tool-using environments.
  • Sleep CLI launch exceptions are swallowed without last_call_error, recreating the silent-zero failure fixed for other backends.
  • Output beginning with Exception can be a legitimate answer but is filtered to empty.
  • Harvesting filters legitimate /tmp projects, caps limit=0 at 200, drops tool-role messages, and never produces feedback signals.
  • New tests cover only a mocked Sleep CLI subset, not core routing, setters, token accounting, message/tool contracts, database schema compatibility, or secret handling.

We recommend splitting this into three reviewable PRs: core model backend, Sleep CLI backend, and transcript harvester. Each should include contract tests and one opt-in real Hermes smoke test. Please ensure backend invocations disable unintended tools/plugins/MCP when processing harvested untrusted text.

We are not rejecting the feature direction; we are asking for a rebase and a narrower, test-backed design so it can be merged safely while keeping your contribution attribution.

@Yif-Yang Yif-Yang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes based on the backend-contract, routing, token-accounting, error-surfacing, harvesting, and test gaps detailed in the maintainer comment above. Please rebase and narrow or split the implementation before re-review.

…at_backend

- Add hermes -> hermes_chat alias so normalize_backend_name() resolves correctly
- Add hermes_chat default model in _BACKEND_DEFAULT_MODELS
- Include hermes_chat in is_optimizer_chat_backend() and is_target_chat_backend()
- Add hermes_chat to validation sets in set_optimizer_backend / set_target_backend

Addresses Yif-Yang review: is_*_chat_backend now returns true for hermes_chat.
…er setters, message API contracts

- Use local _hermes_tracker (TokenTracker) instead of global tracker from
  common.py to prevent token double-counting with Claude/OpenAI
- Make target/optimizer profiles mutable via set_target_deployment /
  set_optimizer_deployment; setters now actually control which profile
  chat_target / chat_optimizer use
- chat_target_messages / chat_optimizer_messages now respect:
  * retries parameter (not hardcoded 3)
  * return_message=True returns CompatAssistantMessage, not str
  * tools/tool_choice serialized into the prompt preamble
- _call_hermes retries on non-zero exit with exponential backoff
- subprocess.run wraps exception in RuntimeError instead of silent empty string

Addresses Yif-Yang review: token accounting, deployment setters, message/tool contract.
26 tests covering:
- Backend alias resolution and default models
- set_target/optimizer_backend acceptance (ValueError boundaries)
- is_*_chat_backend recognition
- Routing dispatch (chat_target, chat_optimizer, chat_with_deployment)
- Token tracker isolation (no double-count with Claude)
- Message API contracts:
  * retries parameter respected
  * return_message=True returns CompatAssistantMessage
  * tools/tool_choice serialized into prompt
  * chat_messages_with_deployment uses custom profile
- Deployment setters (set_target/optimizer_deployment)
- Edge cases (env vars, non-zero exit, routing when not selected)
- One opt-in smoke test (pytest.mark.slow)
@fgscaglioni fgscaglioni force-pushed the feat/hermes-agent-backend branch from ffaa1f1 to 524b864 Compare July 14, 2026 12:32
@fgscaglioni

Copy link
Copy Markdown
Author

Thanks for the thorough review, Yif-Yang. I've split the implementation into three branches as suggested and will open separate PRs shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants