From 8886cfb045f4f6e071be64977f88f5ecebf235bf Mon Sep 17 00:00:00 2001 From: Ajay Thorve Date: Wed, 13 May 2026 10:02:11 -0700 Subject: [PATCH] docs: add coding-agent CLI guidance Signed-off-by: Ajay Thorve --- README.md | 5 +- crates/cli/README.md | 59 +++++-- docs/getting-started/cli.md | 159 ++++++++++++++++++ docs/getting-started/configuration.md | 55 ++++++ docs/getting-started/installation.md | 11 +- docs/getting-started/quick-start.md | 1 + docs/getting-started/rust.md | 8 +- docs/index.md | 1 + .../coding-agent-claude-code.md | 23 ++- .../coding-agent-codex.md | 21 ++- .../coding-agent-cursor.md | 21 ++- .../coding-agent-gateway.md | 146 +++++++++++++--- .../coding-agent-hermes.md | 25 ++- 13 files changed, 458 insertions(+), 77 deletions(-) create mode 100644 docs/getting-started/cli.md diff --git a/README.md b/README.md index dc298bc5..527906b6 100644 --- a/README.md +++ b/README.md @@ -114,7 +114,8 @@ For source builds, testing, and contribution workflow, see [CONTRIBUTING.md](CON End-user documentation lives at [nvidia.github.io/NeMo-Flow](https://nvidia.github.io/NeMo-Flow/). -The primary documentation track covers Rust, Python, and Node.js. +The primary documentation track covers Rust, Python, and Node.js. Experimental +CLI gateway docs are available for local coding-agent observability. The Go, WebAssembly, and raw FFI surfaces are currently experimental and remain source-first under `go/nemo_flow`, `crates/wasm`, and `crates/ffi`. @@ -128,6 +129,7 @@ The table below summarizes the support level for each binding surface. | Python | ✅ Fully Supported | Fully documented with Quick Start and Guides | | Node.js | ✅ Fully Supported | Fully documented with Quick Start and Guides | | Rust | ✅ Fully Supported | Fully documented with Quick Start and Guides | +| Coding-Agent CLI | 🚧 Experimental | Binary crate under `crates/cli`. | | Go | 🚧 Experimental | Source-first under `go/nemo_flow`. | | WebAssembly | 🚧 Experimental | Source-first under `crates/wasm`. | | FFI | 🚧 Experimental | Source-first under `crates/ffi`. | @@ -162,6 +164,7 @@ The following table summarizes maintained third-party integrations and whether e | [opencode](third_party/README-opencode.md) | 🚧 Patch | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | | [OpenClaw](integrations/openclaw/README.md) | `nemo-flow-openclaw` package, `nemo-flow` plugin ID | ✅ Yes | ❌ No | ❌ No | ❌ No | | [Hermes Agent](third_party/README-hermes-agent.md) | 🚧 Patch | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | +| [Coding-Agent CLI](docs/integrate-frameworks/coding-agent-gateway.md) for Claude Code, Codex, Cursor, and Hermes | 🚧 Experimental `nemo-flow-cli` binary | ✅ Yes | 🚧 LLM gateway only | 🚧 LLM gateway only | 🚧 LLM gateway only | Patch-based integrations offer experimental support. Our roadmap includes switching over to first-party plugins and packages where upstream extension points allow it. diff --git a/crates/cli/README.md b/crates/cli/README.md index 5ab4e588..0cbd8b1e 100644 --- a/crates/cli/README.md +++ b/crates/cli/README.md @@ -12,14 +12,15 @@ SPDX-License-Identifier: Apache-2.0 [![npm wasm](https://img.shields.io/npm/v/nemo-flow-wasm?label=nemo-flow-wasm&color=CC3534&logo=npm)](https://www.npmjs.com/package/nemo-flow-wasm) [![Crates.io](https://img.shields.io/crates/v/nemo-flow?label=nemo-flow&color=B7410E&logo=rust)](https://crates.io/crates/nemo-flow) [![Crates.io](https://img.shields.io/crates/v/nemo-flow-adaptive?label=nemo-flow-adaptive&color=B7410E&logo=rust)](https://crates.io/crates/nemo-flow-adaptive) +[![Crates.io](https://img.shields.io/crates/v/nemo-flow-cli?label=nemo-flow-cli&color=B7410E&logo=rust)](https://crates.io/crates/nemo-flow-cli) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/NVIDIA/NeMo-Flow) # nemo-flow-cli -`nemo-flow-cli` is the coding-agent gateway CLI for NeMo Flow observability. -It installs the `nemo-flow` binary, which can configure supported coding-agent -hooks, run agents through an ephemeral gateway, and diagnose local agent and -exporter readiness. +`nemo-flow-cli` is the experimental coding-agent gateway CLI for NeMo Flow +observability. It installs the `nemo-flow` binary, which can configure +supported coding-agent hooks, run agents through an ephemeral gateway, and +diagnose local agent and exporter readiness. The CLI is a Rust package in this repository, but most users should interact with the installed `nemo-flow` command rather than link against the crate. @@ -50,10 +51,10 @@ with the installed `nemo-flow` command rather than link against the crate. ## Installation -Install the CLI from a repository checkout: +Install the CLI: ```bash -cargo install --path crates/cli +cargo install nemo-flow-cli ``` That command installs the binary as: @@ -96,12 +97,47 @@ Use `run --dry-run` to inspect resolved config without spawning the agent: nemo-flow run --agent codex --dry-run ``` +## Benchmark And Eval Runs + +Use `nemo-flow run` for non-interactive benchmark tasks so each run gets an +ephemeral gateway and isolated observability artifacts. Prefer absolute +artifact directories when the benchmark harness runs from a different directory +than the task repository. + +This example uses Codex CLI because it has a non-interactive `exec` mode. Use +the equivalent non-interactive command for another coding agent when one is +available. + +```bash +RUN_ID="$(date +%Y%m%d_%H%M%S)" +ART="/tmp/nemo-flow-eval/$RUN_ID" +mkdir -p "$ART/atif" "$ART/atof" + +nemo-flow run \ + --atif-dir "$ART/atif" \ + --atof-dir "$ART/atof" \ + -- codex \ + --ask-for-approval never \ + exec \ + --cd /path/to/benchmark/repo \ + --sandbox workspace-write \ + --color never \ + "Run the benchmark task and report the result." + +find "$ART" -maxdepth 3 -type f -print +``` + +The expected output includes one ATIF JSON file and one ATOF JSONL file for +the run. Remote or cloud-hosted tasks are outside the local gateway's LLM +capture boundary unless their model traffic reaches the local gateway. + ## Configuration -Project config lives at `./.nemo-flow/config.toml`; user config lives at -`~/.config/nemo-flow/config.toml` or `$XDG_CONFIG_HOME/nemo-flow/config.toml`. -The project layer overrides system config, and the user layer overrides the -project layer. +System config lives at `/etc/nemo-flow/config.toml`, project config lives at +the nearest `./.nemo-flow/config.toml`, and user config lives at +`$XDG_CONFIG_HOME/nemo-flow/config.toml` or `~/.config/nemo-flow/config.toml`. +The CLI loads those layers in that order, then applies `NEMO_FLOW_*` +environment variables and command flags. Exporter config uses nested per-backend tables: @@ -118,6 +154,9 @@ filename_template = "{session_id}.jsonl" endpoint = "http://localhost:6006/v1/traces" ``` +Use `plugin.toml` in the same system, project, or user config locations when +the gateway should activate process-level plugin config. + ## Documentation NeMo Flow Documentation: https://nvidia.github.io/NeMo-Flow diff --git a/docs/getting-started/cli.md b/docs/getting-started/cli.md new file mode 100644 index 00000000..ebd54915 --- /dev/null +++ b/docs/getting-started/cli.md @@ -0,0 +1,159 @@ + + +# CLI Gateway Quick Start + +The `nemo-flow` CLI gateway is experimental. Use it when you want to try local +observability for coding agents without changing the agent implementation. The +CLI can configure supported agent hooks, start a short-lived gateway around one +agent run, run a long-running gateway, and diagnose local setup. + +## Install The CLI + +Install the CLI: + +```bash +cargo install nemo-flow-cli +``` + +That command installs the executable as `nemo-flow`. + +## Use The Wizard + +Run the first-time setup wizard: + +```bash +nemo-flow +``` + +Bare `nemo-flow` launches setup when no config exists. After config exists, the +same command runs `nemo-flow doctor` as a quick health check. + +The wizard detects supported agent binaries on `PATH`, asks which agents to +observe, asks where to save config, and enables local ATIF export by default. +Re-run setup later with: + +```bash +nemo-flow config +nemo-flow config codex +nemo-flow config hermes +``` + +Use the scoped form when you only want to refresh one agent's settings. Use +`nemo-flow config --reset` to remove the project config, or +`nemo-flow config --reset` to remove one project-local agent block. + +## Run An Observed Agent + +Use the agent shortcuts for day-to-day runs: + +```bash +nemo-flow codex +nemo-flow claude -- "summarize this repository" +nemo-flow cursor +nemo-flow hermes +``` + +The shortcut starts an ephemeral gateway on `127.0.0.1`, injects the agent's +hook and provider routing settings for that process, and stops the gateway when +the agent exits. + +Inspect readiness and detected agents with: + +```bash +nemo-flow doctor +nemo-flow doctor codex +nemo-flow agents +``` + +## Config Layers + +When no explicit `--config` path is passed, the CLI loads config from lowest to +highest precedence: + +1. System: `/etc/nemo-flow/config.toml` +2. Project: nearest `.nemo-flow/config.toml`, walking up from the current + directory +3. User: `$XDG_CONFIG_HOME/nemo-flow/config.toml`, or + `~/.config/nemo-flow/config.toml` +4. Environment variables such as `NEMO_FLOW_ATIF_DIR` and + `NEMO_FLOW_OPENAI_BASE_URL` +5. CLI flags on the current command + +Passing `--config path/to/config.toml` disables automatic system, project, and +user discovery for that command. + +Use this minimal project config as a starting point: + +```toml +[exporters.atif] +dir = ".nemo-flow/atif" + +[exporters.atof] +dir = ".nemo-flow/atof" +mode = "append" +filename_template = "{session_id}.jsonl" + +[agents.codex] +command = "codex" + +[agents.claude] +command = "claude" + +[agents.cursor] +command = "cursor-agent" +patch_restore_hooks = true + +[agents.hermes] +command = "hermes" +``` + +Add provider and OpenInference settings only when you need to override defaults: + +```toml +[upstream] +openai_base_url = "https://api.openai.com" +anthropic_base_url = "https://api.anthropic.com" + +[exporters.openinference] +endpoint = "http://localhost:6006/v1/traces" +``` + +Plugin config can live in one of three places: `--plugin-config` JSON, +`[plugins].config` inside `config.toml`, or a sibling/discovered `plugin.toml`. +Do not mix these sources in one effective CLI invocation. + +## Advanced Commands + +Use `run` when you need deterministic, non-interactive behavior: + +```bash +nemo-flow run --agent codex --dry-run --print +nemo-flow run --agent codex -- exec "fix the failing test" +``` + +Use daemon mode when an agent or IDE is launched separately and can point its +provider traffic or hooks at a long-running local gateway: + +```bash +nemo-flow --bind 127.0.0.1:4040 +``` + +Install shell completions when your package manager did not do that for you: + +```bash +nemo-flow completions --install +``` + +`nemo-flow hook-forward` is the internal hook entrypoint generated by setup and +transparent runs. You normally do not call it directly except for smoke tests. + +## Next Steps + +- [Advanced Guide: Coding-Agent Gateway](../integrate-frameworks/coding-agent-gateway.md) +- [Claude Code Gateway Guide](../integrate-frameworks/coding-agent-claude-code.md) +- [Codex Gateway Guide](../integrate-frameworks/coding-agent-codex.md) +- [Cursor Gateway Guide](../integrate-frameworks/coding-agent-cursor.md) +- [Hermes Gateway Guide](../integrate-frameworks/coding-agent-hermes.md) diff --git a/docs/getting-started/configuration.md b/docs/getting-started/configuration.md index 7f62ce0c..bb3adb35 100644 --- a/docs/getting-started/configuration.md +++ b/docs/getting-started/configuration.md @@ -50,3 +50,58 @@ deployment manifests. ## Adaptive Setup Adaptive optimization is enabled through the adaptive plugin component and binding helper APIs. See [Configure Adaptive Optimization](../use-adaptive-optimization/configure.md). + +## CLI Gateway Setup + +The `nemo-flow` CLI gateway is the exception to the "no global config file" +rule because it runs outside an application process. Use it when you want local +observability for coding-agent sessions such as Claude Code, Codex, Cursor, or +Hermes. + +Run the setup wizard: + +```bash +nemo-flow +``` + +The wizard writes project config at `.nemo-flow/config.toml`, user config at +`$XDG_CONFIG_HOME/nemo-flow/config.toml` or `~/.config/nemo-flow/config.toml`, +or both. System config can be supplied at `/etc/nemo-flow/config.toml` for +shared machines and CI images. + +When no explicit `--config` path is passed, CLI config precedence is: + +1. Built-in defaults +2. `/etc/nemo-flow/config.toml` +3. Nearest project `.nemo-flow/config.toml`, walking up from the current + directory +4. `$XDG_CONFIG_HOME/nemo-flow/config.toml`, or + `~/.config/nemo-flow/config.toml` +5. `NEMO_FLOW_*` environment variables +6. Current command flags + +Use the current exporter shape in new config files: + +```toml +[exporters.atif] +dir = ".nemo-flow/atif" + +[exporters.atof] +dir = ".nemo-flow/atof" +mode = "append" +filename_template = "{session_id}.jsonl" + +[exporters.openinference] +endpoint = "http://localhost:6006/v1/traces" + +[agents.codex] +command = "codex" +``` + +For CLI gateway plugin activation, use exactly one source per invocation: +`--plugin-config` JSON, `[plugins].config` inside `config.toml`, or +`plugin.toml` in the discovered system, project, or user config locations. + +See [CLI Gateway Quick Start](cli.md) for wizard commands and +[Advanced Guide: Coding-Agent Gateway](../integrate-frameworks/coding-agent-gateway.md) +for daemon mode, transparent runs, hook forwarding, and per-agent guides. diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index b1273f34..f6ed05e2 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -47,9 +47,9 @@ nemo-flow-adaptive = "0.1.*" - `nemo-flow` provides the core runtime APIs for scopes, middleware, subscribers, plugins, tool calls, and LLM calls. - `nemo-flow-adaptive` provides adaptive runtime primitives and Redis-backed learning components when you want adaptive optimization behavior in Rust. -- `nemo-flow-cli` is a published binary crate for coding-agent hook and LLM - gateway observability. Install it with `cargo install nemo-flow-cli` when - you need the `nemo-flow` executable. +- `nemo-flow-cli` is a binary crate for coding-agent hook and LLM gateway + observability. Install it with `cargo install nemo-flow-cli` when you need + the `nemo-flow` executable. ## Install from Source @@ -103,11 +103,10 @@ nemo-flow = { path = "../NeMo-Flow/crates/core" } nemo-flow-adaptive = { path = "../NeMo-Flow/crates/adaptive" } ``` -Install the local gateway binary from a source checkout when you need to run the -gateway during development: +Install the gateway binary when you need to run the gateway: ```bash -cargo install --path ../NeMo-Flow/crates/cli +cargo install nemo-flow-cli ``` ## Install from the Repository diff --git a/docs/getting-started/quick-start.md b/docs/getting-started/quick-start.md index 4df44eb6..e7bd2ed3 100644 --- a/docs/getting-started/quick-start.md +++ b/docs/getting-started/quick-start.md @@ -13,4 +13,5 @@ Choose the Quick Start guide for the primary binding that you plan to use. Python Quick Start Node.js Quick Start Rust Quick Start +CLI Gateway Quick Start ``` diff --git a/docs/getting-started/rust.md b/docs/getting-started/rust.md index e5a03375..b9090b1d 100644 --- a/docs/getting-started/rust.md +++ b/docs/getting-started/rust.md @@ -25,8 +25,8 @@ serde_json = "1" - `nemo-flow` is the core Rust runtime surface. - `nemo-flow-adaptive` is the companion crate for adaptive runtime primitives and Redis-backed learning components. -- `nemo-flow-cli` is a binary crate. Use `cargo install --path - ../NeMo-Flow/crates/cli` when you need the local coding-agent gateway. +- `nemo-flow-cli` is a binary crate. Use `cargo install nemo-flow-cli` when + you need the local coding-agent gateway. ### Install from a Package Manager @@ -39,8 +39,8 @@ nemo-flow-adaptive = "0.1.*" serde_json = "1" ``` -Install the published gateway binary separately when you need coding-agent hook -and LLM gateway observability: +Install the gateway binary when you need coding-agent hook and LLM gateway +observability: ```bash cargo install nemo-flow-cli diff --git a/docs/index.md b/docs/index.md index 8f182797..146ad56f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -56,6 +56,7 @@ targets. These paths map common reader goals to the most relevant documentation entry points. - **End Users**: Start with [Prerequisites](getting-started/prerequisites.md) and [Quick Start](getting-started/quick-start.md). +- **Coding-Agent CLI Users**: Start with [CLI Gateway Quick Start](getting-started/cli.md), then use the per-agent gateway guide for Claude Code, Codex, Cursor, or Hermes. - **Agent Framework Developers**: Start with [Integrate into Frameworks](integrate-frameworks/about.md). - **Plugin Writers**: Start with [Build Plugins](build-plugins/about.md), then continue to [Basic Guide: Define a Plugin](build-plugins/basic-guide.md). - **Contributors**: Start with [Contribute](contribute/about.md) and the repository root `CONTRIBUTING.md` guide. diff --git a/docs/integrate-frameworks/coding-agent-claude-code.md b/docs/integrate-frameworks/coding-agent-claude-code.md index 13bdfa69..ae925d80 100644 --- a/docs/integrate-frameworks/coding-agent-claude-code.md +++ b/docs/integrate-frameworks/coding-agent-claude-code.md @@ -12,7 +12,13 @@ gateway controls as Claude Code. ## Transparent Run -Use the wrapper for no-install local observability: +After running the setup wizard, use the shortcut for local observability: + +```bash +nemo-flow claude -- "summarize this repository" +``` + +Use `run` when you need explicit, non-interactive launch control: ```bash nemo-flow run --atif-dir .nemo-flow/atif -- claude @@ -34,10 +40,11 @@ nemo-flow run \ -- claude ``` -If a launcher hides the command name, pass the agent explicitly: +If a launcher hides the command name, set that launcher as +`[agents.claude].command`, then pass the agent explicitly: ```bash -nemo-flow run --agent claude -- my-claude-wrapper +nemo-flow run --agent claude ``` ## Shared Config @@ -46,11 +53,10 @@ Create `.nemo-flow/config.toml` for project defaults or `~/.config/nemo-flow/config.toml` for user defaults: ```toml -[observability] -atif_dir = ".nemo-flow/atif" -metadata = { team = "agent-observability" } +[exporters.atif] +dir = ".nemo-flow/atif" -[export.openinference] +[exporters.openinference] endpoint = "http://127.0.0.1:4318/v1/traces" [agents.claude] @@ -58,7 +64,8 @@ command = "claude" ``` Then run `nemo-flow run --agent claude` to use the configured -command. User config takes priority over project and global config. +command, or `nemo-flow claude` for the shortcut. User config takes priority +over project and system config. ## Standalone Gateway diff --git a/docs/integrate-frameworks/coding-agent-codex.md b/docs/integrate-frameworks/coding-agent-codex.md index 5a86eff0..2f7892b9 100644 --- a/docs/integrate-frameworks/coding-agent-codex.md +++ b/docs/integrate-frameworks/coding-agent-codex.md @@ -19,7 +19,13 @@ feature flag. ## Transparent Run -Use the wrapper for no-install local observability: +After running the setup wizard, use the shortcut for local observability: + +```bash +nemo-flow codex +``` + +Use `run` when you need explicit, non-interactive launch control: ```bash nemo-flow run --atif-dir .nemo-flow/atif -- codex @@ -42,10 +48,11 @@ nemo-flow run \ -- codex ``` -If a launcher hides the command name, pass the agent explicitly: +If a launcher hides the command name, set that launcher as +`[agents.codex].command`, then pass the agent explicitly: ```bash -nemo-flow run --agent codex -- my-codex-wrapper +nemo-flow run --agent codex ``` ## Shared Config @@ -57,16 +64,16 @@ Create `.nemo-flow/config.toml` for project defaults or [upstream] openai_base_url = "https://api.openai.com" -[observability] -atif_dir = ".nemo-flow/atif" -metadata = { team = "agent-observability" } +[exporters.atif] +dir = ".nemo-flow/atif" [agents.codex] command = "codex" ``` Then run `nemo-flow run --agent codex` to use the configured command. -User config takes priority over project and global config. +Use `nemo-flow codex` for the shortcut. User config takes priority over +project and system config. ## Standalone Gateway diff --git a/docs/integrate-frameworks/coding-agent-cursor.md b/docs/integrate-frameworks/coding-agent-cursor.md index a258cd45..ad56e3b0 100644 --- a/docs/integrate-frameworks/coding-agent-cursor.md +++ b/docs/integrate-frameworks/coding-agent-cursor.md @@ -21,7 +21,13 @@ model routing is configurable. ## Transparent Run -Use the wrapper for no-install local observability: +After running the setup wizard, use the shortcut for local observability: + +```bash +nemo-flow cursor +``` + +Use `run` when you need explicit, non-interactive launch control: ```bash nemo-flow run --atif-dir .nemo-flow/atif -- cursor-agent @@ -42,10 +48,11 @@ nemo-flow run \ -- cursor-agent ``` -If a launcher hides the command name, pass the agent explicitly: +If a launcher hides the command name, set that launcher as +`[agents.cursor].command`, then pass the agent explicitly: ```bash -nemo-flow run --agent cursor -- my-cursor-wrapper +nemo-flow run --agent cursor ``` ## Shared Config @@ -54,9 +61,8 @@ Create `.nemo-flow/config.toml` for project defaults or `~/.config/nemo-flow/config.toml` for user defaults: ```toml -[observability] -atif_dir = ".nemo-flow/atif" -metadata = { team = "agent-observability" } +[exporters.atif] +dir = ".nemo-flow/atif" [agents.cursor] command = "cursor-agent" @@ -64,7 +70,8 @@ patch_restore_hooks = true ``` Then run `nemo-flow run --agent cursor` to use the configured command. -User config takes priority over project and global config. +Use `nemo-flow cursor` for the shortcut. User config takes priority over +project and system config. ## Standalone Gateway diff --git a/docs/integrate-frameworks/coding-agent-gateway.md b/docs/integrate-frameworks/coding-agent-gateway.md index f693b975..029afa80 100644 --- a/docs/integrate-frameworks/coding-agent-gateway.md +++ b/docs/integrate-frameworks/coding-agent-gateway.md @@ -5,15 +5,43 @@ SPDX-License-Identifier: Apache-2.0 # Advanced Guide: Coding-Agent Gateway -The `nemo-flow` binary observes coding agents that do not expose every -LLM call site directly. It combines agent-specific hook endpoints with a -passthrough LLM gateway so NeMo Flow owns both the agent lifecycle and the model -request lifecycle. +The experimental `nemo-flow` CLI gateway observes coding agents that do not +expose every LLM call site directly. It combines agent-specific hook endpoints +with a passthrough LLM gateway so NeMo Flow can record both the agent lifecycle +and the model request lifecycle. Use the gateway when you need one observability boundary for OpenAI Codex, Claude Code, Cursor, and Hermes without replacing each agent's canonical hook payload. +## Easy Path + +Install the CLI, then run the setup wizard once: + +```bash +cargo install nemo-flow-cli +nemo-flow +``` + +The wizard detects supported agent binaries on `PATH`, asks where to save +config, writes `.nemo-flow/config.toml` or user config, and enables local ATIF +export by default. Re-run it with `nemo-flow config`, or scope setup to one +agent with commands such as `nemo-flow config codex` and +`nemo-flow config hermes`. + +After setup, use the agent shortcuts: + +```bash +nemo-flow codex +nemo-flow claude -- "summarize this repository" +nemo-flow cursor +nemo-flow hermes +``` + +Use `nemo-flow doctor` to inspect config layers, detected agents, hook status, +exporter settings, and shell completions. Use `nemo-flow agents` when you only +need the supported and locally detected agent list. + ## Hook Endpoints Each hook endpoint accepts the agent's native hook JSON directly. Do not wrap @@ -52,10 +80,10 @@ under the active session scope. ## Transparent Run -Use `nemo-flow run` for no-install local observability. The wrapper -starts a gateway on a dynamic `127.0.0.1` port, injects the resolved hook and -gateway configuration into the launched coding agent, and stops the gateway -when the agent exits. +Use `nemo-flow run` for deterministic, non-interactive local observability. The +wrapper starts a gateway on a dynamic `127.0.0.1` port, injects the resolved +hook and gateway configuration into the launched coding agent, and stops the +gateway when the agent exits. ```bash nemo-flow run -- codex @@ -64,13 +92,23 @@ nemo-flow run -- cursor-agent nemo-flow run -- hermes ``` -The wrapper infers the agent from the command basename. Use `--agent` when a -launcher or wrapper hides the real agent name: +The wrapper infers the agent from the command basename. If a launcher or +wrapper hides the real agent name, set that wrapper as the configured command +and pass `--agent`: + +```toml +[agents.codex] +command = "my-codex-wrapper" +``` ```bash -nemo-flow run --agent codex -- my-codex-wrapper +nemo-flow run --agent codex ``` +The agent shortcuts are convenience wrappers over the same transparent launch +path. Use `run` when scripts need `--dry-run`, `--print`, explicit command +vectors, or command-level overrides. + Hermes is different from the other transparent modes: `run --agent hermes` starts the gateway and exports the dynamic `NEMO_FLOW_GATEWAY_URL`, but Hermes shell hooks still need to be installed or otherwise approved in Hermes config. @@ -78,18 +116,55 @@ shell hooks still need to be installed or otherwise approved in Hermes config. Use `--dry-run --print` to inspect the generated hook config, gateway environment, gateway URL, and final command without launching the agent. +## Benchmark And Eval Runs + +Use `nemo-flow run` for non-interactive benchmark tasks so each run gets an +ephemeral gateway and isolated observability artifacts. Prefer absolute +artifact directories when the benchmark harness runs from a different directory +than the task repository. + +The example below uses Codex CLI because it has a non-interactive `exec` mode. +Use the equivalent non-interactive command for another coding agent when one is +available. + +```bash +RUN_ID="$(date +%Y%m%d_%H%M%S)" +ART="/tmp/nemo-flow-eval/$RUN_ID" +mkdir -p "$ART/atif" "$ART/atof" + +nemo-flow run \ + --atif-dir "$ART/atif" \ + --atof-dir "$ART/atof" \ + -- codex \ + --ask-for-approval never \ + exec \ + --cd /path/to/benchmark/repo \ + --sandbox workspace-write \ + --color never \ + "Run the benchmark task and report the result." + +find "$ART" -maxdepth 3 -type f -print +``` + +The expected output includes one ATIF JSON file and one ATOF JSONL file for +the run. This path is suitable for local coding-agent eval workers. Remote or +cloud-hosted tasks are outside the local gateway's LLM capture boundary unless +their model traffic reaches the local gateway. + ## Shared Configuration -Shared TOML config is optional. The gateway loads defaults, then global config, -then project config, then user config. User config takes priority over global -and project config. CLI flags and environment variables override file config. +Shared TOML config is optional. Without an explicit `--config` path, the +gateway loads defaults, then system config, then the nearest project config, +then user config. Environment variables and CLI flags override file config. +Passing `--config path/to/config.toml` disables automatic discovery for that +command. Config file locations are: - `/etc/nemo-flow/config.toml` -- `.nemo-flow/config.toml` -- `$XDG_CONFIG_HOME/nemo-flow/config.toml` -- `~/.config/nemo-flow/config.toml` +- Nearest `.nemo-flow/config.toml`, walking up from the current directory +- `$XDG_CONFIG_HOME/nemo-flow/config.toml`, or + `~/.config/nemo-flow/config.toml` Example: @@ -98,14 +173,15 @@ Example: openai_base_url = "https://api.openai.com" anthropic_base_url = "https://api.anthropic.com" -[observability] -atif_dir = ".nemo-flow/atif" -metadata = { team = "agent-observability" } +[exporters.atif] +dir = ".nemo-flow/atif" -[plugins] -config = { components = [] } +[exporters.atof] +dir = ".nemo-flow/atof" +mode = "append" +filename_template = "{session_id}.jsonl" -[export.openinference] +[exporters.openinference] endpoint = "http://127.0.0.1:4318/v1/traces" [agents.claude] @@ -122,6 +198,24 @@ patch_restore_hooks = true command = "hermes" ``` +Legacy `[observability]`, flat `[exporters]`, and `[export.openinference]` +keys remain readable, but new config should use the nested `[exporters.*]` +shape. Add `[plugins].config` only when the gateway should activate a +process-level plugin config from `config.toml`; otherwise prefer a separate +`plugin.toml` or the `--plugin-config` JSON override. Do not use multiple +plugin config sources in one effective invocation. + +`plugin.toml` follows the same discovery scopes as `config.toml`: + +- `/etc/nemo-flow/plugin.toml` +- Nearest `.nemo-flow/plugin.toml` +- `$XDG_CONFIG_HOME/nemo-flow/plugin.toml`, or + `~/.config/nemo-flow/plugin.toml` + +Discovered plugin TOML files are merged from lowest to highest precedence. +TOML tables merge recursively, and the top-level `components` array merges by +component `kind`. + Transparent runs always bind the managed gateway to `127.0.0.1:0`. The selected port is discovered by the wrapper and exposed to hooks through `NEMO_FLOW_GATEWAY_URL`. @@ -133,14 +227,16 @@ Common environment variables for direct gateway server use are: - `NEMO_FLOW_ANTHROPIC_BASE_URL` - `NEMO_FLOW_OPENINFERENCE_ENDPOINT` - `NEMO_FLOW_ATIF_DIR` +- `NEMO_FLOW_ATOF_DIR` Per-session configuration controls the scope-local OpenInference subscriber, -the ATIF exporter, structured metadata on the top-level agent begin event, and -the plugin configuration metadata associated with the session. +the ATIF and ATOF exporters, structured metadata on the top-level agent begin +event, and the plugin configuration metadata associated with the session. `hook-forward` can also pass per-session configuration through headers: - `x-nemo-flow-atif-dir` +- `x-nemo-flow-atof-dir` - `x-nemo-flow-openinference-endpoint` - `x-nemo-flow-config-profile` - `x-nemo-flow-session-metadata` diff --git a/docs/integrate-frameworks/coding-agent-hermes.md b/docs/integrate-frameworks/coding-agent-hermes.md index 09da2a52..45e8c76b 100644 --- a/docs/integrate-frameworks/coding-agent-hermes.md +++ b/docs/integrate-frameworks/coding-agent-hermes.md @@ -17,8 +17,14 @@ traffic to route through the gateway. ## Transparent Run -Use the wrapper when you want the gateway lifetime managed for a local Hermes -process: +After running the setup wizard, use the shortcut when you want the gateway +lifetime managed for a local Hermes process: + +```bash +nemo-flow hermes +``` + +Use `run` when you need explicit, non-interactive launch control: ```bash nemo-flow run --atif-dir .nemo-flow/atif -- hermes @@ -41,10 +47,11 @@ nemo-flow run \ -- hermes ``` -If a launcher hides the command name, pass the agent explicitly: +If a launcher hides the command name, set that launcher as +`[agents.hermes].command`, then pass the agent explicitly: ```bash -nemo-flow run --agent hermes -- my-hermes-wrapper +nemo-flow run --agent hermes ``` ## Shared Config @@ -53,11 +60,10 @@ Create `.nemo-flow/config.toml` for project defaults or `~/.config/nemo-flow/config.toml` for user defaults: ```toml -[observability] -atif_dir = ".nemo-flow/atif" -metadata = { team = "agent-observability" } +[exporters.atif] +dir = ".nemo-flow/atif" -[export.openinference] +[exporters.openinference] endpoint = "http://127.0.0.1:4318/v1/traces" [agents.hermes] @@ -65,7 +71,8 @@ command = "hermes" ``` Then run `nemo-flow run --agent hermes` to use the configured command. -User config takes priority over project and global config. +Use `nemo-flow hermes` for the shortcut. User config takes priority over +project and system config. ## Hermes Hook Setup