Skip to content

Commit d6e08d3

Browse files
authored
perf(tools): generate serializable tool metadata artifacts (#6153)
* perf(tools): generate serializable tool metadata artifacts Adds `scripts/sync-tool-metadata.ts`, which projects the executable tool registry down to the data half nobody needs a closure for, plus typed accessors over the result. No consumer is rewired yet — that is the next PR. `@/tools/registry` is a ~9,000-line barrel over 4,366 tools. Each `ToolConfig` mixes plain data (`params`, `outputs`, `name`) with closures (`request.headers`, `transformResponse`, `directExecution`, `postProcess`), and those closures reach every integration's SDK client and parser — which is why reaching the barrel costs ~4,700 modules. Every client-reachable caller was audited: none of them need a closure. They need `outputs`, `params`, or an existence check. Two artifacts, not one. `outputs` is ~4 MB of the ~8 MB and has a single consumer, so it is emitted separately and exposed from its own module; callers needing only params never load it. The data is a JSON string parsed at runtime rather than an imported `.json` or an object literal. That is not stylistic — with `resolveJsonModule` (enabled repo-wide) a `.json` import makes TypeScript infer a literal type for all 4,366 entries: tsc --noEmit, baseline 12.6s tsc --noEmit, with `.json` imports 8m07s (38x) tsc --noEmit, with string literals 12.0s An ambient `declare module` does not short-circuit it (measured: 8m18s), and an object literal is the same inference work. A single string literal is one cheap token for the compiler and the bundler, and `JSON.parse` beats evaluating the equivalent literal at runtime. The generator refuses to emit any function value, so shipping executable config to the client fails loudly instead of silently. `hosting` and `schemaEnrichment` are excluded on those grounds — both hold functions and are server-only. Also strips empty param entries: the registry has one (`stt_deepgram_v2`, an `undefined`) which crashes callers that read `param.type` while iterating. `JSON.stringify` drops `undefined` on its own, so the guard is there for an explicit `null` — which serializes faithfully and would reach consumers — and to warn either way. Wires `tool-metadata:check` into CI alongside the other generated-contract gates, and ignores the generated directory in biome (it exceeds the 1 MB limit and was being skipped with a notice on every commit). Adds a `tool-registry-boundary` skill covering which module to import, the three non-obvious properties of the artifacts, and how to verify an edge is actually cut — the canvas route reaches the registry through four redundant paths, so cutting one alone moves the module count by ~1. * fix(tools): harden the metadata accessors against inherited keys Review found two real defects in the generated-metadata layer. `JSON.parse` returns an object with the normal prototype, so a bare bracket lookup resolved inherited members: `getToolMetadata('constructor')` returned a *function* typed as `ToolMetadata`, and `getToolOutputsMetadata('toString')` likewise — silently violating the accessors' documented "undefined if unknown" contract. Guarded with `Object.hasOwn`, with a parameterised regression test over `constructor`, `toString`, `valueOf`, `hasOwnProperty` and `__proto__`. The generator's no-functions scan also gave up past ten levels of nesting. Param and output schemas nest arbitrarily, so a deeper closure would have been dropped silently by `JSON.stringify` while generation reported success — shipping an incomplete schema and defeating the guarantee the scan exists to provide. The depth cap is gone; a `WeakSet` handles the cycles that exposes. * docs(tools): tell tool authors to regenerate the metadata artifacts A new tool now has a second registration step. Client code reads `params` and `outputs` from the generated artifacts rather than from the registry, so a tool added without regenerating them is registered but invisible to the UI — and CI fails on the stale artifacts. `add-tools` and `add-integration` are where someone actually adds a tool, so the step goes in both, next to the registry edit and in each checklist. * docs(blocks): note when a block change needs tool-metadata regeneration Adding a block alone needs no regeneration — it references existing tool IDs and changes no tool's shape. But a change that touches a tool alongside the block does, and this is where that is easy to miss: a block's `outputs` are authored to match its tools' outputs, and the UI now reads those from the generated metadata, so a stale artifact makes the block's declared outputs disagree with what the panel renders (and fails CI). Completes the tool-authoring surface alongside add-tools and add-integration. * docs(tools): cover tool removal in the regeneration guidance The three tool-authoring skills said to regenerate after adding or changing a tool, but not after removing one. Removal is equally breaking and equally guarded: deleting a tool from `tools/registry.ts` without regenerating fails `tool-metadata:check` (verified — exit 1), so a contributor following the skill literally would have hit a CI failure the skill never warned about.
1 parent 9299cee commit d6e08d3

21 files changed

Lines changed: 724 additions & 0 deletions

File tree

.agents/skills/add-block/SKILL.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -919,6 +919,12 @@ Derive templates from the service's real use cases. Each prompt should name a co
919919
- **Ground every skill in operations the block actually exposes** — cross-check each skill's steps against `tools.access`. Never describe an action the integration cannot perform.
920920
- **Derive skills from real, popular use cases found online — never invent them.** Web-search the service's documented use cases (vendor use-case/solutions pages, official docs describing the workflow, reputable "top automations for X" articles) and only add a skill you can source as something people genuinely do with the service. Do not hallucinate skills.
921921

922+
## Generated tool metadata
923+
924+
Adding a block on its own needs **no** regeneration — a block references existing tool IDs through `tools.access` and does not change any tool's shape.
925+
926+
But if the same change also adds, edits **or removes** a tool, run `bun run tool-metadata:generate` and commit the result, or CI fails on stale artifacts. That matters here because a block's `outputs` are authored to match its tools' outputs, and the UI now reads those from the generated metadata rather than the executable registry — an unregenerated tool change makes the block's outputs disagree with what the panel renders. See `.agents/skills/tool-registry-boundary/SKILL.md`.
927+
922928
## Checklist Before Finishing
923929

924930
- [ ] `integrationType` is set to the correct `IntegrationType` enum value
@@ -933,6 +939,7 @@ Derive templates from the service's real use cases. Each prompt should name a co
933939
- [ ] Tools.config.tool returns correct tool ID (snake_case)
934940
- [ ] Outputs match tool outputs
935941
- [ ] Block + meta registered in registry-maps.ts (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`)
942+
- [ ] If any tool was added, changed or removed alongside the block: ran `bun run tool-metadata:generate` and committed the artifacts
936943
- [ ] If icon missing: asked user to provide SVG
937944
- [ ] If triggers exist: `triggers` config set, trigger subBlocks spread
938945
- [ ] Optional/rarely-used fields set to `mode: 'advanced'`

.agents/skills/add-integration/SKILL.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -415,6 +415,16 @@ export const tools: Record<string, ToolConfig> = {
415415
}
416416
```
417417

418+
Then regenerate the generated tool metadata and commit it:
419+
420+
```bash
421+
bun run tool-metadata:generate
422+
```
423+
424+
Client code reads `params`/`outputs` from these artifacts rather than importing
425+
the registry, so a tool you add, change or remove is invisible to the UI until they are regenerated,
426+
and CI fails on stale ones. See `.agents/skills/tool-registry-boundary/SKILL.md`.
427+
418428
### Block Registry (`apps/sim/blocks/registry-maps.ts`)
419429

420430
The data maps (`BLOCK_REGISTRY` + `BLOCK_META_REGISTRY`) live in `registry-maps.ts`; `registry.ts` holds only the accessor functions. Add the import and an entry to each map alphabetically:
@@ -490,6 +500,7 @@ If creating V2 versions (API-aligned outputs):
490500
- [ ] All optional outputs have `optional: true`
491501
- [ ] Created `index.ts` barrel export
492502
- [ ] Registered all tools in `tools/registry.ts`
503+
- [ ] Ran `bun run tool-metadata:generate` and committed the regenerated artifacts
493504

494505
### Block
495506
- [ ] Created `blocks/blocks/{service}.ts`

.agents/skills/add-tools/SKILL.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,17 @@ export const tools = {
296296
}
297297
```
298298

299+
3. Regenerate the tool metadata artifacts:
300+
301+
```bash
302+
bun run tool-metadata:generate
303+
```
304+
305+
Client code reads a tool's `params`/`outputs` from generated metadata rather than
306+
importing the registry, so a tool you add, change or remove is invisible to the UI until
307+
these are regenerated — and CI fails on stale artifacts. Commit the result. See
308+
`.agents/skills/tool-registry-boundary/SKILL.md`.
309+
299310
## Wiring Tools into the Block (Required)
300311

301312
After registering in `tools/registry.ts`, you MUST also update the block definition at `apps/sim/blocks/blocks/{service}.ts`. This is not optional — tools are only usable from the UI if they are wired into the block.
@@ -443,6 +454,7 @@ All tool IDs MUST use `snake_case`: `{service}_{action}` (e.g., `x_create_tweet`
443454
- [ ] Types file has all interfaces
444455
- [ ] Index.ts exports all tools and re-exports types (`export * from './types'`)
445456
- [ ] Tools registered in `tools/registry.ts`
457+
- [ ] `bun run tool-metadata:generate` run and the regenerated artifacts committed
446458
- [ ] Block wired: `tools.access`, dropdown options, subBlocks, `tools.config`, outputs, inputs
447459

448460
## Final Validation (Required)
Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
---
2+
name: tool-registry-boundary
3+
description: Keep the executable tool registry out of client-reachable module graphs — when to read `@/tools/metadata` instead of `getTool`, how to measure whether an import edge pulls the registry, and how to regenerate the metadata artifacts. Use when touching `apps/sim/tools/registry.ts`, `tools/utils.ts`, `tools/params.ts`, or anything that calls `getTool`.
4+
---
5+
6+
# Tool Registry Boundary Skill
7+
8+
You keep the 4,300-tool executable registry out of module graphs that don't execute tools.
9+
10+
## The rule
11+
12+
> Client-reachable code reads tool **metadata**. Only code that actually executes a tool imports the **registry**.
13+
14+
`@/tools/registry` is a ~9,000-line barrel importing every tool. Each `ToolConfig` mixes plain data (`params`, `outputs`, `name`) with closures — `request.url`, `request.headers`, `transformResponse`, `directExecution`, `postProcess`. Those closures reach the SDK clients, API helpers and parsers each integration needs, and that is what makes the barrel expensive: reaching it costs ~4,700 additional modules.
15+
16+
`getTool()` returns the whole `ToolConfig`, so a single `getTool` import anywhere in a client-reachable file drags all of it in.
17+
18+
## Which module to import
19+
20+
| you need | import | notes |
21+
| --- | --- | --- |
22+
| whether a tool id exists | `hasToolMetadata` from `@/tools/metadata` | |
23+
| a tool's params | `getToolParams` / `getToolMetadata` from `@/tools/metadata` | |
24+
| a tool's declared outputs | `getToolOutputsMetadata` from `@/tools/metadata-outputs` | separate module on purpose — see below |
25+
| every tool id | `getToolIds` from `@/tools/metadata` | |
26+
| to **execute** a tool | `getTool` from `@/tools/utils`, or `@/tools/utils.server` | server paths only |
27+
28+
Outputs live in their own module because they are roughly two thirds of the generated data and have a single consumer. Importing `@/tools/metadata` must never pull them — do not "helpfully" re-export one from the other.
29+
30+
## The generated artifacts
31+
32+
`apps/sim/tools/generated/tool-metadata.ts` and `tool-outputs.ts` are produced by `scripts/sync-tool-metadata.ts`:
33+
34+
```bash
35+
bun run tool-metadata:generate # after adding/changing a tool
36+
bun run tool-metadata:check # what CI runs; fails if stale
37+
```
38+
39+
Never hand-edit them. If you add a tool or change a tool's `params`/`outputs`, regenerate and commit the result, or CI fails.
40+
41+
Three non-obvious properties, each of which was measured and is easy to undo by accident:
42+
43+
- **The data is a JSON string parsed at runtime, not an imported `.json` and not an object literal.** With `resolveJsonModule` (which this repo enables), a `.json` import makes TypeScript infer a literal type for all 4,300+ entries and takes `tsc --noEmit` from **12.6s to 8m07s** — a 38x regression. An ambient `declare module` does *not* short-circuit it, and an object literal costs the same. A single string literal is one cheap token for both the compiler and the bundler, and `JSON.parse` beats evaluating the equivalent literal at runtime. Do not "clean this up" into a `.json` import.
44+
- **The generator refuses to emit function values.** If you add a field to `METADATA_FIELDS` that contains a closure, generation fails loudly rather than shipping executable config to the client. `hosting` and `schemaEnrichment` are excluded for exactly this reason (`hosting.enabled`, `pricing`, and `enrichSchema` are functions) — they are server-only.
45+
- **Empty param entries are stripped.** The registry contains one (`stt_deepgram_v2`), which crashes callers that read `param.type` while iterating.
46+
47+
## How to verify an edge actually got cut
48+
49+
Do not eyeball imports — the registry is reached through several redundant paths, so cutting one buys nothing while another survives. Walk the graph:
50+
51+
1. From the entry you care about, follow `import` and `export … from` (skipping `import type`), resolving `@/` against `apps/sim`.
52+
2. Check whether `apps/sim/tools/registry.ts` is in the reachable set, and print the parent chain if it is.
53+
3. Compare the reachable module count before and after.
54+
55+
Reference points measured on this repo:
56+
57+
| entry | modules |
58+
| --- | --- |
59+
| `tools/registry.ts` reachable | ~4,900 |
60+
| `tools/merge-params.ts` (leaf) | 2 |
61+
| `providers/utils.ts` after cutting its `params` edge | 22 |
62+
| `app/workspace/[workspaceId]/w/page.tsx` (canvas) | 6,591, of which 4,689 are the registry |
63+
64+
The canvas route reaches the registry through **four** redundant edges — `providers/utils` (via `tools/params`), `lib/workflows/blocks/block-outputs`, `lib/workflows/sanitization/validation`, and `serializer/index`. Cutting any one alone moves the module count by ~1. They must all be cut before anything improves; measure the route, not the file you edited.
65+
66+
## When adding a new caller
67+
68+
Ask what the caller does with the config. If it reads `params`, `outputs`, `name`, `description` or just checks existence, it belongs on `@/tools/metadata` — no exceptions, even on a path you believe is server-only today, because a future client import will silently re-attach the registry to the graph.
69+
70+
If it genuinely executes — builds a request, transforms a response, runs `directExecution` — use `getTool`, and keep that file off client-reachable paths.

.claude/commands/add-block.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -918,6 +918,12 @@ Derive templates from the service's real use cases. Each prompt should name a co
918918
- **Ground every skill in operations the block actually exposes** — cross-check each skill's steps against `tools.access`. Never describe an action the integration cannot perform.
919919
- **Derive skills from real, popular use cases found online — never invent them.** Web-search the service's documented use cases (vendor use-case/solutions pages, official docs describing the workflow, reputable "top automations for X" articles) and only add a skill you can source as something people genuinely do with the service. Do not hallucinate skills.
920920

921+
## Generated tool metadata
922+
923+
Adding a block on its own needs **no** regeneration — a block references existing tool IDs through `tools.access` and does not change any tool's shape.
924+
925+
But if the same change also adds, edits **or removes** a tool, run `bun run tool-metadata:generate` and commit the result, or CI fails on stale artifacts. That matters here because a block's `outputs` are authored to match its tools' outputs, and the UI now reads those from the generated metadata rather than the executable registry — an unregenerated tool change makes the block's outputs disagree with what the panel renders. See `.agents/skills/tool-registry-boundary/SKILL.md`.
926+
921927
## Checklist Before Finishing
922928

923929
- [ ] `integrationType` is set to the correct `IntegrationType` enum value
@@ -932,6 +938,7 @@ Derive templates from the service's real use cases. Each prompt should name a co
932938
- [ ] Tools.config.tool returns correct tool ID (snake_case)
933939
- [ ] Outputs match tool outputs
934940
- [ ] Block + meta registered in registry-maps.ts (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`)
941+
- [ ] If any tool was added, changed or removed alongside the block: ran `bun run tool-metadata:generate` and committed the artifacts
935942
- [ ] If icon missing: asked user to provide SVG
936943
- [ ] If triggers exist: `triggers` config set, trigger subBlocks spread
937944
- [ ] Optional/rarely-used fields set to `mode: 'advanced'`

.claude/commands/add-integration.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -414,6 +414,16 @@ export const tools: Record<string, ToolConfig> = {
414414
}
415415
```
416416

417+
Then regenerate the generated tool metadata and commit it:
418+
419+
```bash
420+
bun run tool-metadata:generate
421+
```
422+
423+
Client code reads `params`/`outputs` from these artifacts rather than importing
424+
the registry, so a tool you add, change or remove is invisible to the UI until they are regenerated,
425+
and CI fails on stale ones. See `.agents/skills/tool-registry-boundary/SKILL.md`.
426+
417427
### Block Registry (`apps/sim/blocks/registry-maps.ts`)
418428

419429
The data maps (`BLOCK_REGISTRY` + `BLOCK_META_REGISTRY`) live in `registry-maps.ts`; `registry.ts` holds only the accessor functions. Add the import and an entry to each map alphabetically:
@@ -489,6 +499,7 @@ If creating V2 versions (API-aligned outputs):
489499
- [ ] All optional outputs have `optional: true`
490500
- [ ] Created `index.ts` barrel export
491501
- [ ] Registered all tools in `tools/registry.ts`
502+
- [ ] Ran `bun run tool-metadata:generate` and committed the regenerated artifacts
492503

493504
### Block
494505
- [ ] Created `blocks/blocks/{service}.ts`

.claude/commands/add-tools.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -295,6 +295,17 @@ export const tools = {
295295
}
296296
```
297297

298+
3. Regenerate the tool metadata artifacts:
299+
300+
```bash
301+
bun run tool-metadata:generate
302+
```
303+
304+
Client code reads a tool's `params`/`outputs` from generated metadata rather than
305+
importing the registry, so a tool you add, change or remove is invisible to the UI until
306+
these are regenerated — and CI fails on stale artifacts. Commit the result. See
307+
`.agents/skills/tool-registry-boundary/SKILL.md`.
308+
298309
## Wiring Tools into the Block (Required)
299310

300311
After registering in `tools/registry.ts`, you MUST also update the block definition at `apps/sim/blocks/blocks/{service}.ts`. This is not optional — tools are only usable from the UI if they are wired into the block.
@@ -442,6 +453,7 @@ All tool IDs MUST use `snake_case`: `{service}_{action}` (e.g., `x_create_tweet`
442453
- [ ] Types file has all interfaces
443454
- [ ] Index.ts exports all tools and re-exports types (`export * from './types'`)
444455
- [ ] Tools registered in `tools/registry.ts`
456+
- [ ] `bun run tool-metadata:generate` run and the regenerated artifacts committed
445457
- [ ] Block wired: `tools.access`, dropdown options, subBlocks, `tools.config`, outputs, inputs
446458

447459
## Final Validation (Required)
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
---
2+
description: Keep the executable tool registry out of client-reachable module graphs — when to read `@/tools/metadata` instead of `getTool`, how to measure whether an import edge pulls the registry, and how to regenerate the metadata artifacts. Use when touching `apps/sim/tools/registry.ts`, `tools/utils.ts`, `tools/params.ts`, or anything that calls `getTool`.
3+
---
4+
5+
# Tool Registry Boundary Skill
6+
7+
You keep the 4,300-tool executable registry out of module graphs that don't execute tools.
8+
9+
## The rule
10+
11+
> Client-reachable code reads tool **metadata**. Only code that actually executes a tool imports the **registry**.
12+
13+
`@/tools/registry` is a ~9,000-line barrel importing every tool. Each `ToolConfig` mixes plain data (`params`, `outputs`, `name`) with closures — `request.url`, `request.headers`, `transformResponse`, `directExecution`, `postProcess`. Those closures reach the SDK clients, API helpers and parsers each integration needs, and that is what makes the barrel expensive: reaching it costs ~4,700 additional modules.
14+
15+
`getTool()` returns the whole `ToolConfig`, so a single `getTool` import anywhere in a client-reachable file drags all of it in.
16+
17+
## Which module to import
18+
19+
| you need | import | notes |
20+
| --- | --- | --- |
21+
| whether a tool id exists | `hasToolMetadata` from `@/tools/metadata` | |
22+
| a tool's params | `getToolParams` / `getToolMetadata` from `@/tools/metadata` | |
23+
| a tool's declared outputs | `getToolOutputsMetadata` from `@/tools/metadata-outputs` | separate module on purpose — see below |
24+
| every tool id | `getToolIds` from `@/tools/metadata` | |
25+
| to **execute** a tool | `getTool` from `@/tools/utils`, or `@/tools/utils.server` | server paths only |
26+
27+
Outputs live in their own module because they are roughly two thirds of the generated data and have a single consumer. Importing `@/tools/metadata` must never pull them — do not "helpfully" re-export one from the other.
28+
29+
## The generated artifacts
30+
31+
`apps/sim/tools/generated/tool-metadata.ts` and `tool-outputs.ts` are produced by `scripts/sync-tool-metadata.ts`:
32+
33+
```bash
34+
bun run tool-metadata:generate # after adding/changing a tool
35+
bun run tool-metadata:check # what CI runs; fails if stale
36+
```
37+
38+
Never hand-edit them. If you add a tool or change a tool's `params`/`outputs`, regenerate and commit the result, or CI fails.
39+
40+
Three non-obvious properties, each of which was measured and is easy to undo by accident:
41+
42+
- **The data is a JSON string parsed at runtime, not an imported `.json` and not an object literal.** With `resolveJsonModule` (which this repo enables), a `.json` import makes TypeScript infer a literal type for all 4,300+ entries and takes `tsc --noEmit` from **12.6s to 8m07s** — a 38x regression. An ambient `declare module` does *not* short-circuit it, and an object literal costs the same. A single string literal is one cheap token for both the compiler and the bundler, and `JSON.parse` beats evaluating the equivalent literal at runtime. Do not "clean this up" into a `.json` import.
43+
- **The generator refuses to emit function values.** If you add a field to `METADATA_FIELDS` that contains a closure, generation fails loudly rather than shipping executable config to the client. `hosting` and `schemaEnrichment` are excluded for exactly this reason (`hosting.enabled`, `pricing`, and `enrichSchema` are functions) — they are server-only.
44+
- **Empty param entries are stripped.** The registry contains one (`stt_deepgram_v2`), which crashes callers that read `param.type` while iterating.
45+
46+
## How to verify an edge actually got cut
47+
48+
Do not eyeball imports — the registry is reached through several redundant paths, so cutting one buys nothing while another survives. Walk the graph:
49+
50+
1. From the entry you care about, follow `import` and `export … from` (skipping `import type`), resolving `@/` against `apps/sim`.
51+
2. Check whether `apps/sim/tools/registry.ts` is in the reachable set, and print the parent chain if it is.
52+
3. Compare the reachable module count before and after.
53+
54+
Reference points measured on this repo:
55+
56+
| entry | modules |
57+
| --- | --- |
58+
| `tools/registry.ts` reachable | ~4,900 |
59+
| `tools/merge-params.ts` (leaf) | 2 |
60+
| `providers/utils.ts` after cutting its `params` edge | 22 |
61+
| `app/workspace/[workspaceId]/w/page.tsx` (canvas) | 6,591, of which 4,689 are the registry |
62+
63+
The canvas route reaches the registry through **four** redundant edges — `providers/utils` (via `tools/params`), `lib/workflows/blocks/block-outputs`, `lib/workflows/sanitization/validation`, and `serializer/index`. Cutting any one alone moves the module count by ~1. They must all be cut before anything improves; measure the route, not the file you edited.
64+
65+
## When adding a new caller
66+
67+
Ask what the caller does with the config. If it reads `params`, `outputs`, `name`, `description` or just checks existence, it belongs on `@/tools/metadata` — no exceptions, even on a path you believe is server-only today, because a future client import will silently re-attach the registry to the graph.
68+
69+
If it genuinely executes — builds a request, transforms a response, runs `directExecution` — use `getTool`, and keep that file off client-reachable paths.

0 commit comments

Comments
 (0)