docs: harden the OpenAPI sync policy against publishing private surfaces

.agents/skills/sync-openapi-spec/references/sync-policy.md

@@ -1,25 +1,36 @@
 # Sync Policy
 
-This document records what `developers/agent-api-openapi.yaml` keeps from `warp-server/public_api/openapi.yaml`, and why. The exclusion lists live in `scripts/sync_openapi.py` as `EXCLUDED_TAGS` and `EXCLUDED_PATHS`. Update both this document and the script when the policy changes.
+This document records what `developers/agent-api-openapi.yaml` keeps from `warp-server/public_api/openapi.yaml`, and why. The exclusion lists live in `scripts/sync_openapi.py` as `EXCLUDED_TAGS`, `EXCLUDED_PATHS`, and `EXCLUDED_PATH_PREFIXES`. Update both this document and the script when the policy changes.
+
+## Relationship to warp-server's release automation
+
+warp-server publishes the same file through `script/generate-public-openapi` and the `sync_public_openapi_to_docs` workflow, which runs at release-candidate time and opens a PR on the `automation/sync-agent-api-spec` branch. That pipeline is the authoritative publisher, and it filters with `public_api/public-openapi-filter.yaml` (`flagValues: [x-internal: true]`).
+
+This skill is the manual fallback for the same job, so its output has to match that pipeline's public/private decisions. Prefer letting the release automation land routine spec updates. Reach for `--mode apply` only when the docs copy needs a correction the automation won't deliver in time, and re-read this policy before doing so.
 
 ## How filtering works
 
 `scripts/sync_openapi.py` applies these rules, top-down:
 
-1. Drop every tag listed in `EXCLUDED_TAGS`.
-2. Drop every path whose tags are a subset of `EXCLUDED_TAGS`, plus every path listed explicitly in `EXCLUDED_PATHS`.
-3. Keep every surviving path verbatim, including any `x-internal: true` markers on its operations.
+1. Drop every operation marked `x-internal: true`, and drop a path entirely when every one of its operations is internal.
+2. Drop every tag listed in `EXCLUDED_TAGS`.
+3. Drop every path whose tags are a subset of `EXCLUDED_TAGS`, plus every path listed explicitly in `EXCLUDED_PATHS` or matching a prefix in `EXCLUDED_PATH_PREFIXES`.
 4. Keep top-level `openapi`, `info`, `servers`, and `components.securitySchemes` verbatim.
 5. Keep only the `components.schemas` entries that are reachable from the surviving paths via `$ref` walking (recursive over `allOf`/`oneOf`/`anyOf`/`items`/`additionalProperties`/etc.).
 
+Rule 1 mirrors warp-server's own filter, so a surface the server team marks private stays private here without anyone having to maintain a matching allowlist entry.
+
 ## Excluded tags
 
-### `memory_stores`
-Memory stores are gated as `x-internal: true` server-side. They are not part of the public Oz Agent API surface today and are excluded from the docs reference until they ship publicly. If/when this tag goes public, remove it from `EXCLUDED_TAGS` and update this section.
+### `memory_stores` and `memory`
+These tags back Agent Memory, which is a research preview. The tag was renamed `memory_stores` → `memory` upstream, so both names are excluded: keeping only the old name silently reopened the surface after the rename. Remove them from `EXCLUDED_TAGS` when Agent Memory ships publicly.
 
 ### `harness-support`
 The `/harness-support/*` endpoints form the worker-to-server contract used by Oz workers (transcripts, snapshots, finish-task signaling, etc.). They are not part of the public API contract — customers should not call them directly. Excluded permanently.
 
+### `factory`
+Oz Factory has not shipped publicly. Its `FactoryMcp` flag is dogfood and the `@warp/factory` front end is internal, so none of its endpoints belong in the public reference. Remove this tag when Factory goes GA.
+
 ## Excluded paths (within otherwise-public tags)
 
 These four `agent`-tag paths are excluded individually because the `agent` tag itself remains public:
@@ -31,17 +42,24 @@ These four `agent`-tag paths are excluded individually because the `agent` tag i
 
 If any of these become stable public surfaces, remove them from `EXCLUDED_PATHS` and update this list.
 
-## What we deliberately KEEP that you might expect to be hidden
+## Excluded path prefixes
+
+`EXCLUDED_PATH_PREFIXES` drops a path by prefix regardless of how its operations are tagged. Today it holds a single entry, `/factory`, because some Factory operations are tagged `agent` upstream — `GET /factory/scorers/{scorer_id}/results` is one — so a tags-only rule leaks them into the public reference. Use a prefix only when a whole URL namespace is private; prefer a tag or an explicit path everywhere else.
+
+## `x-internal` operations are dropped
+
+Operations marked `x-internal: true` are removed, and a path loses its entry when all of its operations are internal. This covers the `/agent/messages/*` and `/agent/events/*` orchestration-messaging operations, `/agent/runs/{runId}/client-events`, `/agent/conversations/{conversation_id}/rename`, and `/agent/sessions/{sessionUuid}/redirect`.
 
-The script keeps `x-internal: true` operations under public paths. Today this means the `/agent/messages/*` and `/agent/events/*` operations are present in the docs file even though they're flagged `x-internal` in the source. This matches the pre-existing state of `developers/agent-api-openapi.yaml` and the way Scalar already renders the reference. If we want to start stripping `x-internal` operations from the docs spec, change the policy here and update `_should_keep_path`/the operation-level filter in `scripts/sync_openapi.py`.
+An earlier version of this policy kept those operations verbatim so the regenerated file matched the docs copy already on disk. That made this script disagree with warp-server's release filter, which strips them, and meant every newly marked-internal operation would be republished here. Matching the upstream marker is the safer default: it can only ever remove surfaces, never add one.
 
 ## Adding a new exclusion
 
 Use the script's `_unknown_classifications` warnings as the trigger. When the diff flags a new tag or path with `!`:
 1. Read the corresponding handler in `warp-server/router/handlers/public_api/` to determine intent.
 2. If the endpoint should be hidden:
+   - Prefer asking the server team to mark the operation `x-internal: true` upstream, which hides it from both publishers at once.
    - For an entire new tag, add the tag name to `EXCLUDED_TAGS` in `scripts/sync_openapi.py`.
-   - For a single path, add it to `EXCLUDED_PATHS`.
+   - For a single path, add it to `EXCLUDED_PATHS`; for a whole private URL namespace, add a prefix to `EXCLUDED_PATH_PREFIXES`.
 3. Add a short rationale to this document under "Excluded tags" or "Excluded paths."
 4. Re-run `--mode diff` to confirm there are no remaining `!` warnings.
 5. Then run `--mode apply` and proceed with the normal PR flow.

.agents/skills/sync-openapi-spec/scripts/sync_openapi.py

@@ -6,15 +6,22 @@
 `docs/developers/agent-api-openapi.yaml`, which is a curated subset.
 
 This script generates the docs subset deterministically:
+  * operations marked ``x-internal: true`` are removed, and a path whose
+    every operation is internal is dropped entirely
   * tags listed in EXCLUDED_TAGS are removed (and their paths/schemas)
   * paths listed in EXCLUDED_PATHS are removed
-  * surviving paths and operations are kept verbatim, including any
-    ``x-internal: true`` markers
   * components/schemas is pruned to only schemas reachable from the
     surviving paths via $ref walking
   * the regenerated spec is validated for unresolved $refs before
     being written; apply will refuse to write a broken spec
 
+``x-internal`` is warp-server's own public/private marker: its
+``public_api/public-openapi-filter.yaml`` strips those operations when the
+release pipeline publishes the spec. Honoring the same marker here keeps this
+script from publishing a surface the server team has explicitly marked private,
+instead of relying only on a hand-maintained tag allowlist that goes stale
+whenever a new private tag appears.
+
 Modes:
   diff       Print structural drift between source and target. Exits 1
              if drift is found.
@@ -39,9 +46,23 @@
 import yaml
 
 # Tags whose paths and tag entry should be removed entirely.
-# `memory_stores` is gated as `x-internal` server-side.
+# `memory_stores` / `memory` back Agent Memory, which is a research preview.
 # `harness-support` is the worker-to-server contract — not a public API.
-EXCLUDED_TAGS: frozenset[str] = frozenset({"memory_stores", "harness-support"})
+# `factory` is Oz Factory, which has not shipped publicly.
+# These are belt-and-braces on top of the `x-internal` filter below: a tag can
+# be private even when individual operations aren't marked internal yet.
+EXCLUDED_TAGS: frozenset[str] = frozenset(
+    {"memory_stores", "memory", "harness-support", "factory"}
+)
+
+# OpenAPI extension warp-server uses to mark an operation private. Mirrors
+# `flagValues: [x-internal: true]` in warp-server/public_api/public-openapi-filter.yaml.
+INTERNAL_MARKER = "x-internal"
+
+# Path-item keys that are HTTP operations rather than shared path metadata.
+HTTP_METHODS: frozenset[str] = frozenset(
+    {"get", "put", "post", "delete", "options", "head", "patch", "trace"}
+)
 
 # Specific paths under otherwise-public tags that should be hidden from
 # the public API reference. Keep in sync with references/sync-policy.md.
@@ -54,6 +75,12 @@
     }
 )
 
+# Path prefixes that are private no matter how the operation is tagged. Tag
+# checks alone are not enough here: some Factory operations are tagged `agent`
+# upstream (for example `GET /factory/scorers/{scorer_id}/results`), so a
+# tags-only rule would leak them into the public reference.
+EXCLUDED_PATH_PREFIXES: tuple[str, ...] = ("/factory",)
+
 # Default checkout layout: docs/ and warp-server/ as siblings.
 DEFAULT_SOURCE = Path("../warp-server/public_api/openapi.yaml")
 DEFAULT_TARGET = Path("developers/agent-api-openapi.yaml")
@@ -124,12 +151,43 @@ def _path_tags(path_item: dict[str, Any]) -> set[str]:
     return tags
 
 
+def _is_internal_operation(operation: Any) -> bool:
+    """Whether an operation carries warp-server's ``x-internal: true`` marker."""
+    return isinstance(operation, dict) and operation.get(INTERNAL_MARKER) is True
+
+
+def strip_internal_operations(path_item: dict[str, Any]) -> dict[str, Any]:
+    """Return ``path_item`` without any operation marked ``x-internal: true``.
+
+    Non-operation keys (``parameters``, ``summary``, ``servers``, ...) are
+    preserved so a partially-internal path keeps its shared metadata.
+    """
+    return {
+        key: value
+        for key, value in path_item.items()
+        if not (key.lower() in HTTP_METHODS and _is_internal_operation(value))
+    }
+
+
+def _has_public_operation(path_item: dict[str, Any]) -> bool:
+    """Whether a path item still declares at least one non-internal operation."""
+    return any(
+        key.lower() in HTTP_METHODS and not _is_internal_operation(value)
+        for key, value in path_item.items()
+    )
+
+
 def _should_keep_path(path: str, path_item: dict[str, Any]) -> bool:
     if path in EXCLUDED_PATHS:
         return False
+    if path.startswith(EXCLUDED_PATH_PREFIXES):
+        return False
     tags = _path_tags(path_item)
     if tags and tags.issubset(EXCLUDED_TAGS):
         return False
+    # A path whose every operation is marked internal has no public surface.
+    if not _has_public_operation(path_item):
+        return False
     return True
 
 
@@ -240,7 +298,7 @@ def transform(source: dict[str, Any]) -> dict[str, Any]:
 
     src_paths = source.get("paths") or {}
     kept_paths = {
-        path: item
+        path: strip_internal_operations(item)
         for path, item in src_paths.items()
         if isinstance(item, dict) and _should_keep_path(path, item)
     }