diff --git a/scripts/spark-authz-e2e-tests/.gitignore b/scripts/spark-authz-e2e-tests/.gitignore new file mode 100644 index 00000000..eb5097ea --- /dev/null +++ b/scripts/spark-authz-e2e-tests/.gitignore @@ -0,0 +1,5 @@ +config.json +venv/ +.pytest_cache/ +__pycache__/ +*.pyc diff --git a/scripts/spark-authz-e2e-tests/Makefile b/scripts/spark-authz-e2e-tests/Makefile new file mode 100644 index 00000000..460f22cf --- /dev/null +++ b/scripts/spark-authz-e2e-tests/Makefile @@ -0,0 +1,84 @@ +# Spark AuthZ E2E tests for scale-agentex. +# +# Target conventions +# ------------------ +# Targets are grouped so you can run a single test file, all tests for a +# resource, all tests for a logical category, or the whole suite. Future +# resources slot into one of these groups by adding test files and +# updating the matching aggregate target — no new flat targets unless +# the resource is novel. +# +# test all tests, every group +# test-direct-resources everything with its own SpiceDB type +# (agent, task, api_key, …) +# test-sub-resources everything that delegates to a parent +# (event, task_state, message, tracker, checkpoint) +# test- all cases for one resource +# (e.g. test-api-key, test-event) +# test-- one case for one resource +# (e.g. test-api-key-create) +# +# When a parent grouping makes sense (e.g. "all task sub-resources"), add +# a test--sub-resources target alongside the resource targets. + +VENV ?= venv +PYTHON ?= python3 +PIP := $(VENV)/bin/pip +PYTEST := $(VENV)/bin/pytest + +# Test-file globs. Update these (NOT individual case targets) when adding +# a new test file for an existing resource. +EVENT_TESTS := tests/test_event_authz.py + +# Aggregate groupings — extend these as resources are added. +# Direct-resource targets land with their own PRs (e.g. api_key). +SUB_RESOURCE_TESTS := $(EVENT_TESTS) +# Future sub-resource additions go here: +# SUB_RESOURCE_TESTS += $(STATE_TESTS) $(MESSAGE_TESTS) $(TRACKER_TESTS) $(CHECKPOINT_TESTS) + +.PHONY: install test \ + test-sub-resources \ + test-event \ + clean help + +help: + @echo "scale-agentex Spark AuthZ E2E tests" + @echo "" + @echo "Setup:" + @echo " make install venv + deps (one-time)" + @echo "" + @echo "Run everything:" + @echo " make test all tests" + @echo "" + @echo "Run a logical group:" + @echo " make test-sub-resources resources that delegate to a parent" + @echo "" + @echo "Run one resource:" + @echo " make test-event all AGX1-331 cases" + +install: + $(PYTHON) -m venv $(VENV) + $(PIP) install -r requirements.txt + +test: + $(PYTEST) tests/ -v + +# --------------------------------------------------------------------------- +# Logical groupings +# --------------------------------------------------------------------------- + +test-sub-resources: + $(PYTEST) $(SUB_RESOURCE_TESTS) -v + +# --------------------------------------------------------------------------- +# Sub-resources (delegate authz to a parent) +# --------------------------------------------------------------------------- + +# AGX1-331 — events delegate to parent agent +test-event: + $(PYTEST) $(EVENT_TESTS) -v + +clean: + rm -rf $(VENV) .pytest_cache __pycache__ + find . -name __pycache__ -type d -exec rm -rf {} + + find . -name '*.pyc' -delete diff --git a/scripts/spark-authz-e2e-tests/README.md b/scripts/spark-authz-e2e-tests/README.md new file mode 100644 index 00000000..e4ff3296 --- /dev/null +++ b/scripts/spark-authz-e2e-tests/README.md @@ -0,0 +1,183 @@ +# Spark AuthZ E2E tests + +End-to-end tests for FGAC on `scale-agentex` routes. Black-box: every test +hits the real `scale-agentex` HTTP API as one identity, then verifies the +resulting state in SpiceDB via a separate Spark-AuthZ client. + +Modeled after the equivalent KB suite in +`scaleapi/packages/egp-api-backend/scripts/spark-authz-e2e-tests/` (PR +[#142983](https://github.com/scaleapi/scaleapi/pull/142983)). + +## Scope + +This PR establishes the e2e test scaffolding (clients, conftest, factories, +cleanup) and ships the first resource: **events**. Subsequent PRs add new +resource test files on top of this infrastructure (e.g. AGX1-325 api_keys +in a follow-up). + +### AGX1-331 — `events` (read-only, parent-agent-delegated) + +Routes: `GET /events/{id}` and `GET /events?task_id=...&agent_id=...`. + +- Events have **no SpiceDB type of their own** — the check goes against the + parent `agent`. +- No public `POST /events` → the happy-path tests are skipped (see note in + `tests/test_event_authz.py`); only the denied paths are exercised, which + is what the ticket asks for. + +### Scaffolding shipped with this PR + +- `clients/agentex_client.py` — HTTP client for `scale-agentex` routes + (agent + api_key + event surfaces; api_key methods land here ahead of + AGX1-325's tests so the client doesn't grow in two places). +- `clients/spark_authz_client.py` — direct SpiceDB-state client (HTTP- + transcoded). Copied verbatim from the EGP suite — repo-agnostic. +- `conftest.py` — config loader, identity credentials, two `AgentexClient` + instances (user_a / user_b), a `SparkAuthzClient`, an `authz_reachable` + probe with graceful-skip semantics, `parent_agent` fixture that falls + back to a pre-existing `agentex.agent_id` when the test user lacks + `agent.create` on the tenant, function-scoped `cleanup` tracker. +- `helpers/cleanup.py` + `helpers/factories.py` — LIFO teardown + + unique-name generators. + +## Setup + +The suite does not spin up any services itself — it assumes the relevant +backends are already running (same model as the EGP suite this is mirrored +from). Three terminals + the test runner. + +### Terminal 1 — `spark-authz` (authz server + Identity Service + SpiceDB) + +```bash +cd ~/spark-authz +docker compose up +``` + +This brings up everything the authz layer needs in one shot: Postgres, +SpiceDB, Redis, schema migration, dev seed data, the `authz` server on +gRPC `50052` + HTTP `8090`, and `identity-service` on the port its compose +file binds. The suite talks to `localhost:8090` (HTTP-transcoded) for all +direct authz assertions. + +### Terminal 2 — `agentex-auth` (the principal-resolution proxy) + +```bash +cd ~/agentex/agentex-auth +# Start command depends on the repo's own dev-loop — see its README. +# Must be configured with IDENTITY_SERVICE_URL pointing at the one from +# Terminal 1, and SPARK_AUTHZ_URL pointing at localhost:8090. +``` + +`scale-agentex` forwards every request's headers to this service to resolve +the principal context (`user_id`, `service_account_id`, `account_id`). +Without it, `scale-agentex` 401s every request. + +### Terminal 3 — `scale-agentex` itself + +```bash +cd ~/scale-agentex/agentex +# uv run uvicorn ... — see agentex/Makefile for the exact dev target. +# Must be started with AGENTEX_AUTH_URL pointing at the agentex-auth from +# Terminal 2 (otherwise auth is bypassed and the assertions in this suite +# become meaningless). +``` + +### Terminal 4 — run the suite + +```bash +cd ~/scale-agentex/scripts/spark-authz-e2e-tests +make install # one-time: venv + deps +cp config.json.example config.json # one-time +# Edit config.json — fill in real headers + identity_ids + account_id +# (see "Auth model" below for what those need to be). +make test # all tests +# See `make help` for the full list of targets, including logical groups +# (test-sub-resources) and per-resource targets. +``` + +### Minting credentials + +The two `users` in `config.json` need to exist in Identity Service AND be +known to `agentex-auth`. The suite does **not** create them — they're +minted out-of-band, same as the EGP suite's `ssk_is_…` keys. Two paths: + +- **Dev cluster**: grab existing dev API keys / bearer tokens for two real + users in the same account and paste them in. Easiest if you have them. +- **Local stack**: use the seed identities that `spark-authz`'s + `authz-dev-seed` container creates, or mint fresh ones via the local + Identity Service after Terminal 1 comes up. + +`user_b` must **not** be pre-granted access to `user_a`'s resources — the +negative-path tests depend on user_b having no role on user_a's agent / +api_key by default. + +## Run + +Targets are grouped so you can run a single test file, all tests for one +resource, all tests in a logical category, or the whole suite. + +```bash +# Everything +make test + +# Logical groups +make test-sub-resources # resources that delegate to a parent (event, …) + +# One resource (all cases) +make test-event # AGX1-331 — all event cases +``` + +Adding a new resource? Add a `_TESTS` variable in the Makefile, +append it to `DIRECT_RESOURCE_TESTS` or `SUB_RESOURCE_TESTS`, and add a +`test-` target. See the existing entries for the shape. + +## When spark-authz isn't reachable + +Some environments (e.g. `sgp-pubsec-dev`) run scale-agentex without the +spark-authz HTTP frontend — there's raw SpiceDB but no `:8090` REST surface +for direct permission assertions. The suite degrades gracefully: + +- Tests that **only hit scale-agentex HTTP routes** (most of the suite) run + normally and assert on response codes + bodies. +- Tests that **assert directly against SpiceDB** skip with a clear reason + when `spark_authz.host` doesn't answer `/healthz` with 2xx. The event + suite this PR ships doesn't have any SpiceDB-asserting tests, but the + scaffolding is here so future resource PRs land cleanly. +- The factory cleanup falls back to REST-only when spark-authz isn't + reachable (no SpiceDB delete-resource call). Tuples may leak in this + mode, but routes are the unit under test. + +To run the SpiceDB-asserting tests, either point `spark_authz.host` at a +reachable spark-authz instance or set up a port-forward to one. + +## Layout + +``` +clients/ + agentex_client.py # httpx wrapper for /agents + /agent_api_keys + spark_authz_client.py # httpx wrapper for spark-authz REST +helpers/ + cleanup.py # LIFO cleanup tracker honoring config knobs + factories.py # unique_agent_name, unique_api_key_name +tests/ + test_event_authz.py # AGX1-331: GET /events/{id} + /events denied paths +conftest.py # config, identities, clients, factories, cleanup +config.json.example # template — copy to config.json and fill in +``` + +## Auth model + +`scale-agentex`'s middleware forwards request headers to `agentex-auth`, +which resolves them to a principal context (user_id, service_account_id, +account_id). The test client doesn't care what flavor of header the target +environment requires — drop whatever `agentex-auth` accepts into +`users..headers` in `config.json` and the client passes it through. + +## Cleanup model + +Every factory registers a teardown that **first** tries `DELETE` via the REST +route (exercises the dual-write deregister) and **then** issues +`SparkAuthzClient.delete_resource` as a fallback. The second call is +idempotent — `NOT_FOUND` is swallowed server-side — so it's safe to always +run, and it prevents owner-tuple leaks if the REST dual-write deregister +silently fails between tests. diff --git a/scripts/spark-authz-e2e-tests/clients/__init__.py b/scripts/spark-authz-e2e-tests/clients/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/scripts/spark-authz-e2e-tests/clients/agentex_client.py b/scripts/spark-authz-e2e-tests/clients/agentex_client.py new file mode 100644 index 00000000..3f3c48db --- /dev/null +++ b/scripts/spark-authz-e2e-tests/clients/agentex_client.py @@ -0,0 +1,169 @@ +"""REST client for scale-agentex. + +Thin httpx wrapper scoped to the endpoints AGX1-325 exercises: +agent (create / delete) + agent_api_keys (create / get / get-by-name / list / +delete / delete-by-name). Each method returns the raw httpx.Response so +callers can assert on status codes directly. + +Auth: scale-agentex's middleware forwards request headers to ``agentex-auth`` +for verification. Whatever ``agentex-auth`` accepts in the target environment +(an API key, a bearer token, etc.) gets passed through ``IdentityCredentials. +headers`` as-is. +""" + +import logging +from dataclasses import dataclass + +import httpx + +logger = logging.getLogger(__name__) + + +@dataclass(frozen=True) +class IdentityCredentials: + """Headers + account + user identifiers for one identity.""" + + headers: dict[str, str] + identity_id: str + account_id: str + subject_type: str = "identity" + + +class AgentexClient: + """HTTP client for one identity's perspective on scale-agentex.""" + + def __init__( + self, + base_url: str, + credentials: IdentityCredentials, + timeout: float = 30.0, + ) -> None: + self._base_url = base_url.rstrip("/") + self.credentials = credentials + self._client = httpx.Client( + base_url=self._base_url, + timeout=timeout, + headers=credentials.headers, + ) + + # ------------------------------------------------------------------ + # Agents (parent resource — api_keys hang off an agent) + # ------------------------------------------------------------------ + def create_agent(self, name: str) -> httpx.Response: + # ``POST /agents/register-build`` (not /register) — pre-deploy + # registration with no ACP URL or handshake. The agent lands in + # BUILD_ONLY status, which is enough for FGAC-resource tests: + # api_keys still hang off it, events still delegate to it. + # ``name`` must match ``^[a-z0-9-]+$`` (no underscores). + payload = { + "name": name, + "description": "E2E test agent for AGX1-325/331", + } + resp = self._client.post("/agents/register-build", json=payload) + logger.debug("create_agent %s -> %d", name, resp.status_code) + return resp + + def delete_agent(self, agent_id: str) -> httpx.Response: + resp = self._client.delete(f"/agents/{agent_id}") + logger.debug("delete_agent %s -> %d", agent_id, resp.status_code) + return resp + + # ------------------------------------------------------------------ + # Events (AGX1-331 — read-only; delegates authz to parent agent) + # ------------------------------------------------------------------ + def get_event(self, event_id: str) -> httpx.Response: + resp = self._client.get(f"/events/{event_id}") + logger.debug("get_event %s -> %d", event_id, resp.status_code) + return resp + + def list_events( + self, + task_id: str, + agent_id: str, + last_processed_event_id: str | None = None, + limit: int | None = None, + ) -> httpx.Response: + params: dict = {"task_id": task_id, "agent_id": agent_id} + if last_processed_event_id is not None: + params["last_processed_event_id"] = last_processed_event_id + if limit is not None: + params["limit"] = limit + resp = self._client.get("/events", params=params) + logger.debug( + "list_events task=%s agent=%s -> %d", + task_id, + agent_id, + resp.status_code, + ) + return resp + + # ------------------------------------------------------------------ + # API keys + # ------------------------------------------------------------------ + def create_api_key( + self, + agent_id: str, + name: str, + api_key_type: str = "external", + api_key: str | None = None, + ) -> httpx.Response: + payload: dict = { + "agent_id": agent_id, + "name": name, + "api_key_type": api_key_type, + } + if api_key is not None: + payload["api_key"] = api_key + resp = self._client.post("/agent_api_keys", json=payload) + logger.debug("create_api_key %s -> %d", name, resp.status_code) + return resp + + def get_api_key(self, api_key_id: str) -> httpx.Response: + resp = self._client.get(f"/agent_api_keys/{api_key_id}") + logger.debug("get_api_key %s -> %d", api_key_id, resp.status_code) + return resp + + def get_api_key_by_name( + self, + name: str, + agent_id: str, + api_key_type: str = "external", + ) -> httpx.Response: + params = {"agent_id": agent_id, "api_key_type": api_key_type} + resp = self._client.get(f"/agent_api_keys/name/{name}", params=params) + logger.debug("get_api_key_by_name %s -> %d", name, resp.status_code) + return resp + + def list_api_keys( + self, + agent_id: str, + limit: int = 50, + page_number: int = 1, + ) -> httpx.Response: + params = { + "agent_id": agent_id, + "limit": limit, + "page_number": page_number, + } + resp = self._client.get("/agent_api_keys", params=params) + logger.debug("list_api_keys agent=%s -> %d", agent_id, resp.status_code) + return resp + + def delete_api_key(self, api_key_id: str) -> httpx.Response: + resp = self._client.delete(f"/agent_api_keys/{api_key_id}") + logger.debug("delete_api_key %s -> %d", api_key_id, resp.status_code) + return resp + + def delete_api_key_by_name( + self, + name: str, + agent_id: str, + api_key_type: str = "external", + ) -> httpx.Response: + params = {"agent_id": agent_id, "api_key_type": api_key_type} + resp = self._client.delete(f"/agent_api_keys/name/{name}", params=params) + logger.debug("delete_api_key_by_name %s -> %d", name, resp.status_code) + return resp + + def close(self) -> None: + self._client.close() diff --git a/scripts/spark-authz-e2e-tests/clients/spark_authz_client.py b/scripts/spark-authz-e2e-tests/clients/spark_authz_client.py new file mode 100644 index 00000000..31c46913 --- /dev/null +++ b/scripts/spark-authz-e2e-tests/clients/spark_authz_client.py @@ -0,0 +1,219 @@ +"""REST client for Spark AuthZ (SpiceDB) via its HTTP-transcoded gRPC API. + +Production services often use native gRPC to Spark AuthZ; this suite uses the +HTTP-transcoded routes from the proto annotations instead so we can exercise +the same RPCs with httpx only (no generated stubs or extra wiring here). +""" + +import logging +from dataclasses import dataclass +from typing import Any + +import httpx + +logger = logging.getLogger(__name__) + +SUBJECT_TYPE_IDENTITY = "identity" +SUBJECT_TYPE_SERVICE = "service_identity" + + +@dataclass(frozen=True) +class SparkAuthzConfig: + host: str + use_tls: bool = False + + @property + def base_url(self) -> str: + scheme = "https" if self.use_tls else "http" + return f"{scheme}://{self.host}" + + +class SparkAuthzClient: + """HTTP client for the Spark AuthZ ResourceService REST API.""" + + def __init__(self, config: SparkAuthzConfig, timeout: float = 10.0) -> None: + self._client = httpx.Client( + base_url=config.base_url, + timeout=timeout, + headers={"Content-Type": "application/json"}, + ) + + def check_permission( + self, + subject_id: str, + resource_type: str, + resource_id: str, + permission: str, + subject_type: str = SUBJECT_TYPE_IDENTITY, + ) -> dict[str, bool]: + """Check one or more permissions. Returns {permission: bool} map.""" + payload = { + "subject_type": subject_type, + "subject_id": subject_id, + "permission": permission, + "resource_type": resource_type, + "resource_id": resource_id, + } + resp = self._client.post("/v1/resources/check/permission", json=payload) + resp.raise_for_status() + return resp.json().get("permissions", {}) + + def check_permission_bool( + self, + subject_id: str, + resource_type: str, + resource_id: str, + permission: str, + subject_type: str = SUBJECT_TYPE_IDENTITY, + ) -> bool: + """Convenience: check a single permission and return True/False.""" + perms = self.check_permission( + subject_id, resource_type, resource_id, permission, subject_type + ) + return perms.get(permission, False) + + def get_resource_access( + self, + resource_type: str, + resource_id: str, + subject_id: str, + subject_type: str = SUBJECT_TYPE_IDENTITY, + ) -> list[dict[str, str]]: + """Get the access list for a resource. Returns list of entries.""" + payload = { + "resource_type": resource_type, + "resource_id": resource_id, + "subject_type": subject_type, + "subject_id": subject_id, + } + resp = self._client.post("/v1/resources/access/list", json=payload) + resp.raise_for_status() + return resp.json().get("entries", []) + + def grant_access( + self, + resource_type: str, + resource_id: str, + subject_id: str, + relation: str, + grantee_id: str, + grantee_type: str = SUBJECT_TYPE_IDENTITY, + subject_type: str = SUBJECT_TYPE_IDENTITY, + ) -> httpx.Response: + """Grant a relation on a resource. Returns raw response for error checking.""" + payload = { + "resource_type": resource_type, + "resource_id": resource_id, + "subject_type": subject_type, + "subject_id": subject_id, + "relation": relation, + "grantee_type": grantee_type, + "grantee_id": grantee_id, + } + resp = self._client.post("/v1/resources/access/grant", json=payload) + logger.debug( + "grant_access %s:%s %s->%s -> %d", + resource_type, + resource_id, + relation, + grantee_id, + resp.status_code, + ) + return resp + + def revoke_access( + self, + resource_type: str, + resource_id: str, + subject_id: str, + grantee_id: str, + relation: str = "", + grantee_type: str = SUBJECT_TYPE_IDENTITY, + subject_type: str = SUBJECT_TYPE_IDENTITY, + ) -> httpx.Response: + """Revoke a relation (or all relations if relation is empty).""" + payload: dict[str, Any] = { + "resource_type": resource_type, + "resource_id": resource_id, + "subject_type": subject_type, + "subject_id": subject_id, + "grantee_type": grantee_type, + "grantee_id": grantee_id, + } + if relation: + payload["relation"] = relation + resp = self._client.post("/v1/resources/access/revoke", json=payload) + logger.debug( + "revoke_access %s:%s %s from %s -> %d", + resource_type, + resource_id, + relation or "(all)", + grantee_id, + resp.status_code, + ) + return resp + + def create_resource( + self, + resource_type: str, + resource_id: str, + subject_id: str, + tenant_id: str, + subject_type: str = SUBJECT_TYPE_IDENTITY, + ) -> httpx.Response: + """Create a resource in the authorization graph.""" + payload = { + "resource_type": resource_type, + "resource_id": resource_id, + "subject_type": subject_type, + "subject_id": subject_id, + "tenant_id": tenant_id, + } + resp = self._client.post("/v1/resources/create", json=payload) + logger.debug( + "create_resource %s:%s -> %d", resource_type, resource_id, resp.status_code + ) + return resp + + def delete_resource( + self, + resource_type: str, + resource_id: str, + subject_id: str, + subject_type: str = SUBJECT_TYPE_IDENTITY, + ) -> httpx.Response: + """Delete a resource from the authorization graph.""" + payload = { + "resource_type": resource_type, + "resource_id": resource_id, + "subject_type": subject_type, + "subject_id": subject_id, + } + resp = self._client.post("/v1/resources/delete", json=payload) + logger.debug( + "delete_resource %s:%s -> %d", resource_type, resource_id, resp.status_code + ) + return resp + + def lookup_resources( + self, + resource_type: str, + permission: str, + subject_id: str, + tenant_id: str, + subject_type: str = SUBJECT_TYPE_IDENTITY, + ) -> list[str]: + """Lookup all resource IDs accessible by a subject. Returns list of IDs.""" + payload = { + "resource_type": resource_type, + "permission": permission, + "subject_type": subject_type, + "subject_id": subject_id, + "tenant_id": tenant_id, + } + resp = self._client.post("/v1/resources/lookup", json=payload) + resp.raise_for_status() + return resp.json().get("resource_ids", []) + + def close(self) -> None: + self._client.close() diff --git a/scripts/spark-authz-e2e-tests/config.json.example b/scripts/spark-authz-e2e-tests/config.json.example new file mode 100644 index 00000000..f9fea2bf --- /dev/null +++ b/scripts/spark-authz-e2e-tests/config.json.example @@ -0,0 +1,32 @@ +{ + "agentex_api": { + "base_url": "https://scale-agentex.dev.scale.com" + }, + "spark_authz": { + "host": "spark-authz.dev.scale.com:443", + "use_tls": true + }, + "users": { + "user_a": { + "headers": { + "Authorization": "Bearer " + }, + "identity_id": "", + "account_id": "", + "subject_type": "identity" + }, + "user_b": { + "headers": { + "Authorization": "Bearer " + }, + "identity_id": "", + "account_id": "", + "subject_type": "identity" + } + }, + "test_settings": { + "request_timeout_seconds": 30, + "cleanup_on_success": true, + "cleanup_on_failure": true + } +} diff --git a/scripts/spark-authz-e2e-tests/conftest.py b/scripts/spark-authz-e2e-tests/conftest.py new file mode 100644 index 00000000..05b35bef --- /dev/null +++ b/scripts/spark-authz-e2e-tests/conftest.py @@ -0,0 +1,327 @@ +"""Root conftest — session-scoped clients and function-scoped factories. + +Fixtures: + - config (session) — parsed config.json + - authz_client (session) — SparkAuthzClient for direct SpiceDB calls + - agentex_client_a / _b (session) — Agentex REST clients per user identity + - user_a / user_b (session) — IdentityCredentials + - create_agent (function) — creates an agent as user_a and registers cleanup + - create_api_key (function) — creates an api_key as user_a under a given agent + - cleanup (function) — cleanup tracker honoring config knobs + +AGX1-325 scope: same-tenant user_a (owner) + user_b (no permission). No +cross-tenant or service identity here — those will be added if/when the +ticket grows to cover them. +""" + +import json +import logging +from collections.abc import Generator +from pathlib import Path + +import pytest +from clients.agentex_client import AgentexClient, IdentityCredentials +from clients.spark_authz_client import SparkAuthzClient, SparkAuthzConfig +from helpers.cleanup import CleanupTracker +from helpers.factories import unique_agent_name, unique_api_key_name + +logger = logging.getLogger(__name__) + +AGENT_RESOURCE_TYPE = "agent" +API_KEY_RESOURCE_TYPE = "api_key" + + +@pytest.hookimpl(hookwrapper=True, tryfirst=True) +def pytest_runtest_makereport(item, call): + """Attach phase reports to ``item`` so fixtures can read pass/fail in teardown.""" + outcome = yield + rep = outcome.get_result() + setattr(item, f"rep_{rep.when}", rep) + + +# --------------------------------------------------------------------------- +# Config +# --------------------------------------------------------------------------- + + +@pytest.fixture(scope="session") +def config() -> dict: + config_path = Path(__file__).parent / "config.json" + if not config_path.exists(): + pytest.skip("config.json not found. Copy config.json.example and configure.") + with open(config_path) as f: + return json.load(f) + + +# --------------------------------------------------------------------------- +# Identity credentials +# --------------------------------------------------------------------------- + + +def _make_credentials(config: dict, key: str) -> IdentityCredentials: + u = config["users"][key] + return IdentityCredentials( + headers=u["headers"], + identity_id=u["identity_id"], + account_id=u["account_id"], + subject_type=u.get("subject_type", "identity"), + ) + + +@pytest.fixture(scope="session") +def user_a(config) -> IdentityCredentials: + return _make_credentials(config, "user_a") + + +@pytest.fixture(scope="session") +def user_b(config) -> IdentityCredentials: + """Same-tenant user with no permission on user_a's resources by default.""" + if "user_b" not in (config.get("users") or {}): + pytest.skip( + "Add users.user_b to config.json (same account_id as user_a, distinct " + "identity_id) to run negative-permission tests." + ) + return _make_credentials(config, "user_b") + + +# --------------------------------------------------------------------------- +# Clients +# --------------------------------------------------------------------------- + + +@pytest.fixture(scope="session") +def agentex_client_a(config, user_a) -> Generator[AgentexClient, None, None]: + timeout = config.get("test_settings", {}).get("request_timeout_seconds", 30) + client = AgentexClient(config["agentex_api"]["base_url"], user_a, timeout=timeout) + yield client + client.close() + + +@pytest.fixture(scope="session") +def agentex_client_b(config, user_b) -> Generator[AgentexClient, None, None]: + timeout = config.get("test_settings", {}).get("request_timeout_seconds", 30) + client = AgentexClient(config["agentex_api"]["base_url"], user_b, timeout=timeout) + yield client + client.close() + + +def _build_authz_client(config) -> SparkAuthzClient: + authz_cfg = SparkAuthzConfig( + host=config["spark_authz"]["host"], + use_tls=config["spark_authz"].get("use_tls", False), + ) + return SparkAuthzClient(authz_cfg) + + +def _spark_authz_reachable(config) -> bool: + """Best-effort probe. Cached per-session via the calling fixture. + + Returns True only when /healthz returns a 2xx. ``httpx.get`` doesn't + raise on non-2xx by default, so without the explicit status-code check + a 503-Unavailable host would look healthy here; tests would then build + the client and fail with a confusing ``HTTPStatusError`` on the first + real call instead of skipping cleanly. + """ + import httpx + + authz_cfg = SparkAuthzConfig( + host=config["spark_authz"]["host"], + use_tls=config["spark_authz"].get("use_tls", False), + ) + timeout = config.get("test_settings", {}).get("authz_probe_timeout_seconds", 5) + try: + resp = httpx.get(f"{authz_cfg.base_url}/healthz", timeout=timeout) + except httpx.HTTPError: + return False + return resp.is_success + + +@pytest.fixture(scope="session") +def authz_reachable(config) -> bool: + """True iff the configured ``spark_authz.host`` answers /healthz.""" + return _spark_authz_reachable(config) + + +@pytest.fixture(scope="session") +def authz_client(config, authz_reachable) -> Generator[SparkAuthzClient, None, None]: + """Direct spark-authz client. **Skips the test** if the host is unreachable. + + Take this only in tests that assert against SpiceDB directly. Tests that + only exercise scale-agentex HTTP routes should NOT take this fixture — + they degrade gracefully via ``optional_authz_client`` in the factories. + + Not every environment runs spark-authz (e.g. pubsec-dev runs raw SpiceDB + without the spark-authz HTTP frontend). + """ + if not authz_reachable: + host = config["spark_authz"]["host"] + pytest.skip( + f"spark-authz unreachable at {host}. Tests that assert directly " + "against SpiceDB are skipped. To run them, point ``spark_authz.host`` " + "at a reachable spark-authz instance or set up a port-forward." + ) + client = _build_authz_client(config) + yield client + client.close() + + +@pytest.fixture(scope="session") +def optional_authz_client( + config, authz_reachable +) -> Generator[SparkAuthzClient | None, None, None]: + """Like ``authz_client`` but returns ``None`` when unreachable instead of + skipping. For cleanup-fallback paths that should still run when the + SpiceDB side isn't available. + """ + if not authz_reachable: + yield None + return + client = _build_authz_client(config) + yield client + client.close() + + +# --------------------------------------------------------------------------- +# Cleanup +# --------------------------------------------------------------------------- + + +@pytest.fixture() +def cleanup(request, config) -> Generator[CleanupTracker, None, None]: + """Function-scoped cleanup tracker. + + Honors ``test_settings.cleanup_on_success`` and ``cleanup_on_failure`` + in config.json. Defaults: cleanup after pass + after fail. + """ + tracker = CleanupTracker() + yield tracker + + ts = config.get("test_settings", {}) + on_success = ts.get("cleanup_on_success", True) + on_failure = ts.get("cleanup_on_failure", True) + + rep = getattr(request.node, "rep_call", None) + if rep is None: + tracker.execute() + return + + if rep.skipped: + if on_success: + tracker.execute() + return + if rep.failed: + if on_failure: + tracker.execute() + return + if on_success: + tracker.execute() + + +# --------------------------------------------------------------------------- +# Factories +# --------------------------------------------------------------------------- + + +@pytest.fixture() +def create_agent(agentex_client_a, optional_authz_client, user_a, cleanup): + """Factory: creates an agent as user_a and registers cleanup. + + Agents are the parent resource for api_keys; AGX1-325's authz checks + cascade from this agent's tuples. The SpiceDB cleanup fallback only + runs when spark-authz is reachable. + + Note: requires the caller to have ``agent.create`` on the tenant's + ``agent:*`` wildcard. In environments where the test user lacks that + permission (e.g. some shared dev clusters), use ``parent_agent`` + instead — it falls back to a pre-existing ``agentex.agent_id`` from + config.json. + """ + + def _create(name: str | None = None) -> tuple[str, str]: + agent_name = name or unique_agent_name() + resp = agentex_client_a.create_agent(agent_name) + assert resp.status_code in ( + 200, + 201, + ), f"Failed to create agent: {resp.status_code} {resp.text}" + agent_id = resp.json()["id"] + + def _teardown(): + delete_resp = agentex_client_a.delete_agent(agent_id) + if delete_resp.status_code not in (200, 204, 404): + logger.warning( + "REST delete_agent failed (%d); SpiceDB owner tuple may leak", + delete_resp.status_code, + ) + if optional_authz_client is not None: + optional_authz_client.delete_resource( + AGENT_RESOURCE_TYPE, agent_id, user_a.identity_id + ) + + cleanup.add(f"delete agent {agent_id}", _teardown) + return agent_id, agent_name + + return _create + + +@pytest.fixture() +def parent_agent(config, create_agent) -> tuple[str, str]: + """Provides a parent agent for resource-under-test creation. + + Resolution order: + 1. If ``agentex.agent_id`` is set in config.json, use that pre-existing + agent. Returns ``(agent_id, "")``. The user + must already have api_key.create permission on this agent; no + create-time authz is exercised in this mode. + 2. Otherwise, call ``create_agent()`` to mint a fresh one. Requires + ``agent.create`` permission on the tenant. + + Use this fixture in tests that just need *some* agent to attach + api_keys / events to. Take ``create_agent`` directly only when the + test specifically asserts something about agent creation itself. + """ + agentex_cfg = config.get("agentex") or {} + provided_id = agentex_cfg.get("agent_id") + if provided_id: + logger.info( + "Using pre-existing agent_id from config.json (skipping create_agent)" + ) + return provided_id, "" + return create_agent() + + +@pytest.fixture() +def create_api_key(agentex_client_a, optional_authz_client, user_a, cleanup): + """Factory: creates an api_key as user_a under the given agent. + + Returns ``(api_key_id, api_key_name, api_key_secret)``. Cleanup deletes + via REST first; the SpiceDB DeleteResource fallback runs only when + spark-authz is reachable. + """ + + def _create(agent_id: str, name: str | None = None) -> tuple[str, str, str]: + api_key_name = name or unique_api_key_name() + resp = agentex_client_a.create_api_key(agent_id=agent_id, name=api_key_name) + assert resp.status_code in ( + 200, + 201, + ), f"Failed to create api_key: {resp.status_code} {resp.text}" + body = resp.json() + api_key_id = body["id"] + api_key_secret = body["api_key"] + + def _teardown(): + delete_resp = agentex_client_a.delete_api_key(api_key_id) + if delete_resp.status_code not in (200, 204, 404): + logger.warning( + "REST delete_api_key returned %d", delete_resp.status_code + ) + if optional_authz_client is not None: + optional_authz_client.delete_resource( + API_KEY_RESOURCE_TYPE, api_key_id, user_a.identity_id + ) + + cleanup.add(f"delete api_key {api_key_id}", _teardown) + return api_key_id, api_key_name, api_key_secret + + return _create diff --git a/scripts/spark-authz-e2e-tests/helpers/__init__.py b/scripts/spark-authz-e2e-tests/helpers/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/scripts/spark-authz-e2e-tests/helpers/cleanup.py b/scripts/spark-authz-e2e-tests/helpers/cleanup.py new file mode 100644 index 00000000..b75a7b01 --- /dev/null +++ b/scripts/spark-authz-e2e-tests/helpers/cleanup.py @@ -0,0 +1,37 @@ +"""Cleanup utilities for Spark AuthZ E2E tests. + +Provides a context-manager style cleanup tracker so tests can register +resources for teardown. Cleanup runs in LIFO order and never raises — +failures are logged but don't mask test results. +""" + +import logging +from collections.abc import Callable +from dataclasses import dataclass + +logger = logging.getLogger(__name__) + + +@dataclass +class CleanupAction: + description: str + fn: Callable[[], None] + + +class CleanupTracker: + """Register cleanup actions that execute in LIFO order on close.""" + + def __init__(self) -> None: + self._actions: list[CleanupAction] = [] + + def add(self, description: str, fn: Callable[[], None]) -> None: + self._actions.append(CleanupAction(description=description, fn=fn)) + + def execute(self) -> None: + for action in reversed(self._actions): + try: + action.fn() + logger.debug("Cleanup OK: %s", action.description) + except Exception: + logger.warning("Cleanup failed: %s", action.description, exc_info=True) + self._actions.clear() diff --git a/scripts/spark-authz-e2e-tests/helpers/factories.py b/scripts/spark-authz-e2e-tests/helpers/factories.py new file mode 100644 index 00000000..d84ef7b7 --- /dev/null +++ b/scripts/spark-authz-e2e-tests/helpers/factories.py @@ -0,0 +1,12 @@ +"""Unique-name factories. Test runs collide on stale state otherwise.""" + +import time +import uuid + + +def unique_agent_name(prefix: str = "e2e-agent") -> str: + return f"{prefix}-{int(time.time())}-{uuid.uuid4().hex[:8]}" + + +def unique_api_key_name(prefix: str = "e2e-api-key") -> str: + return f"{prefix}-{int(time.time())}-{uuid.uuid4().hex[:8]}" diff --git a/scripts/spark-authz-e2e-tests/requirements.txt b/scripts/spark-authz-e2e-tests/requirements.txt new file mode 100644 index 00000000..03be0a7e --- /dev/null +++ b/scripts/spark-authz-e2e-tests/requirements.txt @@ -0,0 +1,2 @@ +httpx>=0.27.0 +pytest>=8.0.0 diff --git a/scripts/spark-authz-e2e-tests/tests/__init__.py b/scripts/spark-authz-e2e-tests/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/scripts/spark-authz-e2e-tests/tests/test_event_authz.py b/scripts/spark-authz-e2e-tests/tests/test_event_authz.py new file mode 100644 index 00000000..f40e0971 --- /dev/null +++ b/scripts/spark-authz-e2e-tests/tests/test_event_authz.py @@ -0,0 +1,96 @@ +"""AGX1-331 — Events: read-only, delegated to parent ``agent.read``. + +Scope from the ticket: + + > Event get/list delegated to parent agent view (read-only): no agent view, + > 404 on get and filtered/empty list + +Events have no public ``POST`` route in scale-agentex — they're emitted by +ACP streaming and persisted by the worker. So the happy-path side of this +suite ("with view, event is returned") would require either (a) direct DB +seeding, breaking the black-box property, or (b) running an actual agent. +Neither belongs in an e2e PR scoped to authz checks. + +What we CAN black-box, and what the ticket asks for: the denied-path +behavior on both routes. The "with view" happy path is left as a skipped +test below with a clear reason so that whoever later wires up an event- +seeding harness can flip the skip to a real assertion. +""" + +import pytest +from helpers.factories import unique_agent_name + + +@pytest.mark.e2e +class TestEventAuthz: + def test_get_event_nonexistent_returns_404(self, agentex_client_a): + """A get on a nonexistent event id 404s before any authz fires. + + Pure sanity for the route shape: the use case raises + ``ItemDoesNotExist`` on the repo lookup; the parent-agent check + never runs. + """ + resp = agentex_client_a.get_event("00000000-0000-0000-0000-000000000000") + assert ( + resp.status_code == 404 + ), f"expected 404 on nonexistent event id, got {resp.status_code}: {resp.text}" + + def test_list_events_without_agent_view_returns_404( + self, + parent_agent, + agentex_client_b, + ): + """user_b lacks ``read`` on user_a's agent → ``DAuthorizedQuery`` + denies the call → collapsed to 404 (not 403) so the agent's + existence isn't leakable. + """ + agent_id, _ = parent_agent + # Use an arbitrary task id; the DAuthorizedQuery on agent_id fires + # first and short-circuits before the task is even looked at. + denied = agentex_client_b.list_events( + task_id="00000000-0000-0000-0000-000000000001", + agent_id=agent_id, + ) + assert denied.status_code == 404, ( + f"expected 404 (collapsed from denied), got {denied.status_code}: " + f"{denied.text}" + ) + + def test_list_events_denied_on_both_query_params_returns_404( + self, + agentex_client_b, + ): + """When user_b is denied on both ``task_id`` and ``agent_id``, the + route collapses to 404. This verifies the route is gated end-to-end + but does NOT isolate which gate fired: FastAPI evaluates the + ``task_id`` ``Depends`` first, so the ``agent_id`` gate never + executes here. Isolating each gate independently would require + granting one resource but not the other in SpiceDB, which depends + on a reachable spark-authz (see ``authz_client`` skip behavior). + """ + resp = agentex_client_b.list_events( + task_id="00000000-0000-0000-0000-000000000002", + agent_id="00000000-0000-0000-0000-000000000003", + ) + assert resp.status_code == 404, ( + f"expected 404 (collapsed from denied), got {resp.status_code}: " + f"{resp.text}" + ) + + @pytest.mark.skip( + reason=( + "Black-box event seeding isn't possible — no public POST /events " + "and the ACP-stream path requires running an agent. Wire this up " + "once a test-only seeding helper exists (Linear: TODO follow-up)." + ) + ) + def test_get_event_with_view_returns_200(self, create_agent, agentex_client_a): + """Happy path: user_a has ``read`` on the parent agent → ``GET + /events/{id}`` returns 200 and the event payload. + """ + agent_id, _ = create_agent(name=unique_agent_name(prefix="agx1-331-happy")) + # event_id = (agent_id=agent_id) + # resp = agentex_client_a.get_event(event_id) + # assert resp.status_code == 200 + # assert resp.json()["agent_id"] == agent_id + pytest.fail("seeding helper not implemented")