feat(webapp): enforce scopes for environment API keys (#4389)

## Summary

Environment API keys backed by the additional-key table can authenticate
API requests using their stored effective scopes. Revoked and expired
keys are rejected, branch environments retain their existing routing
behavior, and last-used timestamps are updated on a throttled
best-effort basis.

## Design

API route builders receive the resolved ability and reject restricted
keys on routes without an authorization declaration. Existing
deployment, environment variable, queue, run, task, batch, session, and
waitpoint routes declare the resources they access.

Trigger and batch responses return server-signed public access tokens,
so additional keys never need access to the environment signing secret.
Root-key rotation also keeps public tokens valid for the existing grace
window.

## Feature notes
- Root environment keys remain unrestricted for backward compatibility.
Additional keys enforce their persisted scopes and fail closed on routes
   without an authorization declaration.
- Machine-key requests never exchange one credential for another.
Additional keys cannot retrieve the root key, and rotated root keys are
not upgraded
   during their grace window.
- Public JWT validation remains host-owned, while installed RBAC plugins
continue to supply root-key abilities.
- Unfiltered session and run listings preserve existing broad task-read
behavior. Filtered requests enforce the supplied task identifiers.
- Related-run summaries remain embedded in run retrieval for API
compatibility. Retrieving or mutating a related run independently still
requires
   permission for that run.
- Queue management authorizes at collection scope, matching the queue
permissions currently issued.
- Batch responses deliberately include server-signed public access
tokens for all clients. Selected-task credentials continue using their
original
   credential for per-item authorization.
- Two-phase batches authorize declared task identifiers before creation
and authorize every streamed item. Streaming paths that cannot declare
the
   complete task set remain fail closed.
- Authentication telemetry records successful credential resolution
separately from subsequent resource-authorization failures.
- API keys are high-entropy random tokens. SHA-256 is intentionally used
for deterministic indexed lookup, not password hashing.

## Deployment notes

The schema migration must be present before this code is deployed.
Because bearer resolution runs on every authenticated request, deploy
the resolver with additional-key lookup disabled, verify root-key and
public-token parity, then enable lookup before any additional keys can
be issued.

The multi-task authorization tightening changes the result for narrowly
scoped tokens that request tasks outside their grants. Observe
would-deny results before enforcing that check. Request-idempotency keys
are also newly isolated by environment and task, so a retry crossing the
deployment boundary may execute once more before old cache entries
expire.

## Follow-ups

- [x] Add a system-wide kill switch for additional-key lookup, defaulted
off for the initial deployment.
- [x] Add authentication observability by credential kind, result,
latency, and lookup path without recording credential values.
- [ ] ~Add would-deny observability and an independent enforcement
switch for multi-task authorization.~
- [ ] ~Add an independent switch for server-issued batch tokens while
root-key parity is verified.~
- [ ] Confirm every API route reachable by a restricted key has an
explicit authorization declaration or intentionally fails closed.
- [x] Verify root-key rotation, revoked-key grace, and public-token
validation through each bearer resolver path.

Author: carderne

.changeset/friendly-keys-batch-tokens.md

@@ -0,0 +1,6 @@
+---
+"@trigger.dev/core": patch
+"@trigger.dev/sdk": patch
+---
+
+Allow task-scoped environment API keys to run batch operations for their permitted tasks. The SDK declares the batch's task set before creation, and `@trigger.dev/core/v3/apiKeys` now exports the additional-key format helper.

.server-changes/multi-task-scope-enforcement.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: fix
+---
+
+Requests spanning multiple tasks now require permission for every requested task instead of accepting permission for only one task.

.server-changes/scoped-environment-api-key-auth.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: feature
+---
+
+Additional environment API keys can authenticate API requests using their configured permissions, with revoked and expired keys rejected. Batch responses use server-issued public access tokens so additional keys never need the environment signing secret.

apps/webapp/app/models/runtimeEnvironment.server.ts

@@ -5,7 +5,11 @@ import { runStore } from "~/v3/runStore.server";
 import { controlPlaneResolver } from "~/v3/runOpsMigration/controlPlaneResolver.server";
 import { logger } from "~/services/logger.server";
 import { getUsername } from "~/utils/username";
+import { hashApiKey } from "~/utils/apiKeys";
+import { isAdditionalApiKey } from "@trigger.dev/core/v3/apiKeys";
 import { isDefaultDevBranch, sanitizeBranchName } from "@trigger.dev/core/v3/utils/gitBranch";
+import { scopesGrantFullAccess } from "@trigger.dev/rbac";
+import { authFeatureControls } from "~/services/authFeatureControls.server";
 
 export type { RuntimeEnvironment };
 
@@ -93,11 +97,22 @@ export function toAuthenticated(
   };
 }
 
-export async function findEnvironmentByApiKey(
+export type ApiKeyEnvironmentResolution =
+  | { ok: true; environment: AuthenticatedEnvironment }
+  | { ok: false; reason: "not-found" | "restricted" | "disabled" };
+
+/**
+ * Resolve an environment from a raw API key for legacy routes that do not
+ * declare authorization. Additional keys are accepted only when their stored
+ * scopes explicitly grant full access; restricted keys fail closed here
+ * (`reason: "restricted"`, so callers can explain the rejection).
+ */
+async function resolveEnvironmentByApiKey(
   apiKey: string,
   branchName: string | undefined,
-  tx: PrismaClientOrTransaction = $replica
-): Promise<AuthenticatedEnvironment | null> {
+  tx: PrismaClientOrTransaction,
+  additionalApiKeyLookupEnabled: () => boolean
+): Promise<ApiKeyEnvironmentResolution> {
   const branch = sanitizeBranchName(branchName) ?? undefined;
 
   const include = {
@@ -112,80 +127,178 @@ export async function findEnvironmentByApiKey(
       : undefined,
   } satisfies Prisma.RuntimeEnvironmentInclude;
 
-  let environment = await tx.runtimeEnvironment.findFirst({
-    where: {
-      apiKey,
-    },
-    include,
-  });
+  const now = new Date();
+  const routesToAdditionalKey = isAdditionalApiKey(apiKey);
+  if (routesToAdditionalKey && !additionalApiKeyLookupEnabled()) {
+    return { ok: false, reason: "disabled" };
+  }
 
-  // Fall back to keys that were revoked within the grace window
-  if (!environment) {
+  let rootEnvironment = routesToAdditionalKey
+    ? null
+    : await tx.runtimeEnvironment.findFirst({
+        where: {
+          apiKey,
+        },
+        include,
+      });
+
+  // Fall back to root keys that were rotated within the grace window.
+  if (!routesToAdditionalKey && !rootEnvironment) {
     const revokedApiKey = await tx.revokedApiKey.findFirst({
       where: {
         apiKey,
-        expiresAt: { gt: new Date() },
+        expiresAt: { gt: now },
       },
       include: {
         runtimeEnvironment: { include },
       },
     });
 
-    environment = revokedApiKey?.runtimeEnvironment ?? null;
+    rootEnvironment = revokedApiKey?.runtimeEnvironment ?? null;
   }
 
+  // Additional keys are host-owned credentials. Legacy routes cannot apply a
+  // scoped ability, so only an explicit full-access scope is accepted.
+  const match = routesToAdditionalKey
+    ? await tx.apiKey.findFirst({
+        where: {
+          keyHash: hashApiKey(apiKey),
+          revokedAt: null,
+          OR: [{ expiresAt: null }, { expiresAt: { gt: now } }],
+        },
+        select: {
+          id: true,
+          lastUsedAt: true,
+          scopes: true,
+          runtimeEnvironment: { include },
+        },
+      })
+    : null;
+
+  if (match && !scopesGrantFullAccess(match.scopes)) {
+    return { ok: false, reason: "restricted" };
+  }
+
+  const additionalApiKey = match ? { id: match.id, lastUsedAt: match.lastUsedAt } : null;
+  let environment = rootEnvironment ?? match?.runtimeEnvironment ?? null;
+
   if (!environment) {
-    return null;
+    return { ok: false, reason: "not-found" };
+  }
+
+  if (
+    additionalApiKey &&
+    (!additionalApiKey.lastUsedAt ||
+      additionalApiKey.lastUsedAt < new Date(now.getTime() - 300_000))
+  ) {
+    try {
+      // Deliberately the primary `prisma`, not `tx`: `tx` defaults to the
+      // read replica (and may be a caller's transaction), and this last-used
+      // telemetry write must hit the writer. It's throttled to once every 5
+      // minutes per key and best-effort — auth never fails if it can't record.
+      await prisma.apiKey.updateMany({
+        where: {
+          id: additionalApiKey.id,
+          revokedAt: null,
+          OR: [{ expiresAt: null }, { expiresAt: { gt: now } }],
+        },
+        data: { lastUsedAt: now },
+      });
+    } catch (error) {
+      logger.warn("Failed to update API key last-used timestamp", {
+        apiKeyId: additionalApiKey.id,
+        error,
+      });
+    }
   }
 
   //don't return deleted projects
   if (environment.project.deletedAt !== null) {
-    return null;
+    return { ok: false, reason: "not-found" };
   }
 
   if (environment.type === "PREVIEW") {
     if (!branch) {
       logger.warn("findEnvironmentByApiKey(): Preview env with no branch name provided", {
         environmentId: environment.id,
       });
-      return null;
+      return { ok: false, reason: "not-found" };
     }
 
     const childEnvironment = environment.childEnvironments.at(0);
 
     if (childEnvironment) {
-      return toAuthenticated({
-        ...childEnvironment,
-        apiKey: environment.apiKey,
-        orgMember: environment.orgMember,
-        organization: environment.organization,
-        project: environment.project,
-      });
+      return {
+        ok: true,
+        environment: toAuthenticated({
+          ...childEnvironment,
+          apiKey: environment.apiKey,
+          orgMember: environment.orgMember,
+          organization: environment.organization,
+          project: environment.project,
+        }),
+      };
     }
 
     //A branch was specified but no child environment was found
-    return null;
+    return { ok: false, reason: "not-found" };
   }
 
   // If there is a named DEV branch (other than default), return it
   if (environment.type === "DEVELOPMENT" && branch !== undefined && !isDefaultDevBranch(branch)) {
     const childEnvironment = environment.childEnvironments.at(0);
 
     if (childEnvironment) {
-      return toAuthenticated({
-        ...childEnvironment,
-        apiKey: environment.apiKey,
-        orgMember: environment.orgMember,
-        organization: environment.organization,
-        project: environment.project,
-      });
+      return {
+        ok: true,
+        environment: toAuthenticated({
+          ...childEnvironment,
+          apiKey: environment.apiKey,
+          orgMember: environment.orgMember,
+          organization: environment.organization,
+          project: environment.project,
+        }),
+      };
     }
 
     //A branch was specified but no child environment was found
-    return null;
+    return { ok: false, reason: "not-found" };
   }
 
-  return toAuthenticated(environment);
+  return { ok: true, environment: toAuthenticated(environment) };
+}
+
+/**
+ * Resolve an environment from a raw API key. Root and grace-window keys keep
+ * their legacy behavior; additional keys with restricted scopes fail closed.
+ */
+export async function findEnvironmentByApiKey(
+  apiKey: string,
+  branchName: string | undefined,
+  tx: PrismaClientOrTransaction = $replica,
+  additionalApiKeyLookupEnabled = authFeatureControls.additionalApiKeyLookupEnabled
+): Promise<AuthenticatedEnvironment | null> {
+  const resolution = await resolveEnvironmentByApiKey(
+    apiKey,
+    branchName,
+    tx,
+    additionalApiKeyLookupEnabled
+  );
+  return resolution.ok ? resolution.environment : null;
+}
+
+/**
+ * Like `findEnvironmentByApiKey`, but distinguishes a restricted additional
+ * key (fails closed on legacy routes) from an unknown key so callers can
+ * return an accurate error message.
+ */
+export async function findEnvironmentByApiKeyWithResolution(
+  apiKey: string,
+  branchName: string | undefined,
+  tx: PrismaClientOrTransaction = $replica,
+  additionalApiKeyLookupEnabled = authFeatureControls.additionalApiKeyLookupEnabled
+): Promise<ApiKeyEnvironmentResolution> {
+  return resolveEnvironmentByApiKey(apiKey, branchName, tx, additionalApiKeyLookupEnabled);
 }
 
 /**

apps/webapp/app/presenters/v3/ApiRetrieveRunPresenter.server.ts

@@ -289,6 +289,8 @@ export class ApiRetrieveRunPresenter {
         attemptCount:
           taskRun.engine === "V1" ? taskRun.attempts.length : (taskRun.attemptNumber ?? 0),
         attempts: [],
+        // Related runs are an embedded projection of the authorized run, not independent reads.
+        // Preserve the established response shape for run-scoped credentials.
         relatedRuns: {
           root: taskRun.rootTaskRun
             ? await createCommonRunStructure(taskRun.rootTaskRun, this.apiVersion)
@@ -297,7 +299,9 @@ export class ApiRetrieveRunPresenter {
             ? await createCommonRunStructure(taskRun.parentTaskRun, this.apiVersion)
             : undefined,
           children: await Promise.all(
-            taskRun.childRuns.map(async (r) => await createCommonRunStructure(r, this.apiVersion))
+            taskRun.childRuns.map(
+              async (run) => await createCommonRunStructure(run, this.apiVersion)
+            )
           ),
         },
       };

apps/webapp/app/routes/api.v1.artifacts.ts

@@ -4,7 +4,7 @@ import {
   CreateArtifactRequestBody,
   tryCatch,
 } from "@trigger.dev/core/v3";
-import { authenticateRequest } from "~/services/apiAuth.server";
+import { authenticateApiKeyWithScope } from "~/services/apiAuth.server";
 import { logger } from "~/services/logger.server";
 import { ArtifactsService } from "~/v3/services/artifacts.server";
 
@@ -14,17 +14,19 @@ export async function action({ request }: ActionFunctionArgs) {
   }
 
   try {
-    const authenticationResult = await authenticateRequest(request, {
-      apiKey: true,
-      organizationAccessToken: false,
-      personalAccessToken: false,
+    // Artifact uploads are part of the deploy flow (deployment context archive).
+    const authResult = await authenticateApiKeyWithScope(request, {
+      action: "write",
+      resource: { type: "deployments" },
     });
 
-    if (!authenticationResult || !authenticationResult.result.ok) {
+    if (!authResult.ok) {
       logger.info("Invalid or missing api key", { url: request.url });
-      return json({ error: "Invalid or Missing API key" }, { status: 401 });
+      return json({ error: authResult.error }, { status: authResult.status });
     }
 
+    const authenticationResult = { result: authResult.authentication };
+
     const [, rawBody] = await tryCatch(request.json());
     const body = CreateArtifactRequestBody.safeParse(rawBody ?? {});
 

apps/webapp/app/routes/api.v1.deployments.$deploymentId.background-workers.ts

@@ -2,7 +2,7 @@ import type { ActionFunctionArgs } from "@remix-run/server-runtime";
 import { json } from "@remix-run/server-runtime";
 import { CreateBackgroundWorkerRequestBody } from "@trigger.dev/core/v3";
 import { z } from "zod";
-import { authenticateApiRequest } from "~/services/apiAuth.server";
+import { authenticateApiKeyWithScope } from "~/services/apiAuth.server";
 import { logger } from "~/services/logger.server";
 import { ServiceValidationError } from "~/v3/services/baseService.server";
 import { CreateDeclarativeScheduleError } from "~/v3/services/createBackgroundWorker.server";
@@ -26,13 +26,18 @@ export async function action({ request, params }: ActionFunctionArgs) {
 
   try {
     // Next authenticate the request
-    const authenticationResult = await authenticateApiRequest(request);
+    const authResult = await authenticateApiKeyWithScope(request, {
+      action: "write",
+      resource: { type: "deployments" },
+    });
 
-    if (!authenticationResult) {
+    if (!authResult.ok) {
       logger.info("Invalid or missing api key", { url: request.url });
-      return json({ error: "Invalid or Missing API key" }, { status: 401 });
+      return json({ error: authResult.error }, { status: authResult.status });
     }
 
+    const authenticationResult = authResult.authentication;
+
     const authenticatedEnv = authenticationResult.environment;
 
     const { deploymentId } = parsedParams.data;

apps/webapp/app/routes/api.v1.deployments.$deploymentId.cancel.ts

@@ -1,7 +1,7 @@
 import { type ActionFunctionArgs, json } from "@remix-run/server-runtime";
 import { CancelDeploymentRequestBody, tryCatch } from "@trigger.dev/core/v3";
 import { z } from "zod";
-import { authenticateRequest } from "~/services/apiAuth.server";
+import { authenticateApiKeyWithScope } from "~/services/apiAuth.server";
 import { logger } from "~/services/logger.server";
 import { DeploymentService } from "~/v3/services/deployment.server";
 
@@ -21,18 +21,17 @@ export async function action({ request, params }: ActionFunctionArgs) {
   }
 
   try {
-    const authenticationResult = await authenticateRequest(request, {
-      apiKey: true,
-      organizationAccessToken: false,
-      personalAccessToken: false,
+    const authResult = await authenticateApiKeyWithScope(request, {
+      action: "write",
+      resource: { type: "deployments" },
     });
 
-    if (!authenticationResult || !authenticationResult.result.ok) {
+    if (!authResult.ok) {
       logger.info("Invalid or missing api key", { url: request.url });
-      return json({ error: "Invalid or Missing API key" }, { status: 401 });
+      return json({ error: authResult.error }, { status: authResult.status });
     }
 
-    const { environment: authenticatedEnv } = authenticationResult.result;
+    const { environment: authenticatedEnv } = authResult.authentication;
     const { deploymentId } = parsedParams.data;
 
     const [, rawBody] = await tryCatch(request.json());