Skip to content

Integrate DataOps with the shared Cognito browser login #119

Description

@alexeygrigorev

Integrate DataOps with the shared Cognito browser login

Status: in progress
Tags: enhancement, portal, frontend, backend, infra, testing, P1
Depends on: None
Blocks: None

Scope

Replace the production DataOps browser gate at ops.dtcdev.click with the shared serverless authentication service at https://auth.dtcdev.click. The DataOps Lambda remains the relying party: it starts Cognito authorization-code + S256 PKCE, handles https://ops.dtcdev.click/auth/callback, validates the returned identity, maps it to an existing enabled DataOps user by normalized email, and creates an opaque server-side DataOps session carried in a secure cookie.

The shared stack is already deployed as dtcdev-shared-auth in us-east-1. Its DataOps public client is registered for:

  • callback: https://ops.dtcdev.click/auth/callback
  • logout return: https://ops.dtcdev.click/
  • flow/scopes: authorization code with openid email profile
  • issuer: the UserPoolId/IssuerUrl output of that stack

The DataOps application and stack remain in this repository and deploy as dataops-v1 in eu-west-1 through .github/workflows/deploy-dataops-v1.yml with GitHub Actions OIDC. Shared Cognito ownership remains in ../aws-infra/sandbox/auth; this issue only consumes its non-secret contract.

This migration replaces browser Basic auth and the password-derived dtc_auth cookie. It must not weaken or silently replace the independent credentials used by machine-to-machine intake, integrations, webhooks, or existing bearer-session clients. Route-specific layered authorization remains route-specific.

Acceptance Criteria

  • GET /login starts authorization at https://auth.dtcdev.click/oauth2/authorize using the configured DataOps client, exact callback URL, response_type=code, openid email profile, an unpredictable state and nonce, and an S256 PKCE challenge. OAuth transaction state is short-lived, one-time, integrity-protected, and never stored in browser localStorage.
  • GET /auth/callback rejects missing, expired, replayed, or mismatched state; OAuth provider errors; missing codes; failed token exchanges; and invalid tokens without creating a DataOps session.
  • The callback exchanges the code at /oauth2/token with the original PKCE verifier and validates the ID token signature against the issuer JWKS plus issuer, DataOps client audience, expiry/not-before, nonce, email, and email_verified. JWKS caching handles key rotation safely. Tokens, authorization codes, PKCE verifiers, and session identifiers are absent from response bodies, URLs after callback completion, and application logs.
  • A successful callback normalizes the verified email and maps it to exactly one existing DataOps user. Unknown or disabled users receive a non-sensitive 403 response and are not auto-provisioned. The real local user ID is propagated to /api/me, role checks, assignment/notification filtering, and audit fields instead of the current portal-admin identity.
  • A successful callback creates an opaque, cryptographically random, server-side session with a bounded absolute expiry. The browser receives only an HttpOnly; Secure; SameSite=Lax; Path=/ cookie; expired, deleted, fabricated, and revoked sessions are rejected. OAuth-created expiry enforcement must not silently invalidate or change the contract of existing non-browser bearer sessions.
  • An authenticated browser session serves the portal, docs/content routes, and same-origin /api/* calls without Basic credentials or a browser-managed bearer token. Unauthenticated page requests redirect to /login; unauthenticated data/API requests return 401 without HTML or protected data.
  • GET or POST /logout deletes the local server-side session, expires the cookie, and uses the Cognito logout endpoint with the registered DataOps logout URI. A logged-out cookie cannot be reused.
  • Production no longer serves the DataOps username/password form, accepts Basic auth as a browser-session substitute, or requires BASIC_AUTH_USERNAME / the Basic-auth password secret at Lambda runtime. The legacy /api/auth/login endpoint remains unavailable in production portal mode. Deleting historical Secrets Manager resources is not required.
  • Existing dedicated machine credentials and entry points keep their current isolation and authorization semantics, including email-document intake, bookkeeping ingestion/layered authorization, webhook/integration credentials, and existing bearer-session clients. A machine credential alone never becomes a browser session, and browser OAuth alone does not bypass a route's additional credential.
  • infra/template.full.yaml exposes the shared auth base URL, issuer/user-pool contract, DataOps public client ID, callback URL, logout URL, and session-lifetime configuration as non-secret deployment configuration. The Lambda gains only permissions actually required by the implementation; no Cognito client secret or Google secret is copied into this repository or GitHub Actions.
  • .github/workflows/deploy-dataops-v1.yml passes the auth configuration and continues to validate/build/deploy dataops-v1 exclusively through GitHub Actions OIDC. The workflow does not deploy or mutate dtcdev-shared-auth, and the application is not manually deployed for this issue.
  • Automated tests cover authorization parameter construction, PKCE/state/nonce lifecycle and replay rejection, token/JWKS validation failures and key refresh, local-user mapping (enabled/disabled/unknown/case normalization), session creation/expiry/logout, page-vs-API unauthenticated behavior, and authenticated /api/me/role behavior. Token and JWKS HTTP calls are stubbed; automated tests do not require a live Cognito account.
  • Regression tests prove each existing machine credential path and existing bearer-session path retains its prior allow/deny boundary, including invalid/missing credential cases and routes with layered credentials.
  • Tester runs npm --prefix backend test, npm --prefix backend run typecheck, npm --prefix backend run build, npm --prefix backend run test:e2e, make sam-validate, and make sam-build, reporting exact commands, exit codes, and test counts. Playwright captures and inspects screenshots for the signed-out entry/error state and the authenticated DataOps workspace; no screenshot may show a 404, error page, broken layout, overlap, or leaked credential.
  • [HUMAN] After deployment, an administrator-created Cognito username/password user can complete the real hosted-login flow, return to https://ops.dtcdev.click/auth/callback, use the portal as the matching DataOps user, log out, and fail to reuse the old session. Capture the tested identity, time, browser result, and screenshot without recording credentials or tokens.
  • [HUMAN] After the shared stack owner enables Google, a real @datatalks.club Google account can sign in and a non-@datatalks.club Google account is rejected. Record the result without exposing credentials or tokens. Google is enabled on the shared user pool; this real-account allow/deny check keeps the issue open with the human label until a human verifies both account classes.

Test Scenarios

Scenario: Signed-out operator starts secure login

Given: No valid DataOps session cookie exists
When: The operator opens a protected portal page
Then: DataOps redirects through /login to the shared authorize endpoint with fresh state, nonce, and S256 PKCE while an API request receives JSON 401

Scenario: Valid shared identity becomes the correct local operator

Given: Cognito returns a valid token for an enabled DataOps user email
When: The callback validates and exchanges the authorization code
Then: DataOps creates a bounded opaque session, redirects to a clean same-origin page, /api/me returns that local user, and authorization/audit behavior uses that user's ID and role

Scenario: Callback tampering or replay fails closed

Given: State, nonce, PKCE data, token claims, signature, or transaction expiry is invalid, or a completed callback is replayed
When: The callback is requested
Then: No session is created, a safe error is returned, and no code/token/verifier/session value is logged or rendered

Scenario: Unknown or disabled shared identity is denied

Given: A valid Cognito identity has no unique enabled DataOps user mapping
When: The callback completes
Then: DataOps returns 403 without auto-provisioning or revealing whether another account exists

Scenario: Session expiry and logout revoke access

Given: An operator has a valid OAuth-created DataOps session
When: The session expires or the operator logs out
Then: The server-side session and cookie no longer authorize portal or API access and Cognito logout returns to the registered DataOps URI

Scenario: Non-browser credentials retain their boundaries

Given: Existing intake, integration, webhook, layered, and bearer-session clients
When: They call their current routes with valid, missing, and invalid credentials
Then: Their pre-migration allow/deny behavior is preserved, additional credentials are still enforced, and none of those credentials grants browser access

Scenario: CI deploys the relying-party configuration

Given: The auth change reaches main
When: the DataOps deployment workflow runs
Then: tests and SAM validation/build pass, GitHub OIDC deploys dataops-v1 in eu-west-1, and the workflow does not mutate the shared auth stack in us-east-1

Out of Scope

  • Creating, redeploying, or changing the shared Cognito stack in ../aws-infra, including the Dapier, Datamailer, AI Engineering Gym, and AI System Design Studio clients.
  • Creating Google OAuth credentials or enabling the Google identity provider; those require the shared-stack owner and real external credentials.
  • Auto-provisioning DataOps users from Cognito, synchronizing Cognito groups to DataOps roles, or building user lifecycle administration across services.
  • Migrating existing DataOps password hashes into Cognito or deleting historical Basic-auth secrets from AWS Secrets Manager.
  • Replacing route-specific API keys, webhook signatures, integration credentials, or bearer-session APIs with browser OAuth.
  • Redesigning the DataOps workspace beyond the login, callback, logout, and authentication error states needed for this migration.

Dependencies

  • External contract (already available): CloudFormation stack dtcdev-shared-auth in us-east-1, auth base https://auth.dtcdev.click, its IssuerUrl/JWKS outputs, and the DataOps public client registered for https://ops.dtcdev.click/auth/callback.
  • Deployment contract: dataops-v1 remains in eu-west-1 and is deployed by .github/workflows/deploy-dataops-v1.yml through arn:aws:iam::817685572750:role/dataops-github-actions-deploy.
  • Local identity contract: DataOps users, roles, and disabled status in the existing users table remain the authorization source after Cognito proves identity.
  • Google federation is enabled on the shared user pool. The Dapier, DataOps, AI Engineering Gym, AI System Design Studio, and Datamailer clients all support both COGNITO and Google. Infrastructure enablement is complete; the remaining dependency is the unchecked [HUMAN] live allow/deny verification with real in-domain and out-of-domain Google accounts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1ImportantbackendBackend/APIenhancementNew or improved functionalityfrontendFrontend UIhumanCode done or issue blocked on human verificationinfraDeployment and infrastructureportalShared portal shell and UXtestingTests and QA

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions