Read Dapier-managed Mailchimp credential from DynamoDB

[alexeygrigorev]

Read Dapier-managed Mailchimp credential from DynamoDB

Status: pending
Tags: enhancement, backend, infra, data, P1
Depends on: Dapier CredentialsTable deployment/export and a populated mailchimp record (external repository)
Blocks: None

Scope

Replace only the mailing-list export service's Mailchimp Secrets Manager lookup with a consistent GetItem from the dedicated DynamoDB credentials table owned and populated by Dapier. DynamoDB's normal server-side encryption at rest is the protection mechanism: DataOps does not perform client-side encryption/decryption and does not call KMS or another credential service for this value.

The cross-repository contract is:

• DataOps receives the Dapier credentials table name and ARN as deployment parameters; it does not create, migrate, scan, or write the table.
• The table partition key is credential_id (String).
• The Mailchimp item has credential_id: "mailchimp", provider: "mailchimp", value as a map containing apiKey and server, and non-secret updated_at metadata as an ISO timestamp. No DataOps config, CloudFormation parameter, environment variable, log, job, artifact, task, or API response contains value or apiKey.
• DATAOPS_MAILING_EXPORTS_CONFIG references credentialId: "mailchimp" instead of secretName for each Mailchimp export configuration.
• DataOps uses GetItem with ConsistentRead: true and the configured credential ID. It accepts only a matching Mailchimp record with a non-empty string value.apiKey; value.server, when present, must remain subject to the provider's existing validation.

Keep the existing operator-visible authorization behavior. Missing, unreadable, wrong-provider, or malformed credential records must produce the existing sanitized authorization failure and fix-authorization action, without exposing DynamoDB errors, table identifiers, item contents, API keys, or Mailchimp responses.

Update the SAM/CloudFormation contract, mailing export documentation, focused backend tests, and infrastructure tests. Remove the mailing-export-specific MailchimpSecretArn parameter/condition and Secrets Manager IAM grant after the DynamoDB path is active; do not change Secrets Manager use by unrelated DataOps features.

Acceptance Criteria

• Mailing export configuration requires a non-empty credentialId and no longer accepts or exposes secretName; public configuration responses continue to omit credential identifiers.
• The production credential loader performs exactly one consistent DynamoDB GetItem against the configured Dapier table and requested credentialId, with no Scan, Query, BatchGet, write, KMS, or Secrets Manager call.
• A valid mailchimp item supplies value.apiKey and optional value.server to the existing provider registry and the export continues through the current request/poll/download flow.
• Missing table configuration, missing item, key/provider mismatch, missing or non-string API key, malformed value map, and DynamoDB read failure all persist a sanitized authorization failure with nextAction: "fix-authorization".
• Credential values are absent from logs, exceptions returned to callers, mailing-export jobs, artifacts, task links, API responses, and CloudFormation/environment configuration. Automated tests use obvious placeholders rather than a real credential.
• DataOps infrastructure accepts the Dapier credential table name and ARN, passes only the table name to the backend runtime, and grants only dynamodb:GetItem on that exact table ARN, constrained with dynamodb:LeadingKeys to mailchimp. It grants no Scan/Query/BatchGet/write access to the credential table.
• The mailing-export-specific MailchimpSecretArn, HasMailchimpSecret, and conditional secretsmanager:GetSecretValue resource are removed without altering unrelated DataOps secret permissions.
• DataOps documentation describes the DynamoDB contract, transparent at-rest encryption, write-only ownership by Dapier, rotation by replacing the same item, safe failure behavior, and deployment/rollback order. It must not claim that DynamoDB prevents an authorized GetItem caller from receiving plaintext.
• Existing mailing export idempotency, provider polling, private ZIP storage, artifact/task handling, daily limit behavior, and authenticated API behavior remain unchanged.
• [HUMAN] In the target AWS account, after Dapier has populated the item, an authorized operator triggers one Mailchimp export and confirms it reaches requested/pending/completed rather than credential authorization failure; no credential value is inspected or copied as evidence.
• [HUMAN] After the DataOps deployment, IAM policy inspection confirms the runtime role has only GetItem for the dedicated Dapier credential table and no longer has access to the legacy Mailchimp secret.

Test Scenarios

Scenario: Valid Dapier credential

Given: A configured Mailchimp export and a credential loader returning the contracted mailchimp item with placeholder values
When: The export starts
Then: DataOps constructs the Mailchimp provider and preserves the existing durable export workflow without persisting or returning the credential

Scenario: Missing or malformed credential

Given: The table setting is absent, the item is absent, the provider does not match, or value.apiKey is absent/non-string
When: The scheduled or manual export runs
Then: The job is failed with category authorization, a sanitized operator message, and fix-authorization, with no sensitive item data in logs or responses

Scenario: DynamoDB denies the read

Given: GetItem raises an access, throttling, or service error that may include internal details
When: The export runs
Then: The job records the same sanitized authorization failure and neither the original exception nor item/table details cross the service boundary

Scenario: Public API remains credential-free

Given: A configuration contains credentialId: "mailchimp" and runs exist
When: An authenticated operator requests GET /api/mailing-exports
Then: The response contains operator-facing configuration and safe run state but no credential ID, credential record, API key, or server derived from the credential

Scenario: Least-privilege infrastructure

Given: The SAM template is rendered with the Dapier table name and ARN
When: Infrastructure tests inspect backend environment and IAM statements
Then: The runtime receives the table name, can only call GetItem on the exact ARN for leading key mailchimp, and has no mailing-export Mailchimp Secrets Manager permission

Required Verification

• npm --prefix backend test
• npm --prefix backend run typecheck
• npm --prefix backend run build
• uv run --with pytest python -m pytest tests/infra
• make sam-validate
• make sam-build
• No Playwright screenshot is required because this issue changes no UI; existing route tests must prove the API remains sanitized.

Deployment Sequence

1. Dapier deploys the dedicated credential table, write-only admin API/UI behavior, stable item contract above, and output/documentation for the table name and ARN.
2. [HUMAN] An operator saves/re-saves the real Mailchimp API key in Dapier so the mailchimp item exists before the reader switches. Do not copy the value through DataOps deployment parameters or logs.
3. Deploy DataOps with the Dapier table name/ARN parameters, the credentialId: "mailchimp" mailing config, exact-key GetItem IAM permission, and removal of its legacy Mailchimp secret permission.
4. Run the human smoke check. On failure, roll DataOps back to its previous stack version while leaving the Dapier table/item intact; do not paste the credential into rollback commands.
5. After successful verification, Dapier may remove the obsolete Secrets Manager copy under its own issue/change. DataOps must not delete or mutate it.

Out of Scope

• Creating or writing the credential table from DataOps.
• Dapier UI/API implementation, credential migration, or deletion of Dapier/Secrets Manager resources.
• Client-side/application-layer encryption, direct KMS calls, SSM Parameter Store, Secrets Manager fallback, automatic rotation, or secret history/version UI.
• Moving DataOps admin/session, Typefully, GitHub, bookkeeping, email intake, or any other credentials to DynamoDB.
• Changing Mailchimp export scope, scheduling, storage, artifact visibility, task linkage, or portal UI.
• Reading arbitrary credential IDs or providers from the Dapier table.

Dependencies

• Deployment dependency: Dapier defines and owns the dedicated CredentialsTable and exact item contract above. Dapier must deploy/export the table and an operator must populate credential_id: "mailchimp" before the DataOps reader switches; this sequencing requirement does not block DataOps implementation and automated testing.
• The Dapier stack must expose a stable table name and ARN for DataOps deployment. DataOps should consume parameters rather than hardcode an AWS account ID or physical table name.
• The dedicated table deliberately stores credential values as ordinary DynamoDB attributes protected by DynamoDB's transparent encryption at rest. Any principal with authorized GetItem access receives plaintext, so IAM isolation and application-level redaction are mandatory privacy boundaries.

[alexeygrigorev]

TEST PASS

Findings

No bugs, security regressions, or acceptance-criterion gaps found in the uncommitted implementation.

Verification Evidence

• NODE_ENV=test SKIP_AUTH=true node --import tsx --test backend/tests/mailing-exports.test.ts -> exit 0; 15 passed, 0 failed. Covers the exact consistent GetItem, all malformed/missing/read-failure cases, registry handoff, durable request/poll/download behavior, redaction, idempotency, storage, artifacts, daily limits, and authenticated APIs.
• npm --prefix backend test -> exit 0; 1,203 passed, 0 failed.
• npm --prefix backend run typecheck -> exit 0.
• npm --prefix backend run build -> exit 0.
• uv run --with pytest python -m pytest tests/infra -> exit 0; 20 passed, 0 failed.
• make sam-validate -> exit 0.
• make sam-build -> exit 0; SAM build succeeded.
• sam validate --lint --template-file infra/template.full.yaml with isolated AWS config -> exit 0; valid SAM template.
• uv run --project tools/content_tools python -m content_tools.validate_docs_links --repo-root . --content-root content -> exit 0; docs link validation passed.
• Workflow YAML parsed successfully with PyYAML and contains the deploy job.
• git diff --check -> exit 0.

Acceptance Criteria

1. PASS: credentialId is required, legacy secretName is rejected, and public configs remove the identifier at runtime.
2. PASS: the production loader issues exactly one low-level GetItemCommand with ConsistentRead: true; static inspection finds no Scan, Query, BatchGet, write, KMS, or Secrets Manager call in the mailing credential path.
3. PASS: a contracted Mailchimp item returns only apiKey and optional server to the existing registry; existing request/poll/download tests remain green.
4. PASS: absent table config, absent item, key/provider mismatch, malformed values, invalid API-key types, invalid server types, and read exceptions become the same sanitized persisted authorization failure with fix-authorization.
5. PASS: credential values are not written to jobs, artifacts, task links, logs, APIs, errors, environment configuration, or CloudFormation; tests use explicit placeholder values.
6. PASS: infrastructure accepts table name and ARN separately, exposes only the name to runtime, and grants only dynamodb:GetItem on the exact supplied ARN with leading key mailchimp.
7. PASS: the mailing-specific MailchimpSecretArn, HasMailchimpSecret, and conditional Secrets Manager resource are removed; unrelated secret grants are unchanged.
8. PASS: documentation accurately describes ownership, transparent at-rest encryption, plaintext visibility to an authorized reader, rotation, safe failures, deployment order, and rollback.
9. PASS: the full backend suite confirms existing idempotency, polling, private ZIP storage, artifact/task behavior, daily-limit handling, and authenticated APIs remain green.
10. HUMAN REMAINS: trigger a real Mailchimp export after Dapier has populated the target item and confirm requested/pending/completed without inspecting the credential.
11. HUMAN REMAINS: inspect the deployed runtime role and confirm exact-table GetItem only and no legacy Mailchimp secret access.

No screenshots were required because this change has no UI surface.

[alexeygrigorev]

ACCEPTED

PM final acceptance confirms the implementation matches issue #120 and the deployed Dapier contract:

• Operator flow: Dapier remains the credential entry point. DataOps does not add a second credential-management surface, and its existing mailing-export API/operator flow continues to return safe run state and the sanitized fix-authorization action.
• Contract consistency: DataOps reads the Dapier-owned table by credential_id: "mailchimp", verifies provider: "mailchimp", and passes only value.apiKey plus optional value.server to the provider. Dapier's implementation writes that schema and exports the table name/ARN.
• Privacy: The implementation and documentation accurately state that DynamoDB encryption protects data at rest while an IAM-authorized GetItem receives plaintext. Credential values and identifiers do not enter public configs, jobs, artifacts, tasks, logs, API responses, or deployment parameters.
• Authorization boundary: DataOps receives only dynamodb:GetItem on the supplied table ARN, constrained to leading key mailchimp; it does not scan, query, batch-read, write, call KMS, or use the legacy Mailchimp Secrets Manager path.
• Deployment and rollback: Documentation and CI inputs require Dapier's table and populated item before switching DataOps. Rollback preserves the Dapier item and private export data and never places the credential in commands.
• Verification: Tester reports 1,203 backend tests and 20 infrastructure tests passing, plus typecheck, build, SAM validation/build/lint, docs-link validation, workflow parsing, and diff checks. No UI changed in DataOps, so screenshots are not required.

The agent-verifiable acceptance criteria are complete. Ship with Refs #120, keep the issue open with the human label, and complete these production checks after deployment:

• [HUMAN] Trigger one real Mailchimp export after Dapier has populated the item; confirm requested/pending/completed without inspecting or copying the credential.
• [HUMAN] Inspect the deployed DataOps runtime role; confirm exact-table GetItem only for the Dapier credential table and no access to the legacy Mailchimp secret.

[alexeygrigorev]

ON-CALL DEPLOYMENT PASS

Commit 0267050fe386d64139f850311f95d5cbfa77bb0a deployed successfully.

Workflow evidence

• Deploy DataOps V1 Lambda run 29343824649: success. Backend tests, typecheck, build, Playwright E2E, SAM validation/build, CloudFormation deployment, runtime seeding, and the deployed single-origin smoke test all passed.
• Validate DataOps Content run 29343824652: success.
• Validate Planning Docs run 29343824845: success.
• CloudFormation stack dataops-v1: UPDATE_COMPLETE at 2026-07-14T15:12:01Z.
• Deployed Lambda dataops-v1-BackendFunction-tl4mKPLYblY3: Active, last modified 2026-07-14T15:12:39Z, using runtime role dataops-v1-BackendFunctionRole-tMuHGfvC2M55.

Production IAM inspection

The deployed role has one dedicated statement for the Dapier credential table:

• Action: dynamodb:GetItem only.
• Resource: exact ARN arn:aws:dynamodb:eu-west-1:817685572750:table/dapier-CredentialsTable-1BNOH3MMNO5UA.
• Condition: ForAllValues:StringEquals on dynamodb:LeadingKeys with only mailchimp.

IAM policy simulation against that exact table confirmed:

• GetItem with leading key mailchimp: allowed.
• GetItem with leading key slack: implicit deny.
• Scan, Query, BatchGetItem, PutItem, UpdateItem, and DeleteItem: implicit deny.
• secretsmanager:GetSecretValue on the legacy dataops-v1/mailchimp-* secret path: implicit deny.

Direct inspection of every deployed inline role policy found no Mailchimp Secrets Manager resource. The remaining Secrets Manager grants are unrelated DataOps integrations. The runtime environment contains only the Dapier credentials table name for this path. No credential value was read or inspected.

The production IAM acceptance check is complete. The real Mailchimp export smoke check remains open and was intentionally not run because the Dapier item is currently empty.