feat(toolkit-lib): surface CloudTrail control-plane errors for failed resources#1676
Open
iankhou wants to merge 11 commits into
Open
feat(toolkit-lib): surface CloudTrail control-plane errors for failed resources#1676iankhou wants to merge 11 commits into
iankhou wants to merge 11 commits into
Conversation
aws-cdk-automation
temporarily deployed
to
run-tests
GitHub Actions
Inactive
aws-cdk-automation
temporarily deployed
to
run-tests
GitHub Actions
Inactive
iankhou
commented
Jun 25, 2026
iankhou
commented
Jun 26, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1676 +/- ##
=======================================
Coverage 89.59% 89.59%
=======================================
Files 77 77
Lines 11758 11758
Branches 1651 1651
=======================================
Hits 10534 10534
Misses 1195 1195
Partials 29 29
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR enhances @aws-cdk/toolkit-lib’s post-failure diagnosis for custom resources by optionally consulting CloudTrail to surface underlying control-plane errors (e.g., AccessDenied) that typically don’t show up clearly in the custom resource Lambda logs. It also fixes a pre-existing diagnosis gap for rolled-back stacks by expanding the stack event poll range to cover both the failed operation and its rollback.
Changes:
- Add a CloudTrail lookup (scoped to the custom resource’s backing Lambda “Username”) to surface errored API calls during
cdk diagnose, while keepingcdk deployon a hint-only path. - Fix rolled-back stack diagnosis by introducing
PollRange.mostRecentDeploymentAttempt()and using it in fresh diagnosis. - Add/update tests and SDK mocking to cover CloudTrail behavior, pagination, and the rolled-back regression.
Reviewed changes
Copilot reviewed 12 out of 15 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| yarn.lock | Adds the CloudTrail AWS SDK v3 client (and transitive updates) to the workspace lockfile. |
| packages/aws-cdk/THIRD_PARTY_LICENSES | Updates bundled third-party attributions to include the newly bundled CloudTrail client and related deps. |
| packages/@aws-cdk/toolkit-lib/test/api/diagnosing/stack-diagnoser.test.ts | Adds regression test ensuring diagnosis finds failures across rollback + failed create operations. |
| packages/@aws-cdk/toolkit-lib/test/api/diagnosing/resource-investigation.test.ts | Adds tests for CloudTrail integration (gating, scoping, pagination, and “no errors” behavior). |
| packages/@aws-cdk/toolkit-lib/test/_helpers/mock-sdk.ts | Adds CloudTrail client mocking support for unit tests. |
| packages/@aws-cdk/toolkit-lib/package.json | Declares @aws-sdk/client-cloudtrail as a runtime dependency of toolkit-lib. |
| packages/@aws-cdk/toolkit-lib/lib/api/stack-events/stack-event-poller.ts | Introduces PollRange.mostRecentDeploymentAttempt() to span rollback + triggering operation. |
| packages/@aws-cdk/toolkit-lib/lib/api/diagnosing/stack-diagnoser.ts | Threads cloudTrailEnabled and uses the new poll range for diagnose-from-fresh. |
| packages/@aws-cdk/toolkit-lib/lib/api/diagnosing/resource-investigation.ts | Extends the investigation dispatcher to pass options through to custom-resource investigation. |
| packages/@aws-cdk/toolkit-lib/lib/api/diagnosing/investigate-ecs-service.ts | Clarifies shared vs resource-specific investigation options in comments. |
| packages/@aws-cdk/toolkit-lib/lib/api/diagnosing/investigate-custom-resource.ts | Implements CloudTrail lookup + deploy-path hint for custom resource investigation. |
| packages/@aws-cdk/toolkit-lib/lib/api/aws-auth/sdk.ts | Adds ICloudTrailClient and SDK.cloudTrail() wrapper. |
| packages/@aws-cdk/toolkit-lib/.projen/deps.json | Records the new runtime dependency for projen-managed dependency tracking. |
| .projenrc.ts | Adds the CloudTrail SDK dependency to the toolkit-lib project configuration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
|
Why do this only for custom resources? Can we not do it in a way that works generically for all resources? I haven't though too deeply about how we should get "useful information" from CloudTrail; I was hoping you would. I was hoping we could do something like the following: 
|
Contributor
Mostly true, but not exactly: CloudTrail records control plane events, not data plane events. Usually that means: low-traffic configuration events are recorded, high-traffic events are not. So 2 counterexamples to your statement above:
The point still stands that not everything will be recorded, but you need to know what will and won't be, and for what reasons, to give accurate guidance to users. Speaking of, can you make sure that you understand the Pull Request and write the PR body yourself, rather than extrude some text via AI? I would like some proof of understanding. Thank you. |
Contributor
Author
Will do, rewritten. Missed the errant caveats at the end, and thanks for the catch. Removed that section because it's evident from the main section of the PR body anyway. I'll outsource less for the PR descriptions. |
Contributor
|
Total lines changed 7257 is greater than 1000. Please consider breaking this PR down. |
Contributor
Author
It's from THIRD_PARTY_LICENSES, which should be ignored. See #1680 for fix. |
Contributor
Author
Yep this is what we are doing for custom resources. I'll explore how to do this generically. |
… custom resources When a custom resource fails because its backing Lambda was denied an AWS API call (a control-plane failure), the function's own logs often don't show it — but CloudTrail records the AccessDenied. This adds a CloudTrail lookup to the custom-resource investigation. CloudTrail events are delivered with several minutes of latency, so they aren't available when a `cdk deploy` fails. Since deploy and diagnose share the diagnosis code, this is gated on a cloudTrailEnabled flag: - diagnoseFromFresh (`cdk diagnose`, run after the fact) enables it and looks up CloudTrail, scoped to the function server-side via the Username attribute (the Lambda execution role's session name is the function name), paginated and bounded to a window around the failure. Errored events are surfaced (e.g. "AccessDenied on s3:CreateBucket"). - diagnoseFromErrorCollection (`cdk deploy`) leaves it off and adds a hint to re-run `cdk diagnose` in a few minutes. Also fixes diagnoseFromFresh reporting "no issues found" on any rolled-back stack: a rolled-back deployment spans two CloudFormation operations (the failed create and the rollback); the poll range only covered the most recent (the rollback's successful deletes), excluding the real CREATE_FAILED. Adds PollRange.mostRecentDeploymentAttempt() spanning both. - Add @aws-sdk/client-cloudtrail and an ICloudTrailClient.lookupEvents wrapper. - Thread cloudTrailEnabled through DiagnoseOptions/InvestigateOptions. Validated end-to-end against a denied s3:CreateBucket. Known limitation: CloudTrail LookupEvents records write/mutating denials, not most read calls. Tests cover both paths, Username scoping, pagination, and the rolled-back regression; new behaviors are mutation-verified.
…enied When the lookup role lacks cloudtrail:LookupEvents (common in locked-down accounts), the custom-resource investigation previously swallowed the AccessDenied to debug and returned nothing, leaving the user with the same unhelpful message CloudTrail was meant to improve. Detect the authorization failure specifically and emit a one-line hint pointing at the missing permission; other lookup failures still degrade silently (best-effort).
…ources Rework the custom-resource-only CloudTrail lookup into a stack-level investigation, per review feedback on #1676: - One unfiltered LookupEvents sweep per diagnosis, bounded to the failure window, correlated client-side against a stack identity set (physical IDs, IAM roles referenced by *Role* properties, custom-resource backing functions). This covers calls by customer code AND by services acting on a resource's behalf (e.g. Lambda creating ENIs with the function's role), which the previous Username-scoped lookup could never match. - No ranking: correlated events are deduplicated by (eventSource, eventName, errorCode) and shown earliest-first, capped at 5 with disclosure. The window ends shortly after the last failure event so rollback teardown noise never enters the sweep. - The deploy/diagnose fork (cloudTrailEnabled) is removed: both paths run the same investigation. A recent failure with zero correlated events yields a single 'events may not be delivered yet' note. - Lookup failures are IoHost warnings, not diagnosis results. Correlation design validated against real failed stacks (custom-resource AccessDenied and VPC-Lambda ENI denial): zero false attributions with two stacks failing concurrently in the same account.
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…ation - mostRecentDeploymentAttempt only spans into a second operation when the newest operation is a rollback; a --no-rollback failure stops at the first operation boundary instead of pulling in a previous deployment's events - correlation matches identity keys as whole ARN segments (never bare substrings) and only against the calling principal's ARNs — resources[].ARN is excluded, so foreign principals' calls against stack resources can no longer be attributed to the stack - findings attach to problems by (stackArn, logicalId), not bare logicalId, so nested stacks with repeated logical IDs attribute correctly; findings for resources that are not themselves problems are labeled with the resource they matched - role discovery walks nested template properties (Firehose destination configs, EventBridge targets) and resolves Fn::Sub role ARNs with literal name segments - the truncation note reports events checked, not errored events kept; the overflow disclosure moved from a fake error entry to the notes channel; dedup keeps the earlier event's own attribution; the delivery-latency hint is no longer suppressed by partial delivery - drop the unused InvestigateResourceOptions alias and dead _options parameter
…sification getStackTemplate was copied between the custom-resource and CloudTrail investigations; it now lives once in diagnosing/stack-template.ts. isAccessDeniedError moves to aws-auth/util.ts and replaces the three ad-hoc AccessDenied/AccessDeniedException checks in cloudtrail-investigation, environment-resources, and stack-activity-monitor.
Contributor
Author
|
Logs from VpcEniDeniedStack Here, we captured the CloudTrail event at the end of Before (CDK CLI on
|
Contributor
Author
|
Logs from IamDeniedCustomResourceStack In this case, the lambda handler returns a useful message, so much of the CloudTrail log is redundant. We also get useful CloudTrail logs from Before (CDK CLI on
|
…match a CloudTrail event An assumed-role session ARN contains both the role name and (for Lambda-made calls) the function name, so a single event can match two stack identities; first-match made the attribution depend on DescribeStackResources response order. The most specific match now wins: a key equal to the session-name segment (the actual caller) beats any other match, and key length breaks ties.
Contributor
Author
|
Logs from CrashingCustomResourceStack Before (CDK CLI on
|
Contributor
Author
|
Logs from FailingCustomResourceStack This is the negative case, where CloudTrail events don't add additional utility. Before (CDK CLI on
|
…identifiers identityMatches, nameSegment, and assumedRoleSessionName are pure identifier parsers, so they belong in resource-identifiers.ts with the other ones — and as exports they get direct unit tests for the segment-anchoring boundary cases instead of being reachable only through the full investigation with four mocked clients. The attribution policy (mostSpecificMatch) stays with correlateEvents in cloudtrail-investigation.ts.
Contributor
Author
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Today, some cases of resource deployment failures do not present useful error messages to users in the CDK CLI:
Solution
In all of these cases, CloudTrail records failed management (control-plane) events, with the error code and message. This PR looks for those events in deployment diagnosis and attributes them to the right stack resource.
Prerequisite fix for
cdk diagnosecdk diagnosecurrently reports "no issues found" on any rolled-back stack, because it sees the rollback operation as a success.Output on a failed stack deployment that contained a custom resource, but rolled back successfully:
The same command on this branch (after our changes) finds the failure and adds the CloudTrail attribution:
The fix was in
PollRange.mostRecentDeploymentAttempt(), which, after our changes, spans the rollback plus the operation that triggered it. It only spans two operations when the newest one actually is a rollback, i.e. a--no-rollbackfailure stops at the first operation boundary, so a previous attempt's stale failures are not re-reported.Implementation details
cdk deployandcdk diagnosepaths run the same investigation, once per failed deployment. Right after a failed deploy, the events may not be delivered yet. The output says so and suggests runningcdk diagnose.The lookup is unfiltered server-side.
LookupEventsaccepts at most one attribute per call. So the sweep is scoped only by time, and attribution happens client-side: an event belongs to the stack when a known stack identity (physical IDs, IAM roles referenced byRole/RoleArnproperties at any depth,ServiceTokenfunctions) appears as a whole ARN segment in the event's calling-principal fields (userIdentity.arn,sessionIssuer.arn,inScopeOf.credentialsIssuedTo). Only the caller is matched so that we do not misattribute calls.The time window we use is derived from the failed operation's stack events and closes shortly after the last failure event, so rollback-teardown noise isn't captured (e.g. failures in rollback).
Pagination is capped at 10 pages (~500 events); truncation is disclosed in the output.
Only events carrying an
errorCodeare considered. Duplicates collapse by(eventSource, eventName, errorCode)to the earliest occurrence (the call closest to the root cause); up to 5 are shown earliest-first, with a note for anything collapsed or cut.Best-effort: failures of the lookup itself are not diagnosis results. A missing
cloudtrail:LookupEventspermission presents as an IoHost warning with the grant hint; throttles and other errors degrade to debug logging. Nothing blocks or fails the deploy/diagnose.Considerations
CloudTrail management events are available for 90 days by default, and incur no charges to the customer. Data-plane events are not recorded by default.
The sweep is page-capped (10 pages ≈ 500 events) and we disclose truncation.
Everything is best-effort via the lookup role; no new permissions are required beyond the default
ReadOnlyAccess(cloudtrail:LookupEventsis included).Validations
The correlation design was validated against failed stacks in a test account before implementation, and the final build was compared end-to-end against a
main-built CLI across four scenarios (custom resource with swallowed error, silently-crashing handler, VPC/ENI denial, app-level failure as the negative case):iankhou/cdk-app-with-problems @
iankhou-cr-main:s3:CreateBucket, reports a vague FAILEDec2:CreateNetworkInterfaceon the function's behalf (the generic case)Results
No false attributions, including testing with two stacks failing concurrently in the same account (each run correctly excluded the other stack's errored events).
Custom-resource scenario: 110 events swept, 80 errored, 2 correlated — the root-cause S3 denial plus one benign
CreateLogStream404, both attributed to the handler.VPC/ENI scenario: 68 events correlated, but 67 were identical post-rollback polling noise (
ResourceNotFoundExceptionon the already-deleted function) vs 1 root cause — which is why the window ends at the failure and duplicates collapse to the earliest occurrence, rather than showing "the last 5" chronologically.The negative case (handler throws an app-level error; no control-plane call fails) correctly produces no CloudTrail output.
On the
mainbranch,cdk diagnosereturns a false "no issues found" for every rolled-back stack (shown above) and deploy output shows StatusReason only.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license