direct: retry 504 errors by denik · Pull Request #5349 · databricks/cli

denik · 2026-05-27T15:03:04Z

Changes

Retry resource methods that return an error that has http_code equal to 504 but that has not been retried by SDK.

This affects all DoRead, DoUpdate(WithID), DoDelete. This affects DoCreate for grants and permissions.

Note, for DoCreate the retry functionality is opt-in - implementations need to wrap error with retrySafe(). For other methods the retry is always enabled.

Why

We've seen reports where deploy fails with

Error: cannot create resources.pipelines..permissions: The service at /api/2.0/permissions/pipelines/<pipeline_id> is taking too long to process your request. (504 TEMPORARILY_UNAVAILABLE)

We also saw that terraform does custom retries for 504/GET databricks/terraform-provider-databricks#4355

Note, the two cases are different - the first one is "cannot create" so it refers to PUT.

Tests

New testserver feature that allows injecting expiring faults in a given endpoint. See fault.py.
New acceptance tests make use of fault.py to check failures in plan/create/update for permissions.

Wraps adapter methods DoRead, DoDelete, DoUpdate, DoUpdateWithID, DoResize, WaitAfterCreate, and WaitAfterUpdate with retry logic that retries on HTTP 408/500/502/503/504 up to 2 times with a 30s interval (overridable via DATABRICKS_BUNDLE_RETRY_INTERVAL_MS for tests). DoCreate is intentionally not retried. Adds fault injection support to testserver (POST /__testserver/fault) so acceptance tests can inject transient errors dynamically, and two acceptance tests verifying retry on permissions PUT (update deploy) and GET (plan). Co-authored-by: Isaac

…DoCreate - Narrow retry condition to 504 errors SDK did not already handle - Add retrySafeError/retrySafe: resource impls wrap DoCreate errors to opt in to transient retries when the operation is idempotent - Wire DoCreate in adapter to retry only when both retrySafe and isTransient - permissions and grants DoCreate opt in (both delegate to a PUT/PATCH) - Update 504/create acceptance test: now expects retry success + 2 PUTs Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

pietern · 2026-05-28T07:53:45Z

+// IsRetrySafe reports whether err was marked as safe to retry from DoCreate.
+func IsRetrySafe(err error) bool {
+	var safe *retrySafeError
+	return errors.As(err, &safe)


I added a linter earlier this week to suggest errors.AsType[T]. Maybe the base SHA is not up to date?

that was it, updated.

pietern · 2026-05-28T07:55:44Z

 	if err != nil {
-		return fmt.Errorf("waiting after creating id=%s: %w", newID, err)
+		if isTransient(ctx, err) {
+			log.Warnf(ctx, "waiting after creating id=%s: %s", newID, err)


waitRemoteState is not up to date but this falls through. Must either retry or fail hard.

good catch. changed to retry.

…an.go Adapter is a type-adaptation layer; retries are an operational concern. - New bundle/direct/retry.go: isTransient, retryWith/retryOnTransient/retryErr - dresources/retry.go: only retrySafe/IsRetrySafe/UnwrapRetrySafe (opt-in signal) - adapter.go: stripped to pure type adaptation - apply.go, bundle_plan.go: retries applied at each adapter call site Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

…Update The create/update already succeeded; a 504 during status polling should not abort the deployment. Non-transient errors still fail hard. Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

…site Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

…warn+fallthrough Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

eng-dev-ecosystem-bot · 2026-05-28T10:45:26Z

Commit: 42f0cc1

Run: 26568534085

Follow up to #5349

## Changes Retry resource methods that return an error that has http_code equal to 504 but that has not been retried by SDK. This affects all DoRead, DoUpdate(WithID), DoDelete. This affects DoCreate for grants and permissions. Note, for DoCreate the retry functionality is opt-in - implementations need to wrap error with retrySafe(). For other methods the retry is always enabled. ## Why We've seen reports where deploy fails with > Error: cannot create resources.pipelines.<pipeline>.permissions: The service at /api/2.0/permissions/pipelines/<pipeline_id> is taking too long to process your request. (504 TEMPORARILY_UNAVAILABLE) We also saw that terraform does custom retries for 504/GET databricks/terraform-provider-databricks#4355 Note, the two cases are different - the first one is "cannot create" so it refers to PUT. ## Tests New testserver feature that allows injecting expiring faults in a given endpoint. See fault.py. New acceptance tests make use of fault.py to check failures in plan/create/update for permissions.

eng-dev-ecosystem-bot · 2026-05-28T13:01:28Z

Commit: 813b754

Run: 26570032233

denik changed the title ~~WIP retry 504~~ direct: retry 504 errors May 27, 2026

denik marked this pull request as ready for review May 27, 2026 16:40

denik added 6 commits May 28, 2026 10:01

update tests

c955b5c

tests

a176d26

clean up

4edfda4

rename doRefresh to doRead

d5e1599

pietern reviewed May 28, 2026

View reviewed changes

denik added 5 commits May 28, 2026 10:03

Log and fall through on transient errors in WaitAfterCreate/WaitAfter…

05f5bd7

…Update The create/update already succeeded; a 504 during status polling should not abort the deployment. Non-transient errors still fail hard. Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

Rename retryErr to retryOnTransientErr

82e3e2c

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

Fix log prefix in Destroy: "destroying" not "deploying"

73d0b05

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

Reduce default retry interval from 30s to 15s

c68f966

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik force-pushed the denik/retrier branch from 0752251 to c68f966 Compare May 28, 2026 08:16

denik temporarily deployed to test-trigger-is May 28, 2026 08:17 — with GitHub Actions Inactive

Use errors.AsType[T] instead of errors.As for Go 1.26+ compatibility

1720072

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 08:20 — with GitHub Actions Inactive

Revert unrelated cosmetic changes in adapter.go

dd2b0e4

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 08:25 — with GitHub Actions Inactive

Inline IsRetrySafe: export RetrySafeError, use errors.AsType at call …

fe5f124

…site Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 08:32 — with GitHub Actions Inactive

Retry WaitAfterCreate/WaitAfterUpdate on transient errors instead of …

5219571

…warn+fallthrough Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 08:33 — with GitHub Actions Inactive

Guard apiErr nil in retryWith log line

564f68e

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 08:36 — with GitHub Actions Inactive

Export FaultRules/FaultRule and add unit tests for fault injection logic

0059f64

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 09:04 — with GitHub Actions Inactive

restructure

95d6a8a

denik temporarily deployed to test-trigger-is May 28, 2026 09:08 — with GitHub Actions Inactive

denik requested a review from pietern May 28, 2026 09:09

Mark acceptance/bin/fault.py as executable

bf19721

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 09:30 — with GitHub Actions Inactive

Fix uniq -c count width portability in 504/plan acceptance test

80a9f58

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 09:44 — with GitHub Actions Inactive

Add NEXT_CHANGELOG entry for 504 retry

e7b037f

Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>

denik temporarily deployed to test-trigger-is May 28, 2026 10:02 — with GitHub Actions Inactive

pietern approved these changes May 28, 2026

View reviewed changes

Comment thread bundle/direct/apply.go

Comment thread bundle/direct/apply.go

add a comment about DoCreate opt-in

42f0cc1

denik temporarily deployed to test-trigger-is May 28, 2026 10:12 — with GitHub Actions Inactive

denik enabled auto-merge May 28, 2026 10:17

denik disabled auto-merge May 28, 2026 10:45

denik merged commit 813b754 into main May 28, 2026
24 of 26 checks passed

denik deleted the denik/retrier branch May 28, 2026 10:45

denik added a commit that referenced this pull request May 28, 2026

Extend retries to 5xx and 408

29dc23b

Follow up to #5349

denik mentioned this pull request May 28, 2026

Extend retries to 5xx and 408 #5364

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

direct: retry 504 errors#5349

direct: retry 504 errors#5349
denik merged 22 commits into
mainfrom
denik/retrier

denik commented May 27, 2026 •

edited

Loading

Uh oh!

pietern May 28, 2026

Uh oh!

denik May 28, 2026

Uh oh!

pietern May 28, 2026

Uh oh!

denik May 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented May 28, 2026

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

denik commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

pietern May 28, 2026

Choose a reason for hiding this comment

Uh oh!

denik May 28, 2026

Choose a reason for hiding this comment

Uh oh!

pietern May 28, 2026

Choose a reason for hiding this comment

Uh oh!

denik May 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented May 28, 2026

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

denik commented May 27, 2026 •

edited

Loading