Skip to content

fix(api): include exhaustion reason in "Node exhausted" placement warning#3279

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/placement-log-exhaustion-reason
Open

fix(api): include exhaustion reason in "Node exhausted" placement warning#3279
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/placement-log-exhaustion-reason

Conversation

@AdaAibaby

@AdaAibaby AdaAibaby commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #3278

The "Node exhausted, trying another node" Warn log in PlaceSandbox discarded the gRPC error from the orchestrator, making it impossible to tell from API logs which resource limit was hit.

Change

One-line fix in packages/api/internal/orchestrator/placement/placement.go:

// Before
logger.L().Warn(ctx, "Node exhausted, trying another node",
    logger.WithSandboxID(...), logger.WithNodeID(failedNode.ID))

// After
logger.L().Warn(ctx, "Node exhausted, trying another node",
    logger.WithSandboxID(...), logger.WithNodeID(failedNode.ID),
    zap.Error(utils.UnwrapGRPCError(err)))

This matches the pattern already used in the default branch two lines below.

Before / After

Before:

WARN  Node exhausted, trying another node  {"sandbox.id": "i5ksfunc...", "node.id": "cpu-nat-398"}

After:

WARN  Node exhausted, trying another node  {"sandbox.id": "i5ksfunc...", "node.id": "cpu-nat-398", "error": "too many sandboxes starting on this node, please retry"}

or

WARN  Node exhausted, trying another node  {"sandbox.id": "i5ksfunc...", "node.id": "cpu-nat-398", "error": "max number of running sandboxes on node reached (350), please retry"}

Now you can tell at a glance whether the node hit its concurrent-start limit (MaxStartingInstancesPerNode, default 3) or its total-sandbox limit (MaxSandboxesPerNode).

Test plan

  • Trigger sandbox creation against a node at capacity; confirm "Node exhausted" Warn log now includes the error field with the specific reason

PlaceSandbox logs "Node exhausted, trying another node" whenever the
orchestrator returns codes.ResourceExhausted, but the gRPC error was
silently dropped. The orchestrator has two distinct limits:

  - max running sandboxes per node (MaxSandboxesPerNode flag)
  - max concurrent sandbox starts (MaxStartingInstancesPerNode, default 3)

Both return ResourceExhausted with different messages, and without the
error there is no way to tell which limit was hit from the API logs.

Add zap.Error(utils.UnwrapGRPCError(err)) to the Warn call, matching
the pattern already used in the default (hard failure) branch below.

Fixes: e2b-dev#3278

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds error logging to the sandbox placement warning when a node is exhausted by using utils.UnwrapGRPCError(err). Feedback indicates that utils.UnwrapGRPCError uses github.com/gogo/status instead of the standard google.golang.org/grpc/status, which prevents standard gRPC errors from being unwrapped correctly. It is recommended to update the helper function to use the standard gRPC status package.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread packages/api/internal/orchestrator/placement/placement.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

api: "Node exhausted" placement warning drops the exhaustion reason from logs

1 participant