fix: actionable error when model weights bucket is inaccessible

[dm36]

Summary

Diagnosed from a real failed deploy: an endpoint create returned the opaque {"error":"Internal error occurred. Our team has been notified.","request_id":...} 500. The logged traceback showed the true cause was an S3 AccessDenied during model-weight discovery:

create_model_endpoint → create_vllm_bundle → load_model_weights_sub_commands_s3
  → S3LLMArtifactGateway.list_files → boto3 ListObjects
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling
ListObjects ... not authorized to perform: s3:ListBucket on "arn:aws:s3:::scale-ml"

list_files let the raw ClientError propagate. Because it isn't a DomainException, it bypassed every per-route exception handler and fell through to the global catch-all 500 — so the deploy reported a generic error and the real cause (a bucket the deployment role can't read) was only visible in logs by request_id.

Change

Catch ClientError in S3LLMArtifactGateway.list_files. For access/missing errors (AccessDenied, NoSuchBucket, 403, 404), raise ObjectHasInvalidValueException with a path-only message:

Could not read model weights at '<path>'. Check that the path exists and that the deployment has read access to the bucket.

The create/update endpoint handlers already map ObjectHasInvalidValueException → 400, so this surfaces as an actionable client error with no new wiring. The IAM role ARN from the raw error is never surfaced (only the path); full detail is still logged server-side. Non-access ClientErrors (e.g. throttling/SlowDown) are re-raised unchanged so transient/internal failures still behave as before.

Test plan

• list_files success path (baseline).
• list_files with S3 AccessDenied raises ObjectHasInvalidValueException, message contains the path and does not contain the IAM role ARN.
• black / ruff / isort clean on changed files.

Context

Companion to #840 (async-deploy status_reason) and #841 (middleware error surfacing). This one fixes the specific opaque-500 at its source — the most common shape being an inaccessible/incorrect checkpoint bucket.

🤖 Generated with Claude Code

Greptile Summary

• Converts selected S3 model-weight access and missing-path ClientErrors into sanitized ObjectHasInvalidValueExceptions.
• Keeps transient or non-access S3 ClientErrors propagating unchanged.
• Adds unit coverage for successful listing, sanitized access failures, config download failures, and passthrough behavior.

Confidence Score: 5/5

The change is narrowly scoped to S3 listing error translation and preserves existing behavior for successful and transient failure paths.

Unit coverage exercises the intended sanitized client error behavior, unchanged passthrough for non-access S3 errors, and the existing successful listing path.

T-Rex Logs

What T-Rex did

• Ran the base-commit S3 list-files error contract to capture the command, working directory, full script, and the initial output, where AccessDenied and NoSuchBucket appeared as ClientError.
• Ran the head-commit S3 list-files error contract to compare against the base run, where AccessDenied and NoSuchBucket appeared as ObjectHasInvalidValueException and referenced the s3://scale-ml/models/checkpoint path.
• Both executions exited with exit code 0.
• The head-commit run returned the expected keys ['models/checkpoint/a.bin', 'models/checkpoint/sub/b.json'].

_{Ran code and verified through T-Rex}

_{Reviews (3): Last reviewed commit: "fix: also map inaccessible model config ..." | Re-trigger Greptile}