Skip to content

Replaced all instances of gsutil with gcloud - #334

Open
kvo3 wants to merge 11 commits into
devfrom
kristenliu/BENCH-9580
Open

Replaced all instances of gsutil with gcloud#334
kvo3 wants to merge 11 commits into
devfrom
kristenliu/BENCH-9580

Conversation

@kvo3

@kvo3 kvo3 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Replace all gsutil CLI calls with gcloud storage equivalents across production code, tests, docs, and examples, in preparation for gsutil deprecation
  • Translate gsutil-specific flags to their gcloud equivalents
  • Rename internal bash functions and Python constants
  • Bump version to 0.5.3.dev0
  • Updated: All integration tests and test utilities to use gcloud storage commands
  • Updated README.md, docs/code.md, docs/providers/README.md, and all example READMEs/scripts to reference gcloud storage commands and link to the gcloud storage docs
  • Bump Cloud SDK Docker image from 294.0.0-slim to 499.0.0-slim since gcloud storage requires SDK >= 407
  • Updated examples folder to use google-batch instead of google-cls-v2 due to the deprecation of the latter

Production code changes

  • google_utils.py: Rewrote gcloud_cp() and gcloud_rsync() bash helper functions with gcloud storage syntax and
    translated flag formats. Updated LOCALIZATION_LOOP and DELOCALIZATION_LOOP to call the renamed functions.
  • local.py: Updated _localize_inputs_command(), _delocalize_outputs_commands(), and _delocalize_logging_command() to use gcloud storage cp with --billing-project instead of gsutil -u ... -mq cp.
  • providers_util.py: Changed _LOCALIZE_COMMAND_MAP[P_GCS] from gsutil -m rsync -r to gcloud storage rsync -r.
  • google_batch.py / google_v2_base.py: Updated template variable references to use renamed constants.

Example code changes

  • --provider google-cls-v2 → --provider google-batch
  • --zones "us-central1-*" → --regions us-central1 (google-batch doesn't expand zone wildcards)

Test code changes

  • Updated all integration test scripts and Python test utilities to use equivalent gcloud storage commands
  • Updated test_util.py functions naming: gsutil_ls_check() to gcloud_ls_check(), gsutil_cat() to gcloud_cat()
  • Updated retry config in script_block_external_network.sh: gsutil -o 'Boto:num_retries=0' to --retry-max-attempts=0
  • Updated e2e_io_recursive.sh sed pattern from s#/:$## to s#/$## to match gcloud storage ls directory marker format
  • Removed obsolete pass_credentials_to_gsutil diagnostic line from test_setup_e2e.sh

Documentation changes

  • Updated README.md, docs/code.md, docs/providers/README.md, and all example READMEs and scripts
  • Updated links from gsutil docs to gcloud storage reference docs
  • Updated commands: gsutil mb to gcloud storage buckets create, gsutil iam ch to gcloud storage buckets
    add-iam-policy-binding

BENCH-9580

@kvo3
kvo3 changed the base branch from main to dev August 19, 2026 22:39
@kvo3
kvo3 requested a review from PeterSu92 August 19, 2026 22:40

@PeterSu92 PeterSu92 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TLDR two breaking issues identified, one around the local-provider breaking for non GCS input/output (gcloud storage cp doesn't work for local files), another for the retry-disable flag. Another comment is more on drift and shouldn't block PR merge, but should be addressed nonetheless

Comment thread dsub/providers/local.py Outdated
Comment thread test/integration/script_block_external_network.sh Outdated
Comment thread test/integration/e2e_io_recursive.sh
@kvo3
kvo3 requested a review from PeterSu92 August 21, 2026 20:07

@PeterSu92 PeterSu92 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - non-blocking ask, could you add .ipynb_checkpoints/ to .gitignore?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants