chore: [WIP] experimental parallel kokoro system test by chalmerlowe · Pull Request #17608 · googleapis/google-cloud-python

chalmerlowe · 2026-07-01T17:50:37Z

Warning

Something in this PR is off. Hold off on reviewing it.

Note

This is an experimental PR intended to test parallelization of system tests in CI/CD.
It is paired with a non-parallelized version to benchmark any improvement. chore: [WIP] experimental baseline kokoro system test #17645

Results

Baseline: 2 hours 5 mins
With parallelization: 42 mins

This is blocked on: #17826

gemini-code-assist

Code Review

This pull request introduces parallel execution for system tests in .kokoro/system.sh with concurrency limits and log segregation, and isolates gcloud configurations per run using unique temporary directories. The feedback suggests robust cleanup of the temporary directories using shell traps to prevent leaks if commands fail under set -e, sanitizing package names used in directory creation, and refactoring the duplicate process reaping logic into a helper function to improve maintainability.

…andling

chalmerlowe · 2026-07-16T18:59:36Z

Applied Do Not Merge, simply because we will want to remove the throw-away comments just prior to merging.

chalmerlowe · 2026-07-17T18:20:35Z

Something in this PR is off. Hold off on reviewing it.

daniel-sanche · 2026-07-17T17:56:59Z

+
+printf '%s\n' "${PACKAGES_TO_TEST[@]}" | xargs -P "$MAX_JOBS" -I {} bash -c 'run_package_test "{}" > "$LOG_DIR/{}.log" 2>&1 || touch "$LOG_DIR/{}.failed"'
+
+reap_parallel_results || RETVAL=1


It looks like right now, the main cloud-devrel/client-libraries/python/googleapis/google-cloud-python/presubmit/system target always returns 0. Is that intended? The individual sub-targets are still reporting properly, but it would probably be better to have the main target fail if any others do

Maybe the exit ${RETVAL} line was removed by mistake?

Fixed. We accidentally deleted it.

daniel-sanche · 2026-07-17T17:57:34Z

+  local CLOUDSDK_CONFIG="${gcloud_config_dir}"
+
+  # 🪤 TRAP: Ensure cleanup of THIS specific temp dir on exit of this subshell
+  trap 'rm -rf "$gcloud_config_dir"' EXIT


Out of curiosity, is this required? I assumed the environment would just be destroyed after completion

It may not be required, but felt like best practice: don't want space issues to bog things down mid test if we end up changing a lot of packages.

daniel-sanche · 2026-07-17T18:09:16Z

+export -f run_package_test
+export system_test_script PROJECT_ROOT KOKORO_GFILE_DIR
+
+printf '%s\n' "${PACKAGES_TO_TEST[@]}" | xargs -P "$MAX_JOBS" -I {} bash -c 'run_package_test "{}" > "$LOG_DIR/{}.log" 2>&1 || touch "$LOG_DIR/{}.failed"'


nit: This is a pretty dense line. A comment explaining that this triggers run_package_test for each package, and writes to LOG_DIR would be helpful

Done: We broke the line into steps and added some comments here and elsewhere in the script.

daniel-sanche · 2026-07-17T18:30:46Z

+MAX_JOBS=${MAX_JOBS:-4}
+
+# Temporary directory for clean log segregation
+LOG_DIR=$(mktemp -d -t test-logs-XXXXXX)


IIRC, the expectation is to write individual package logs to ${KOKORO_ARTIFACTS_DIR}/${pkg}/sponge_log.log".

In theory, this will persist the logs as an artifact, which would let us view logs for each individual target under the "Target Log" tab.

This doesn't work properly in the default BTX view. But if you click through to Sponge, it renders nicely there (invocation details > Sponge URL)

I don't think this really matters, but consider saving individual logs to that artifact directory any way, just in case BTX starts working for us some day

This is a good point, I was not sure why my logs weren't showing up where I expected them to be and this was a great question. Helped me a lot.

We added logic to write to KOKORO_ARTIFACTS_DIR by default and to fallback to a local dir if needed during a local test (I know we don't do many of those cause: "system tests", but it felt like good practice to have it).

chalmerlowe · 2026-07-23T14:31:31Z

This is blocked on: #17826

## The Problem Two unit tests in `google-cloud-bigquery` (`test_result_w_retry_wo_state` and `test_result_w_custom_retry`) were failing intermittently with a `TimeoutError` or `RetryError`. This flakiness occurs because the tests use a very short inner retry deadline of 0.1 seconds. Under heavy load (such as during parallel testing or on constrained continuous integration runners), this deadline can be exceeded, causing the test to fail confusingly. ## The Solution Increased the `deadline` parameter for the custom retry objects in both tests from 0.1 seconds to 1.0 seconds. This provides enough buffer for the test assertions to complete without timing out, aligning with other similar tests in the suite. ## Notes to Reviewers - This change only affects unit tests and does not alter production code. - This issue is blocking: - #17642 - #17608 Here is a snippet of the failure: ```python def test_result_w_retry_wo_state(global_time_lock): from google.cloud.bigquery.retry import DEFAULT_GET_JOB_TIMEOUT begun_job_resource = helpers._make_job_resource( job_id=JOB_ID, project_id=PROJECT, location="EU", started=True ) done_job_resource = helpers._make_job_resource( job_id=JOB_ID, project_id=PROJECT, location="EU", started=True, ended=True, ) conn = helpers.make_connection( exceptions.NotFound("not normally retriable"), begun_job_resource, exceptions.NotFound("not normally retriable"), done_job_resource, ) client = helpers._make_client(project=PROJECT, connection=conn) job = google.cloud.bigquery.job._AsyncJob( google.cloud.bigquery.job._JobReference(JOB_ID, PROJECT, "EU"), client ) custom_predicate = mock.Mock() custom_predicate.return_value = True custom_retry = google.api_core.retry.Retry( predicate=custom_predicate, initial=0.001, maximum=0.001, deadline=0.1, ) > assert job.result(retry=custom_retry) is job ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tests/unit/job/test_async_job_retry.py:115: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ google/cloud/bigquery/job/base.py:1047: in result return super(_AsyncJob, self).result(timeout=timeout, retry=retry) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .nox/unit-3-12-test_type-unit/lib/python3.12/site-packages/google/api_core/future/polling.py:256: in result self._blocking_poll(timeout=timeout, retry=retry, polling=polling) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = _AsyncJob<project=test-project, location=EU, id=test-job-id> timeout = None retry = <google.api_core.retry.retry_unary.Retry object at 0x7ff2beda32c0> polling = <google.api_core.retry.retry_unary.Retry object at 0x7ff2beda01d0> def _blocking_poll(self, timeout=_DEFAULT_VALUE, retry=None, polling=None): """Poll and wait for the Future to be resolved.""" if self._result_set: return polling = polling or self._polling if timeout is not PollingFuture._DEFAULT_VALUE: polling = polling.with_timeout(timeout) try: polling(self._done_or_raise)(retry=retry) except exceptions.RetryError: > raise concurrent.futures.TimeoutError( f"Operation did not complete within the designated timeout of " f"{polling.timeout} seconds." ) E TimeoutError: Operation did not complete within the designated timeout of None seconds. ```

chalmerlowe added 2 commits July 1, 2026 13:46

feat: enable parallel system tests in Kokoro and trigger 4 packages

692e1ff

refactor: refine gcloud isolation with local variable

461580e

chalmerlowe changed the title ~~[WIP] chore/experimental parallel kokoro system test~~ [WIP] chore: experimental parallel kokoro system test Jul 1, 2026

chalmerlowe changed the title ~~[WIP] chore: experimental parallel kokoro system test~~ chore: [WIP] experimental parallel kokoro system test Jul 1, 2026

gemini-code-assist Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread .kokoro/system.sh

Comment thread .kokoro/system.sh Outdated

Comment thread .kokoro/system.sh Outdated

Comment thread .kokoro/system.sh Outdated

Comment thread .kokoro/system.sh Outdated

parthea assigned chalmerlowe Jul 2, 2026

chalmerlowe added 5 commits July 7, 2026 05:44

chore: add VM diagnostics and temporarily skip sqlalchemy-bigquery

b8795e4

chore: revert sqlalchemy-bigquery trigger

54f8a72

chore: add 10 more packages to parallel test trigger

4206da2

refactor(ci): use xargs for parallel execution and add reaping function

2363d05

refactor(ci): improve reliability with trap EXIT and explicit error h…

c7a9d99

…andling

chalmerlowe mentioned this pull request Jul 16, 2026

chore: [WIP] experimental baseline kokoro system test #17645

Draft

chalmerlowe marked this pull request as ready for review July 16, 2026 18:43

chalmerlowe requested review from a team as code owners July 16, 2026 18:43

chalmerlowe requested review from sycai and removed request for a team July 16, 2026 18:43

chore(ci): remove temporary skip for sqlalchemy-bigquery

0e2a314

chalmerlowe added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jul 16, 2026

chalmerlowe added kokoro:force-run Add this label to force Kokoro to re-run the tests. and removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jul 16, 2026

chalmerlowe changed the title ~~chore: [WIP] experimental parallel kokoro system test~~ chore: experimental parallel kokoro system test Jul 17, 2026

chalmerlowe added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 17, 2026

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 17, 2026

chalmerlowe marked this pull request as draft July 17, 2026 18:19

chalmerlowe changed the title ~~chore: experimental parallel kokoro system test~~ chore: [WIP] experimental parallel kokoro system test Jul 17, 2026

fix(ci): ensure script exits with failure code when tests fail

5d1e078

daniel-sanche reviewed Jul 17, 2026

View reviewed changes

chalmerlowe added 2 commits July 20, 2026 08:23

feat(ci): enhance log artifacts for Sponge and improve readability

77b3856

feat(ci): list passed packages in test summary for better visibility

dde517d

chalmerlowe commented Jul 21, 2026

View reviewed changes

Comment thread .kokoro/system.sh Outdated

updates word choice

b399688

chalmerlowe mentioned this pull request Jul 22, 2026

chore(bigquery): fix timeout flakiness in job tests #17826

Merged


		printf '%s\n' "${PACKAGES_TO_TEST[@]}" \| xargs -P "$MAX_JOBS" -I {} bash -c 'run_package_test "{}" > "$LOG_DIR/{}.log" 2>&1 \|\| touch "$LOG_DIR/{}.failed"'

		reap_parallel_results \|\| RETVAL=1

Uh oh!

Conversation

chalmerlowe commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Results

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chalmerlowe commented Jul 16, 2026

Uh oh!

chalmerlowe commented Jul 17, 2026

Uh oh!

daniel-sanche Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniel-sanche Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

daniel-sanche Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniel-sanche Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chalmerlowe commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chalmerlowe commented Jul 1, 2026 •

edited

Loading

chalmerlowe Jul 17, 2026 •

edited

Loading

chalmerlowe Jul 20, 2026 •

edited

Loading