Multiple quickstarts crashing with memory corruption (double free) and networksecurity timeouts

[colinmoy]

While triaging GCB failures, I found a recurring failure in fedora-latest-cmake-quickstart-production.

The build officially fails because networksecurity_quickstart repeatedly times out after 300 seconds:

CMake Error at /workspace/cmake/quickstart-runner.cmake:90 (message):
  Non-zero exit status (Process terminated due to timeout) running
  /workspace/cmake-out/google/cloud/networksecurity/networksecurity_quickstart

However, the raw logs show a much deeper systemic memory corruption issue. Several other quickstarts are crashing violently but showing as "Passed" because of the --repeat until-pass:3 CTest flag.

Examples from the logs:

• backupdr_quickstart & alloydb_quickstart: double free or corruption (fasttop)
• video_quickstart: corrupted double-linked list
• servicedirectory_quickstart: malloc_consolidate(): unaligned fastbin chunk detected
• gkebackup_quickstart & vpcaccess_quickstart: Segmentation fault

This looks like a regression in a core library or a dependency ABI mismatch causing heap corruption.

Failed Build Link: https://pantheon.corp.google.com/cloud-build/builds;region=us-east1/30f1aa53-9157-4939-ba8e-a2790abc278b?project=cloud-cpp-testing-resources

[scotthart]

Similar issue: #14411

[colinmoy]

Update: This same memory corruption issue is also crashing core unit tests.

While triaging further, I found that the gcc-oldest-ci build failed because bigtable_data_connection_test aborted with the exact same error:

[==========] 10 tests from 1 test suite ran. (412 ms total)
[  PASSED  ] 10 tests.
double free or corruption (fasttop)

This confirms the regression is not isolated to the quickstarts or the Fedora environment.

Failed Build Link: https://pantheon.corp.google.com/cloud-build/builds;region=us-east1/6ffb59b7-2898-4f59-a617-43251f06eed0?project=cloud-cpp-testing-resources