ci: restore EL7 stresstest coverage, add nightly EL7 functional tests - #805
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: daedd84e8c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
The EL7 stress job cannot pull its image. The nightly functional jobs also cannot start the script or load the required native test library.
🤖 Datadog Autotest · Commit daedd84 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
❌ ErrorsYour PR has failed checks. Please review the issues below and take necessary action before merging. 🚦 1 Pipeline job failed
Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: cb4d5a1 | Docs | View more details | Give us feedback! |
CI Test ResultsRun: #35973884240 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-09-24 08:31:31 UTC |
83a2eb8 to
8a59fb1
Compare
|
Checked all 6 review threads — all point at the very first commit (daedd84), before the follow-up fixes landed. Each is already addressed:
Resolving all six. |
functional:x64-el7-jdk* previously just ran gradlew directly and let its raw exit code decide the job -- ddprof-test/quarantine.txt had no effect on it at all, unlike the GH Actions matrix (#777). Wraps the gradlew invocation in run_tests_with_retry.sh with cell "el7-<jdk>-<config>-amd64". MAX_ATTEMPTS defaults to 1 (no retry) here specifically: the default of 2 means a full second suite run, and this runner has already OOMKilled the container outright once at the current heap/concurrency (PR #805). Losing the flaky-vs-broken distinction is an acceptable trade for not doubling memory pressure on a runner already this tight; revisit once there's more headroom. Extends .functional_job's artifact paths with flake-evidence/, ci-outcome/ and build/logs/attempt.log so a quarantine decision on this job has evidence to show for it. Verified end-to-end inside the real Oracle Linux 7 image with a stub gradlew: a quarantined failure on its own cell now exits 0 through the full functional-tests.sh-shaped invocation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PROF-16014. Root-caused in PR #805: NativeSocketSampler's hooks install and fire correctly (confirmed via a debug-config diagnostic build), but the rate limiter rejects nearly every sample because NativeSocketTestBase.doTcpTransfer()'s assumption that the send buffer fills and blocks after ~32 iterations does not hold on this runner's network stack. Scoped to el7-8-release-amd64, the only cell this has actually been observed on -- not el7-*-release-amd64, so the nightly el7-17/-21 cells still gate for real if this doesn't reproduce there the same way. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kaahos
left a comment
There was a problem hiding this comment.
I've left 2 comments, but nothing blocking. Thanks!
functional:x64-el7-jdk* previously just ran gradlew directly and let its raw exit code decide the job -- ddprof-test/quarantine.txt had no effect on it at all, unlike the GH Actions matrix (#777). Wraps the gradlew invocation in run_tests_with_retry.sh with cell "el7-<jdk>-<config>-amd64". MAX_ATTEMPTS defaults to 1 (no retry) here specifically: the default of 2 means a full second suite run, and this runner has already OOMKilled the container outright once at the current heap/concurrency (PR #805). Losing the flaky-vs-broken distinction is an acceptable trade for not doubling memory pressure on a runner already this tight; revisit once there's more headroom. Extends .functional_job's artifact paths with flake-evidence/, ci-outcome/ and build/logs/attempt.log so a quarantine decision on this job has evidence to show for it. Verified end-to-end inside the real Oracle Linux 7 image with a stub gradlew: a quarantined failure on its own cell now exits 0 through the full functional-tests.sh-shaped invocation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PROF-16014. Root-caused in PR #805: NativeSocketSampler's hooks install and fire correctly (confirmed via a debug-config diagnostic build), but the rate limiter rejects nearly every sample because NativeSocketTestBase.doTcpTransfer()'s assumption that the send buffer fills and blocks after ~32 iterations does not hold on this runner's network stack. Scoped to el7-8-release-amd64, the only cell this has actually been observed on -- not el7-*-release-amd64, so the nightly el7-17/-21 cells still gate for real if this doesn't reproduce there the same way. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
4285688 to
bcd7b1b
Compare
functional:x64-el7-jdk* previously just ran gradlew directly and let its raw exit code decide the job -- ddprof-test/quarantine.txt had no effect on it at all, unlike the GH Actions matrix (#777). Wraps the gradlew invocation in run_tests_with_retry.sh with cell "el7-<jdk>-<config>-amd64". MAX_ATTEMPTS defaults to 1 (no retry) here specifically: the default of 2 means a full second suite run, and this runner has already OOMKilled the container outright once at the current heap/concurrency (PR #805). Losing the flaky-vs-broken distinction is an acceptable trade for not doubling memory pressure on a runner already this tight; revisit once there's more headroom. Extends .functional_job's artifact paths with flake-evidence/, ci-outcome/ and build/logs/attempt.log so a quarantine decision on this job has evidence to show for it. Verified end-to-end inside the real Oracle Linux 7 image with a stub gradlew: a quarantined failure on its own cell now exits 0 through the full functional-tests.sh-shaped invocation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PROF-16014. Root-caused in PR #805: NativeSocketSampler's hooks install and fire correctly (confirmed via a debug-config diagnostic build), but the rate limiter rejects nearly every sample because NativeSocketTestBase.doTcpTransfer()'s assumption that the send buffer fills and blocks after ~32 iterations does not hold on this runner's network stack. Scoped to el7-8-release-amd64, the only cell this has actually been observed on -- not el7-*-release-amd64, so the nightly el7-17/-21 cells still gate for real if this doesn't reproduce there the same way. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PR #790 moved build:x64 onto AlmaLinux 8, which took stresstest:x64 with it since it ran in the same image — runtime coverage at the shipped binary's glibc 2.17 floor went to zero. This follows up on the two items proposed there: - .stresstest_job now takes RUNTIME_IMAGE independently of BUILD_IMAGE, and stresstest:x64 points it at a new Oracle Linux 7 image so it goes back to running against glibc 2.17 while build:x64 stays on EL8. - adds a nightly functional:x64-el7-jdk{8,17,21} job matrix that runs the ddprof-test debug suite against the shipped x64 artifact on the same EL7 image, covering the libc-sensitive suites without multiplying the per-PR matrix. The new .gitlab/base/el7/Dockerfile is runtime-only (no compiler): both jobs consume the AlmaLinux-8-built .so via -Pskip-native -Pwith-libs. functional-tests.sh fetches the JDK under test from Adoptium into JAVA_TEST_HOME so Gradle itself keeps running on JDK 21 regardless of which JDK is under test. BUILD_IMAGE_X64_EL7 is a placeholder pin — the image still needs to be built and pushed via a real pipeline (REBUILD_IMAGES="x64-el7") before this can merge, same as #790's own image rebuild. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Keeps the 3-JDK sweep nightly but lets one JDK's worth of EL7 functional coverage run on every pipeline, so it isn't gated entirely on the nightly schedule. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Built and pushed successfully in pipeline 138399627, job 2056785023. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…r probe stresstest:x64 failed with "No C++ compiler found": ConfigurationPresets. setupStandardConfigurations() calls PlatformUtils.findCompiler() unconditionally at project-configuration time, before -Pskip-native is consulted, so an image with no compiler at all fails to even configure the build. The stock EL7 gcc-c++ (4.8.5, base repo, no EPEL) satisfies that --version probe; -Pskip-native still gates every real compile/link task onlyIf, so it's never invoked to build anything. Verified the image still builds and g++ --version succeeds. This needs another `REBUILD_IMAGES="x64-el7"` run and a re-pin of BUILD_IMAGE_X64_EL7 before the fix takes effect. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rebuilt in pipeline 138438605 with the gcc-c++ fix from the previous commit; pushed and signed successfully before the rebuild-images log cuts off (mid-way through the unrelated datadog-ci image build). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
functional:x64-el7-jdk8 failed with "Permission denied" — the script was committed as mode 100644 instead of 100755. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ContendedCallTraceStorageTest OOM'd with "GC overhead limit exceeded" on jdk8: it drives availableProcessors()*2 threads of CPU/allocation load against the standard -Xmx512m test heap, and the GitLab shared runner pod apparently gives it less headroom than GitHub's ubuntu-latest runners do. Adds an opt-in -PtestMaxHeap gradle property to ddprof-test (appended after the plugin's default -Xmx512m, so the last -Xmx wins) and passes -PtestMaxHeap=1536m from functional-tests.sh only. The shared default used by every other caller (GH Actions matrix, stresstest, etc.) is untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Several ddprof-test suites (NativeThread, DynamicNativeThread, ThreadEntryDetectionTest, ...) need ddprof-test-native's small JNI test helper on java.library.path regardless of which profiler binary is under test. Bare -Pskip-native disabled ALL native compilation project-wide, so those suites failed with UnsatisfiedLinkError/ NoClassDefFoundError on the EL7 functional job, which needs to skip only ddprof-lib's own build (substituted via -Pwith-libs) while still compiling that helper for real. PlatformUtils.isNativeSkipped() now supports -Pskip-native=<comma list of project names>, skipping only those; bare -Pskip-native (no value) keeps skipping everywhere, unchanged for every existing caller (stresstests.sh, build.sh, GH Actions). SimpleNativeLibPlugin (used by ddprof-test-native and malloc-shim) and ddprof-lib/build.gradle.kts's own skip-native check now go through it. functional-tests.sh passes -Pskip-native=ddprof-lib,malloc-shim (malloc-shim excluded too: it's unrelated to ddprof-test and its -std=c++17 flag doesn't compile under EL7's stock gcc 4.8.5). Verified locally: with -Pskip-native=ddprof-lib,malloc-shim, ddprof-test-native:linkLib actually compiles and links, while ddprof-lib:assembleDebugJar only runs copyExternalLibs (no native compile task in the graph at all). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Now that -Pskip-native=ddprof-lib,malloc-shim actually compiles ddprof-test-native for the EL7 functional job, nativealloc.c failed: error: 'for' loop initial declarations are only allowed in C99 mode EL7's stock gcc (4.8.5) still defaults to gnu89; every other Linux gcc/ clang this project builds on already defaults to a C99+ dialect, so this was latent until a compiler this old actually built the file. Adds an explicit -std=gnu99. Verified against the real EL7 toolchain (oraclelinux:7 + gcc 4.8.5): all three ddprof-test-native source files compile and link. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
build:x64 (build.sh) only ever produces and ships the release-config
libjavaProfiler.so -- that IS the shipped artifact this job exists to
exercise. Running testDebug set ddprof_test.config=debug while the
loaded library was actually release-built, so debug-only assertions
failed outright instead of assumeTrue-skipping correctly, e.g.
JVMAccessTest.agentOnLoadVMFlagDetectionTest checking for
[TEST::INFO] log lines that only exist in debug builds
("initLibrary not found", "DebugNonSafepoints flag lookup log line
was not observed").
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
functional:x64-el7-jdk8 has NativeSocketEvent count: 0 across every retry (zero, not just fewer-than-expected), suggesting the send/recv/ write/read hooks (libraryPatcher_linux.cpp's GOT patching) never actually get installed in this environment -- but that can't be told apart from "installed but never firing" without the debug-only hook-fire counters, which only exist in a debug-config build. build:x64-debug-diag builds a real debug-config libjavaProfiler.so on the same AlmaLinux 8 toolchain build:x64 uses (release configs don't carry NativeSocketSampler's #ifdef DEBUG counters/TEST_LOG calls). functional:x64-el7-debug-diag runs just the NativeSocket* suite against it on the EL7 runtime (via functional-tests.sh's new TESTS_FILTER -> -Ptests passthrough), so --info output shows whether send_hook/recv_hook/write_hook/read_hook ever get called at all. Both are marked TEMPORARY and should be removed once this is root-caused. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
build:x64-debug-diag's first attempt failed to load on EL7: UnsatisfiedLinkError: libstdc++.so.6: version GLIBCXX_3.4.20 not found Debug configs link libstdc++ dynamically (only release gets -static-libstdc++), so the .so needed a newer GLIBCXX than EL7's stock gcc 4.8.5 provides. Statically linking (gated behind -Dddprof.debugStaticLibstdcxx=true, diagnostic-only) then surfaced a second, unrelated issue: -static-libstdc++ transitively pulls in libstdc++.a's cow-string-inst.o (legacy ABI string instantiations), which references std::random_device::_M_init -> getentropy@GLIBC_2.25 -- traced via the linker map (-Wl,-Map). That's dead code configureRelease's -Wl,--gc-sections already strips; debug's linkerArgs never had that flag. Adding --gc-sections and --exclude-libs,ALL alongside the static libstdc++ flags (same diagnostic gate) fixes both. Verified end to end: rebuilt in the real AlmaLinux 8 image, confirmed via objdump that the result has no libstdc++.so.6 NEEDED entry, no getentropy reference, and a max GLIBC_2.17 requirement, then loaded it inside the real EL7 image and saw VM::initLibrary log successfully. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
build:x64-debug-diag and functional:x64-el7-debug-diag did their job: they showed the socket hooks fire correctly on EL7 (send/recv/write/ read all called, PLT-patched libjvm.so/libnio.so/libnet.so), and that the real cause of NativeSocketEvent count: 0 is the sampler's rate limiter legitimately rejecting nearly every call because NativeSocketTestBase.doTcpTransfer()'s "the send buffer fills after ~32 iterations and blocks" assumption doesn't hold on this runner's network stack -- a real test bug, not a CI/EL7 problem. That will be quarantined separately now that #777 is merged. Drops the diagnostic-only -Dddprof.debugStaticLibstdcxx gate in ConfigurationPresets.kt and the now-unused TESTS_FILTER passthrough in functional-tests.sh along with the jobs that were the only callers of either. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
"(rebuilt with gcc-c++ added, see el7/Dockerfile)" and "restoring the runtime coverage build:x64 lost when it moved to AlmaLinux 8 (PR #790)" tell a reader what changed relative to old code, not why the current code is the way it is -- and the -std=gnu99 comment named the exact source construct that motivated it, which goes stale the moment that code changes. State the current rationale only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…sistency
- .gitlab-ci.yml: drop "unchanged from before" -- describes the diff
against the file's prior state, not the current mechanism.
- ddprof-lib/build.gradle.kts: its own skip-native check still used
bare hasProperty("skip-native"), not PlatformUtils.isNativeSkipped().
Both agree today (every caller either omits the property, uses it
bare, or lists "ddprof-lib" whenever it's meant to be skipped), so
this isn't a behavior change -- confirmed locally: assembleDebugJar
still runs only copyExternalLibs under
-Pskip-native=ddprof-lib,malloc-shim, and still depends on the real
compile/link without it. But a future -Pskip-native=<list not
containing ddprof-lib> would silently skip ddprof-lib's own build
too under the old bare check, contradicting isNativeSkipped()'s own
doc comment. Fixes the one call site the new scoped semantics
didn't reach.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Flagged in review: functional-tests.sh's .jdk-cache/ was never in any cache paths -- .cache-config/.cache-config-pull only cover .gradle/caches/, .gradle/wrapper/, .m2/repository/ -- so every run re-downloaded the JDK tarball from Adoptium. Gives .functional_job its own two-entry cache instead of extending .cache-config-pull: the JDK tarball's natural cache key is the major version (TEST_JDK), not the branch/commit the Gradle cache is keyed on, and it doesn't change per-branch the way the Gradle cache's gradle-wrapper.properties hash does, so it gets pull-push with a static, ref-independent key while the Gradle entry stays pull-only exactly as before. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Flagged in review: only ddprof-lib/build.gradle.kts and
SimpleNativeLibPlugin.kt went through PlatformUtils.isNativeSkipped()
after the earlier fix; GtestPlugin.kt, GtestTaskBuilder.kt,
FuzzTargetsPlugin.kt and ddprof-lib/benchmarks/build.gradle.kts still
used the bare hasProperty("skip-native") check, so
-Pskip-native=<list not containing ddprof-lib> would silently skip
gtest/fuzz/benchmarks too, contradicting the scoped semantics
everywhere else now documents.
GtestPlugin.kt/GtestTaskBuilder.kt are applied directly to ddprof-lib
(project.name == "ddprof-lib" there already), so those are a direct
swap. FuzzTargetsPlugin.kt and benchmarks/build.gradle.kts are applied
to the :ddprof-lib:fuzz and :ddprof-lib:benchmarks *subprojects*
(project.name == "fuzz"/"benchmarks") -- fuzzing and benchmarking are
part of ddprof-lib's native surface, not something a caller names on
its own, so those go through project.parent's identity instead of
their own.
Verified locally: -Pskip-native=ddprof-lib,malloc-shim still skips
:ddprof-lib:benchmarks:compileBenchmark (SKIPPED, not executed), and
it still compiles normally without the flag -- matching the old bare
check's behavior for every current caller, while now correctly
generalizing for a caller that doesn't name ddprof-lib.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bcd7b1b to
cb4d5a1
Compare
functional:x64-el7-jdk* previously just ran gradlew directly and let its raw exit code decide the job -- ddprof-test/quarantine.txt had no effect on it at all, unlike the GH Actions matrix (#777). Wraps the gradlew invocation in run_tests_with_retry.sh with cell "el7-<jdk>-<config>-amd64". MAX_ATTEMPTS defaults to 1 (no retry) here specifically: the default of 2 means a full second suite run, and this runner has already OOMKilled the container outright once at the current heap/concurrency (PR #805). Losing the flaky-vs-broken distinction is an acceptable trade for not doubling memory pressure on a runner already this tight; revisit once there's more headroom. Extends .functional_job's artifact paths with flake-evidence/, ci-outcome/ and build/logs/attempt.log so a quarantine decision on this job has evidence to show for it. Verified end-to-end inside the real Oracle Linux 7 image with a stub gradlew: a quarantined failure on its own cell now exits 0 through the full functional-tests.sh-shaped invocation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PROF-16014. Root-caused in PR #805: NativeSocketSampler's hooks install and fire correctly (confirmed via a debug-config diagnostic build), but the rate limiter rejects nearly every sample because NativeSocketTestBase.doTcpTransfer()'s assumption that the send buffer fills and blocks after ~32 iterations does not hold on this runner's network stack. Scoped to el7-8-release-amd64, the only cell this has actually been observed on -- not el7-*-release-amd64, so the nightly el7-17/-21 cells still gate for real if this doesn't reproduce there the same way. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
functional:x64-el7-jdk* previously just ran gradlew directly and let its raw exit code decide the job -- ddprof-test/quarantine.txt had no effect on it at all, unlike the GH Actions matrix (#777). Wraps the gradlew invocation in run_tests_with_retry.sh with cell "el7-<jdk>-<config>-amd64". MAX_ATTEMPTS defaults to 1 (no retry) here specifically: the default of 2 means a full second suite run, and this runner has already OOMKilled the container outright once at the current heap/concurrency (PR #805). Losing the flaky-vs-broken distinction is an acceptable trade for not doubling memory pressure on a runner already this tight; revisit once there's more headroom. Extends .functional_job's artifact paths with flake-evidence/, ci-outcome/ and build/logs/attempt.log so a quarantine decision on this job has evidence to show for it. Verified end-to-end inside the real Oracle Linux 7 image with a stub gradlew: a quarantined failure on its own cell now exits 0 through the full functional-tests.sh-shaped invocation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PROF-16014. Root-caused in PR #805: NativeSocketSampler's hooks install and fire correctly (confirmed via a debug-config diagnostic build), but the rate limiter rejects nearly every sample because NativeSocketTestBase.doTcpTransfer()'s assumption that the send buffer fills and blocks after ~32 iterations does not hold on this runner's network stack. Scoped to el7-8-release-amd64, the only cell this has actually been observed on -- not el7-*-release-amd64, so the nightly el7-17/-21 cells still gate for real if this doesn't reproduce there the same way. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(ci): make the quarantine machinery run under Python 3.6 Follow-up to #777, for wiring the GitLab EL7 functional job into quarantine: EL7's base-repo python3 is 3.6.8 (no SCL/newer-Python repo exists for Oracle Linux 7), and running the real scripts inside that image crashed twice: add_subparsers(dest="command", required=True) TypeError: __init__() got an unexpected keyword argument 'required' datetime.date.fromisoformat(review_by) AttributeError: type object 'datetime.date' has no attribute 'fromisoformat' Both need Python 3.7+. Drops required=True (checking args.command is None manually afterward) and replaces fromisoformat with strptime(s, "%Y-%m-%d").date() everywhere it was called. Verified end-to-end inside the real Oracle Linux 7 image: a quarantined failure on its own cell now goes green under python3 3.6.8, and the existing 59 test_quarantine.sh assertions (which run under whatever python3 is on the host, typically 3.9+) still pass unchanged. Also adds "el7" to KNOWN_LIBCS: the GitLab functional job's Oracle Linux 7 runtime is technically glibc but must not share the "glibc" cell token with the GitHub Actions Ubuntu matrix -- confirmed by a second smoke test that a quarantine entry scoped to el7-8-release-amd64 does not excuse the identical failure on glibc-8-release-amd64. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * ci: add python3 to the EL7 image for the quarantine classifier functional-tests.sh is about to wrap its gradlew invocation in run_tests_with_retry.sh, which shells out to quarantine.py and flake_report.py. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * ci: wire functional-tests.sh into the quarantine machinery functional:x64-el7-jdk* previously just ran gradlew directly and let its raw exit code decide the job -- ddprof-test/quarantine.txt had no effect on it at all, unlike the GH Actions matrix (#777). Wraps the gradlew invocation in run_tests_with_retry.sh with cell "el7-<jdk>-<config>-amd64". MAX_ATTEMPTS defaults to 1 (no retry) here specifically: the default of 2 means a full second suite run, and this runner has already OOMKilled the container outright once at the current heap/concurrency (PR #805). Losing the flaky-vs-broken distinction is an acceptable trade for not doubling memory pressure on a runner already this tight; revisit once there's more headroom. Extends .functional_job's artifact paths with flake-evidence/, ci-outcome/ and build/logs/attempt.log so a quarantine decision on this job has evidence to show for it. Verified end-to-end inside the real Oracle Linux 7 image with a stub gradlew: a quarantined failure on its own cell now exits 0 through the full functional-tests.sh-shaped invocation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * ci: quarantine the NativeSocket* flake on el7-8-release-amd64 PROF-16014. Root-caused in PR #805: NativeSocketSampler's hooks install and fire correctly (confirmed via a debug-config diagnostic build), but the rate limiter rejects nearly every sample because NativeSocketTestBase.doTcpTransfer()'s assumption that the send buffer fills and blocks after ~32 iterations does not hold on this runner's network stack. Scoped to el7-8-release-amd64, the only cell this has actually been observed on -- not el7-*-release-amd64, so the nightly el7-17/-21 cells still gate for real if this doesn't reproduce there the same way. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs: keep the quarantine.txt example even with real entries present The example line was written to be deleted once the first real entry landed, but it's worth keeping as a permanent format reference -- the NativeSocket entries are tied to a bug that will get fixed and removed, and the file shouldn't go back to having no example once that happens. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * style: move _parse_date next to its first use Inserting it in the module-level constants block left it wedged between two densely-packed constant groups with no blank-line separation, breaking the file's own two-blank-lines-around-defs convention. Moves it next to is_expired(), its first caller, matching how the file already places its other small helpers near their use. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix: quarantine NativeSocket* by class, not by exact method Flagged in review, and confirmed against both the test source and quarantine.txt's own documented format: every affected class's single test method is @RetryingTest, which -- like @ParameterizedTest and @testtemplate -- reports in JUnit XML as "Class.[1]", "[2]", ... with no method name at all. The exact-method entries could never match anything; only the already-class-wide NativeSocketStackTraceTest.* entry ever worked. Converts the other 8 to <class>.*. Also drops the NativeSocketUdpExcludedTest entry entirely rather than widening it: that test asserts *zero* events, which the bug's own symptom (near-zero events everywhere) would make pass, not fail. Its earlier observed failure has an unconfirmed, likely different cause, and quarantining it on this evidence risks hiding an unrelated regression later. Verified: find_entry() now matches the real "NativeSocketEnabledTest.[1]"-shaped id against the class-wide entry, and confirms the UDP test is not quarantined. All 59 test_quarantine.sh assertions and quarantine.py validate still pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * ci: repin BUILD_IMAGE_X64_EL7 to the python3-enabled image Built and pushed successfully in pipeline 139300629 (the job's overall failure was two unrelated images, arm64-musl and benchmarks-amd64). Confirmed python3 3.6.8 installed in the build log. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * ci: raise container memory limit for EL7 functional job The x64-el7 functional job was getting OOMKilled (exit 137) at the container level during pod cleanup, after the test suite had already finished and uploaded its artifacts. The JVM under test already has its own -Xmx1536m cap; the pod itself needed more headroom to run Gradle's daemon/workers, the gtest builds, and the JVM together. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * build: compile build-logic in-process instead of a Kotlin daemon The Kotlin compile daemon outlives the build-logic compile and stays resident for the rest of the build, including the whole test run, next to the Gradle process and the test JVMs. Measured on a --no-daemon build: ~580MB Gradle + ~660MB Kotlin daemon with the default strategy, versus ~920MB Gradle and no daemon in-process. The out-of-process strategy was removed in Kotlin 2.4.0. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Revert "build: compile build-logic in-process instead of a Kotlin daemon" This reverts commit 7864e74. * ci: compile build-logic in-process only in the EL7 functional job The Kotlin compile daemon (~660MB RSS) stays resident through the whole test run in this 6GB pod. Setting in-process globally via build-logic/gradle.properties broke the CodeQL Java job: its Kotlin extractor runs inside the compiler, and in-process that exhausted Gradle's default 512m heap. Scope the setting to this job instead. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Revert "ci: raise container memory limit for EL7 functional job" This reverts commit a794f69. * fix: gate a quarantined failure when the final attempt was killed by a signal A signal that kills the Gradle JVM itself (e.g. the OOM killer taking the --no-daemon build process) leaves nothing alive to print a cut-short marker, and with MAX_ATTEMPTS=1 there is no earlier attempt to measure a shortfall against. If a quarantined test had already written its failure XML, the report excused the run and the wrapper turned exit 137 into 0. Treat an exit status above 128 as the attempt being cut short. Gradle never exits above 128 on its own, however many tests fail. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: honor skip-native=true as a global native skip gradle.properties.template documents skip-native=true, but isNativeSkipped() only treated an empty value as the global skip and read anything else as a list of project names. "true" matched no project, so native compilation ran anyway. Treat "true" like a bare -Pskip-native, and "false" as skipping nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Summary
Follow-up to #790's "One coverage regression this introduces, and how to undo it" section.
Moving
build:x64to AlmaLinux 8 (#790) tookstresstest:x64with it, since it ran in the same image — runtime coverage at the shipped binary's actual glibc floor (2.17) went from thin to zero. This implements the two follow-ups #790 proposed:.stresstest_jobgets its own runtime image, independent of the image the artifact was built in.RUNTIME_IMAGEreplaces the reusedBUILD_IMAGEvariable in the job template;stresstest:x64now points it at a new Oracle Linux 7 image (glibc 2.17), whilebuild:x64stays on AlmaLinux 8 — matching ci: move the shipped binaries to AlmaLinux 8, get the test images off EOL Debian #790's own note that this "is better than the old arrangement, because it would exercise the el8-built artifact on 2.17 rather than a 2.17-built one." Every otherstresstest:*job keeps its existing image, just renamed to the new variable.functional:x64-el7-jdk{8,17,21}, runs theddprof-testsuite against the shipped x64 artifact on the same EL7 image — the "libc-sensitive suites" coverage ci: move the shipped binaries to AlmaLinux 8, get the test images off EOL Debian #790 called out.jdk17andjdk21are nightly-only (CI_PIPELINE_SOURCE == "schedule", same pattern as the existingchaos:buildjob);jdk8also runs on every pipeline — it's the oldest/most different from the JDK 21 that runs Gradle, so EL7 functional coverage isn't gated entirely on the nightly schedule.Details
.gitlab/base/el7/Dockerfile(new): Oracle Linux 7. Built and pushed for real viarebuild-images(BUILD_IMAGE_X64_EL7, currently pinned to the pipeline-138438605 digest,gcc-c++included — see below)..gitlab/scripts/functional-tests.sh(new): fetches the JDK under test from Adoptium intoJAVA_TEST_HOME, so Gradle itself keeps running on JDK 21 (Gradle 9's minimum) regardless of which JDK the suite targets. RunstestRelease, nottestDebug—build:x64only ever ships a release-config.so, sotestDebugwould setddprof_test.config=debugagainst a library that was actually built release, and debug-only assertions would fail outright instead of correctlyassumeTrue-skipping..gitlab/build-deploy/images.yml,rebuild-images.sh,check-image-updates.sh: wiredBASE_IMAGE_EL7/x64-el7through the same image-build, repin and staleness-check machinery as the existing images.PlatformUtils.isNativeSkipped()(build-logic):-Pskip-nativenow also accepts a comma-separated project list (-Pskip-native=ddprof-lib,malloc-shim) instead of only the bare, skip-everywhere form.functional-tests.shneeds to skip onlyddprof-lib's own native build (substituted via-Pwith-libs) while still compilingddprof-test-native's small JNI test helper for real — severalddprof-testsuites need it onjava.library.pathregardless of which profiler binary is under test, and the bare flag disabled that unconditionally.ddprof-test-native/build.gradle.kts: added-std=gnu99— one of its.cfiles uses a C99 for-loop-scoped declaration, which EL7's stock gcc 4.8.5 rejects under its defaultgnu89dialect (every other gcc/clang this project builds on already defaults to C99+, so this was latent until a compiler this old actually compiled the file).ddprof-test/build.gradle.kts: added an opt-in-PtestMaxHeapproperty (appended after the plugin's default-Xmx512m, so the last-Xmxwins), andfunctional-tests.shpasses-PtestMaxHeap=1536m. One suite (ContendedCallTraceStorageTest) drives more concurrent CPU/allocation load than this shared runner's pod apparently gives it headroom for at the default heap. Every other caller (GH Actions matrix,stresstest.sh) is unaffected..gitlab/base/el8/Dockerfile: addedgcc-c++. Gradle'sConfigurationPresets.setupStandardConfigurations()callsfindCompiler()unconditionally at configuration time, before-Pskip-nativeis even consulted — an image with no compiler at all fails to configure the build, even though-Pskip-nativemeans it's never actually invoked. (Applies to the EL7 imagefunctional-tests.shruns in, not the shipped AlmaLinux 8 build.)A real bug this surfaced (not fixed here)
With all of the above green,
functional:x64-el7-jdk8still fails — but now for a legitimate reason, not a CI bug. EveryNativeSocket*test reportsdatadog.NativeSocketEvent count: 0.Built a temporary debug-config diagnostic build (statically linked, since debug configs don't get
-static-libstdc++and needed it to even load on EL7 — see the now-removed diagnostic commits for the full trail) to getNativeSocketSampler's debug-only hook-fire counters. Confirmed the hooks are not the problem:send_hook/recv_hook/write_hook/read_hookall fire correctly, PLT patching succeeds acrosslibjvm.so/libnio.so/libnet.so. Thestop summarylines show the sampler's rate limiter rejecting nearly every single call (accept=0on almost every line, out of hundreds to thousands of calls) because none of them are actually slow —NativeSocketTestBase.doTcpTransfer()'s comment claims the persistent connection's send buffer fills after ~32 iterations and blocks for >1ms, but on this runner's network stack that assumption doesn't hold, so the time-weighted sampler correctly rejects everything as "too fast to be worth a sample."That's a real test-portability bug in
NativeSocketTestBase, unrelated to EL7/glibc-2.17 specifically. Tracked separately for quarantine (ddprof-test/quarantine.txt) now that #777 is merged, rather than worked around here.Test plan
.gitlab/base/el7/Dockerfilebuilds locally and the resulting image runsjava -versionagainst glibc 2.17x64-el7image built and pushed via a real pipeline, digest pinnedstresstest:x64andfunctional:x64-el7-jdk8run in real GitLab pipelines through several rounds of fixes (compiler probe, exec permission, C99, debug/release config, heap) — each fix verified against the actual failure it targetsfunctional:x64-el7-jdk8failure (NativeSocket*) root-caused via a temporary diagnostic build/job pair, now removed🤖 Generated with Claude Code