Conversation
Contributor
❌ ErrorsYour PR has failed checks. Please review the issues below and take necessary action before merging. 🚦 2 Pipeline jobs failed
Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: 2bd422e | Docs | View more details | Give us feedback! |
Contributor
CI Test ResultsRun: #35248781237 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-09-17 17:06:27 UTC |
Contributor
Reliability & Chaos Results❌ 1 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/127431974 ❌ chaos: profiler gmalloc aarch64 21 0 3 temXchaos |
Contributor
Benchmark Results (commit cfec595)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/123822759 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1949 / 2055 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · | ✅ / · |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 8727 / 8389 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8166 / 8396 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1292 / 1256 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1288 / 1290 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2830 / 3023 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2976 / 2821 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | · / ✅ | · / ✅ | · / 3 | · / 3555 | · / ✅ | · / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3499 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1592 / 1697 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1922 / 1854 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 10, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Benchmark Results (commit 0714bd3)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124004132 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10184 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +1.8% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8891 ms (24 iters) | ✅ 8865 ms (24 iters) | ≈ -0.3% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6003 ms (33 iters) | ✅ 6034 ms (33 iters) | ≈ +0.5% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5507 ms (36 iters) | ✅ 5426 ms (36 iters) | ≈ -1.5% (±23.8%) | |
| fj-kmeans | 21 | ✅ 2834 ms (66 iters) | ✅ 2761 ms (67 iters) | 🟢 -2.6% | — / — |
| fj-kmeans | 25 | ✅ 2824 ms (66 iters) | ✅ 2838 ms (66 iters) | ≈ +0.5% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2113 ms (88 iters) | ✅ 2065 ms (89 iters) | ≈ -2.3% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2059 ms (90 iters) | ✅ 2102 ms (89 iters) | ≈ +2.1% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1283 ms (134 iters) | ✅ 1260 ms (135 iters) | ≈ -1.8% (±32%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1017 ms (168 iters) | ≈ +0.1% (±31.5%) | — / — |
| reactors | 21 | ✅ 16658 ms (15 iters) | ✅ 16371 ms (16 iters) | ≈ -1.7% (±9%) | — / — |
| reactors | 25 | ✅ 18507 ms (15 iters) | ✅ 18056 ms (15 iters) | ≈ -2.4% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1929 / 2034 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2310 / 2113 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 8529 / 8852 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8635 / 8390 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 5 | 1284 / 1281 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 2 | 1283 / 1285 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2930 / 2864 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2963 / 2953 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 6 | 3558 / 3510 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3463 / 3469 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1564 / 1782 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1905 / 1681 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 700d838)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124275960 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10425 ms (21 iters) | ✅ 10259 ms (21 iters) | ≈ -1.6% (±11.1%) | — / — |
| akka-uct | 25 | ✅ 8748 ms (24 iters) | ✅ 8797 ms (24 iters) | ≈ +0.6% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5970 ms (33 iters) | ✅ 5982 ms (33 iters) | ≈ +0.2% (±25.3%) | |
| finagle-chirper | 25 | ✅ 5469 ms (36 iters) | ✅ 5468 ms (36 iters) | ≈ -0% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2598 ms (73 iters) | ✅ 2715 ms (69 iters) | 🔴 +4.5% | — / — |
| fj-kmeans | 25 | ✅ 2784 ms (67 iters) | ✅ 2847 ms (66 iters) | ≈ +2.3% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2122 ms (87 iters) | ✅ 2126 ms (88 iters) | ≈ +0.2% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2035 ms (91 iters) | ✅ 2057 ms (91 iters) | ≈ +1.1% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1304 ms (131 iters) | ✅ 1253 ms (136 iters) | ≈ -3.9% (±32.2%) | — / — |
| naive-bayes | 25 | ✅ 1015 ms (169 iters) | ✅ 1018 ms (168 iters) | ≈ +0.3% (±31.8%) | — / — |
| reactors | 21 | ✅ 16107 ms (15 iters) | ✅ 16847 ms (15 iters) | ≈ +4.6% (±7.9%) | — / — |
| reactors | 25 | ✅ 18469 ms (15 iters) | ✅ 19075 ms (14 iters) | ≈ +3.3% (±4.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 1950 / 1991 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2228 / 2453 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8686 / 8417 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8561 / 8290 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1272 / 1289 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 1287 / 1298 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2912 / 3017 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2907 / 2872 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 3515 / 3489 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 7 | 3498 / 3463 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1732 / 1701 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1780 / 1924 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 14, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 14, 2026 12:03
700d838 to
9972bff
Compare
Contributor
Benchmark Results (commit 9972bff)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124511782 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit a636398)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124700626 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit f2d8978)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124747648 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit a48899b)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124773626 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10376 ms (21 iters) | ✅ 10228 ms (21 iters) | ≈ -1.4% (±11%) | — / — |
| akka-uct | 25 | ✅ 8829 ms (24 iters) | ✅ 8859 ms (24 iters) | ≈ +0.3% (±10%) | — / — |
| finagle-chirper | 21 | ✅ 6001 ms (33 iters) | ✅ 5928 ms (33 iters) | ≈ -1.2% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5469 ms (36 iters) | ✅ 5475 ms (36 iters) | ≈ +0.1% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2687 ms (70 iters) | ✅ 2761 ms (68 iters) | 🔴 +2.8% | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2805 ms (66 iters) | ≈ -0.2% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2047 ms (91 iters) | ✅ 2116 ms (88 iters) | 🔴 +3.4% | — / — |
| future-genetic | 25 | ✅ 2000 ms (93 iters) | ✅ 2083 ms (89 iters) | 🔴 +4.2% | — / — |
| naive-bayes | 21 | ✅ 1256 ms (136 iters) | ✅ 1253 ms (136 iters) | ≈ -0.2% (±32.8%) | — / — |
| naive-bayes | 25 | ✅ 1016 ms (169 iters) | ✅ 1016 ms (169 iters) | ≈ 0% (±31.3%) | — / — |
| reactors | 21 | ✅ 16870 ms (15 iters) | ✅ 16463 ms (15 iters) | ≈ -2.4% (±7.3%) | — / — |
| reactors | 25 | ✅ 18270 ms (15 iters) | ✅ 18333 ms (15 iters) | ≈ +0.3% (±4.5%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 1988 / 2024 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 2263 / 2395 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 6 | 8440 / 8679 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 8283 / 8410 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1288 / 1260 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 1250 / 1295 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 2968 / 3003 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2843 / 2902 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 3474 / 3510 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 3495 / 3488 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1726 / 1721 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1893 / 1750 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit 787f29e)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124811024 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10215 ms (21 iters) | ✅ 10475 ms (21 iters) | ≈ +2.5% (±11.9%) | — / — |
| akka-uct | 25 | ✅ 8843 ms (24 iters) | ✅ 8861 ms (24 iters) | ≈ +0.2% (±10.5%) | — / — |
| finagle-chirper | 21 | ✅ 5986 ms (33 iters) | ✅ 5971 ms (33 iters) | ≈ -0.3% (±25.5%) | |
| finagle-chirper | 25 | ✅ 5515 ms (36 iters) | ✅ 5479 ms (36 iters) | ≈ -0.7% (±24.2%) | |
| fj-kmeans | 21 | ✅ 2693 ms (70 iters) | ✅ 2836 ms (66 iters) | 🔴 +5.3% | — / — |
| fj-kmeans | 25 | ✅ 2820 ms (66 iters) | ✅ 2837 ms (66 iters) | ≈ +0.6% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2059 ms (90 iters) | ✅ 2070 ms (89 iters) | ≈ +0.5% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2089 ms (89 iters) | ✅ 2130 ms (87 iters) | ≈ +2% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1284 ms (133 iters) | ✅ 1313 ms (131 iters) | ≈ +2.3% (±33%) | — / — |
| naive-bayes | 25 | ✅ 1009 ms (170 iters) | ✅ 988 ms (173 iters) | ≈ -2.1% (±31.4%) | — / — |
| reactors | 21 | ✅ 16540 ms (15 iters) | ✅ 16155 ms (15 iters) | ≈ -2.3% (±7.1%) | — / — |
| reactors | 25 | ✅ 18545 ms (15 iters) | ✅ 18816 ms (15 iters) | ≈ +1.5% (±4.2%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1987 / 1990 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 4 | 2238 / 2323 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8290 / 8529 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 8323 / 8386 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1299 / 1277 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1274 / 1299 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2920 / 2982 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 2886 / 2864 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 3505 / 3499 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1593 / 1669 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1849 / 1940 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit fede524)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124833099 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit ef0f5c6)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124961539 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit 1299316)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124967829 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10336 ms (21 iters) | ✅ 10245 ms (21 iters) | ≈ -0.9% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8868 ms (24 iters) | ✅ 8789 ms (24 iters) | ≈ -0.9% (±10.7%) | — / — |
| finagle-chirper | 21 | ✅ 5968 ms (33 iters) | ✅ 5971 ms (33 iters) | ≈ +0.1% (±25.8%) | |
| finagle-chirper | 25 | ✅ 5510 ms (36 iters) | ✅ 5537 ms (36 iters) | ≈ +0.5% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2696 ms (69 iters) | ✅ 2822 ms (66 iters) | 🔴 +4.7% | — / — |
| fj-kmeans | 25 | ✅ 2842 ms (66 iters) | ✅ 2746 ms (68 iters) | 🟢 -3.4% | — / — |
| future-genetic | 21 | ✅ 2068 ms (90 iters) | ✅ 2051 ms (90 iters) | ≈ -0.8% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2134 ms (87 iters) | ✅ 2059 ms (90 iters) | 🟢 -3.5% | — / — |
| naive-bayes | 21 | ✅ 1318 ms (130 iters) | ✅ 1253 ms (136 iters) | ≈ -4.9% (±31.8%) | — / — |
| naive-bayes | 25 | ✅ 998 ms (171 iters) | ✅ 1025 ms (167 iters) | ≈ +2.7% (±32.1%) | — / — |
| reactors | 21 | ✅ 16264 ms (15 iters) | ✅ 16625 ms (15 iters) | ≈ +2.2% (±7.4%) | — / — |
| reactors | 25 | ✅ 17926 ms (15 iters) | ✅ 18917 ms (15 iters) | ≈ +5.5% (±5.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1917 / 1934 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 2285 / 2434 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 2 | 8503 / 8628 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 8331 / 8664 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1287 / 1266 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1277 / 1290 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 3037 / 2981 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 2897 / 2943 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 12 | 3507 / 3493 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 5 | 3446 / 3481 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 1765 / 1665 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1855 / 1920 | ✅ / ✅ | ✅ / ✅ |
jbachorik
added a commit
that referenced
this pull request
Jul 16, 2026
…racking Addresses review findings from PR #644 (reference chains for surviving live-heap samples): FrontierTable's shared-lock mutation race, a single-byte JFR event size prefix that silently truncates chains longer than 255 bytes, blocking sample-lock retries in writeReferenceChain() now bounded by a shared per-batch deadline with a drop counter, unvalidated referencechains sub-option values, a startThread()/ pthread_kill() race publishing _running before the thread handle is initialized, a null-JNIEnv leak in threadLoop(), releaseSearchTags() now surfacing GetObjectsWithTags() failures so restartSearch() never resets tag state prematurely, resolveLoadedClasses() skipping its per-class scan only when the loaded-class count is unchanged (not just non-decreasing), a spurious COMPLETED state after a failed first-pass FollowReferences call, and removal of a leftover debug helper in ExternalProcessReferenceChainTest. Adds regression test coverage for release-failure and negative-value option paths. Verified via the full ddprof-lib gtestDebug suite (149/149 tasks, 57/57 referenceChains_ut tests). Environment: Datadog workspace Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 16, 2026 11:31
1299316 to
baba070
Compare
Contributor
Benchmark Results (commit baba070)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124983919 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Benchmark Results (commit 35c49c2)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/124997895 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10367 ms (21 iters) | ✅ 10371 ms (21 iters) | ≈ +0% (±11.3%) | — / — |
| akka-uct | 25 | ✅ 8917 ms (24 iters) | ✅ 8766 ms (24 iters) | ≈ -1.7% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 5986 ms (33 iters) | ✅ 5972 ms (33 iters) | ≈ -0.2% (±25.3%) | |
| fj-kmeans | 21 | ✅ 2801 ms (67 iters) | ✅ 2750 ms (68 iters) | ≈ -1.8% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2842 ms (66 iters) | ✅ 2756 ms (68 iters) | 🟢 -3% | — / — |
| future-genetic | 21 | ✅ 2142 ms (87 iters) | ✅ 2080 ms (89 iters) | 🟢 -2.9% | — / — |
| future-genetic | 25 | ✅ 1991 ms (94 iters) | ✅ 2035 ms (91 iters) | ≈ +2.2% (±2.7%) | — / — |
| naive-bayes | 25 | ✅ 1025 ms (167 iters) | ✅ 1010 ms (169 iters) | ≈ -1.5% (±31.4%) | — / — |
| reactors | 21 | ✅ 16244 ms (15 iters) | ✅ 16474 ms (15 iters) | ≈ +1.4% (±7.9%) | — / — |
| reactors | 25 | ✅ 18288 ms (15 iters) | ✅ 18332 ms (15 iters) | ≈ +0.2% (±3.9%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 1950 / 1991 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2303 / 2180 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 6 | 8767 / 8881 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 8034 / 8300 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1316 / 1287 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1289 / 1282 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2889 / 2882 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2935 / 2838 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 8 / 2 | 3496 / 3495 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 3477 / 3510 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1642 / 1586 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1802 / 1759 | ✅ / ✅ | ✅ / ✅ |
Contributor
Benchmark Results (commit a9fb05f)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125008807 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10309 ms (21 iters) | ✅ 10453 ms (21 iters) | ≈ +1.4% (±11%) | — / — |
| akka-uct | 25 | ✅ 8814 ms (24 iters) | ✅ 8779 ms (24 iters) | ≈ -0.4% (±10.4%) | — / — |
| finagle-chirper | 21 | ✅ 6006 ms (33 iters) | ✅ 6047 ms (33 iters) | ≈ +0.7% (±25.8%) | |
| finagle-chirper | 25 | ✅ 5467 ms (36 iters) | ✅ 5455 ms (36 iters) | ≈ -0.2% (±24.8%) | |
| fj-kmeans | 21 | ✅ 2772 ms (68 iters) | ✅ 2687 ms (70 iters) | 🟢 -3.1% | — / — |
| fj-kmeans | 25 | ✅ 2810 ms (66 iters) | ✅ 2801 ms (66 iters) | ≈ -0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2087 ms (88 iters) | ✅ 2055 ms (90 iters) | ≈ -1.5% (±2.7%) | — / — |
| future-genetic | 25 | ✅ 2088 ms (89 iters) | ✅ 2046 ms (91 iters) | ≈ -2% (±2.5%) | — / — |
| naive-bayes | 21 | ✅ 1233 ms (139 iters) | ✅ 1257 ms (136 iters) | ≈ +1.9% (±32.9%) | — / — |
| naive-bayes | 25 | 💥 0 ms (0 iters) | 💥 0 ms (0 iters) | — | — / — |
| reactors | 21 | ✅ 16947 ms (15 iters) | ✅ 15936 ms (15 iters) | ≈ -6% (±8.2%) | — / — |
| reactors | 25 | ✅ 18681 ms (15 iters) | ✅ 18731 ms (15 iters) | ≈ +0.3% (±5.3%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2008 / 2140 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2058 / 2262 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8437 / 8783 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 8064 / 8228 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 6 / ✅ | 1275 / 1291 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 1250 / 1261 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2876 / 3036 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 2912 / 2869 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 3492 / 3491 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1892 / 1633 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1914 / 1937 | ✅ / ✅ | ✅ / ✅ |
3 tasks
Contributor
Benchmark Results (commit 2033006)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125072159 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10363 ms (21 iters) | ✅ 10147 ms (21 iters) | ≈ -2.1% (±11.6%) | — / — |
| akka-uct | 25 | ✅ 8966 ms (24 iters) | ✅ 8873 ms (24 iters) | ≈ -1% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6007 ms (33 iters) | ✅ 5986 ms (33 iters) | ≈ -0.3% (±26.1%) | |
| finagle-chirper | 25 | ✅ 5488 ms (36 iters) | ✅ 5433 ms (36 iters) | ≈ -1% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2726 ms (68 iters) | ✅ 2760 ms (68 iters) | ≈ +1.2% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2848 ms (66 iters) | ✅ 2743 ms (69 iters) | 🟢 -3.7% | — / — |
| future-genetic | 21 | ✅ 2062 ms (90 iters) | ✅ 2047 ms (90 iters) | ≈ -0.7% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 1994 ms (93 iters) | ✅ 2057 ms (90 iters) | 🔴 +3.2% | — / — |
| naive-bayes | 21 | ✅ 1269 ms (135 iters) | ✅ 1258 ms (136 iters) | ≈ -0.9% (±32.5%) | — / — |
| naive-bayes | 25 | ✅ 1012 ms (169 iters) | ✅ 1016 ms (168 iters) | ≈ +0.4% (±31.6%) | — / — |
| reactors | 21 | ✅ 16554 ms (15 iters) | ✅ 16630 ms (15 iters) | ≈ +0.5% (±8.5%) | — / — |
| reactors | 25 | ✅ 18407 ms (15 iters) | ✅ 18308 ms (15 iters) | ≈ -0.5% (±4.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2101 / 2026 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 2336 / 2365 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 8772 / 8497 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8397 / 8470 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1260 / 1274 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1303 / 1307 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2893 / 2882 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2958 / 2912 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 3540 / 3527 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 6 | 3458 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1736 / 1754 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1990 / 1872 | ✅ / ✅ | ✅ / ✅ |
jbachorik
force-pushed
the
jb/reference-chains
branch
from
July 17, 2026 11:39
2033006 to
02a0a93
Compare
Contributor
Benchmark Results (commit 02a0a93)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125255579 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
Contributor
Bits has a CI fix ready🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready
View in Datadog | Reviewed commit ed6525a · Any feedback? Reach out in #deveng-pr-agent |
jbachorik
force-pushed
the
jb/reference-chains
branch
from
August 6, 2026 13:41
92d65f2 to
da35063
Compare
Contributor
jbachorik
force-pushed
the
jb/reference-chains
branch
from
August 10, 2026 12:41
da35063 to
9b5fa79
Compare
Collaborator
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9b5fa79851
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return; | ||
| } | ||
| _thread = thread; | ||
| _running.store(true, std::memory_order_release); |
There was a problem hiding this comment.
If pthread_create schedules the new thread before the parent reaches this store, threadLoop() attaches and immediately evaluates while (_running.load(...)) against the initial false, then detaches and exits. The parent subsequently publishes true, leaving the tracker reporting that it is running even though no BFS passes will execute for the entire recording; use a startup handshake or otherwise ensure the worker cannot test the flag before startup is published.
Useful? React with 👍 / 👎.
jbachorik
force-pushed
the
jb/reference-chains
branch
from
August 13, 2026 08:24
64a79a6 to
cba3ca6
Compare
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
jbachorik
force-pushed
the
jb/reference-chains
branch
2 times, most recently
from
August 19, 2026 12:56
c033074 to
b5dc65f
Compare
Add the reference-chain engine for the live-heap profiler: - Track reference chains from at-risk/leak-tag candidate anchors via bounded descend walks, with retention-edge labels on every hop - Correlate leak-tagged allocations with anchor discoveries; cache and re-emit chains for repeated leak-tag targets - Tier anchor selection by class shape (leak/container/other), index root-attached anchors, and cover JNI-global and static-field roots - Guard self-edge chain demotion and contain FIFO floods with a per-class quota on the at-risk queue - Add a heap-wide time-to-OOM predictor to the fast-path leak gate - Make reference-chains debug logs runtime-configurable via rcDebugLevel, silent by default
Unit tests for the reference-chain engine: anchor selection and admission, walk tiers and quotas, self-edge guards, leak-tag correlation, JFR roundtrip of ReferenceChain events, and debug seams for the OOM-bypass gate (arguments, jvmHeap, stale-leaf coverage).
Integration tests for ReferenceChain events on real recordings: leak-tag correlation, aggressive-leak, external-process, and thread-local scenarios, with JFR parsing, assertions, and test-seam coverage. Migrate ReferenceChainTrackingTest to the JfrEvents API and extend the shared test harness (external launcher, profiler test base).
Stresstest wiring for reference-chain validation: a leak antagonist that grows collections and holds thread-locals, a standalone leak demo for repro runs, and chaos-harness build integration.
Scripts to run the reference-chains repro, sweep budget/behavior matrices across builds, compare repro output, and render single/multi run reports from JFR recordings (with vendored chart.js).
Design doc for reference chains on surviving live-heap samples, architecture notes, explanation of the emitted signals, and a collection summary of what lands in JFR recordings.
Full record of the investigation that produced the reference-chain engine: STATE/INDEX memory, per-round pod evidence, root-cause analyses, and repro artifacts.
Ignore local build/repro artifacts; extend the build-and-summarize command with reference-chains build steps.
Close out the investigation record: status active -> done, completed 2026-09-16. last_commit stamps the content the investigation actually concluded at (9282d5f, the pre-history-optimization branch head, preserved locally by backup/pre-history-optimize-rc-pi).
jbachorik
force-pushed
the
jb/reference-chains
branch
from
September 17, 2026 11:59
3a2fc0d to
e507b4f
Compare
Collaborator
Author
|
@arp review |
Detailed record of the walk engine as implemented: pass structure, root discovery (including the static-field sweep), candidate-scoped reach, canary tagging mechanics, rotation tiers, pacing/budget arithmetic, and the LivenessTracker admission boost. Cross-references the design, component-summary, and signals documents instead of repeating them.
The walk is correct on every collector, not only ZGC: JVMTI's iterators apply the active collector's own barriers, and the walk reads no raw oop. ZGC's concurrent relocation is the worst case a raw-oop reader would face, not the only collector the walk supports.
- registerThreadObject() was never called from Profiler::onThreadStart (only the one-time registerExistingThreads() snapshot), so threads born mid-recording never entered the tid->Thread registry and their ThreadLocalMap subgraphs were unreachable by walkCandidateThreadLocals(). - unregisterThreadObject()/register-replace DeleteGlobalRef'd under the registry lock while walkCandidateThreadLocals() can still hold a copied ref as a FollowReferences anchor - JNI use-after-free. Deletion is now deferred to _thread_refs_pending_delete and drained by releaseEndedThreadRefs() on the BFS thread (start of each pass) and from Profiler::stop() after stopThread() joined it. - tagAsRootForTest(): record the directly-tagged root as a discovered instance when a candidate slot watches its klass - the leak-tag redesign left a representative without a sampler-tracked instance no discovery channel; the seam test's tag now runs after the first poll. - getLeakTagInfo(): drop the bogus free-list membership check (a LIFO stack of indices is not an index-bounded region). - gtest: ThreadRefUnregisterDefersGlobalRefDeletion; failure-path logging in tagAsRootForTest.
- arguments.h: 0 first-pass budget auto-scales from budget x AUTO_FIRST_PASS_BUDGET_MULTIPLIER, it does not fall back to the plain per-pass budget. - flightRecorder.h: recordReferenceChain is called from Profiler::dump()'s drain loop, not continuously from pollWatchedTargets(). - collection-summary.md: a terminal search state is not final - shouldRunPass() restarts from it. - ThreadLocalLeakScenario gains a lateThread mode (thread created after recording start) with a new threadlocal-leak-late launcher mode and shouldCorrelateLeakOnThreadCreatedAfterRecordingStart - the regression shape for the onThreadStart registration wiring.
On a cyclic heap reach the deeper path's referrer can be a descendant of the already-admitted entry being improved; re-parenting to it made the parent chains mutually recursive, and every chain through the cycle then failed to reconstruct (reconstructChain ran to its hop bound and reported failure). Observed as the thread-local leak scenario producing zero chain events: all discovered instances of the watched klass sat inside a cycle rooted at an early ThreadGroup/Thread admission. improveChain now walks the new parent's ancestor chain and refuses when it reaches the entry (bounded at 4096 hops - recorded depths can be understated by later ancestor improves). reparentToDurableRoot cannot create cycles (its new parent must be root-attached). Also: reconstructChain logs the failing hop and the first tag->parent pairs on the broken-chain/hop-bound exits, and improveChain logs its refusals (all level-gated diagnostics).
The chain's root-side end was the static field's holder instance; the declaring class - the chain's actual root - was only present as the holder hop's edge label and the event's rootKind field. reconstructChain() now optionally returns the root-attached entry, and buildChainEvent()/buildCanaryChainEvent() append the declaring class (resolved from the root-attached entry's referrer_class_tag, captured at admission from the sweep's referrer_tag_ptr) as the chain's root-side terminal element, with one matching kind-only root edge so the _edges.size() == _chain.size() invariant recordReferenceChain() relies on to emit labels is preserved. Thread-rooted chains already end at the Thread instance; other root kinds have no further expressible root object.
The test asserted shouldRunPassForTest0() directly, but opening the gate spends the episode's one-shot urgency entitlement on the FIRST evaluation - and the BFS thread, whose cadence ramps to ~10ms once urgency latches, races the test's own call for it. The direct call only returned true when it won that race, failing deterministically on fast runners. Assert through the thread instead: seed the rising floor, then wait for referenceChainPassesRunForTest0() to advance. A pass only runs after shouldRunPass() returned true, and with an empty per-klass population table the urgent-OOM projection is the only possible trigger.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?:
This PR adds reference chains for live-heap leak candidates (PROF-15341).
LivenessTrackertracks class populations across GC generations. A classwith a sustained growth trend becomes a leak candidate. For each candidate,
a background BFS thread walks the heap. It records the referrer chain from a
GC root to a live instance. The profiler writes each chain as a
datadog.ReferenceChainJFR event.Key properties:
referencechains=trueto enable it.flowchart LR A[LivenessTracker leak candidates] --> B[marker-tag representative objects] B --> C[BFS pass: roots + static sweep + expand + descend walks] C --> D[chain found and pruned] D --> E[chain cached and re-stamped into each JFR dump]Main design decisions. The full detail lives in the design documents listed
under "Additional Notes":
FollowReferencescall. One BFS level is expanded per safepoint operation. No pass walks the whole heap at once.ThreadLocalMapsubgraphs. One prong walks root-attached static holders.static finalcollection is already visited by the walk. The search stays open while a watched class grows. Rotation re-expands the flagged containers and picks up new elements.datadog.ReferenceChainAbandonedevent. Restarts need a live leak signal and pain-budget headroom.Motivation:
PROF-15341: give live-heap samples that survive long enough to look
leak-shaped an actual referrer chain, not just "this object is still alive",
without regressing safepoint-pause behavior on low-pause collectors.
Additional Notes:
Design documents shipped with this PR:
doc/reference-chains-design.md: why bounded BFS-from-roots was chosen over the alternatives.doc/reference-chains-collection-summary.md: the four components. These are detection signal, resumable walk, latency budgets, and rotation.doc/architecture/LiveHeapReferenceChains.md: the full architecture. It covers the frontier and tag lifecycle, triggering, and termination.doc/architecture/LiveHeapReferenceChains-Implementation.md: the as-built implementation reference. It covers the pass structure, root discovery, candidate-scoped reach, canary tagging, rotation tiers, and pacing arithmetic.doc/architecture/ReferenceChains-SignalsExplained.md: a guided tour of the scheduling side. It covers cadence, the leak-signal gate, the pain budget, the PID controller, canary backoff, the OOM ramp, and the abort path.Other notes:
utils/tooling measures the real quantities directly: per-pass safepoint durations from JFR and the tracker pass-cost telemetry (seeLiveHeapReferenceChains-Implementation.md§11). A coarse JMH on/off A/B remains useful as an end-to-end throughput regression guard.How to test the change?:
./gradlew :ddprof-lib:gtestDebug: the native unit suite. It covers the walk engine, frontier table, target selection, rotation, pacing, JFR roundtrip, argument parsing, and the leak-candidate ranking../gradlew :ddprof-test:testDebug --tests "*referencechains.*": the Java suite:ExternalProcessReferenceChainTest: separate-process end-to-end. A child JVM runs a leaking app. The test asserts the chain's leaf class. It also asserts the chain re-emits into three fresh JFR dumps. One case appends to an already-discoveredstatic finalcollection and asserts the chain for a later element.ThreadLocalLeakReferenceChainTest: separate-process. The leak is reachable only through a worker thread'sThreadLocalMap.LeakTagCorrelationReferenceChainTest: leak-tag correlation, plus rotation under pod-like tight pass windows.AggressiveLeakReferenceChainTest: the search gate opens on heap-wide growth with no single candidate. It stays closed on a flat heap floor.ReferenceChainTrackingTest: in-process coverage for the walk engine, target selection, and abandonment reporting.ReferenceChainJfrParserTest: JFR-side parsing.ReferenceChainTestSeamsTest: C++ seam exposure.utils/run-refchains-repro.shwithddprof-stresstest'sReferenceChainLeakDemo;utils/sweep-refchains-*.shwithutils/refchains-report*.py;utils/run-chaos-harness.shwithReferenceChainLeakAntagonist.For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!