Tachyon Oracle or Solaris (was: Bye ABBA and friends!) by maurycy · Pull Request #5 · maurycy/cpython

maurycy · 2026-06-19T12:43:23Z

(Removed the old description, but it should be available under versions here.)

The branch, relative to python#151614:

increases last_profiled_frame_seq on every stack pop, not only last_profiled_frame (when sampling), assuming that the churn can happen anywhere,
adds the Tools/inspection/oracle_external_inspection.py with reproductions inspired by gh-151613: Fix remote debugging frame cache ABA python/cpython#151614 (called deep_alternating), the shared_leaf we spoke about and infra{page,chunk} (called deep_recursion) and gi_iframe races (called gen_alternating).

There's very important usability gotcha, ie: --rate 0 is a freerunning version, which does not fully match the -m profiling.sampling behaviour.

Fixes python#151613. The remote debugging frame cache previously used only the last_profiled_frame address as its cache anchor. If a frame returned and a later frame reused the same _PyInterpreterFrame address, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack. This adds a last_profiled_frame_seq counter next to last_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches.

This reverts commit 6024fc7.

maurycy · 2026-06-20T19:10:15Z

Comparison between https://github.com/python/cpython/pull/151614/changes and the improved helper from https://github.com/python/cpython/pull/151614/changes#r3445008629 (b792693 here):

maurycy@gimel cpython (maurycy/bye-abba a7b5ee0?+) % time sudo ~/work/cpython/python.exe Tools/inspection/oracle_external_inspection.py --rate 0 --duration 3 --runs 8
3.16.0a0 (heads/maurycy/bye-abba-dirty:640a2288531, Jun 20 2026, 22:00:44) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=deep_alternating,shared_leaf modes=blocking-cache,blocking-nocache,live-cache,live-nocache reference=blocking-nocache runs=8 duration=3.0 rate_khz=0.0 warmup=0.7 poisson_sampling=False

deep_alternating
metric       mode               runs  samples   events     mean    stdev      min      max
errors%      blocking-cache        8  2639913        0     0.00     0.00     0.00     0.00
errors%      blocking-nocache      8  2024672        0     0.00     0.00     0.00     0.00
errors%      live-cache            8  4686474        3     0.00     0.00     0.00     0.00
errors%      live-nocache          8  2619022      818     0.03     0.00     0.03     0.04
impossible%  blocking-cache        8  2639913        0     0.00     0.00     0.00     0.00
impossible%  blocking-nocache      8  2024672        0     0.00     0.00     0.00     0.00
impossible%  live-cache            8  4686474        0     0.00     0.00     0.00     0.00
impossible%  live-nocache          8  2619022        2     0.00     0.00     0.00     0.00
tvd          blocking-cache        8  2639913        -    0.070    0.026    0.028    0.102
tvd          blocking-nocache      8  2024672        -      ref      ref      ref      ref
tvd          live-cache            8  4686474        -    0.163    0.042    0.108    0.234
tvd          live-nocache          8  2619022        -    0.179    0.038    0.128    0.246

shared_leaf
metric       mode               runs  samples   events     mean    stdev      min      max
errors%      blocking-cache        8  2604867        0     0.00     0.00     0.00     0.00
errors%      blocking-nocache      8  2323044        0     0.00     0.00     0.00     0.00
errors%      live-cache            8  5780332       11     0.00     0.00     0.00     0.00
errors%      live-nocache          8  2932092      482     0.02     0.00     0.01     0.02
impossible%  blocking-cache        8  2604867        0     0.00     0.00     0.00     0.00
impossible%  blocking-nocache      8  2323044        0     0.00     0.00     0.00     0.00
impossible%  live-cache            8  5780332        0     0.00     0.00     0.00     0.00
impossible%  live-nocache          8  2932092        1     0.00     0.00     0.00     0.00
tvd          blocking-cache        8  2604867        -    0.022    0.020    0.003    0.067
tvd          blocking-nocache      8  2323044        -      ref      ref      ref      ref
tvd          live-cache            8  5780332        -    0.140    0.020    0.099    0.166
tvd          live-nocache          8  2932092        -    0.138    0.021    0.096    0.165
maurycy@gimel cpython (maurycy/bye-abba a7b5ee0?+) % time sudo ~/work/cpython-pablo/python.exe Tools/inspection/oracle_external_inspection.py --rate 0 --duration 3 --runs 8
3.16.0a0 (heads/gh-151613-remote-debugging-frame-cache-aba:fc9fafd7699, Jun 20 2026, 00:31) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=deep_alternating,shared_leaf modes=blocking-cache,blocking-nocache,live-cache,live-nocache reference=blocking-nocache runs=8 duration=3.0 rate_khz=0.0 warmup=0.7 poisson_sampling=False

deep_alternating
metric       mode               runs  samples   events     mean    stdev      min      max
errors%      blocking-cache        8  2279548        0     0.00     0.00     0.00     0.00
errors%      blocking-nocache      8  1807981        0     0.00     0.00     0.00     0.00
errors%      live-cache            8  3727568        1     0.00     0.00     0.00     0.00
errors%      live-nocache          8  2106869      662     0.03     0.01     0.03     0.04
impossible%  blocking-cache        8  2279548        0     0.00     0.00     0.00     0.00
impossible%  blocking-nocache      8  1807981        0     0.00     0.00     0.00     0.00
impossible%  live-cache            8  3727568       83     0.00     0.01     0.00     0.02
impossible%  live-nocache          8  2106869        8     0.00     0.00     0.00     0.00
tvd          blocking-cache        8  2279548        -    0.068    0.047    0.017    0.137
tvd          blocking-nocache      8  1807981        -      ref      ref      ref      ref
tvd          live-cache            8  3727568        -    0.162    0.045    0.094    0.249
tvd          live-nocache          8  2106869        -    0.176    0.061    0.071    0.283
 
shared_leaf
metric       mode               runs  samples   events     mean    stdev      min      max
errors%      blocking-cache        8  2594844        0     0.00     0.00     0.00     0.00
errors%      blocking-nocache      8  2341066        0     0.00     0.00     0.00     0.00
errors%      live-cache            8  5601216       13     0.00     0.00     0.00     0.00
errors%      live-nocache          8  2811284      477     0.02     0.00     0.01     0.02
impossible%  blocking-cache        8  2594844        0     0.00     0.00     0.00     0.00
impossible%  blocking-nocache      8  2341066        0     0.00     0.00     0.00     0.00
impossible%  live-cache            8  5601216     1146     0.02     0.03     0.00     0.07
impossible%  live-nocache          8  2811284        0     0.00     0.00     0.00     0.00
tvd          blocking-cache        8  2594844        -    0.030    0.032    0.006    0.095
tvd          blocking-nocache      8  2341066        -      ref      ref      ref      ref
tvd          live-cache            8  5601216        -    0.146    0.029    0.090    0.187
tvd          live-nocache          8  2811284        -    0.150    0.042    0.086    0.234
maurycy@gimel cpython (maurycy/bye-abba 5c4015e?) %

using the newly added oracle_external_inspection.py:

maurycy@gimel cpython (maurycy/bye-abba 5c4015e?) % time sudo ~/work/cpython-pablo/python.exe Tools/inspection/oracle_external_inspection.py --help                       
usage: oracle_external_inspection.py [-h] [--case {deep_alternating,shared_leaf}] [--mode {blocking-cache,blocking-nocache,live-cache,live-nocache}]
                                     [--reference-mode {blocking-cache,blocking-nocache,live-cache,live-nocache}] [--duration DURATION] [--runs RUNS]
                                     [--rate-khz RATE_KHZ] [--warmup WARMUP] [--poisson-sampling]

options:
  -h, --help            show this help message and exit
  --case {deep_alternating,shared_leaf}
                        case to run; may be passed more than once; omit to run all cases (default: None)
  --mode {blocking-cache,blocking-nocache,live-cache,live-nocache}
                        mode to run; may be passed more than once; omit to run all modes (default: None)
  --reference-mode {blocking-cache,blocking-nocache,live-cache,live-nocache}
                        mode used as the distribution reference (default: blocking-nocache)
  --duration DURATION   seconds to sample each mode (default: 3.0)
  --runs RUNS           number of independent runs per case (default: 1)
  --rate-khz RATE_KHZ   target sampling rate in kHz; 0 samples as fast as possible (default: 100.0)
  --warmup WARMUP       seconds to let the target run before sampling (default: 0.7)
  --poisson-sampling    sample with exponential inter-arrival times instead of a fixed period (default: False)
maurycy@gimel cpython (maurycy/bye-abba 5c4015e?) %

maurycy · 2026-06-20T22:30:39Z

I was able to nicely reproduce python#151424 with 60.53% impossible rate, that directionally matches that finding:

2026-06-21T01:13:11.679863000+0200 maurycy@gimel /Users/maurycy/work/cpython (maurycy/bye-abba ecd60c9?) % sudo ~/work/cpython-old-main/python.exe Tools/inspection/oracle_external_inspection.py --case flat_alternating --rate 1 --duration 3 --runs 8
3.16.0a0 (heads/old-g6679ac07d88-dirty:6679ac07d88,) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=flat_alternating modes=blocking-cache,blocking-nocache,live-cache,live-nocache reference=blocking-nocache runs=8 duration=3.0 rate_khz=1.0 warmup=0.7 poisson_sampling=False

flat_alternating
metric       mode               runs  samples   events     mean    stdev      min      max
errors%      blocking-cache        8    24008        0     0.00     0.00     0.00     0.00
errors%      blocking-nocache      8    24008        0     0.00     0.00     0.00     0.00
errors%      live-cache            8    20942     3066    12.77     0.78    11.96    14.10
errors%      live-nocache          8    20625     3383    14.09     0.67    13.30    15.29
impossible%  blocking-cache        8    24008        0     0.00     0.00     0.00     0.00
impossible%  blocking-nocache      8    24008        0     0.00     0.00     0.00     0.00
impossible%  live-cache            8    20942    12677    60.53     1.98    56.80    63.46
impossible%  live-nocache          8    20625       62     0.30     0.09     0.23     0.51
tvd          blocking-cache        8    24008        -    0.019    0.013    0.006    0.044
tvd          blocking-nocache      8    24008        -      ref      ref      ref      ref
tvd          live-cache            8    20942        -    0.295    0.037    0.218    0.333
tvd          live-nocache          8    20625        -    0.017    0.007    0.005    0.029

This reverts commit 3207d12.

maurycy · 2026-06-20T23:15:55Z

The full run with all the current repros.

1KHz:

maurycy@gimel cpython (maurycy/bye-abba e6d5f5c*?) % sudo ~/work/cpython-main/python.exe Tools/inspection/oracle_external_inspection.py --rate 1 --duration 3 --runs 4
3.16.0a0 (heads/main:1fb874cc076, Jun 21 2026, 00:51:43) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=deep_alternating,deep_recursion,flat_alternating,gen_alternating,shared_leaf modes=blocking-cache,blocking-nocache,live-cache,live-nocache reference=blocking-nocache runs=4 duration=3.0 rate_khz=1.0 warmup=0.7 poisson_sampling=False
 
deep_alternating
metric       mode               runs  samples   events     mean    stdev      min      max
errors%      blocking-cache        4    12004        0     0.00     0.00     0.00     0.00
errors%      blocking-nocache      4    12004        0     0.00     0.00     0.00     0.00
errors%      live-cache            4    12001        3     0.02     0.02     0.00     0.03
errors%      live-nocache          4    11999        5     0.04     0.04     0.00     0.10
impossible%  blocking-cache        4    12004        0     0.00     0.00     0.00     0.00
impossible%  blocking-nocache      4    12004        0     0.00     0.00     0.00     0.00
impossible%  live-cache            4    12001        0     0.00     0.00     0.00     0.00
impossible%  live-nocache          4    11999        0     0.00     0.00     0.00     0.00
tvd          blocking-cache        4    12004        -    0.012    0.002    0.010    0.013
tvd          blocking-nocache      4    12004        -      ref      ref      ref      ref
tvd          live-cache            4    12001        -    0.188    0.006    0.180    0.194
tvd          live-nocache          4    11999        -    0.195    0.019    0.169    0.211

deep_recursion
metric       mode               runs  samples   events     mean    stdev      min      max
errors%      blocking-cache        4    12004        0     0.00     0.00     0.00     0.00
errors%      blocking-nocache      4    12004        0     0.00     0.00     0.00     0.00
errors%      live-cache            4     6911     5093    42.43     1.14    40.85    43.52
errors%      live-nocache          4     5092     6912    57.58     1.03    56.15    58.41
impossible%  blocking-cache        4    12004        0     0.00     0.00     0.00     0.00
impossible%  blocking-nocache      4    12004        0     0.00     0.00     0.00     0.00
impossible%  live-cache            4     6911     3389    49.04     1.10    47.52    50.00
impossible%  live-nocache          4     5092     1721    33.78     1.76    32.00    35.87
tvd          blocking-cache        4    12004        -    0.259    0.016    0.238    0.276
tvd          blocking-nocache      4    12004        -      ref      ref      ref      ref
tvd          live-cache            4     6911        -    0.711    0.024    0.675    0.726
tvd          live-nocache          4     5092        -    0.707    0.019    0.678    0.719
 
flat_alternating
metric       mode               runs  samples   events     mean    stdev      min      max
errors%      blocking-cache        4    12004        0     0.00     0.00     0.00     0.00
errors%      blocking-nocache      4    12004        0     0.00     0.00     0.00     0.00
errors%      live-cache            4    10279     1725    14.37     0.26    14.06    14.63
errors%      live-nocache          4    10198     1806    15.04     1.06    14.20    16.56
impossible%  blocking-cache        4    12004        0     0.00     0.00     0.00     0.00
impossible%  blocking-nocache      4    12004        0     0.00     0.00     0.00     0.00
impossible%  live-cache            4    10279       46     0.45     0.12     0.27     0.55
impossible%  live-nocache          4    10198       55     0.54     0.22     0.43     0.86
tvd          blocking-cache        4    12004        -    0.022    0.010    0.011    0.032
tvd          blocking-nocache      4    12004        -      ref      ref      ref      ref
tvd          live-cache            4    10279        -    0.037    0.016    0.022    0.051
tvd          live-nocache          4    10198        -    0.035    0.017    0.021    0.060

gen_alternating
metric       mode               runs  samples   events     mean    stdev      min      max
errors%      blocking-cache        4    12004        0     0.00     0.00     0.00     0.00
errors%      blocking-nocache      4    12004        0     0.00     0.00     0.00     0.00
errors%      live-cache            4     9985     2019    16.82     0.58    15.96    17.23
errors%      live-nocache          4     9959     2045    17.04     0.22    16.73    17.23
impossible%  blocking-cache        4    12004        0     0.00     0.00     0.00     0.00
impossible%  blocking-nocache      4    12004        0     0.00     0.00     0.00     0.00
impossible%  live-cache            4     9985     1519    15.22     1.99    14.13    18.20
impossible%  live-nocache          4     9959     1365    13.71     0.98    12.93    15.00
tvd          blocking-cache        4    12004        -    0.036    0.015    0.022    0.056
tvd          blocking-nocache      4    12004        -      ref      ref      ref      ref
tvd          live-cache            4     9985        -    0.236    0.014    0.223    0.254
tvd          live-nocache          4     9959        -    0.250    0.022    0.231    0.279

shared_leaf
metric       mode               runs  samples   events     mean    stdev      min      max
errors%      blocking-cache        4    12004        0     0.00     0.00     0.00     0.00
errors%      blocking-nocache      4    12004        0     0.00     0.00     0.00     0.00
errors%      live-cache            4    12004        0     0.00     0.00     0.00     0.00
errors%      live-nocache          4    12004        0     0.00     0.00     0.00     0.00
impossible%  blocking-cache        4    12004        0     0.00     0.00     0.00     0.00
impossible%  blocking-nocache      4    12004        0     0.00     0.00     0.00     0.00
impossible%  live-cache            4    12004        1     0.01     0.02     0.00     0.03
impossible%  live-nocache          4    12004        0     0.00     0.00     0.00     0.00
tvd          blocking-cache        4    12004        -    0.019    0.011    0.008    0.030
tvd          blocking-nocache      4    12004        -      ref      ref      ref      ref
tvd          live-cache            4    12004        -    0.136    0.022    0.104    0.151
tvd          live-nocache          4    12004        -    0.139    0.023    0.120    0.172

This reverts commit e6d5f5c.

pablogsal and others added 2 commits June 17, 2026 23:06

poor man's epoch

b792693

maurycy self-assigned this Jun 19, 2026

maurycy changed the title ~~Bye, ABBA!~~ Bye, AB(B)A! Jun 19, 2026

maurycy changed the title ~~Bye, AB(B)A!~~ Bye, AB(B)A and friends! Jun 19, 2026

maurycy changed the title ~~Bye, AB(B)A and friends!~~ Bye AB(B)A and friends! Jun 19, 2026

seq1 != seq2

6024fc7

maurycy mentioned this pull request Jun 20, 2026

gh-151613: Fix remote debugging frame cache ABA python/cpython#151614

Open

Revert "seq1 != seq2"

3207d12

This reverts commit 6024fc7.

maurycy marked this pull request as ready for review June 20, 2026 08:38

maurycy added 3 commits June 20, 2026 20:48

simple oracle

ef8fa79

improve oracle help

16ffc77

rm the verdict

640a228

maurycy added 9 commits June 20, 2026 22:22

khz, duration, runs

2f409ea

better impossible classify, ci95 is noise

9904421

popen; list not set

c77e588

shared_leaf

a00608d

bool classify, tvd from real observations

bd010dd

filename is wrong here

5213e4d

--samples bye

a7b5ee0

show number of events

f32d294

100khz default rate

5c4015e

maurycy changed the title ~~Bye AB(B)A and friends!~~ ABBA Oracle Jun 20, 2026

maurycy changed the title ~~ABBA Oracle~~ Tachyon Oracle (was: Bye ABBA and friends!) Jun 20, 2026

maurycy added 2 commits June 21, 2026 00:49

Reapply "seq1 != seq2"

e6d5f5c

This reverts commit 3207d12.

more repros

ecd60c9

Revert "Reapply "seq1 != seq2""

99c2fa6

This reverts commit e6d5f5c.

maurycy changed the title ~~Tachyon Oracle (was: Bye ABBA and friends!)~~ Tachyon Oracle or Solaris (was: Bye ABBA and friends!) Jun 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tachyon Oracle or Solaris (was: Bye ABBA and friends!)#5

Tachyon Oracle or Solaris (was: Bye ABBA and friends!)#5
maurycy wants to merge 19 commits into
mainfrom
maurycy/bye-abba

maurycy commented Jun 19, 2026 •

edited

Loading

Uh oh!

maurycy commented Jun 20, 2026 •

edited

Loading

Uh oh!

maurycy commented Jun 20, 2026 •

edited

Loading

Uh oh!

maurycy commented Jun 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

maurycy commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maurycy commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maurycy commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maurycy commented Jun 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

maurycy commented Jun 19, 2026 •

edited

Loading

maurycy commented Jun 20, 2026 •

edited

Loading

maurycy commented Jun 20, 2026 •

edited

Loading