Skip to content

test: guard same-job compile ratios on CPU time with wall backstops #1162

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kvlonge merged 2 commits into from
Oct 19, 2026
Merged

test: guard same-job compile ratios on CPU time with wall backstops #1162

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
24a1aae
test: guard same-job compile ratios on CPU time with wall backstops
kvlonge Oct 19, 2026
3634829
Merge branch 'main' into test/cpu-ratio-guards
kvlonge Oct 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions tests/e2e/src/sqlbuild/cli/commands/main/compile/COMPILE_BENCHMARKS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,9 +75,10 @@ behind a loose timing budget. It runs in the 3,000-model fresh-process compile j

The same inspection project also bounds `sqb plan --json` against an empty DuckDB warehouse. The
plan budget is a ratio to a warm `sqb compile --json` measured in the same test on the same runner,
plus a peak-RSS cap. A second case plans a 300-model copy of the benchmark and fails when the
planning CPU of the large project (plan CPU minus compile CPU) exceeds twice its linear projection
from the small one, so super-linear planning work fails on any runner.
plus a peak-RSS cap. The CPU-time ratio is the tight limit; a looser wall-time ratio backstops lost
parallelism and lock or I/O waits. A second case plans a 300-model copy of the benchmark and fails
when the planning CPU of the large project (plan CPU minus compile CPU) exceeds twice its linear
projection from the small one, so super-linear planning work fails on any runner.

`test_build_performance.py` builds a 1,000-model DuckDB benchmark without tests or audits from an
empty warehouse once per module, with wall and peak-RSS limits. Its existing-state cases copy the
Expand Down
3 changes: 2 additions & 1 deletion tests/e2e/src/sqlbuild/cli/commands/main/compile/_test_types.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -368,7 +368,8 @@ class InspectionCommandPerformanceGuardTestCase:
@dataclass(frozen=True)
class PlanComparedToCompileGuardTestCase:
description: str
expected_max_compile_ratio: float
expected_max_compile_cpu_ratio: float
expected_max_compile_wall_ratio: float
expected_max_plan_wall_seconds: float
expected_max_rss_bytes: int

Expand Down
14 changes: 8 additions & 6 deletions ...2e/src/sqlbuild/cli/commands/main/compile/test_fresh_process_compile_cache_performance.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,10 @@
_MAX_AFTER_PROJECT_REWRITE_TO_COLD_RATIO: float = 0.8
_MAX_EDIT_TO_COLD_RATIO: float = 0.70
# Writing the cache during a cold compile must stay a small same-runner cost over --no-cache.
_MAX_CACHE_WRITE_CPU_OVERHEAD_RATIO: float = 1.10
_MAX_CACHE_WRITE_WALL_OVERHEAD_RATIO: float = 1.20
_MAX_LARGE_CACHE_WRITE_CPU_OVERHEAD_RATIO: float = 1.15
# CPU time is the tight guard because it is steady on shared runners; the looser wall-time
# backstop still catches lost parallelism, lock waits and write I/O regressions.
_MAX_CACHE_WRITE_CPU_OVERHEAD_RATIO: float = 1.25
_MAX_CACHE_WRITE_WALL_OVERHEAD_RATIO: float = 1.35


@pytest.mark.performance
Expand Down Expand Up @@ -123,7 +124,7 @@
expected_edit_max_wall_seconds=13.5,
expected_max_warm_to_cold_ratio=_MAX_WARM_TO_COLD_RATIO,
expected_max_edit_to_cold_ratio=_MAX_EDIT_TO_COLD_RATIO,
expected_max_cache_write_cpu_overhead_ratio=_MAX_LARGE_CACHE_WRITE_CPU_OVERHEAD_RATIO,
expected_max_cache_write_cpu_overhead_ratio=_MAX_CACHE_WRITE_CPU_OVERHEAD_RATIO,
expected_max_cache_write_wall_overhead_ratio=_MAX_CACHE_WRITE_WALL_OVERHEAD_RATIO,
# Temporary memory exception (2026-10-10); remove after conversion optimisation.
expected_max_rss_bytes=3008 * _MIB,
Expand Down Expand Up @@ -180,11 +181,12 @@ def test_given_semantic_project_when_compiling_across_processes_then_cache_is_in
}
for label, measurement in measurements.items():
_LOGGER.info(
"fresh-process cache models=%d path=%s wall=%.3fs peak_rss_bytes=%d "
"fresh-process cache models=%d path=%s wall=%.3fs cpu=%.3fs peak_rss_bytes=%d "
"fingerprint=%s metrics=%s",
test_case.model_count,
label,
measurement.elapsed_seconds,
measurement.cpu_seconds,
measurement.peak_rss_bytes,
measurement.semantic_fingerprint,
fresh_process_compile_cache_metrics(measurement),
Expand All @@ -204,8 +206,8 @@ def test_given_semantic_project_when_compiling_across_processes_then_cache_is_in
cache_write_wall_ratio,
cache_write_cpu_ratio,
)
assert cache_write_wall_ratio <= test_case.expected_max_cache_write_wall_overhead_ratio
assert cache_write_cpu_ratio <= test_case.expected_max_cache_write_cpu_overhead_ratio
assert cache_write_wall_ratio <= test_case.expected_max_cache_write_wall_overhead_ratio
for measurement in (
result.warm,
result.after_leaf_edit,
Expand Down
18 changes: 13 additions & 5 deletions tests/e2e/src/sqlbuild/cli/commands/main/compile/test_inspection_command_performance.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,8 @@ def test_given_shared_dependency_benchmark_when_inspecting_then_preserves_resour
[
PlanComparedToCompileGuardTestCase(
description="models_3000_plan_json_relative_to_warm_compile",
expected_max_compile_ratio=2.6,
expected_max_compile_cpu_ratio=3.1,
expected_max_compile_wall_ratio=3.3,
expected_max_plan_wall_seconds=45.0,
expected_max_rss_bytes=1_280 * _MIB,
)
Expand All @@ -227,18 +228,25 @@ def test_given_shared_dependency_benchmark_when_planning_then_stays_near_compile
expected_max_wall_seconds=test_case.expected_max_plan_wall_seconds,
)

ratio: float = planned.elapsed_seconds / compiled.elapsed_seconds
# CPU time is the tight guard because it is steady on shared runners; the looser wall-time
# backstop still catches lost parallelism and lock or I/O waits.
wall_ratio: float = planned.elapsed_seconds / compiled.elapsed_seconds
cpu_ratio: float = planned.cpu_seconds / compiled.cpu_seconds
_LOGGER.info(
"plan %s wall=%.3fs cpu=%.3fs peak_rss_bytes=%d compile_wall=%.3fs ratio=%.2f",
"plan %s wall=%.3fs cpu=%.3fs peak_rss_bytes=%d compile_wall=%.3fs compile_cpu=%.3fs "
"ratio=%.2f cpu_ratio=%.2f",
test_case.description,
planned.elapsed_seconds,
planned.cpu_seconds,
planned.peak_rss_bytes,
compiled.elapsed_seconds,
ratio,
compiled.cpu_seconds,
wall_ratio,
cpu_ratio,
)
assert len(cast(dict[str, list[object]], planned.payload)["models"]) == _MODEL_COUNT
assert ratio < test_case.expected_max_compile_ratio
assert cpu_ratio < test_case.expected_max_compile_cpu_ratio
assert wall_ratio < test_case.expected_max_compile_wall_ratio
assert planned.peak_rss_bytes < test_case.expected_max_rss_bytes


Expand Down
Loading