Skip to content

feat: make async row-group horizon configurable#745

Merged
andreatgretel merged 4 commits into
NVIDIA-NeMo:mainfrom
eric-tramel:etramel/fix/741-row-group-admission
Jul 8, 2026
Merged

feat: make async row-group horizon configurable#745
andreatgretel merged 4 commits into
NVIDIA-NeMo:mainfrom
eric-tramel:etramel/fix/741-row-group-admission

Conversation

@eric-tramel

@eric-tramel eric-tramel commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

📋 Summary

Exposes a single advanced RunConfig.max_concurrent_row_groups setting for the async scheduler's fixed row-group horizon. The historical default remains 3; adaptive admission, active-row budgets, and completion/checkpoint changes are intentionally out of scope.

🔗 Related Issue

Refs #741

🔄 Changes

  • Adds a validated fixed row-group cap (>= 1) to RunConfig and forwards it to the existing async scheduler setting.
  • Documents how the row-group horizon interacts with buffer_size, request concurrency, memory use, and checkpoint timing.
  • Adds focused config-validation and builder-wiring tests.
  • Keeps adaptive row-group admission, row-budget controls, and completion-state changes out of the public API and this PR.
  • Merges the latest main and reconciles the concurrency documentation with the current request-admission terminology.

🧪 Testing

  • .venv/bin/pytest packages/data-designer-config/tests/config/test_run_config.py (31 passed)
  • .venv/bin/pytest packages/data-designer-engine/tests/engine/dataset_builders/test_async_builder_integration.py (10 passed)
  • Ruff check and format check on all changed Python files
  • make check-fern-docs (fern check: 0 errors, 1 warning)
  • No new E2E coverage required; the scheduler algorithm is unchanged and its existing tests cover fixed-cap enforcement.

✅ Checklist

  • Follows commit message conventions
  • Commits are signed off (DCO)
  • Architecture docs updated

Description updated with AI

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants