Skip to content

feat(mri): expose skip initial run on leaderboard creation - #832

Merged
RapidPoseidon merged 2 commits into
mainfrom
feat(mri)/expose-skip-initial-run-on-leaderboard-creation
Aug 21, 2026
Merged

feat(mri): expose skip initial run on leaderboard creation#832
RapidPoseidon merged 2 commits into
mainfrom
feat(mri)/expose-skip-initial-run-on-leaderboard-creation

Conversation

@RapidPoseidon

Copy link
Copy Markdown
Contributor

Exposes the backend's skipInitialRun flag on benchmark.create_leaderboard, added in rapidata-backend#5141.

leaderboard = benchmark.create_leaderboard(
    name="Staged Evaluation",
    instruction="Which image do you prefer?",
    skip_initial_run=True,
)

Defaults to False, so every existing caller keeps queueing the initial run exactly as before. The parameter is appended last, so positional callers are unaffected too.

The leaderboard then starts with no responses and therefore no standings. Everything else keeps working: a model added afterwards is compared against the whole existing field, and the leaderboard can still be boosted from the app.

Notes

  • The generated client already had the fieldCreateLeaderboardEndpointInput.skip_initial_run arrived with the automated regeneration in Update OpenAPI Schemas #831 (pinned to the backend merge commit f4aff59). This PR is purely the high-level wrapper.
  • Nothing is stored on RapidataLeaderboard. The backend treats it as a creation-time decision and does not record it, so there is no property to read back — same as included_tags being create-only, except that one at least echoes locally.
  • No boost method to point at. The docstring says "boosting the leaderboard still works" rather than referencing a method, because boosting is not exposed in the SDK.

Tests

tests/rapidata_client/benchmark/test_leaderboard_skip_initial_run.py — the flag reaches the wire when set, and is explicitly False (not omitted) by default, so the request never leans on the server-side default to preserve today's behaviour.

Also ran: pyright src/rapidata/rapidata_client clean, pytest tests/rapidata_client/benchmark 65/65.

Four tests in tests/rapidata_client/audience/ fail on this branch, but they fail identically on a clean main — pre-existing and unrelated (test_assign_job_*).

🔗 Session: https://poseidon.rapidata.internal/chat/session-66b1bde9

Threads the backend's skipInitialRun flag through create_leaderboard, defaulting
to False so existing callers keep queueing the initial run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
@LinoGiger
LinoGiger marked this pull request as ready for review August 21, 2026 07:52
@LinoGiger
LinoGiger self-requested a review as a code owner August 21, 2026 07:52
Comment thread docs/mri_advanced.md Outdated
Comment on lines +137 to +155
### Skipping the Initial Run

Creating a leaderboard normally starts an initial run that evaluates the models
already in the benchmark against each other. Pass `skip_initial_run=True` to
create the leaderboard without it:

```python
leaderboard = benchmark.create_leaderboard(
name="Staged Evaluation",
instruction="Which image do you prefer?",
skip_initial_run=True,
)
```

The leaderboard starts with no responses and therefore no standings. Everything
else still works: a model added afterwards is compared against the whole existing
field, and the leaderboard can be boosted from the app. Use this when you want to
decide what gets evaluated first rather than paying for a full round up front.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not something we wanna encourage so just the code doc should be fine instead of the main mri docs

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 6991e07 — dropped the section from docs/mri_advanced.md, so the flag is documented only in the create_leaderboard docstring (and therefore the API reference). The diff against main is now just the wrapper parameter and its tests.

Keeping it to the create_leaderboard docstring: the flag is there for people who
already know they want it, not a pattern to point users at.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
@RapidPoseidon
RapidPoseidon merged commit d9f1cfb into main Aug 21, 2026
4 checks passed
@RapidPoseidon
RapidPoseidon deleted the feat(mri)/expose-skip-initial-run-on-leaderboard-creation branch August 21, 2026 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants