feat(mri): expose skip initial run on leaderboard creation - #832
Merged
RapidPoseidon merged 2 commits intoAug 21, 2026
Merged
Conversation
Threads the backend's skipInitialRun flag through create_leaderboard, defaulting to False so existing callers keep queueing the initial run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
LinoGiger
marked this pull request as ready for review
August 21, 2026 07:52
LinoGiger
reviewed
Aug 21, 2026
Comment on lines
+137
to
+155
| ### Skipping the Initial Run | ||
|
|
||
| Creating a leaderboard normally starts an initial run that evaluates the models | ||
| already in the benchmark against each other. Pass `skip_initial_run=True` to | ||
| create the leaderboard without it: | ||
|
|
||
| ```python | ||
| leaderboard = benchmark.create_leaderboard( | ||
| name="Staged Evaluation", | ||
| instruction="Which image do you prefer?", | ||
| skip_initial_run=True, | ||
| ) | ||
| ``` | ||
|
|
||
| The leaderboard starts with no responses and therefore no standings. Everything | ||
| else still works: a model added afterwards is compared against the whole existing | ||
| field, and the leaderboard can be boosted from the app. Use this when you want to | ||
| decide what gets evaluated first rather than paying for a full round up front. | ||
|
|
Collaborator
There was a problem hiding this comment.
not something we wanna encourage so just the code doc should be fine instead of the main mri docs
Contributor
Author
There was a problem hiding this comment.
Done in 6991e07 — dropped the section from docs/mri_advanced.md, so the flag is documented only in the create_leaderboard docstring (and therefore the API reference). The diff against main is now just the wrapper parameter and its tests.
Keeping it to the create_leaderboard docstring: the flag is there for people who already know they want it, not a pattern to point users at. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
LinoGiger
approved these changes
Aug 21, 2026
RapidPoseidon
deleted the
feat(mri)/expose-skip-initial-run-on-leaderboard-creation
branch
August 21, 2026 07:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Exposes the backend's
skipInitialRunflag onbenchmark.create_leaderboard, added in rapidata-backend#5141.Defaults to
False, so every existing caller keeps queueing the initial run exactly as before. The parameter is appended last, so positional callers are unaffected too.The leaderboard then starts with no responses and therefore no standings. Everything else keeps working: a model added afterwards is compared against the whole existing field, and the leaderboard can still be boosted from the app.
Notes
CreateLeaderboardEndpointInput.skip_initial_runarrived with the automated regeneration in Update OpenAPI Schemas #831 (pinned to the backend merge commitf4aff59). This PR is purely the high-level wrapper.RapidataLeaderboard. The backend treats it as a creation-time decision and does not record it, so there is no property to read back — same asincluded_tagsbeing create-only, except that one at least echoes locally.Tests
tests/rapidata_client/benchmark/test_leaderboard_skip_initial_run.py— the flag reaches the wire when set, and is explicitlyFalse(not omitted) by default, so the request never leans on the server-side default to preserve today's behaviour.Also ran:
pyright src/rapidata/rapidata_clientclean,pytest tests/rapidata_client/benchmark65/65.Four tests in
tests/rapidata_client/audience/fail on this branch, but they fail identically on a cleanmain— pre-existing and unrelated (test_assign_job_*).🔗 Session: https://poseidon.rapidata.internal/chat/session-66b1bde9