Add meta evaluation task - #76
Conversation
Introduce the core judge meta-evaluation building blocks and package prompt resources so later CLI wiring can reuse them unchanged.
Add the annotation orchestration path and expose meta-eval as a flat --task route alongside generate+judge and ELO workflows.
Add unit/integration coverage for meta-eval behavior and ensure prompt resources remain loadable from wheel and sdist installs.
Describe the new meta-eval CLI path, prompt modes, swap-mode semantics, and temporary SQLite cache limitations for operators.
ErlisLushtaku
changed the title
Reconcile meta-eval with the hierarchical RunConfig CLI, prompt registry, and Bradley-Terry APIs introduced on main.
Store rendered prompt text rather than LangChain prompt objects so SQLite-backed meta-eval runs complete after annotation. Includes-AI-Code: true
|
|
||
| def parse_pairscore_pref(judge_completion: str, *, temperature: float) -> float: | ||
| score = pair_score_parser(temperature).parse_model_raw(judge_completion) | ||
| if score is None or np.isnan(score): |
There was a problem hiding this comment.
Why we return tie=0.5 for parsing errors? We have to make sure this does not change the Cohen's Kappa or any accuracy
| replace=False, | ||
| random_state=int(rng.integers(0, 2**32 - 1)), | ||
| ) | ||
| if exclude_ties: |
There was a problem hiding this comment.
Sampling num_battles row then excluding tie means that our battles will have n <= num_battles
| args.prompt_mode, | ||
| provide_explanation=args.provide_explanation, | ||
| ) | ||
| judge_chat_model = make_model( |
There was a problem hiding this comment.
Should we call build_default_judge_model_kwargs before? Or this is already enough
| annotation_cache: AnnotationCache | None = None, | ||
| ) -> pd.DataFrame: | ||
| n_total = len(df_sample) | ||
| n_batches = (n_total + args.batch_size - 1) // args.batch_size |
There was a problem hiding this comment.
Not so problematic but we never check args.batch_size > 0. It can raise division by zero by mistake.
|
Apart from these I feel like there are some duplications: (1) The existing function already supports continuous preferences: fit_bradley_terry(df, pref_col="pref_llm")(2) The PR introduces: PromptModeSpec
resolve_prompt_mode()
_read_prompt()JudgeArena already has: JudgePromptPreset
ResolvedJudgePrompt
resolve_judge_prompt()Are these exchangable or we are required to implement this specifically? |
2 participants
Summary
--task meta-evalfor evaluating judge models against human-labeled arena battles.RunConfigCLI.Usage
judgearena \ --task meta-eval \ --judge.model OpenRouter/deepseek/deepseek-v3.2 \ --meta_eval.reference_arena LMArena-140k \ --meta_eval.prompt_mode standard \ --meta_eval.languages '["en", "es"]' \ --meta_eval.top_models 20 \ --meta_eval.battles_per_model 50 \ --meta_eval.n_bootstraps 1000 \ --run.seed 0Outputs
Runs produce:
annotations.parquetresults.jsonsummary.csvargs.jsonrun-metadata.v1.jsonWith
--judge.swap_mode both, annotations contain one row per judge pass with normalized model ordering and anorientationcolumn.Cache note
Meta-eval temporarily retains the existing per-battle SQLite WAL cache. It should be used from a single host and not concurrently across NFS-mounted compute nodes. Cache unification remains deferred to the dedicated caching work.