(6/n) Package MT-Bench as a declarative multi-turn task

[kargibora]

Summary

This PR migrates MT-Bench into the declarative task registry and gives it a dedicated benchmark runner.

MT-Bench differs from the existing pairwise tasks because it uses two-turn conversations, category-specific generation
temperatures, reference answers for selected categories, and FastChat-specific judge prompts and verdict parsing.

Previously, the generic pairwise runner contained a special if task == "mt-bench" execution path. MT-Bench
configuration was also distributed across dataset utilities, prompt constants, judge helpers, and baseline mappings.

This PR moves the stable MT-Bench protocol into one validated task definition and routes it through a specialized
runner.

Task definition

The MT-Bench task YAML declares:

• The pinned lmsys/mt-bench Hugging Face Space source
• The pinned FastChat reference-answer source
• Question ID, turns, and category field mappings
• Multi-turn generation mode
• Category-specific generation temperatures
• Default gpt-4 baseline
• FastChat pairwise judge prompt and parser
• Single-turn and multi-turn judging policy
• Categories requiring reference answers
• Pairwise win-rate scoring

The task remains configurable at runtime. Candidate, baseline, judge, instruction count, and other experiment-level
settings may still be supplied through the CLI or run configuration.

Protocol schemas

The task schema now distinguishes between generic pairwise tasks and MT-Bench:

ProtocolSpec
├── PairwiseProtocol
└── MTBenchProtocol

Shared concepts such as baseline and scoring remain reusable, while MT-Bench can declare fields that only its runner
understands:

MultiTurnGeneration
├── mode
└── category_temperatures

MTBenchJudgeSpec
├── turns_mode
├── FastChat prompt preset
├── judge temperature
└── reference-answer categories

This avoids adding MT-Bench-only fields to every pairwise task.

Runtime flow

MT-Bench now follows this execution path:

mt-bench task ID
-> resolve validated task definition
-> select the mt_bench runner
-> load questions and reference answers
-> generate both conversation turns
-> apply category-specific temperatures
-> select the appropriate FastChat judge prompt
-> parse pairwise verdicts
-> compute and save the result

The generic pairwise runner no longer checks for or executes MT-Bench directly.

Dataset handling

The mt_bench dataset adapter owns the source-specific operations:

• Downloading the pinned question and baseline-answer files
• Downloading pinned FastChat reference answers
• Normalizing questions into the task’s canonical fields
• Loading existing answers when available
• Providing reference answers for the configured categories

The runner receives a resolved protocol and normalized data rather than reading global MT-Bench constants.

Baseline ownership

Native baseline resolution now comes exclusively from validated task definitions. The remaining legacy baseline registry
and its compatibility tests are removed.

For MT-Bench, the task default remains gpt-4, but a runtime baseline can override it when required.

Why this helps

MT-Bench demonstrates that the task registry can support benchmarks that share common concepts without forcing them
through the same execution function.

The task YAML owns stable benchmark policy, the dataset adapter owns source normalization, and the specialized runner
owns multi-turn execution.