(3/n) Introduce declarative YAML task definitions

[kargibora]

Summary

Task-specific configuration is currently distributed across Python constants, dataset loaders, baseline mappings, prompt
registries, and runner conditions. Adding or changing a task therefore requires modifying several unrelated modules.

This PR introduces declarative YAML task definitions, inspired by the task packaging approach used by lm-evaluation-
harness.

AlpacaEval is migrated as the first packaged task. Existing benchmarks continue using their current fallback behavior
until they are migrated in later PRs.

Task definitions

A task YAML describes the stable contract of a benchmark:

task: alpaca-eval
task_version: 1

dataset:
  adapter: judgearena_tables
  sources: ...
  fields: ...

protocol:
  runner: pairwise
  generation: ...
  baseline: ...
  judge: ...
  scoring: ...

The main sections are:

• task: stable task identity, version, description, and tags.
• dataset: source locations, pinned revisions, loader adapter, and canonical field mappings.
• protocol: how the task is generated, judged, and scored.
• metadata: links to the reference implementation or paper.

The protocol does not contain implementation code. It references registered components such as pairwise,
judgearena_tables, pairwise_preference, and pairwise_win_rate.

Main components

The new task package contains three main responsibilities:

judgearena/tasks/
├── definitions/   # YAML task definitions
├── schema.py      # Valid task structure
├── loader.py      # YAML loading and inheritance
├── registry.py    # Task discovery and component validation
└── cli.py         # list, show, and validate commands

• The schema defines the supported dataset sources, baseline strategies, judge settings, and scoring configuration.
  Unknown fields are rejected.

• The loader reads YAML safely, resolves optional _base.yaml inheritance, and calculates hashes for the source and
  resolved definitions.

• The registry discovers packaged tasks by ID and verifies that every referenced runner, dataset adapter, prompt,
  parser, and scorer exists.

• The CLI allows users to inspect and validate task definitions before running them.

Runtime behavior

When a task is executed, JudgeArena resolves it through the registry:

Task ID
-> load and validate task YAML
-> resolve dataset source and field mappings
-> apply task protocol defaults
-> select the registered runner
-> generate, judge, and score

Task definitions provide benchmark defaults, while users can still configure experiment-specific values.

The effective precedence is:

CLI flags -> run config YAML -> task-defined defaults -> framework defaults

For example:

• The task selects the runner and dataset adapter.
• The task provides the default baseline, prompt, swap mode, and optional judge temperature.
• --model.baseline may override the task baseline when the task permits it.
• Explicit judge settings from the CLI or run config remain unchanged.
• Stable task identity remains in the task YAML, while model and judge choices remain run-level configuration.

Additional changes

• AlpacaEval dataset loading now uses the source revision and field mappings declared in its task YAML.
• Task definitions are included in the installed Python package.
• Run metadata records the task version and resolved definition hash without copying the complete YAML into every
  result.
• Legacy fallback behavior is preserved for tasks that have not yet been packaged.