(2/n) Organize Benchmark Execution Into Responsibility-Owned Packages

[kargibora]

Summary

The previous PR introduced a shared benchmark runner interface, but benchmark implementations and related utilities are still distributed across the root of judgearena.

For example, pairwise evaluation, ELO execution, MT-Bench, datasets, and run metadata currently live in unrelated root-
level modules. This makes ownership unclear and increases the number of unrelated files that must be changed when adding or modifying a benchmark.

This PR reorganizes the existing implementation into responsibility-owned packages and moves benchmark dispatch out of the pairwise runner.

Take a look at SoP to understand the reason and shortcomings of the current approach.

What changed

Benchmark implementations are now grouped under:

judgearena/benchmarks/
├── elo/
├── mt_bench/
├── pairwise/
├── execution.py
├── registry.py
└── runner.py

Other shared components are also separated:

judgearena/
├── artifacts/
│   ├── metadata.py
│   └── run.py
└── datasets/
    ├── arena_hard.py
    ├── m_arenahard.py
    └── mt_bench.py

Important changes include:

• Moves pairwise, MT-Bench, and ELO implementations into dedicated packages.
• Adds a benchmark-owned registry and thin shared dispatcher.
• Keeps the CLI responsible only for configuration and top-level dispatch.
• Separates ELO rating calculations from the ELO execution workflow.
• Centralizes result-directory creation and safe metadata writing.
• Moves dataset loading utilities under judgearena/datasets.
• Updates imports and tests to follow the new package ownership.

Most of the diff consists of file moves and import updates. Existing benchmark protocols, prompts, configuration, and
evaluation behavior are preserved.

Why this helps

A benchmark can now own its runner and utilities without adding more logic to the pairwise implementation or the package
root. It also gives datasets and artifacts stable locations that can be reused by future benchmark integrations.

Notes

• As can be seen on diffs, most of the LoC coming from moving codes from certain scripts so each script is more readable and does one job.