update sdk with new adds - #467

Summary — v0.19.0: EvaluationV2, benchmark paradigm

Brings the SDK to parity with the Nucleus EvaluationV2 UI. Evaluation creation is benchmark-scoped only — dataset/slice-scoped creation (an unreleased earlier iteration of this branch) was removed before release as the platform moved to the benchmark paradigm.

Added

• Benchmarks. create_benchmark() (members from item_ids, (dataset_id, ref_id) items pairs, a slice_id, or a dataset_id — exactly one required; membership frozen at creation), list_benchmarks(), get_benchmark(), update_benchmark(), delete_benchmark(), list_benchmark_items(), plus the Benchmark resource (refresh() / update() / delete() / items() / create_evaluation_v2()) in nucleus/benchmark.py.
• Benchmark evaluations. create_benchmark_evaluation_v2(benchmark_id, model_run_id, ...) — every benchmark item is scored (uncovered items count as FN, keeping leaderboard scores comparable). EvaluationV2 exposes benchmark_id, rollup_groups, exclusion_rules, exclusion_stats. No only_items_with_predictions / eval-scope slice_id (invalid in the benchmark paradigm; the server rejects the former).
• Rollup groups. RollupGroup (class_name + labels) as the primary label configuration on benchmark eval create and presets (create/update_evaluation_v2_preset(rollup_groups=...), mutually exclusive with legacy allowed_label_matches; parsed from both key casings). Preset seeding prefers rollup_groups, falling back to legacy matches.
• Exclusion rules. MetadataExclusionRule / LabelExclusionRule / BoxAreaExclusionRule via exclusion_rules on eval create and presets.
• Presets. list/create/update/delete_evaluation_v2_preset + EvaluationV2Preset resource; preset= seeds benchmark eval creation (explicit args override).
• Results. EvaluationV2.charts() (mAP, per-class AP, confusion matrix, PR/F1 curves, TIDE, AP by size) and examples() (paginated TP/FP/FN; match_type optional) with EvaluationV2FilterArgs filtering; cancel() / retry(); Dataset.evaluation_label_schema().
• Benchmark leaderboards. leaderboard_ranking(metric_type, benchmark_ids, ...) (metrics: MAP_50, MAP_50_95, AP_SMALL/MEDIUM/LARGE, PRECISION, RECALL, F1; scope/collapse) and leaderboard_f1_curve(benchmark_ids, ..., top_n=5).
• Filter schema discovery. EvaluationV2.filter_schema() / get_evaluation_v2_filter_schema() return the evaluation's filter vocabulary (gt_labels, pred_labels, metadata_fields with inferred value types).
• New DTOs: EvaluationV2FilterSchema, LeaderboardRankingEntry, LeaderboardF1CurveEntry, BenchmarkItemsPage.

Removed (relative to earlier commits on this branch — never released)

• create_evaluation_v2 (dataset/slice-scoped), create_evaluations_v2_batch, BatchEvaluationResult, only_items_with_predictions. Reading existing evals (including old slice-scoped ones) still works.

Fixed

• CI build_test pylint failure (pre-existing W0718 on the per-job catch) — suppressed with a black-stable # pylint: disable-next line.

Tests / Version

• tests/test_benchmarks.py, tests/test_leaderboard.py, plus rollup/benchmark coverage in the eval/preset test files (59 unit tests, mock-based).
• pyproject.toml → 0.19.0; single consolidated CHANGELOG entry (the unreleased 0.18.9 section was folded in).

Server dependency: the leaderboard and filter-schema methods call new REST mirrors (POST /nucleus/leaderboard/ranking, POST /nucleus/leaderboard/f1Curve, GET /nucleus/evaluationsV2/:id/filterSchema) that ship separately in scaleapi. Unit tests pass regardless; live calls 404 until that deploys. All benchmark CRUD / benchmark-eval / rollup-group / preset endpoints are already live (DE-8209).

resolves https://linear.app/scale-epd/issue/DE-8209

Greptile Summary

This PR brings the Python SDK to parity with the Nucleus EvaluationV2 UI under a benchmark paradigm: benchmarks are frozen ground-truth item sets that model runs are evaluated against, enabling comparable leaderboard scores across runs.

• Benchmarks (nucleus/benchmark.py, NucleusClient): full CRUD (create_benchmark, list_benchmarks, get_benchmark, update_benchmark, delete_benchmark, list_benchmark_items) plus a Benchmark resource object with refresh/update/delete/items/create_evaluation_v2.
• Benchmark evaluations (create_benchmark_evaluation_v2): supports rollup_groups (new primary label config), legacy allowed_label_matches*, exclusion rules, and preset seeding; adds cancel, retry, filter_schema to EvaluationV2; migrates charts() from GET to POST.
• Presets, exclusion rules, leaderboard (evaluation_v2_preset.py, evaluation_v2_exclusions.py): new EvaluationV2Preset with per-user CRUD; MetadataExclusionRule, LabelExclusionRule, BoxAreaExclusionRule; leaderboard_ranking and leaderboard_f1_curve client methods backed by new server endpoints.

sequenceDiagram
    participant User
    participant NucleusClient
    participant API

    User->>NucleusClient: "create_benchmark(name, slice_id=...)"
    NucleusClient->>API: POST /benchmarks
    API-->>NucleusClient: "{benchmark_id, ...}"
    NucleusClient-->>User: Benchmark

    User->>NucleusClient: "create_benchmark_evaluation_v2(benchmark_id, model_run_id, rollup_groups=..., preset=...)"
    Note over NucleusClient: Seed from preset if no explicit label config
    NucleusClient->>API: "POST /benchmarks/{id}/evaluationsV2"
    API-->>NucleusClient: "{evaluation_id}"
    NucleusClient->>API: "GET /evaluationsV2/{evaluation_id}"
    API-->>NucleusClient: EvaluationV2 payload
    NucleusClient-->>User: EvaluationV2

    User->>NucleusClient: evaluation.wait_for_completion()
    loop Poll status
        NucleusClient->>API: "GET /evaluationsV2/{id}"
        API-->>NucleusClient: status
    end

    User->>NucleusClient: "evaluation.charts(iou_threshold=0.5)"
    NucleusClient->>API: "POST /evaluationsV2/{id}/charts"
    API-->>NucleusClient: EvaluationV2Charts

    User->>NucleusClient: leaderboard_ranking(metric_type, benchmark_ids)
    NucleusClient->>API: POST /leaderboard/ranking
    API-->>NucleusClient: List[LeaderboardRankingEntry]
    NucleusClient-->>User: ranked entries

_{Reviews (10): Last reviewed commit: "Route benchmark eval create through the ..." | Re-trigger Greptile}