fix(benchmark): fail loudly on missing AMI annotations instead of scoring placeholder ground truth#754
Conversation
…of scoring placeholder ground truth (#752) The 80.8% DER reported on PR #744 was not a diarizer regression: the AMI annotation download from groups.inf.ed.ac.uk transiently failed in that CI run, AMIParser silently fell back to a synthetic placeholder reference (8 evenly-sliced round-robin segments), and the benchmark scored real diarization output against fake ground truth. The pipeline itself was healthy (4/4 speakers, normal timings); every surrounding run scored 15.1%. Changes: - AMIParser: remove the placeholder ground-truth fallbacks; the search-based loaders now throw AMIParserError.annotationsNotFound (or propagate parse errors) instead of returning generateSimplifiedGroundTruth output. Loaders become synchronous throwing functions; generateSimplifiedGroundTruth is deleted. - DiarizationBenchmark: exit(1) when no meeting produced a result so CI goes red instead of exiting 0 with empty metrics. - DatasetDownloader: retry the annotation zip download (3 attempts, backoff). - Workflows: cache Datasets/ami_public_1.6.2 in diarizer-benchmark, offline-pipeline, and sortformer-benchmark so the flaky Edinburgh server is only needed on cache miss; only post the PR benchmark comment when metric extraction succeeded. - Tests: cover the new throwing behavior via injectable search roots. Verified locally: without annotations the benchmark logs annotationsNotFound and exits 1 with no results file; with annotations it reproduces the healthy baseline (DER 15.1%, JER 24.8% on ES2004a).
…6-meeting split Offline: unchanged at 10.6% avg DER with --threshold 0.7 (bit-identical across runs; reproduces the #533-era table exactly). Documented that the CLI default clustering threshold changed from 0.7 to 0.6 (community-1 preset) in #616, which undercounts speakers on 4 meetings and yields 15.5% avg DER on this split — the table now pins --threshold 0.7. Streaming: all four configs re-run on the official 16-meeting split (previous tables used a 7-meeting subset). No-overlap configs beat overlap configs: 10s/0s 38.2%, 5s/0s 39.0%, 3s/1s 53.3%, 5s/2s 55.9% avg DER. Noted the IOSurface/E5RT buffer exhaustion when running many overlapping-chunk meetings in one process. Hardware: Apple M5 Pro, 2026-07-03. Also fixed missing line continuations in the documented streaming commands.
Supertonic3 Smoke Test ✅
Runtime: 0m23s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
PocketTTS Smoke Test ✅
Runtime: 1m34s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 2m 38s • 2026-07-04T03:19:52.402Z |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 2m8s • 07/03/2026, 11:20 PM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 10m11s • 07/03/2026, 11:26 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 117.8s processing • Test runtime: 2m 2s • 07/03/2026, 11:31 PM EST |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 32.9s diarization time • Test runtime: 2m 12s • 07/03/2026, 11:32 PM EST |
1 participant
Closes #752
Root cause of the 80.8% DER report
The 80.8% DER / 80.9% JER on PR #744's CI run was not a diarizer regression. From that run's logs:
The pipeline itself was healthy (4/4 speakers, normal stage timings, normal speaker mapping). The AMI annotation download from
groups.inf.ed.ac.uktransiently failed,AMIParsersilently substituted a synthetic placeholder reference (8 evenly-sliced round-robin segments), and the benchmark scored real diarization output against fake ground truth — then posted the result as if genuine.Supporting evidence it was a one-off harness artifact:
KMeansClustering.swiftChanges
AMIParserError.annotationsNotFound(or propagate parse errors) instead of returninggenerateSimplifiedGroundTruthoutput — benchmarks can no longer silently score against fake data.generateSimplifiedGroundTruthis deleted; the loaders become synchronous throwing functions.exit(1)when no meeting produced a result, so CI goes red instead of exiting 0 with empty metrics.diarizer-benchmark,offline-pipeline,sortformer-benchmark): cacheDatasets/ami_public_1.6.2so the flaky Edinburgh server is only hit on cache miss; only post the PR benchmark comment when metric extraction succeeded (previouslyif: always(), which would post garbage numbers on failure).AMIParserTestscases covering the throwing behavior via injectable search roots.Verification
annotationsNotFound(subdirectory: "segments"), writes no results file, exits 1Also: Documentation/Benchmarks.md re-run (second commit)
All diarization tables re-measured 2026-07-03 on an Apple M5 Pro against the official AMI-SDM 16-meeting split:
--threshold 0.7, bit-identical across consecutive runs, reproducing the existing table exactly. Documented that feat(cli): expose all unexposed config fields across 6 commands #616 changed the effective CLI default clustering threshold from 0.7 to 0.6 (community-1 preset), which undercounts speakers on 4 meetings (e.g. EN2002a 2/4 → 41.9% DER) and yields 15.5% avg on this split — the documented command now pins--threshold 0.7.E5RT: Failed to allocate memory IOSurface object(SIGABRT in MLE5Engine) — the overlap tables were measured one meeting per process.🤖 Generated with Claude Code