All issues

SamplingMetric string `normalize` argument always raises: getmembers() list treated as a dict

#1365

· Abelo9996 opened

on Aug 29, 2026

Prediction cache identity is the model config, not the model that ran: weights moving under the same revision name silently reuse the old responses

#1364

· AUTHENSOR opened

on Aug 29, 2026

Sample cache: doc.id collides across evaluation splits, crashing runs and silently serving one split's answers for another split's questions

#1363

· AUTHENSOR opened

on Aug 29, 2026

OpenAI-compat via endpoint litellm: openai/ prefix and /v1 base_url

#1362

· cursor opened

on Aug 28, 2026

F1_score collapses nltk's undefined result onto zero, including when the gold reference is empty

#1353

· buzzedaldrinx86-creator opened

on Aug 23, 2026

[BUG] Why isn't apply_chat_template called with add_generation_prompt=True in LLM as a judge?

#1346

· Jeronymous opened

on Aug 18, 2026

[BUG] Incompatibility with latest version of xxhash

#1330

· Giuseppe5 opened

on Aug 13, 2026

[BUG] dyck_language scores with inspect_ai's exact(), which ignored word order in every release up to 0.3.252

#1329

· LippInc opened

on Aug 13, 2026

lighteval eval crashes on fresh install: GenerateConfig rejects frequence_penalty / log_probs / response_format with current inspect-ai

#1327

· sk8ordie84 opened

on Aug 11, 2026

[Proposal] OpenEval Import/Export Support

#1318

· adhabnr-ux opened

on Jul 29, 2026

Lighteval TriviaQA evaluation not giving results as per smolLM-360M reported numbers

#1316

· Kun-AI-san opened

on Jul 27, 2026

Balanced few-shot sampling truncates on a falsy class label and uses the global random module

#1309

· ErenAta16 opened

on Jul 20, 2026