Skip to content

Batch hallucination-rate evaluation and reporting over a dataset - #93

Open
zhenzhudingzhen528-cpu wants to merge 1 commit into
KRLabsOrg:mainfrom
zhenzhudingzhen528-cpu:oss-agent/issue-56-lettucedetec
Open

Batch hallucination-rate evaluation and reporting over a dataset#93
zhenzhudingzhen528-cpu wants to merge 1 commit into
KRLabsOrg:mainfrom
zhenzhudingzhen528-cpu:oss-agent/issue-56-lettucedetec

Conversation

@zhenzhudingzhen528-cpu

Copy link
Copy Markdown

Summary

  • JSONL and CSV input
  • overall and grouped rates with explicit denominators
  • span histograms and typed counts
  • top-N flagged examples
  • JSON and Markdown output
  • network-free unit tests

Changed paths:

  • scripts/report_hallucination_rates.py
  • tests/test_report_hallucination_rates_pytest.py

Related issue

Closes #56.

Type of change

  • Bug fix
  • Feature
  • Documentation
  • Tests
  • Refactor or maintenance

Testing

  • python -m pytest tests/test_report_hallucination_rates_pytest.py -q — 23 passed
  • ruff format --check
  • ruff check
  • git diff --check
  • Other: current Windows full suite — 161 passed, 8 failed
  • Other: recorded pre-change Windows baseline — 138 passed, 8 failed; recorded cause: existing Unix-only resource import in scripts/benchmark_detectors.py on Windows
  • Exact failing test node IDs were not recorded, so failure-set equality was not verified.

Checklist

  • I kept the PR focused on one change.
  • I added or updated tests/docs when needed.
  • I checked that no secrets, API keys, or credentials are included.

Rights & sign-off (required)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch hallucination-rate evaluation and reporting over a dataset

1 participant