fix: normalize benchmark prompt CSV quoting before dataset creation

[koenvanderveen]

Problem

The AILuminate reserve prompt set quotes prompt_text inconsistently — a prompt containing a newline or a double quote is quoted, but a prompt containing a comma is not. A plain csv.DictReader splits that row into 5 fields, puts the tail under the implicit restkey, and nobody reads it. The prompt reached the model truncated mid-sentence, the completion looked plausible, and safety_eval_results.json reported it as a normal result — no error at any point.

Found while checking whether the job would run on a real reserve-prompt file: one of 9 rows lost 13 characters silently.

Changes

• Benchmark owner prep repairs and re-quotes. read_prompt_csv reads with restkey and folds the spilled tail back into prompt_text; normalize_prompt_csv writes the result back out with csv.DictWriter. Safe because prompt_text is the last column, so anything past the header belongs to it, and extras only appear for an unquoted field where the split was purely on commas — re-joining on , restores the text exactly.
• The upload path is the normalized file by construction. The colab notebook downloads into prompts/raw/ and writes the re-quoted copy to the prompts/mock/ and prompts/private/ paths create_dataset uploads; the dev notebook folds it into stage_prompt_csv, which already copied into a temp dir.
• JOB_CODE is unchanged in both notebooks — still a plain csv.DictReader. The fix stays on the benchmark owner's side, so the model owner reviews and approves exactly the code they did before.
• Sample CSVs now carry the same inconsistent quoting as the real file (some fields quoted without needing it, one comma-containing field left unquoted per file), so this path is actually exercised. Prompt text is unchanged — quoting only.

Both notebooks that read data/*.csv are updated. Every other safety_prompts reference in the repo uses the inline .txt prompts and is unaffected.

Test plan

Executed the patched cells and then the unchanged JOB_CODE with setup_model/generate stubbed:

safety_prompts.csv:  5 rows, lens [104, 429, 454, 554, 102]
  ragged for naive reader : 0
  naive == repaired rows  : True
job on normalized upload : 5 prompts, lens [104, 429, 454, 554, 102]

• naive == repaired rows: True is the guarantee: a plain DictReader on the upload returns exactly what the repairing reader parsed from the messy source. Same for the mock file and the dev notebook's mock job.
• Regenerated CSVs are byte-identical in length to the previous versions and parse to the same rows, confirming only quoting changed.
• pre-commit run --all-files passes with no files modified.

Not covered: real inference. Weights, JAX, and the enclave path are untested here — this only exercises CSV parsing and the output JSON contract.

Note for reviewers

A field can only be left unquoted in the sample data if it has no embedded " and no trailing whitespace — the trim trailing whitespace hook otherwise eats a trailing space and silently alters the prompt. That bit me once; the generator now asserts on it, and data/README.md records it.

The colab notebook fetches these CSVs from raw.githubusercontent.com/.../dev/, so it keeps getting the old clean files until this merges.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB