Skip to content

Fix sample-limited dataset extractors - #307

Open
neubig wants to merge 7 commits into
mainfrom
fix-full-dataset-extractors
Open

Fix sample-limited dataset extractors#307
neubig wants to merge 7 commits into
mainfrom
fix-full-dataset-extractors

Conversation

@neubig

@neubig neubig commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • remove hardcoded AgentTuning sample ID filters so extract_raw emits all valid examples for AgentInstruct configs
  • remove the Android-in-the-Wild five-record per split cap
  • make AndroidControl ignore the bundled tf_record_sample and use local full TFRecords or the public GCS full-corpus glob

Validation

  • python -m py_compile on all touched extract_raw.py files
  • static grep confirmed the AgentTuning SAMPLE_IDS gates and Android-in-the-Wild i >= 5 cap are gone
  • runtime AgentTuning smoke test was attempted, but local Hugging Face datasets loading failed in the existing venv while reading cached feature metadata before extractor iteration

Context

The ADP v2 upload included several configs with only 3-5 rows because these extractors were sample/demo-limited while the upload pipeline treated completed artifacts as full datasets.

Tracking

Related to #218.

Live evidence

  • 2026-07-14: at PR head 90dbcff, ran the AgentInstruct AlfWorld extractor against the public Hugging Face parquet source. The source generated 336 rows, and the extractor emitted consecutive valid records alfworld_0 through alfworld_5. This exercises the new full-corpus path and demonstrates that the previous five hard-coded IDs are no longer the output limit.
  • Command: uv run --with datasets python datasets/agenttuning_alfworld/extract_raw.py piped to head for six emitted records; the extractor handled the closed pipe cleanly.

Closes #310

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix sample-limited dataset extractors

1 participant