You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is there somewhere were the file counts are listed in full for each dataset by language (dedup and full)?
Essentially I am looking to make sure that I have accessed the entirety of the dataset, so I either need to understand the dataset size difference, or know how many files there should be for each language so I can validate my download. Ideally both.
The dataset was compressed with parquet + snappy when uploaded to the hub. Here is a before and after deduplication comparison in terms of physical size (w/o compression) and number of files: bquxjob_25a65048_188085aa72f.csv
The last line is the total change, here is the screenshot for quick reference:
Activity
ChenghaoMou commentedon May 11, 2023
The dataset was compressed with parquet + snappy when uploaded to the hub. Here is a before and after deduplication comparison in terms of physical size (w/o compression) and number of files:
bquxjob_25a65048_188085aa72f.csv
The last line is the total change, here is the screenshot for quick reference:
