fix(services/hf): correct the refs/convert revision split - #8045
Conversation
parse_revision sliced rev_and_path[..14 + slash] to recover the revision, but
"refs/convert/" is 13 bytes, so the separator that terminates the revision
segment was pulled into the revision itself. A URI like
datasets/squad@refs/convert/parquet/default/train/0000.parquet
yielded the revision "refs/convert/parquet/" rather than "refs/convert/parquet",
and that trailing slash is carried into the operator config and percent-encoded
into every request path as refs%2Fconvert%2Fparquet%2F.
Rebuild the revision with format! the way the refs/pr/ arm immediately below
already does, so there is no offset to get wrong.
erickguan
left a comment
There was a problem hiding this comment.
Good investigation!
minor comment so approve already.
| if let Some(rest) = rev_and_path.strip_prefix("refs/convert/") { | ||
| return if let Some(slash) = rest.find('/') { | ||
| ( | ||
| rev_and_path[..14 + slash].to_string(), | ||
| rest[slash + 1..].to_string(), | ||
| ) | ||
| let revision = format!("refs/convert/{}", &rest[..slash]); | ||
| (revision, rest[slash + 1..].to_string()) | ||
| } else { | ||
| (rev_and_path.to_string(), String::new()) | ||
| }; |
There was a problem hiding this comment.
| if let Some(rest) = rev_and_path.strip_prefix("refs/convert/") { | |
| return if let Some(slash) = rest.find('/') { | |
| ( | |
| rev_and_path[..14 + slash].to_string(), | |
| rest[slash + 1..].to_string(), | |
| ) | |
| let revision = format!("refs/convert/{}", &rest[..slash]); | |
| (revision, rest[slash + 1..].to_string()) | |
| } else { | |
| (rev_and_path.to_string(), String::new()) | |
| }; | |
| if let Some(rest) = rev_and_path.strip_prefix("refs/convert/") { | |
| return match rest.split_once('/') { | |
| Some((segment, path)) => ( | |
| format!("refs/convert/{segment}"), | |
| path.to_string(), | |
| ), | |
| None => (rev_and_path.to_string(), String::new()), | |
| }; | |
| } |
Applies the reviewer's suggestion. Both the refs/convert/ and refs/pr/ arms now use split_once instead of find + manual slicing, which removes the index arithmetic entirely and keeps the two adjacent arms symmetric. No behaviour change: split_once yields exactly the same two halves the find + slice pair did.
|
Thanks — applied in 35f23d3. One thing beyond your suggestion, easy to drop if you'd rather not have it: I applied the same shape to the adjacent
Say the word and I'll drop that hunk. 57 unit tests pass, |
Which issue does this PR close?
None — filing directly, per CONTRIBUTING. I searched first: no open issue mentions the HF revision parsing, and the only two open PRs touching
core/services/hfare the v0.58.2 release prep and #7801'sbuild_abs_pathrename, neither of which overlaps.Rationale of this change
parse_revisionrecovers the revision from a URI by slicing:"refs/convert/"is 13 bytes, not 14, so the/that terminates the revision segment is pulled into the revision:The trailing slash is copied into the operator config by
HfConfig::from_uriand then percent-encoded bypercent_encode_revision— which usesNON_ALPHANUMERICand does not trim it — intopaths_info_url,git_commit_url,resolve_urlandfile_tree_url, so requests go out withrefs%2Fconvert%2Fparquet%2F. The HF API resolves revisions exactly, and that is not a valid ref.The repo already agrees on the correct shape:
test_hf_path_info_url_custom_endpointpins the URL as.../paths-info/refs%2Fconvert%2Fparquet, with no trailing%2F.Why it was not caught
resolve_refs_convert_revisiononly coversdatasets/squad@refs/convert/parquet, which has no path after the revision. That takes theelsearm and never reaches the slice, so the off-by-one was invisible.What changed
The
refs/pr/arm ten lines below already avoids the whole class of bug by rebuilding withformat!:This adopts the same idiom for
refs/convert/rather than changing14to13— a corrected magic offset is still a magic offset.Tests
New
resolve_refs_convert_revision_with_pathnext to the existing case. Onmain:The existing test passing alongside it is the point — it shows the gap rather than the change moving any goalposts.
With this change:
cargo test -p opendal-service-hfis 57 passed / 0 failed, andcargo fmt --checkandcargo clippy --all-targetsare clean.Scope note
HfConfig::from_urialso discards the parsedpath_in_repo—rootcomes only fromopts.get("root")— so in this exact scenario the path is dropped regardless. That looked like a separate change and I have left it alone; happy to follow up if you would like it fixed.This only affects URI-based construction (
Operator::from_uri("hf://...")). Anyone callingHuggingfaceBuilder::revision("refs/convert/parquet")directly never reachesparse_revision.