Skip to content

fix(services/webhdfs): encode startAfter and abort the blocks that were written - #8075

Merged
Xuanwo merged 1 commit into
apache:mainfrom
PDGGK:fix-webhdfs-paths
Aug 14, 2026
Merged

fix(services/webhdfs): encode startAfter and abort the blocks that were written#8075
Xuanwo merged 1 commit into
apache:mainfrom
PDGGK:fix-webhdfs-paths

Conversation

@PDGGK

@PDGGK PDGGK commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

None — found while reading the service. Two independent path defects, both one expression.

1. The batched lister's startAfter is not encoded

webhdfs_list_status_batch encodes the path and then appends the marker raw:

let mut url = format!(
    "{}/webhdfs/v1/{}?op=LISTSTATUS_BATCH",
    self.endpoint,
    percent_encode_path(&p),                                 // encoded
);
if !start_after.is_empty() {
    url += format!("&startAfter={start_after}").as_str();    // not encoded
}

start_after is ctx.token, which the lister sets verbatim from the last entry's pathSuffix — a raw HDFS file name:

ctx.token.clone_from(&file_statuses.last().unwrap().path_suffix);

Measured by feeding the resulting strings to http::Request::get:

name what is sent
report 2024.csv BUILD ERROR: invalid uri character — the listing aborts
note#1.txt startAfter=note — the rest is a fragment, so the batch boundary rewinds and the page repeats
a&b=c startAfter=a plus a stray b=c parameter

This is the same defect just fixed for the marker parameter across obs/swift/cos/azblob/azfile in #8073 — it was outside that PR's scope only because it is a format! concatenation rather than a QueryPairsWriter::push.

startAfter is the one query value in this file that carries data the server chose. user.name is configuration, appears at eleven sites, and is a separate question; &{auth} is deliberately a whole query fragment and must stay verbatim. Neither is touched here.

2. abort_block deletes paths that were never written

write_block creates each block under the atomic write dir:

.webhdfs_create_object(&self.ctx, &format!("{atomic_write_dir}{block_id}"), ...)

and complete_block concatenates from the same strings. But abort_block asked for the bare id:

.webhdfs_delete(&self.ctx, &block_id.to_string())

So aborting a multi-block write — Writer::abort(), or any mid-write failure — deleted a path that does not exist and left every uploaded block sitting in atomic_write_dir for ever. WebHDFS answers DELETE on a missing path with 200 and {"boolean": false}, so the leak is silent.

It now resolves atomic_write_dir the same way write_block does, which also makes the unsupported case explicit instead of deleting a top-level path named after a UUID.

Tests

No new tests, deliberately: both fixes are single expressions inside async methods whose only seam is an HTTP round trip, and asserting on either would mean restructuring the URL builder and the writer — which I would rather not fold into a fix. The 7 existing unit tests pass; cargo fmt --all -- --check and cargo clippy -p opendal-service-webhdfs --all-targets are clean with zero warnings.

I kept the two together because they are one-liners in one service; happy to split if you would prefer them reviewed separately.

Are there any user-facing changes?

Yes, for services-webhdfs: a batched listing whose boundary file name contains a reserved character now resumes correctly instead of failing to build the request or repeating a page, and an aborted multi-block write removes its blocks instead of leaving them behind.

…re written

Two independent path defects in this service.

1. The batched lister's startAfter is not encoded.

    let mut url = format!(
        "{}/webhdfs/v1/{}?op=LISTSTATUS_BATCH",
        self.endpoint,
        percent_encode_path(&p),      // the path is encoded
    );
    if !start_after.is_empty() {
        url += format!("&startAfter={start_after}").as_str();   // this is not
    }

start_after is ctx.token, which lister.rs sets verbatim from the last
entry's pathSuffix -- a raw HDFS file name. Measured against
http::Request::get: a name with a space is a hard "invalid uri
character" build error, one with # truncates the marker so the batch
boundary rewinds and a page repeats, and one with & grafts a stray
parameter onto the query.

This is the same defect just fixed for the marker parameter across
obs/swift/cos/azblob/azfile in apache#8073; it was out of that PR's scope only
because it is a format! concatenation rather than a QueryPairsWriter
push. startAfter is the one query value in this file carrying data the
server chose. user.name is config, at eleven sites, and is a separate
question; &{auth} is deliberately a whole query fragment and must stay
verbatim.

2. abort_block deletes paths that were never written.

write_block creates each block at {atomic_write_dir}{block_id} and
complete_block concatenates from the same strings, but abort_block asked
for {block_id} alone. So aborting a multi-block write -- Writer::abort,
or any mid-write failure -- deleted a path that does not exist and left
every uploaded block sitting in atomic_write_dir for ever.

It now resolves atomic_write_dir the same way write_block does, which
also makes the unsupported case explicit rather than deleting a
top-level path named after a UUID.

No new tests: both are single expressions inside async methods whose
seams are an HTTP round trip, and asserting on either would mean
restructuring the URL builder and the writer. The seven existing unit
tests pass, fmt and clippy are clean.
@PDGGK
PDGGK requested a review from Xuanwo as a code owner August 14, 2026 16:53
@dosubot dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. releases-note/fix The PR fixes a bug or has a title that begins with "fix" labels Aug 14, 2026
@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Aug 14, 2026
@Xuanwo
Xuanwo merged commit 1797b05 into apache:main Aug 14, 2026
105 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer releases-note/fix The PR fixes a bug or has a title that begins with "fix" size:S This PR changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants