Skip to content

fix(services/vercel-blob): send the pagination cursor so listing terminates - #8071

Merged
erickguan merged 2 commits into
apache:mainfrom
PDGGK:fix-vercel-blob-cursor
Aug 15, 2026
Merged

fix(services/vercel-blob): send the pagination cursor so listing terminates#8071
erickguan merged 2 commits into
apache:mainfrom
PDGGK:fix-vercel-blob-cursor

Conversation

@PDGGK

@PDGGK PDGGK commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

None — found while reading the service.

Rationale for this change

VercelBlobLister::next_page stores the cursor the server returns:

ctx.done = !resp.has_more;

if let Some(cursor) = resp.cursor {
    ctx.token = cursor;
}

and nothing ever reads it back — that assignment is the only mention of ctx.token in the crate. VercelBlobCore::list took only a prefix and a limit:

let mut url = format!(
    "https://blob.vercel-storage.com?prefix={}",
    percent_encode_path(prefix)
);

if let Some(limit) = limit {
    url.push_str(&format!("&limit={limit}"))
}

so page two is byte-identical to page one. PageLister::next loops while ctx.done is false:

loop {
    if let Some(entry) = self.ctx.entries.pop_front() { return Ok(Some(entry)); }
    if self.ctx.done { return Ok(None); }
    self.inner.next_page(&mut self.ctx).await?;
}

A truncated listing therefore never advances: the same page is fetched and pushed again for as long as has_more stays true. Operator::list collects into a Vec and grows without bound; lister() streams the same entries forever.

Vercel documents cursor alongside prefix and limit"A string obtained from a previous list response to be used for reading the next page of results" — and notes the default limit is 1000, so any prefix holding more than that reaches this with the caller setting no options at all.

What changes are included in this PR?

list now takes the cursor and appends it when non-empty, and the URL construction moves into build_list_url so it can be asserted directly. The four internal lookups pass "" — each asks for Some(1) to resolve a single blob's URL and never paginates.

One extra guard in the lister: has_more true with no cursor now ends the listing rather than leaving ctx.done false. That truncates, but the only other option is re-requesting the same page indefinitely. Happy to drop it if you would rather not carry the branch.

Tests

Four unit tests. Removing the cursor from the URL again fails exactly the two that involve one:

the_first_page_carries_no_cursor a_later_page_resumes_from_the_cursor the_cursor_is_percent_encoded the_root_prefix_is_sent_empty
cursor not sent (current main) ok FAILED FAILED ok
this PR ok ok ok ok

The cursor is percent-encoded because it is an opaque server token: an unescaped & or = would silently graft extra parameters onto the query, the same reason prefix is encoded one line above.

cargo test -p opendal-service-vercel-blob --lib → 6 passed. cargo fmt --all -- --check and cargo clippy -p opendal-service-vercel-blob --all-targets both clean.

Are there any user-facing changes?

Yes — a vercel-blob listing that the server truncates now advances through its pages and terminates, instead of re-emitting the first page indefinitely. VercelBlobCore::list gains a cursor parameter, but mod core is private and the crate re-exports only VercelBlobBuilder and VercelBlobConfig, so nothing outside the crate can name it.

…inates

The lister stores the cursor the server returns:

    ctx.done = !resp.has_more;

    if let Some(cursor) = resp.cursor {
        ctx.token = cursor;
    }

and nothing ever reads it back. VercelBlobCore::list took only a prefix
and a limit, so page two was byte-identical to page one:

    https://blob.vercel-storage.com?prefix={prefix}&limit={limit}

ctx.token is the only write of that field in the crate. Meanwhile
PageLister::next loops while ctx.done is false, so a listing the server
truncates never advances: the same page is fetched and pushed again for
as long as has_more stays true. Operator::list collects into a Vec and
grows without bound; lister() streams the same entries forever.

Vercel documents cursor beside prefix and limit -- "a string obtained
from a previous list response to be used for reading the next page of
results" -- and the default limit is 1000, so any prefix holding more
than that reaches this without the caller setting anything.

list now takes the cursor and appends it when non-empty, and the URL
construction moves into build_list_url so it can be asserted directly.
The four internal lookups pass "": each asks for Some(1) to resolve a
single blob's URL and never paginates.

One extra guard in the lister: has_more true with no cursor now ends the
listing rather than leaving ctx.done false. That truncates, but the only
other option is re-requesting the same page indefinitely.

Four unit tests. Dropping the cursor from the URL again fails exactly
the two that involve one; the first-page and root-prefix tests hold
either way.
@PDGGK
PDGGK requested a review from Xuanwo as a code owner August 14, 2026 14:40
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. releases-note/fix The PR fixes a bug or has a title that begins with "fix" labels Aug 14, 2026
Comment thread core/services/vercel-blob/src/core.rs Outdated
ctx: &OperationContext,
prefix: &str,
limit: Option<usize>,
cursor: &str,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use Option<&str>

Comment thread core/services/vercel-blob/src/core.rs Outdated
if let Some(limit) = limit {
url.push_str(&format!("&limit={limit}"))
}
let url = build_list_url(prefix, limit, cursor);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inline function.

Comment thread core/services/vercel-blob/src/core.rs Outdated
pub(super) use error::*;

#[cfg(test)]
mod tests {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove tests

Review points from @erickguan, all three applied:

- cursor is Option<&str> rather than an empty string standing in for
  "no cursor"; the four single-blob lookups pass None, and the lister
  turns an empty ctx.token into None
- build_list_url removed, the format! goes back inline
- test module removed

The functional change is unchanged: the cursor is sent when there is
one, percent-encoded because it is an opaque server token.
@dosubot dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Aug 15, 2026
@PDGGK

PDGGK commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

All three applied in 7f1812c.

Option<&str> is the better shape — an empty string standing in for "no cursor" was encoding absence in the value. The four single-blob lookups pass None, and the lister turns an empty ctx.token into None:

let cursor = if ctx.token.is_empty() {
    None
} else {
    Some(ctx.token.as_str())
};

build_list_url is gone and the format! is back inline, with the cursor appended when present:

if let Some(cursor) = cursor {
    url.push_str(&format!("&cursor={}", percent_encode_path(cursor)));
}

Flagging one thing that goes with the tests, not to argue for keeping them: one of the four asserted that the cursor is percent-encoded. A Vercel cursor is an opaque server token, so an unescaped & or = in it would graft extra parameters onto the query — the same failure mode as the marker parameter in #8073. The percent_encode_path call itself stays; it is just no longer pinned by anything. Your call entirely.

cargo fmt --all -- --check and cargo clippy -p opendal-service-vercel-blob --all-targets are clean.

PDGGK added a commit to PDGGK/opendal that referenced this pull request Aug 15, 2026
Matching the shape asked for on apache#8068, apache#8069 and apache#8071 rather than
waiting to be asked again: the extracted helper goes, the construction
is inline, and the test module goes with it.

The fix is unchanged -- the concatenation is already relative to the
root, so the second build_rel_path is what truncated it, and "/" as the
listed path must not become a leading slash on the entry.
PDGGK added a commit to PDGGK/opendal that referenced this pull request Aug 15, 2026
Matching the shape asked for on apache#8068, apache#8069 and apache#8071 rather than
waiting to be asked again. The function goes back to being a method as
it was, and the three tests added with it go.

The change is now the two etag lines: a listed directory reports the
etag stored with its record, and the synthesised directory placeholder
reports none.

The relative_to_root tests already in this file are from apache#8048 and are
untouched.
PDGGK added a commit to PDGGK/opendal that referenced this pull request Aug 15, 2026
Matching the shape asked for on apache#8068, apache#8069 and apache#8071 rather than
waiting to be asked again: the two extracted helpers go and the branch
is written inline, leaving the existing folder loop untouched.

resp becomes mut so embedded can be taken out of it, which keeps the
rest of the response available for the file case below.
@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Aug 15, 2026
@erickguan
erickguan merged commit 2ce124a into apache:main Aug 15, 2026
110 checks passed
erickguan pushed a commit that referenced this pull request Aug 15, 2026
#8070)

* fix(services/ipmfs): stop stripping the root twice from listed entries

The listing loop concatenates the path being listed with a files/ls
name, then runs the result through build_rel_path a second time:

    let path = match object.mode() {
        EntryMode::FILE => format!("{}{}", self.path, object.name),
        ...
    };
    let path = build_rel_path(&self.root, &path);

self.path is what the operator handed the service, so it is already
relative to the root -- IpmfsCore::ipmfs_ls is what turns it into a
rooted absolute path for the request. The concatenation is therefore
root-relative before that call, and the call removes a prefix that is
not there.

Under the default root "/" the second strip happens to cancel out, which
is why this has gone unnoticed. Under any other root, in release:

    root "/abc/", listing "dir/", name "a"
        build_rel_path("/abc/", "dir/a") -> "a"
        the directory is dropped from every entry path

    root "/abc/", listing the root itself (self.path is "/"), name "a"
        build_rel_path("/abc/", "/a")
        panicked: start byte index 5 is out of bounds for string of
        length 2

In a debug build both cases trip the debug_assert! inside build_rel_path
instead.

The construction moves into build_entry_path, which takes no root at
all -- an entry path does not depend on one. The only thing it has to
handle is that self.path is "/" when the root itself is listed, and an
entry path carries no leading slash.

The self-entry twenty lines above is left alone: it does
build_abs_path then build_rel_path, a deliberate round trip, and is
correct.

Three unit tests. Putting the old expression back passes all of them
with a root of "/" and fails three of them with a root of "/abc/" --
which is the shape of the bug.

Note for rebasing: #7801 renames this very call to
build_relative_path. If that lands first this needs a one-word rebase;
the call itself goes away here.

* Inline the entry-path construction and drop the test module

Matching the shape asked for on #8068, #8069 and #8071 rather than
waiting to be asked again: the extracted helper goes, the construction
is inline, and the test module goes with it.

The fix is unchanged -- the concatenation is already relative to the
root, so the second build_rel_path is what truncated it, and "/" as the
listed path must not become a leading slash on the entry.

* Drop the inline comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer releases-note/fix The PR fixes a bug or has a title that begins with "fix" size:S This PR changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants