Skip to content

feat: add iter()/aiter() for lazy filter-based key iteration - #682

Open
Aryan-Pardeshi wants to merge 3 commits into
redis:mainfrom
Aryan-Pardeshi:feat/index-lazy-key-iteration
Open

feat: add iter()/aiter() for lazy filter-based key iteration#682
Aryan-Pardeshi wants to merge 3 commits into
redis:mainfrom
Aryan-Pardeshi:feat/index-lazy-key-iteration

Conversation

@Aryan-Pardeshi

@Aryan-Pardeshi Aryan-Pardeshi commented Aug 9, 2026

Copy link
Copy Markdown

Fixes #489

Adds iter() and aiter() for lazy, filter-based iteration over the keys in an index.

Neither existing API covers this: paginate() yields full document records rather than keys, and scan_by_pattern() works on raw Redis key patterns and materialises a list. For index maintenance over a large index you want keys only, streamed, and selectable by filter expression.

for key in index.iter(Tag("category") == "A"):
    ...

Both take an optional FilterExpression (defaulting to match-all) and a batch_size defaulting to DEFAULT_BULK_BATCH_SIZE, page through _query with return_fields=["id"], and yield keys one at a time without ever building the full list. The async version mirrors the sync one exactly.

tests/integration/test_index_iteration.py covers full iteration, filtered iteration, laziness (the first key arrives without draining the index), and a batch_size smaller than the document count so the paging loop is actually exercised rather than short-circuiting on a single batch. 7 passed against Redis in Docker.

Two things I want to flag rather than have you find:

iter shadows the builtin as a method name. That is what the issue asked for, but if you would rather have iter_keys/aiter_keys I will rename without argument.

I did not add __iter__/__aiter__ dunders. Making a SearchIndex directly iterable would mean list(index) silently issues a full paged scan against Redis, which felt like a surprising thing to attach to a plain for loop. Easy to add if you want it.


Note

Low Risk
Additive public API on top of an existing bulk helper; no changes to auth, persistence, or bulk mutation semantics beyond new iteration entry points.

Overview
Adds iter() on SearchIndex and aiter() on AsyncSearchIndex so callers can walk document Redis keys (not full records) with an optional FilterExpression and configurable batch_size, defaulting to match-all when the filter is omitted.

Both methods are thin wrappers around the existing _iter_keys_by_filter path (FT.AGGREGATE + WITHCURSOR), so iteration avoids MAXSEARCHRESULTS caps that affect FT.SEARCH + LIMIT, with the same de-duplication and memory characteristics documented on that helper.

Integration tests in test_index_iteration.py cover full and filtered scans, lazy consumption, and paging when batch_size is smaller than the index size for sync and async.

Reviewed by Cursor Bugbot for commit 69dd491. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 441a51d. Configure here.

Comment thread redisvl/index/index.py Outdated
…H+LIMIT

FT.SEARCH + LIMIT is capped by MAXSEARCHRESULTS and non-deterministic without
a unique sort, which is exactly the large-index case this API targets. The
repo already has _iter_keys_by_filter for this reason -- it pages with
FT.AGGREGATE ... WITHCURSOR and always releases the cursor. Delegate to it
instead of reimplementing offset-based paging.

Caught by Cursor Bugbot on review, verified against the existing
_iter_keys_by_filter docstring and callers (drop_by_filter, update_by_filter).
@Aryan-Pardeshi

Copy link
Copy Markdown
Author

Good catch — pushed a fix. iter()/aiter() now delegate to _iter_keys_by_filter(), which pages with FT.AGGREGATE ... WITHCURSOR instead of FT.SEARCH + LIMIT, so this is no longer subject to MAXSEARCHRESULTS. Tests still pass (7/7), full unit suite green (1292 passed).

@vishal-bala vishal-bala self-assigned this Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add iter()/aiter() for filter-based lazy iteration

2 participants