From ef51726d8d1fc960489e85edad164776d3635408 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 10 Jul 2026 08:48:21 +0200 Subject: [PATCH 1/2] [FSTORE-2059][FSTORE-2060] Document collect, aggregate, and snowflake REST serving https://hopsworks.atlassian.net/browse/FSTORE-2059 https://hopsworks.atlassian.net/browse/FSTORE-2060 Document the collect(N) and aggregate query operations and the REST serving of snowflake-schema feature views. concepts/fs/feature_view/online_api.md introduces the two operations and how the online API serves them (primary-key reads, ordered index scans, RonSQL pushdown). user_guides/fs/feature_view/query.md adds the "Collect recent rows" and "Aggregate rows by entity" sections: order_by/ascending semantics, the (entity..., order column) primary-key requirement for online feature groups, filter-before-limit semantics and the online filter shape restriction, label and mutual-exclusion rules, the N and N-times-width caps, TTL-implied lookback, aggregation functions and output naming, and the window/TTL bound. The nested-join section now states the REST gates: full-primary-key hops, one feature store, and uniform join types (all inner or all left; mixed subtrees stay on the SQL client). user_guides/fs/feature_view/feature-vectors.md adds "Retrieve collected rows" (folded array-of-structs feature, scan_vectors) and "Retrieve aggregate features", and updates the client-choice section with the REST serving paths and the snowflake gates. The filter note now distinguishes point-read filters (not applied) from collect and aggregate filters (applied before folding). All claims verified against the implementation: query.py and feature_view.py signatures, QueryController collect/aggregate validation, PreparedStatementBuilder join gates and filter rendering, ConstructorController output naming, and both clients' empty-history fold. Examples use return_type="pandas" (get_feature_vector has no dict return type). markdownlint, snakeoil, and the strict mkdocs build pass. Signed-off-by: Jim Dowling Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01L5mBKFhoVJBS7J7kCSKQVN --- docs/concepts/fs/feature_view/online_api.md | 8 ++ .../fs/feature_view/feature-vectors.md | 59 +++++++++++++- docs/user_guides/fs/feature_view/query.md | 79 +++++++++++++++++++ 3 files changed, 145 insertions(+), 1 deletion(-) diff --git a/docs/concepts/fs/feature_view/online_api.md b/docs/concepts/fs/feature_view/online_api.md index 6fa9f453c4..af539171ae 100644 --- a/docs/concepts/fs/feature_view/online_api.md +++ b/docs/concepts/fs/feature_view/online_api.md @@ -2,6 +2,14 @@ The Feature View provides an Online API to return an individual feature vector, To retrieve a feature vector, a client needs to provide the primary key(s) for the feature groups backing the feature view. For example, if you have `customer_profile` and `customer_purchases` Feature Groups both with `customer_id` as a primary key, and a Feature View made up from features from both Feature Groups, then, you would use `customer_id` to retrieve a feature vector using the Feature View object. +A feature-view query can also produce historical or aggregate online features. +The `collect` operation returns the most recent N matching rows for an entity as one ordered array of row structs. +The `aggregate` operation returns scalar count, sum, minimum, maximum, or average features over an entity's matching rows and optional event-time window, as well as the greatest or least value across several columns. +Hopsworks applies the same operations when creating offline training data to prevent training-serving skew. + +The online API serves regular features with primary-key reads, collected history with ordered index scans, and aggregate or supported nested-join features with pushdown queries in RonDB. +The entity key identifies the feature vector, while a collected feature's event-time ordering column remains inside the collected rows and is not required from the caller. + ## Feature Vectors A feature vector is a row of features (without the primary key(s) and event timestamp): diff --git a/docs/user_guides/fs/feature_view/feature-vectors.md b/docs/user_guides/fs/feature_view/feature-vectors.md index 125e3615b5..121359ba99 100644 --- a/docs/user_guides/fs/feature_view/feature-vectors.md +++ b/docs/user_guides/fs/feature_view/feature-vectors.md @@ -14,7 +14,8 @@ If you need to get more familiar with the concept of feature vectors, you can re ## Retrieval You can get back feature vectors from either python or java client by providing the primary key value(s) for the feature view. -Note that filters defined in feature view and training data will not be applied when feature vectors are returned. +Filters on regular point-read features are not applied when feature vectors are returned. +Filters attached to a collected or aggregated query node are applied before collecting or aggregating its rows. If you need to retrieve a complete value of feature vectors without missing values, the required `entry` are [FeatureView.primary_keys][hsfs.feature_view.FeatureView.primary_keys]. Alternative, you can provide the primary key of the feature groups as the key of the entry. It is also possible to provide a subset of the entry, which will be discussed [below](#partial-feature-retrieval). @@ -47,6 +48,59 @@ It is also possible to provide a subset of the entry, which will be discussed [b featureView.getFeatureVectors(Lists.newArrayList(entry1, entry2)); ``` +### Retrieve collected rows + +A query that uses `collect` contributes one array-of-struct feature to the returned feature vector. +The feature is named `_collect`, with any join prefix applied to avoid name collisions. +Entity-key features remain scalar, and the collected ordering feature is included inside each struct. + +```python +vector = feature_view.get_feature_vector( + entry={"user_id": 7}, + return_type="pandas", +) +transactions = vector["transactions_collect"].iloc[0] + +most_recent_amount = transactions[0]["amount"] +most_recent_event_time = transactions[0]["event_time"] +``` + +The array contains at most the N rows configured by the feature-view query. +It is ordered newest first unless the query used `ascending=True`. +An entity with no matching history returns an empty collected array. + +Use `scan_vectors` when you need the collected source rows as a table instead of one folded feature. + +```python +recent_rows = feature_view.scan_vectors( + entry={"user_id": 7}, + limit=10, + return_type="pandas", +) +``` + +The optional `limit` must be a positive integer and can narrow, but cannot widen, the N configured by `collect`. +Set `return_type` to `"list"`, `"pandas"`, or `"polars"`. + +### Retrieve aggregate features + +A query that uses `aggregate` returns one scalar feature for each configured feature and aggregation function. + +```python +vector = feature_view.get_feature_vector( + entry={"user_id": 7}, + return_type="pandas", +) + +amount_sum = vector["amount_sum"].iloc[0] +amount_average = vector["amount_avg"].iloc[0] +transaction_count = vector["count"].iloc[0] +``` + +The SQL and REST online clients compute the aggregation in the online store. +With the REST client, batch retrieval groups entities into bounded requests when the serving key contains one feature. +Composite serving keys use per-entry requests when the online query cannot safely group them. + ### Required entry Starting from python client v3.4, you can specify different values for the primary key of the same name which exists in multiple feature groups but are not joint by the same name. @@ -280,6 +334,9 @@ It requires a direct SQL connection to your RonDB cluster and uses python asynci The REST client is an alternative implementation connecting to [RonDB Feature Vector Server](./feature-server.md). Perfect if you want to avoid exposing ports of your database cluster directly to clients. This implementation is available as of Hopsworks 3.7. +Collected rows use an ordered RonDB index scan, while aggregate and supported nested-join queries are pushed down through RonSQL. +Snowflake feature views can use the REST client when every nested hop joins the complete primary key, all nested joins use the same join type (all inner or all left), and the subtree stays within one feature store. +Use the SQL client for right or full joins, subtrees mixing inner and left joins, partial-primary-key hops, cross-feature-store nested joins, or another query shape that cannot be served through REST. Initialise the client by calling the `init_serving` method on the Feature View object before starting to fetch feature vectors. This will initialise the chosen client, test the connection, and initialise the transformation functions registered with the Feature View. diff --git a/docs/user_guides/fs/feature_view/query.md b/docs/user_guides/fs/feature_view/query.md index 108c505eea..79723bcae9 100644 --- a/docs/user_guides/fs/feature_view/query.md +++ b/docs/user_guides/fs/feature_view/query.md @@ -220,6 +220,12 @@ Now, you have the benefit that in online inference you only need to pass two ser }) ``` +The Python SQL client executes each nested subtree as one SQL statement. +The Python REST client pushes supported nested subtrees to RonDB through RonSQL, so callers provide the same root serving keys for either client. +REST serving requires every nested join to cover the child feature group's complete primary key and all feature groups in the subtree to belong to the same feature store. +Inner and left joins are supported, and all nested joins in a subtree must use the same join type, because a subtree mixing them cannot be split into independent REST queries without changing which rows an inner-join miss removes. +Right joins, full joins, subtrees mixing inner and left joins, partial-primary-key hops, and cross-feature-store nested joins require the SQL client. + #### Filter In the same way as joins, applying filters to feature groups creates a query with the applied filter. @@ -302,6 +308,79 @@ The filters can be applied at any point of the query: .filter(creditCardTransactionsFg.getFeature("category").eq("Grocery"))) ``` +#### Collect recent rows + +Use `collect` to include the most recent matching rows for each entity in a feature view. +The operation produces one array-of-struct feature whose elements contain the selected value features and the ordering feature. +This preserves the relationship between values in the same source row, including null values. + +`collect` is a non-terminal query operation, so you can filter the source rows before collecting them and join other feature groups afterward. +Filters are applied before the row limit, which means the result contains the most recent N rows that match the filter. +For online serving, filters on a collected or aggregated feature group must be AND-combined comparisons of a feature against a value, because OR conditions and feature-to-feature comparisons cannot be applied to the online statement without breaking the match with offline training data. + +```python +transactions_fg = fs.get_feature_group(name="transactions", version=1) +customers_fg = fs.get_feature_group(name="customers", version=1) + +query = ( + transactions_fg.select(["user_id", "event_time", "amount", "category"]) + .filter(transactions_fg.amount > 0) + .collect(100, order_by="event_time") + .join(customers_fg.select(["country", "tier"]), on=["user_id"]) +) +``` + +The `order_by` argument defaults to the feature group's event-time feature. +The default output order is newest first. +Set `ascending=True` to return the same most recent N rows from oldest to newest. + +For online feature groups, the primary key must contain the entity key followed by the ordering feature. +For example, a per-user transaction history ordered by event time uses the primary key `(user_id, event_time)`. +The ordering feature is not supplied as a serving key because the lookup identifies an entity rather than one event. + +Labels cannot be collected because a collected output is an input feature containing several historical rows. +The value of `n` must be positive and cannot exceed the maximum configured by the Hopsworks administrator. +The product of `n` and the number of selected features is also capped, because point-in-time training data materializes that many values for every source event. + +When time-to-live is enabled on the source feature group, offline training data uses the same lookback horizon as online serving. +An explicit lookback can narrow that horizon but cannot make it wider than the feature group's time-to-live. + +#### Aggregate rows by entity + +Use `aggregate` to define scalar features computed from all matching rows for each entity. +The operation supports `count`, `sum`, `min`, `max`, and `avg`. +Use the special `"*"` key with `count` to count rows. +Use a comma-separated feature key with `greatest` or `least` to aggregate the row-wise greatest or least value. + +```python +from datetime import timedelta + +transactions_fg = fs.get_feature_group(name="transactions", version=1) + +query = ( + transactions_fg.select(["user_id", "event_time", "amount", "fee"]) + .filter(transactions_fg.category == "grocery") + .aggregate( + { + "amount": ["count", "sum", "avg"], + "*": ["count"], + "amount,fee": ["greatest"], + }, + window=timedelta(days=30), + ) +) +``` + +Each feature and function pair creates one scalar output feature. +For example, the query above creates `amount_count`, `amount_sum`, `amount_avg`, `count`, and `amount_fee_greatest`. + +The optional `window` is a trailing event-time interval. +If the source feature group has time-to-live enabled, the aggregation window cannot exceed the time-to-live period because older rows are unavailable during online serving. + +`aggregate` and `collect` are mutually exclusive on the same query node. +Sub-entity grouping through `group_by` is not supported. +You can apply either operation independently to different feature groups in the same feature-view query. + #### Joins and/or Filters on feature view query The query retrieved from a feature view can be extended with new joins and/or new filters. From bf050b74e8dac3defba48228668bbbc54c6691e6 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 10 Jul 2026 18:03:37 +0200 Subject: [PATCH 2/2] [FSTORE-2059][FSTORE-2060] Document windowed-aggregation semantics and gates https://hopsworks.atlassian.net/browse/FSTORE-2059 https://hopsworks.atlassian.net/browse/FSTORE-2060 Updates the collect/aggregate guide for behavior that landed in the later review rounds. The aggregate section now states the creation-time type checks (sum and avg numeric-only, min and max non-complex, greatest and least integer-only), the whole-second window with a required TIMESTAMP event-time feature, the window anchoring contract (read time online, each training row's own event time in point-in-time training data), the (entity..., event_time) primary-key layout online aggregations share with collect, and the point-in-time gates: windowed aggregations join the root directly with an inner or left join, and shapes that cannot be computed point-in-time correct fail at feature view creation instead of producing training data with future leakage. The collect section notes that complex-typed value features cannot be collected on online-enabled feature groups and that n must be an integer. The feature-vectors guide documents empty aggregation windows (count zero, other functions missing) for single and batch reads. markdownlint, snakeoil, and the strict mkdocs build pass. Signed-off-by: Jim Dowling Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01L5mBKFhoVJBS7J7kCSKQVN --- docs/user_guides/fs/feature_view/feature-vectors.md | 2 ++ docs/user_guides/fs/feature_view/query.md | 12 ++++++++++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/user_guides/fs/feature_view/feature-vectors.md b/docs/user_guides/fs/feature_view/feature-vectors.md index 121359ba99..d0c4e6ac60 100644 --- a/docs/user_guides/fs/feature_view/feature-vectors.md +++ b/docs/user_guides/fs/feature_view/feature-vectors.md @@ -97,6 +97,8 @@ amount_average = vector["amount_avg"].iloc[0] transaction_count = vector["count"].iloc[0] ``` +An entity with no rows in the aggregation window returns zero for count features and missing values for the other functions, in single and batch reads alike. + The SQL and REST online clients compute the aggregation in the online store. With the REST client, batch retrieval groups entities into bounded requests when the serving key contains one feature. Composite serving keys use per-entry requests when the online query cannot safely group them. diff --git a/docs/user_guides/fs/feature_view/query.md b/docs/user_guides/fs/feature_view/query.md index 79723bcae9..b163717d63 100644 --- a/docs/user_guides/fs/feature_view/query.md +++ b/docs/user_guides/fs/feature_view/query.md @@ -339,7 +339,8 @@ For example, a per-user transaction history ordered by event time uses the prima The ordering feature is not supplied as a serving key because the lookup identifies an entity rather than one event. Labels cannot be collected because a collected output is an input feature containing several historical rows. -The value of `n` must be positive and cannot exceed the maximum configured by the Hopsworks administrator. +Complex-typed value features (arrays, maps, structs, and binary) cannot be collected on an online-enabled feature group, because the online clients cannot yet decode them inside the collected rows. +The value of `n` must be a positive integer and cannot exceed the maximum configured by the Hopsworks administrator. The product of `n` and the number of selected features is also capped, because point-in-time training data materializes that many values for every source event. When time-to-live is enabled on the source feature group, offline training data uses the same lookback horizon as online serving. @@ -373,10 +374,17 @@ query = ( Each feature and function pair creates one scalar output feature. For example, the query above creates `amount_count`, `amount_sum`, `amount_avg`, `count`, and `amount_fee_greatest`. +The functions are type-checked when the feature view is created: `sum` and `avg` require numeric features, `min` and `max` accept any non-complex feature, and `greatest` and `least` require integer features. -The optional `window` is a trailing event-time interval. +The optional `window` is a trailing event-time interval, given as a whole number of seconds or a timedelta. +A windowed aggregation requires the feature group to declare a TIMESTAMP event-time feature. +Online serving anchors the window at the read time, while point-in-time training data anchors it at each training row's own event time, so a training row never includes source events that had already expired at that row's time. If the source feature group has time-to-live enabled, the aggregation window cannot exceed the time-to-live period because older rows are unavailable during online serving. +For online feature groups, the primary key must contain the entity key followed by the event-time feature, exactly like `collect`, so the online store keeps per-entity history. +In a feature view with point-in-time joins, a windowed aggregation must be joined directly to the root feature group with an inner or left join. +Query shapes that cannot be computed point-in-time correct are rejected with an error when the feature view is created, instead of silently producing training data with future leakage. + `aggregate` and `collect` are mutually exclusive on the same query node. Sub-entity grouping through `group_by` is not supported. You can apply either operation independently to different feature groups in the same feature-view query.