From f0d51f6b4a869477b4f34901968e1826699cf7ab Mon Sep 17 00:00:00 2001 From: bubriks Date: Thu, 2 Jul 2026 17:40:48 +0300 Subject: [PATCH 1/4] init --- docs/concepts/fs/feature_view/offline_api.md | 7 ++++++ .../fs/feature_view/training-data.md | 24 +++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/docs/concepts/fs/feature_view/offline_api.md b/docs/concepts/fs/feature_view/offline_api.md index 48d783ee1f..be0ac3a4e3 100644 --- a/docs/concepts/fs/feature_view/offline_api.md +++ b/docs/concepts/fs/feature_view/offline_api.md @@ -19,6 +19,13 @@ You can apply filters when creating training data from a feature view: Note that filters are not applied when retrieving feature vectors using feature views, as we only look up features for a specific entity, like a customer. In this case, the application should know that predictions for this customer should be made on the model trained on customers in USA, for example. +### Incremental Training Data + +Materialized training data does not have to be rewritten when new data arrives. +A training dataset materialized in the parquet format can be appended to, keeping the same training dataset version: each append materializes only the new time range and stores it as a new increment of the dataset. +This allows very large training datasets to grow with, for example, a daily batch, without rewriting the data already materialized. +See the [training data guide](../../../user_guides/fs/feature_view/training-data.md#appending-to-a-training-dataset) for details. + ### Point-in-time Correct Training Data When you create training data from features in different feature groups, it is possible that the feature groups are updated at different cadences. diff --git a/docs/user_guides/fs/feature_view/training-data.md b/docs/user_guides/fs/feature_view/training-data.md index df1c35d939..bf72a14cde 100644 --- a/docs/user_guides/fs/feature_view/training-data.md +++ b/docs/user_guides/fs/feature_view/training-data.md @@ -147,6 +147,30 @@ X_train, X_test, y_train, y_test = feature_view.train_test_split( ) ``` +## Appending to a Training Dataset + +A materialized training dataset can grow incrementally: `insert_training_data` appends a new batch of data to an existing training dataset version instead of rewriting it. +Only the new batch is computed and written, as a separate increment under the same version, so a large (multi-terabyte) training dataset can grow with, for example, a new daily batch, without rewriting the data already materialized. +The training dataset version and its metadata stay the same, and `get_training_data` returns all increments together. + +```python +# append yesterday's batch to training dataset version 1 +job = feature_view.insert_training_data( + training_dataset_version=1, + start_time="2026-07-01 00:00:00", + end_time="2026-07-01 23:59:59", +) +``` + +Passing `overwrite=True` rewrites the entire training dataset version for the given time range instead of appending. +From a Python client the append is executed by the [ArrowFlight Server with DuckDB](../../../setup_installation/common/arrow_flight_duckdb.md) service if enabled, otherwise a `PySparkJob` is launched, as for `create_training_data`. + +!!! note "Requirements and behavior" + - Appending is only supported for the `parquet` data format. + - Statistics are not recomputed on append; they are left as computed when the training dataset version was created. + - Training datasets with splits are appended per split: a random split re-splits each batch (for example 80/20), while a time-series split assigns the batch by the split's fixed time boundaries, so a recent batch usually lands entirely in the last split and a warning is raised. + - Training datasets materialized with an older Hopsworks version are not appendable; recreate the training dataset once, after which it can be appended to. + ## Read Training Data Once you have created a training dataset, all its metadata are saved in Hopsworks. From ad5fdde5c10d47f60035f866613fbca228a39abe Mon Sep 17 00:00:00 2001 From: bubriks Date: Mon, 6 Jul 2026 11:27:44 +0300 Subject: [PATCH 2/4] Address review feedback: use heading-ID cross-references Co-Authored-By: Claude Fable 5 --- docs/concepts/fs/feature_view/offline_api.md | 4 ++-- docs/user_guides/fs/feature_view/training-data.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/concepts/fs/feature_view/offline_api.md b/docs/concepts/fs/feature_view/offline_api.md index be0ac3a4e3..11c6f13f1f 100644 --- a/docs/concepts/fs/feature_view/offline_api.md +++ b/docs/concepts/fs/feature_view/offline_api.md @@ -22,9 +22,9 @@ In this case, the application should know that predictions for this customer sho ### Incremental Training Data Materialized training data does not have to be rewritten when new data arrives. -A training dataset materialized in the parquet format can be appended to, keeping the same training dataset version: each append materializes only the new time range and stores it as a new increment of the dataset. +A training dataset materialized in the `.parquet` format can be appended to, keeping the same training dataset version: each append materializes only the new time range and stores it as a new increment of the dataset. This allows very large training datasets to grow with, for example, a daily batch, without rewriting the data already materialized. -See the [training data guide](../../../user_guides/fs/feature_view/training-data.md#appending-to-a-training-dataset) for details. +See the [training data guide][appending-to-a-training-dataset] for details. ### Point-in-time Correct Training Data diff --git a/docs/user_guides/fs/feature_view/training-data.md b/docs/user_guides/fs/feature_view/training-data.md index bf72a14cde..a445d18adf 100644 --- a/docs/user_guides/fs/feature_view/training-data.md +++ b/docs/user_guides/fs/feature_view/training-data.md @@ -163,7 +163,7 @@ job = feature_view.insert_training_data( ``` Passing `overwrite=True` rewrites the entire training dataset version for the given time range instead of appending. -From a Python client the append is executed by the [ArrowFlight Server with DuckDB](../../../setup_installation/common/arrow_flight_duckdb.md) service if enabled, otherwise a `PySparkJob` is launched, as for `create_training_data`. +From a Python client, the append is executed by the [ArrowFlight Server with DuckDB][arrowflight-server-with-duckdb] service if enabled, otherwise a `PySparkJob` is launched, as for `create_training_data`. !!! note "Requirements and behavior" - Appending is only supported for the `parquet` data format. From aa0a92ce89aa265436d86b82e45df3a0165649ae Mon Sep 17 00:00:00 2001 From: bubriks Date: Wed, 8 Jul 2026 15:34:12 +0300 Subject: [PATCH 3/4] even_time td partitions --- docs/concepts/fs/feature_view/offline_api.md | 3 ++ .../fs/feature_view/training-data.md | 33 +++++++++++++++++-- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/docs/concepts/fs/feature_view/offline_api.md b/docs/concepts/fs/feature_view/offline_api.md index 11c6f13f1f..6c9dbf57a7 100644 --- a/docs/concepts/fs/feature_view/offline_api.md +++ b/docs/concepts/fs/feature_view/offline_api.md @@ -24,6 +24,9 @@ In this case, the application should know that predictions for this customer sho Materialized training data does not have to be rewritten when new data arrives. A training dataset materialized in the `.parquet` format can be appended to, keeping the same training dataset version: each append materializes only the new time range and stores it as a new increment of the dataset. This allows very large training datasets to grow with, for example, a daily batch, without rewriting the data already materialized. + +Each increment is keyed by its batch start time, making the dataset time-addressable: reads can select a time range of increments instead of the whole dataset. +For time-series models this replaces materialized time-series splits — the train/test boundary is chosen at read time, so both windows grow or shift automatically as new batches arrive (for example, a test set that is always the most recent 30 days, or a training window shifted after detecting model drift). See the [training data guide][appending-to-a-training-dataset] for details. ### Point-in-time Correct Training Data diff --git a/docs/user_guides/fs/feature_view/training-data.md b/docs/user_guides/fs/feature_view/training-data.md index a445d18adf..e44af31d93 100644 --- a/docs/user_guides/fs/feature_view/training-data.md +++ b/docs/user_guides/fs/feature_view/training-data.md @@ -151,7 +151,10 @@ X_train, X_test, y_train, y_test = feature_view.train_test_split( A materialized training dataset can grow incrementally: `insert_training_data` appends a new batch of data to an existing training dataset version instead of rewriting it. Only the new batch is computed and written, as a separate increment under the same version, so a large (multi-terabyte) training dataset can grow with, for example, a new daily batch, without rewriting the data already materialized. -The training dataset version and its metadata stay the same, and `get_training_data` returns all increments together. +The training dataset version and its metadata stay the same, and `get_training_data` returns all increments together by default. + +The batch `start_time` also becomes the increment's key, making the training dataset time-addressable: `get_training_data` can read just a time range of increments, as shown in [Reading a time range](#reading-a-time-range-of-an-incremental-training-dataset). +Because the increments are keyed by time, batches must be appended in event-time order — a batch whose `start_time` is not later than the newest increment is rejected. ```python # append yesterday's batch to training dataset version 1 @@ -168,9 +171,33 @@ From a Python client, the append is executed by the [ArrowFlight Server with Duc !!! note "Requirements and behavior" - Appending is only supported for the `parquet` data format. - Statistics are not recomputed on append; they are left as computed when the training dataset version was created. - - Training datasets with splits are appended per split: a random split re-splits each batch (for example 80/20), while a time-series split assigns the batch by the split's fixed time boundaries, so a recent batch usually lands entirely in the last split and a warning is raised. + - Randomly split training datasets are appended per split: each batch is re-split (for example 80/20), which stays sound over many appends. + - Appending to a time-series-split training dataset is not supported and raises an error: the batch would land entirely in the last split (for example, test) while the earlier splits stay frozen, skewing the dataset. For time-series data, grow an unsplit training dataset and derive the train and test sets at read time instead, as shown in [Reading a time range](#reading-a-time-range-of-an-incremental-training-dataset). - Training datasets materialized with an older Hopsworks version are not appendable; recreate the training dataset once, after which it can be appended to. +### Reading a time range of an incremental training dataset + +Each appended batch is stored as an increment keyed by its `start_time`, so `get_training_data` can read only the increments whose batch start time falls inside a given range, without scanning the rest of the data. +This replaces materialized time-series splits for growing datasets: the train/test boundary is chosen at read time, so both windows grow or shift automatically as new batches arrive — for example, a sliding training window after detecting model drift, or a test set that is always the most recent 30 days. + +```python +# train on everything up to 30 days ago +X_train, y_train = feature_view.get_training_data( + training_dataset_version=1, + end_time="2026-06-01", +) + +# test on the most recent 30 days +X_test, y_test = feature_view.get_training_data( + training_dataset_version=1, + start_time="2026-06-02", +) +``` + +!!! note "Range semantics" + - Both bounds are inclusive and select whole increments by their batch `start_time`; rows are not filtered individually, so align the range with the appended batch boundaries (for example, day boundaries for daily batches). + - Only increments appended with a `start_time` can be matched by a time range; increments appended without one are never matched. + ## Read Training Data Once you have created a training dataset, all its metadata are saved in Hopsworks. @@ -195,6 +222,8 @@ X_train, X_val, X_test, y_train, y_val, y_test = ( ) ``` +For an incrementally grown training dataset, `get_training_data` also accepts `start_time`/`end_time` to read only a time range of increments — see [Reading a time range](#reading-a-time-range-of-an-incremental-training-dataset). + ## Passing Context Variables to Transformation Functions Once you have [defined a transformation function using a context variable](../transformation_functions.md#passing-context-variables-to-transformation-function), you can pass the required context variables using the `transformation_context` parameter when generating IN-MEMORY training data or materializing a training dataset. From 5ae407e2271f2bdfa80434c56067e9ead8d1bb12 Mon Sep 17 00:00:00 2001 From: bubriks Date: Wed, 8 Jul 2026 16:17:25 +0300 Subject: [PATCH 4/4] compute_training_dataset_statistics --- docs/user_guides/fs/feature_view/training-data.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/user_guides/fs/feature_view/training-data.md b/docs/user_guides/fs/feature_view/training-data.md index e44af31d93..48cbfd3616 100644 --- a/docs/user_guides/fs/feature_view/training-data.md +++ b/docs/user_guides/fs/feature_view/training-data.md @@ -170,7 +170,8 @@ From a Python client, the append is executed by the [ArrowFlight Server with Duc !!! note "Requirements and behavior" - Appending is only supported for the `parquet` data format. - - Statistics are not recomputed on append; they are left as computed when the training dataset version was created. + - Statistics are not recomputed on append by default, since that reads the whole dataset back every time. Pass `compute_statistics=True` to `insert_training_data` to refresh the descriptive statistics over all increments after the batch is written, or call `feature_view.compute_training_dataset_statistics(training_dataset_version)` explicitly when fresh statistics are needed, for example periodically or right before retraining. + - Model-dependent transformation functions transform each appended batch with the statistics computed when the training dataset version was created, so all increments and serving stay consistent with each other. To refit those statistics, rebuild the version with `overwrite=True` or create a new training dataset version. - Randomly split training datasets are appended per split: each batch is re-split (for example 80/20), which stays sound over many appends. - Appending to a time-series-split training dataset is not supported and raises an error: the batch would land entirely in the last split (for example, test) while the earlier splits stay frozen, skewing the dataset. For time-series data, grow an unsplit training dataset and derive the train and test sets at read time instead, as shown in [Reading a time range](#reading-a-time-range-of-an-incremental-training-dataset). - Training datasets materialized with an older Hopsworks version are not appendable; recreate the training dataset once, after which it can be appended to.