Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/concepts/fs/feature_view/offline_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,13 @@ You can apply filters when creating training data from a feature view:
Note that filters are not applied when retrieving feature vectors using feature views, as we only look up features for a specific entity, like a customer.
In this case, the application should know that predictions for this customer should be made on the model trained on customers in USA, for example.

### Incremental Training Data

Materialized training data does not have to be rewritten when new data arrives.
A training dataset materialized in the `.parquet` format can be appended to, keeping the same training dataset version: each append materializes only the new time range and stores it as a new increment of the dataset.
This allows very large training datasets to grow with, for example, a daily batch, without rewriting the data already materialized.
See the [training data guide][appending-to-a-training-dataset] for details.

### Point-in-time Correct Training Data

When you create training data from features in different feature groups, it is possible that the feature groups are updated at different cadences.
Expand Down
24 changes: 24 additions & 0 deletions docs/user_guides/fs/feature_view/training-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,30 @@ X_train, X_test, y_train, y_test = feature_view.train_test_split(
)
```

## Appending to a Training Dataset

A materialized training dataset can grow incrementally: `insert_training_data` appends a new batch of data to an existing training dataset version instead of rewriting it.
Only the new batch is computed and written, as a separate increment under the same version, so a large (multi-terabyte) training dataset can grow with, for example, a new daily batch, without rewriting the data already materialized.
The training dataset version and its metadata stay the same, and `get_training_data` returns all increments together.

```python
# append yesterday's batch to training dataset version 1
job = feature_view.insert_training_data(
training_dataset_version=1,
start_time="2026-07-01 00:00:00",
end_time="2026-07-01 23:59:59",
)
```

Passing `overwrite=True` rewrites the entire training dataset version for the given time range instead of appending.
From a Python client, the append is executed by the [ArrowFlight Server with DuckDB][arrowflight-server-with-duckdb] service if enabled, otherwise a `PySparkJob` is launched, as for `create_training_data`.

!!! note "Requirements and behavior"
- Appending is only supported for the `parquet` data format.
- Statistics are not recomputed on append; they are left as computed when the training dataset version was created.
- Training datasets with splits are appended per split: a random split re-splits each batch (for example 80/20), while a time-series split assigns the batch by the split's fixed time boundaries, so a recent batch usually lands entirely in the last split and a warning is raised.
- Training datasets materialized with an older Hopsworks version are not appendable; recreate the training dataset once, after which it can be appended to.

## Read Training Data

Once you have created a training dataset, all its metadata are saved in Hopsworks.
Expand Down
Loading