From 59a85f302b75b3bcc6637aa4abd3eee0b37c5ebd Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 7 Aug 2026 08:43:31 +0200 Subject: [PATCH 1/9] [FSTORE-2075] Document the tag lifecycle, job and dataset tags, and keywords https://hopsworks.atlassian.net/browse/FSTORE-2075 Tags now cover jobs and datasets alongside the feature store artifacts, tag schemas can be deprecated and deleted rather than living for ever, every attachment records when it was made, and the keyword API is a documented surface rather than three CLI commands. The pages here were written for the five artifact types that existed before and told readers that a schema is immutable, which is no longer true. The tags guide gains the two new artifact types, the attachment time and the methods that expose it, and the freeze on tags attached to files inside a dataset. Its immutability warning becomes a statement about the JSON definition, which is still fixed, and points at the new lifecycle page for what an administrator can do instead. Two new user guides cover the schema lifecycle (deprecate, restore, usage, delete, force delete, and why a plain delete against an older backend is refused by the client) and keywords. The mandatory tags guide gains datasets, which are enforced at creation and exempt the datasets Hopsworks provisions for a project. The search concept page lists the four asset types that became searchable. A new administration page covers dataset tag storage: the write window the upgrade takes for the migration, the activation gate that holds the two cluster-wide operations until every node runs the new code, the admission policy that fences a rollback, and the cut-over itself, which pauses dataset tag writes while the API restarts and leaves reads untouched. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/concepts/projects/search.md | 5 + .../admin/dataset_tag_storage.md | 121 ++++++++++++++++ docs/user_guides/fs/tags/keywords.md | 94 ++++++++++++ docs/user_guides/fs/tags/mandatory_tags.md | 48 ++++++- .../fs/tags/tag_schema_lifecycle.md | 134 ++++++++++++++++++ docs/user_guides/fs/tags/tags.md | 111 ++++++++++++++- mkdocs.yml | 3 + 7 files changed, 505 insertions(+), 11 deletions(-) create mode 100644 docs/setup_installation/admin/dataset_tag_storage.md create mode 100644 docs/user_guides/fs/tags/keywords.md create mode 100644 docs/user_guides/fs/tags/tag_schema_lifecycle.md diff --git a/docs/concepts/projects/search.md b/docs/concepts/projects/search.md index 4750bde12f..e1b30d3ab3 100644 --- a/docs/concepts/projects/search.md +++ b/docs/concepts/projects/search.md @@ -10,9 +10,14 @@ Hopsworks supports free-text search to discover machine-learning assets: - feature groups - feature views - training data +- models +- deployments +- jobs +- datasets You can use the search bar at the top of your project to free-text search for the names or descriptions of any ML asset. You can also search using keywords or tags that are attached to an ML asset. +Keywords apply to feature groups, feature views and training data only, so a keyword filter never matches a job or a dataset. You can search for assets within a specific project or across all projects in a Hopsworks deployment, including those you are not a member of. This allows for easier discoverability and reusability of assets within an organization. diff --git a/docs/setup_installation/admin/dataset_tag_storage.md b/docs/setup_installation/admin/dataset_tag_storage.md new file mode 100644 index 0000000000..4ad1bfd7e4 --- /dev/null +++ b/docs/setup_installation/admin/dataset_tag_storage.md @@ -0,0 +1,121 @@ +# Dataset Tag Storage + +## Introduction + +Tags attached to a dataset used to be stored as HopsFS extended attributes. +They are now rows in the Hopsworks database. +The move makes dataset tags searchable, countable when an administrator asks what still references a tag schema, and deleted in the same transaction as the dataset they belong to. + +Nothing about this is visible to a project member: the same tags are read and written through the same API either way. +It is visible to an administrator, because moving the store on a running cluster is a one-way step that has to be taken deliberately, and because tags attached to individual files inside a dataset do not move. + +This page covers what an upgrade does on its own, what the cut-over is, and how to run it. + +## What the upgrade does on its own + +The upgrade that introduces this feature applies a database migration that audits the tag tables for duplicates and then adds unique keys over them. +A tag or keyword write landing between the audit and the index creation can be the row that makes the index creation fail, so the chart stops those writes for the length of the migration. + +A pre-upgrade hook scales both Payara deployments to zero, waits until their pods are gone, runs the migration, and leaves the deployments at zero for the upgrade's own apply to bring back on the new image. +The hook fires on the one upgrade that applies the migration and on no other. +It is measured at 70 seconds against a database holding a million tag values. + +Two failure paths are deliberately different. +A refusal by the pre-migration audit happens before any schema change, so the original replica counts are restored and the upgrade aborts with the cluster running as it was. +A failure after the schema change has begun leaves the cluster scaled to zero, because starting the old nodes over a half-applied schema is worse than an outage. + +Set `tagLifecycle.writeWindow.enabled=false` to skip it, only if you are taking the write window yourself. + +After the upgrade the cluster keeps reading dataset tags from the extended attributes and writes them to both stores. +Nothing is lost while you stay in that state, and you can stay in it indefinitely. + +## Per-file tags are frozen + +Tags could previously be attached to any file or directory inside a dataset. +Attaching a new tag to a path inside a dataset is now rejected with HTTP 400. +Per-file tags were stored outside the database, could not be searched or counted, and would have made the cut-over unbounded. + +Tags that were already attached to such paths remain readable and deletable, and the file browser keeps showing them. +They are not migrated, and they are the reason the tag schema usage report states that its counts cover database-backed references only. + +## The activation gate + +Two operations are only safe once every node in the cluster runs the new code: replacing a cluster-wide mandatory tag policy, and the dataset tag cut-over. +Both are refused with HTTP 503 and `ACTIVATION_PENDING` while a rolling upgrade is in progress. + +The chart's post-upgrade hook verifies against the Kubernetes API that every pod serving the Hopsworks API belongs to the new rollout, then activates the lifecycle. +Activation is what the refusal waits for; no timer and no node decides it for itself. +A fresh install is activated by the post-install hook instead, since there is no old node to wait for. + +Check the state at any time: + +```bash +curl -s -H "Authorization: ApiKey $API_KEY" \ + https:///hopsworks-api/api/admin/tag-lifecycle +``` + +## Fencing a rollback + +Once the database is the canonical store for dataset tags, a node running the previous release writes extended attributes that nothing reads any more, and those writes are lost silently. +Rolling back is therefore not made safe; it is refused. + +`tagLifecycle.admissionPolicy.enabled=true` installs a `ValidatingAdmissionPolicy` that refuses to admit an API pod below the current capability epoch. +It is **off by default**, because turning it on means an emergency downgrade requires restoring the pre-cut-over database and deleting the policy, in that order. +A cluster that never cuts over never needs it. + +The cut-over refuses to run while the policy is absent, and the operator who accepts the risk says so explicitly with `--accept-unfenced-rollback`, which is logged with the decision. + +## The cut-over + +The cut-over moves the canonical store from the extended attributes to the database. +It is a scheduled maintenance action, not an administrative call to make at an arbitrary moment: dataset tag **writes** are refused with a retryable HTTP 503 from the start of the window until it commits. +Dataset tag **reads** are never interrupted, and nothing else on the cluster is affected. + +The window is about five minutes on a three-node cluster, dominated by the API restart. +The verification over several hundred datasets takes seconds. + +Run it by setting `tagLifecycle.cutover.run=true` on a `helm upgrade`. +The value is off by default and the Job runs once per upgrade that sets it. +The Job: + +1. Reads the admission policy and its binding from the Kubernetes API and checks the whole contract: that it denies rather than warns, that it fails closed, that it selects this namespace and both API deployments, and that it names the current epoch. + A policy with the right name but any of those wrong is not a fence. +2. Refuses to start unless the rolling upgrade has been activated, the background migration of existing tags reports done, and no tag was quarantined for failing validation. +3. Records the current replica count, scales the API deployment to zero, and waits until its pods are gone. + At that instant every write that was in flight has either reached the file system or never will, so nothing is left to drain. +4. Sets the state to `cutting_over` while nothing is running, then restores the replica count. + The API comes back refusing dataset tag writes with `CUTOVER_IN_PROGRESS`. +5. Runs a final migration pass and then a verification, and stops if the verification reports any difference between the two stores. +6. Commits. + Every precondition is checked again inside the transaction that publishes the new state. + +If the Job stops at step 5, the cluster stays in `cutting_over` with reads unaffected. +Investigate the reported differences, then either fix them and re-run, or abort: + +```bash +curl -s -X POST -H "Authorization: ApiKey $API_KEY" \ + https:///hopsworks-api/api/admin/dataset-tags/cutover/abort +``` + +Aborting returns the cluster to the extended attributes and resumes writes. +It is never blocked on anything: getting out of a paused state must not itself require the state to be healthy. + +Ask what the cluster thinks at any point: + +```bash +curl -s -H "Authorization: ApiKey $API_KEY" \ + https:///hopsworks-api/api/admin/dataset-tags/cutover +``` + +The response reports the current state, when the window opened, and what would block a commit right now. + +!!! warning "Do not edit the state directly" + Setting the canonical store through the cluster variables endpoint is refused in both directions, and the error names the cut-over endpoints. + A direct database edit bypasses every check here, as it always has. + +## After the cut-over + +The extended attributes are left in place after the cut-over. +Readers holding a cached copy of the cluster state for up to one cache interval would otherwise read a store that had already been emptied, so the extended attributes are removed no earlier than one full cache interval afterwards, and only after the verification has passed. + +Removing them is the point of no return for a downgrade, so keep them until the release is known good. diff --git a/docs/user_guides/fs/tags/keywords.md b/docs/user_guides/fs/tags/keywords.md new file mode 100644 index 0000000000..f23a01cad8 --- /dev/null +++ b/docs/user_guides/fs/tags/keywords.md @@ -0,0 +1,94 @@ +# Keywords + +## Introduction + +A keyword is a single user-defined word attached to a feature group, feature view or training dataset. +Keywords are free text with no schema behind them, which makes them the lighter alternative to [tags](tags.md): use a keyword to make an artifact easier to find, and a tag when the metadata has to be structured and validated. + +Keywords are indexed for free-text search alongside names, descriptions and tags. +They can also be used as a filter in the search API. + +Keywords are not available on models, deployments, jobs or datasets. +A keyword filter passed to a job or dataset search therefore matches nothing. + +## Read the keywords of an artifact + +=== "Python" + + ```python + fg = fs.get_feature_group("transactions_4h_aggs_fraud_batch_fg", version=1) + + fg.get_keywords() # ["fraud", "eu_region"] + ``` + +Keywords record the time they were added, from the release that introduced this guide onwards. + +=== "Python" + + ```python + for keyword, added in fg.get_keywords_metadata().items(): + print(keyword, added) + ``` + +`added` is `None` for keywords attached before that release, and against an older backend that does not report the time. + +## Add, replace and delete keywords + +=== "Python" + + ```python + # Add to the existing set + fg.add_keywords("pii") + fg.add_keywords(["fraud", "eu_region"]) + + # Replace the whole set + fg.set_keywords(["fraud", "pii"]) + + # Remove one + fg.delete_keyword("pii") + ``` + +Each of these returns the resulting set of keywords. + +!!! warning "add_keywords is not atomic" + `add_keywords` reads the current set, adds to it, and writes the whole set back. + Two calls running at the same time can therefore lose one of the additions. + Use `set_keywords` when the caller already knows the full set it wants. + +The same five methods are available on `FeatureView` and `TrainingDataset`. +On a feature view, the training datasets it owns are reached through the `*_training_dataset_keywords` variants, which take the training dataset version as their first argument. + +=== "Python" + + ```python + fv = fs.get_feature_view("transactions_fv", version=1) + + fv.get_training_dataset_keywords(1) + fv.add_training_dataset_keywords(1, "backfill") + ``` + +## The cluster vocabulary + +`get_all_keywords` returns every keyword used on the cluster, not only those in the current project. +Use it to offer autocompletion, or to check whether a word is already in use before introducing a variant of it. + +=== "Python" + + ```python + fs.get_all_keywords() + ``` + +## Command line + +```bash +hops fg keywords transactions_4h_aggs_fraud_batch_fg +hops fg add-keyword transactions_4h_aggs_fraud_batch_fg fraud +hops fg remove-keyword transactions_4h_aggs_fraud_batch_fg fraud +``` + +The same three commands exist for feature views (`hops fv`) and training datasets (`hops td`). + +!!! warning "`--value` on the keyword commands is being removed" + `--value` used to make `hops fg add-keyword` write a tag rather than a keyword. + It is now rejected with a message naming `hops fg add-tag` as the replacement, and it is removed in the following release. + Attach a tag with the tag commands, described in the [Tags](tags.md) guide. diff --git a/docs/user_guides/fs/tags/mandatory_tags.md b/docs/user_guides/fs/tags/mandatory_tags.md index e1f43000d3..b15516e4b6 100644 --- a/docs/user_guides/fs/tags/mandatory_tags.md +++ b/docs/user_guides/fs/tags/mandatory_tags.md @@ -6,7 +6,7 @@ Mandatory tags let a Hopsworks administrator require that specific tag schemas a They build on top of [tags](tags.md) and are used to enforce governance rules, such as requiring every model to declare a data owner. A mandatory tag is a tag schema that has been marked as required for one or more artifact types. -The supported artifact types are feature groups, feature views, training datasets, models and deployments. +The supported artifact types are feature groups, feature views, training datasets, models, deployments and datasets. ## Prerequisites @@ -31,12 +31,15 @@ Mandatory tags are configured in two scopes. Configure project-specific mandatory tags under Cluster settings, Projects, by editing a project For each mandatory tag you select the artifact types it applies to. -A tag schema can be mandatory for any combination of feature groups, feature views, training datasets, models and deployments. -For example, a `data_owner` schema can be marked mandatory for models and deployments only, leaving feature groups, feature views and training datasets unaffected. +A tag schema can be mandatory for any combination of feature groups, feature views, training datasets, models, deployments and datasets. +For example, a `data_owner` schema can be marked mandatory for models and deployments only, leaving the others unaffected. + +A [deprecated](tag_schema_lifecycle.md) schema cannot be registered as mandatory, because nobody is allowed to attach it. +Conversely, a schema that is registered as mandatory cannot be deprecated until the registrations are removed. ## Enforcement per artifact type -All five artifact types, feature groups, feature views, training datasets, models and deployments, enforce mandatory tags the same way. +All six artifact types, feature groups, feature views, training datasets, models, deployments and datasets, enforce mandatory tags the same way. The create request is validated against the configured mandatory tags. If any mandatory tag is missing from the tags provided at creation, the artifact is not created and the request is rejected with an HTTP 400 error that lists the missing tag names. @@ -129,8 +132,45 @@ The `tags` argument takes the same shape as feature group tags: a `{"name": ..., deployment.save() ``` +=== "Dataset (Python)" + + ```python + dataset_api = project.get_dataset_api() + + # data_owner is mandatory for datasets; pass it when creating the dataset + dataset_api.mkdir( + "transactions_raw", + tags=[{"name": "data_owner", "value": "email@hopsworks.ai"}], + ) + ``` + Omitting a mandatory tag from the `tags` argument rejects the create request with an HTTP 400 error that lists the missing tag names. +### Datasets + +Mandatory tags on datasets apply to the dataset itself, which is a top-level directory in the project's file system. +They are not applied per file: a policy that had to be satisfied by every uploaded file would make every upload a violation. + +The policy applies to datasets a user creates. +The datasets Hopsworks creates when it provisions a project, such as `Resources`, `Logs` and the training dataset and deployment directories, are exempt, because a cluster-wide policy would otherwise make project creation fail. + +A Hopsworks client older than the release that introduced this feature cannot send tags on the create call, so it receives the HTTP 400 listing the missing tags once a dataset policy is enabled. +This is the same contract feature groups have, and it is why enabling a dataset policy is an explicit administrator action. + +A dataset that predates the policy stays valid. +Its missing tags are reported on read through the `missingMandatoryTags` field, and shown on the dataset page in the UI. + +=== "Python" + + ```python + dataset_api = project.get_dataset_api() + + dataset = dataset_api.get("transactions_raw") + print(dataset.get("missingMandatoryTags", [])) + + dataset_api.add("transactions_raw", "data_owner", "email@hopsworks.ai") + ``` + ## Missing mandatory tags on pre-existing artifacts Marking a tag mandatory does not retroactively reject artifacts that already exist without it. diff --git a/docs/user_guides/fs/tags/tag_schema_lifecycle.md b/docs/user_guides/fs/tags/tag_schema_lifecycle.md new file mode 100644 index 0000000000..0918d05e94 --- /dev/null +++ b/docs/user_guides/fs/tags/tag_schema_lifecycle.md @@ -0,0 +1,134 @@ +# Tag Schema Lifecycle + +## Introduction + +A tag schema is defined once and then attached to artifacts across every project on the cluster, as described in the [Tags](tags.md) guide. +This guide covers what an administrator can do with a schema afterwards: retire it so that nothing new uses it, and remove it once nothing does. + +The JSON definition of a schema cannot be changed after it is created. +The values already attached were validated against the original definition, and editing it would leave those values unvalidated. +Create a new schema instead, deprecate the old one, and migrate at your own pace. + +Deprecating, restoring and deleting a schema are administrator actions, performed in the `Cluster settings` > `Tag schemas` section or through the REST API. +Attaching and detaching tag values stays with the project members. + +## Deprecate a schema + +Deprecation stops a schema from being attached to anything new while leaving everything already attached untouched. +Use it when a schema is being replaced and you want existing owners to migrate off it without breaking their pipelines. + +| Operation on a deprecated schema | Allowed | +| --- | --- | +| Read the schema, list schemas, read attached values | Yes | +| Attach it to an artifact that does not have it | No, rejected with HTTP 400 and error code 370011 | +| Update the value already attached to an artifact | Yes, so a team can correct data while migrating off | +| Detach it from an artifact | Yes | +| Register it as a mandatory tag | No, because a mandatory tag that nobody may attach cannot be satisfied | +| Delete the schema | Yes, under the rules below | +| Restore it | Yes | + +A schema that is currently registered as a [mandatory tag](mandatory_tags.md) cannot be deprecated. +The request is rejected with HTTP 409 and error code 370012. +Remove the mandatory registrations first, then deprecate. + +In the UI, deprecated schemas carry a muted `Deprecated` badge with the date, and the schema detail drawer shows which administrator deprecated it. +The tag picker on an artifact page no longer offers them, while a deprecated tag that is already attached still renders so it can be edited or removed. + +=== "Python" + + ```python + import hopsworks + from hopsworks_common.core.tag_schemas_api import TagSchemasApi + + + hopsworks.login() + tag_schemas = TagSchemasApi() + + tag_schemas.deprecate("data_privacy_v1") + + # Existing attachments keep working; new ones are refused. + schema = tag_schemas.get("data_privacy_v1") + print(schema["deprecated"], schema["deprecatedOn"], schema["deprecatedBy"]) + ``` + +## Restore a schema + +Restoring clears the deprecation and the record of who set it and when. +Those three fields describe the current state rather than a history, so a restored schema looks like one that was never deprecated. +The audit log remains the record of the full sequence of actions. + +=== "Python" + + ```python + tag_schemas.restore("data_privacy_v1") + ``` + +## Check what uses a schema + +Before deleting a schema, ask what still references it. + +=== "Python" + + ```python + usage = tag_schemas.usage("data_privacy_v1") + + print(usage["attachmentCount"]) # values attached across all projects + print(usage["mandatoryRegistrationCount"]) # mandatory registrations + print(usage["deletable"]) # both counts are zero + for ref in usage.get("references", []): + print(ref["kind"], ref["projectName"], ref["artifactName"], ref["version"]) + ``` + +`references` lists the artifacts holding a value, up to a limit. +Above that limit the list is omitted and `truncated` is `true`, leaving `attachmentCount` as the answer. + +`complete` is always `false`. +Tags attached to individual files inside a dataset before per-file tags were frozen live in the file system rather than the database, and no query can enumerate them without walking the file system. +The counts therefore cover database-backed references only. +This is why the UI wording is "No database-backed references found" rather than "not used". + +## Delete a schema + +A schema can be deleted once nothing references it. +The request is refused with HTTP 409 and error code 370010 while any value is attached or any mandatory registration exists, and the error names what to detach. + +The check and the deletion run in one transaction against the database, and foreign keys refuse the deletion even if a value is attached between the check and the commit. +The dialog in the UI is advisory; the database is authoritative, so a delete that looked safe a moment earlier can still be refused. + +=== "Python" + + ```python + usage = tag_schemas.usage("data_privacy_v1") + if usage["deletable"]: + tag_schemas.delete("data_privacy_v1") + ``` + +### Force delete + +`force=True` deletes the schema together with every value attached to it, in a single transaction, and queues the affected artifacts for reindexing so that search stops returning the deleted tag. +It is exposed on the REST API and in the Python client, and deliberately not in the UI. + +=== "Python" + + ```python + tag_schemas.delete("data_privacy_v1", force=True) + ``` + +Above a threshold of attached values, configured cluster-wide and defaulting to 5000, a force delete is refused with HTTP 409 rather than started. +The error states the count and names the reindex endpoint. +A partial delete that leaves stale values in the search index is worse than a refusal, and reindexing that many documents is an operation an administrator should schedule. + +### Deleting against an older backend + +A plain `delete` against a Hopsworks version older than the one described here is refused by the client rather than sent. +That backend ignores the unknown `force` parameter and deletes every attached value for any delete call, so the safest-looking call would get the most destructive behaviour the old server has. +The client detects it through the usage endpoint, which shipped in the same release as the refusal, and raises a `RuntimeError` naming `force=True` as the way to proceed deliberately. + +## Command line + +The same two lifecycle actions are available in the CLI. + +```bash +hops tags deprecate data_privacy_v1 +hops tags restore data_privacy_v1 +``` diff --git a/docs/user_guides/fs/tags/tags.md b/docs/user_guides/fs/tags/tags.md index 6fc582829c..db5dc21cf8 100644 --- a/docs/user_guides/fs/tags/tags.md +++ b/docs/user_guides/fs/tags/tags.md @@ -2,7 +2,7 @@ ## Introduction -Hopsworks feature store enables users to attach tags to artifacts, such as feature groups, feature views, training datasets, models or deployments. +Hopsworks enables users to attach tags to artifacts, such as feature groups, feature views, training datasets, models, deployments, jobs or datasets. A tag is a `{key: value}` pair which provides additional information about the data managed by Hopsworks. Tags allow you to design custom metadata for your artifacts. @@ -18,9 +18,10 @@ Schemas are defined globally across all projects. When users attach tags to an artifact, the tag will be validated against a specific schema. This allows tags to be consistent no matter the project or the team generating them. -!!! warning "Immutable" - Tag schemas are immutable. - Once defined, a tag schema cannot be edited nor deleted. +!!! warning "Schema definitions cannot be edited" + The JSON schema of a tag schema cannot be changed after it is created, because the values already attached were validated against the original definition. + A schema can be deprecated so that it accepts no new attachments, and it can be deleted once nothing references it. + Both are administrator actions and are described in the [Tag schema lifecycle](tag_schema_lifecycle.md) guide. ## Step 1: Define a tag schema @@ -73,7 +74,7 @@ Where the type is a valid primitive type: `string`, `boolean`, `integer`, `numbe ## Step 2: Attach a tag to an artifact -Once the tag schema has been created, you can attach a tag with that schema to a feature group, feature view, training dataset, model or deployment either using the APIs, or by using the UI. +Once the tag schema has been created, you can attach a tag with that schema to a feature group, feature view, training dataset, model, deployment, job or dataset, either using the APIs or the UI. ### Using the API @@ -122,6 +123,62 @@ Finally you can remove a tag from a given artifact by calling the `delete_tag()` The same APIs work for feature views, training datasets, models and deployments alike. +#### Jobs + +Jobs carry tags through the same three methods, on the `Job` object returned by the job API. + +=== "Python" + + ```python + job_api = project.get_job_api() + job = job_api.get_job("transactions_ingestion") + + job.add_tag("data_privacy", {"business_unit": "Fraud", "pii": True}) + job.get_tags() + job.delete_tag("data_privacy") + ``` + +A job also has a free-text description, which is indexed for search alongside its tags. + +=== "Python" + + ```python + job.description = "Hourly ingestion of card transactions" + job.save() + ``` + +#### Datasets + +Tags are attached to a dataset, which is a top-level directory in the project's file system. +They are reached through the dataset API by path. + +=== "Python" + + ```python + dataset_api = project.get_dataset_api() + + dataset_api.add("Resources", "data_privacy", {"business_unit": "Fraud", "pii": True}) + dataset_api.get_tags("Resources") + dataset_api.delete("Resources", "data_privacy") + ``` + +Tags can also be attached when the dataset is created. + +=== "Python" + + ```python + dataset_api.mkdir( + "transactions_raw", + tags=[{"name": "data_privacy", "value": {"business_unit": "Fraud", "pii": True}}], + ) + ``` + +!!! warning "Tags on files inside a dataset are frozen" + Tags could previously be attached to any file or directory inside a dataset. + Attaching a new tag to a path inside a dataset is now rejected with HTTP 400, because per-file tags were stored outside the database and could not be searched, counted, or governed. + Tags that were already attached to such paths remain readable and deletable, and the file browser keeps showing them. + Attach the tag to the dataset instead. + ### Using the UI You can attach tags to feature groups and feature views directly from the UI. @@ -135,11 +192,33 @@ From there you can select the tag schema of the tag you want to attach and popul

+## When a tag was attached + +Every tag attached after the upgrade that introduced this feature records the time it was attached. +`get_tags` returns values only, so the attachment time is exposed through a second pair of methods that keep the tag objects. + +=== "Python" + + ```python + fg = fs.get_feature_group("transactions_4h_aggs_fraud_batch_fg", version=1) + + tags = fg.get_tags_metadata() # dict[str, Tag] + for name, t in tags.items(): + print(name, t.value, t.created_on) + + one = fg.get_tag_metadata("data_privacy") + ``` + +`get_tag_metadata` and `get_tags_metadata` are available on feature groups, feature views, training datasets, models, deployments and jobs, and on the dataset API as `get_tag_metadata(path, name)` and `get_tags_metadata(path)`. +`created_on` is `None` for tags that were attached before the upgrade, because the attachment time was not recorded then. +The UI shows the same value under the tag name on the artifact page. + ## Step 3: Search -Hopsworks indexes the tags attached to feature groups, feature views and training datasets. +Hopsworks indexes the tags attached to feature groups, feature views, training datasets, models, deployments, jobs and datasets. The tags will then be searchable using the free text search box located at the top of the UI. -Tags attached to models and deployments are stored and retrievable through the APIs and the UI, but they are not indexed for free text search. +For jobs the search also covers the job name and description, and for datasets the dataset name and description. +Model and deployment documents are written by Hopsworks itself, so a tag change on a model or a deployment is reflected in search rather than leaving a stale answer behind.

@@ -147,3 +226,21 @@ Tags attached to models and deployments are stored and retrievable through the A
Search for tags in the feature store

+ +Jobs and datasets can also be searched from the Python client. + +=== "Python" + + ```python + search_api = project.get_search_api() + + for job_meta in search_api.jobs("ingestion"): + print(job_meta.name, job_meta.job_type) + job = job_meta.get() # the Job object, resolved in the hit's own project + + for ds_meta in search_api.datasets("transactions"): + print(ds_meta.name, ds_meta.path) + ``` + +Both accept a `tag_filter` and a `global_search` flag, like the feature group search. +Jobs and datasets carry no keywords, so a `keyword_filter` never matches them. diff --git a/mkdocs.yml b/mkdocs.yml index b53bc4becc..946ca20691 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -134,6 +134,8 @@ nav: - Sharing: user_guides/fs/sharing/sharing.md - Tags: user_guides/fs/tags/tags.md - Mandatory Tags: user_guides/fs/tags/mandatory_tags.md + - Tag Schema Lifecycle: user_guides/fs/tags/tag_schema_lifecycle.md + - Keywords: user_guides/fs/tags/keywords.md - Provenance: user_guides/fs/provenance/provenance.md - Feature Monitoring: - user_guides/fs/feature_monitoring/index.md @@ -272,6 +274,7 @@ nav: - IAM Role Chaining: setup_installation/admin/roleChaining.md - Configure Project Mapping: setup_installation/admin/configure-project-mapping.md - Airflow 3 operator notes: setup_installation/admin/airflow3.md + - Dataset Tag Storage: setup_installation/admin/dataset_tag_storage.md - Monitoring: - Services Dashboards: setup_installation/admin/monitoring/grafana.md - Export metrics: setup_installation/admin/monitoring/export-metrics.md From c6ebb5c8836069889057165215f57a391232233d Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 7 Aug 2026 16:15:03 +0200 Subject: [PATCH 2/9] [FSTORE-2075] Align the cut-over page with the shipped Job https://hopsworks.atlassian.net/browse/FSTORE-2075 The page was written from the specification before the Job existed, and the implementation settled three details differently. The Job reads the cut-over status before anything else, so a database that is already canonical exits immediately and a window an earlier attempt left open is resumed rather than opened twice; both matter to an operator deciding whether re-running is safe, so they are now step 1. It scales both API deployments, not one. And accepting an unfenced rollback is the tagLifecycle.cutover.acceptUnfencedRollback value, not a command line flag. Also a note that Helm 4 needs --force-conflicts on the upgrade: server side apply refuses fields owned by other field managers, which a running cluster always has, and the failure lands before any hook runs. Pre-existing Helm 4 behaviour, but the operator following this page is the one who hits it. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- .../admin/dataset_tag_storage.md | 26 ++++++++++++------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/docs/setup_installation/admin/dataset_tag_storage.md b/docs/setup_installation/admin/dataset_tag_storage.md index 4ad1bfd7e4..f391297855 100644 --- a/docs/setup_installation/admin/dataset_tag_storage.md +++ b/docs/setup_installation/admin/dataset_tag_storage.md @@ -63,7 +63,7 @@ Rolling back is therefore not made safe; it is refused. It is **off by default**, because turning it on means an emergency downgrade requires restoring the pre-cut-over database and deleting the policy, in that order. A cluster that never cuts over never needs it. -The cut-over refuses to run while the policy is absent, and the operator who accepts the risk says so explicitly with `--accept-unfenced-rollback`, which is logged with the decision. +The cut-over refuses to run while the policy is absent, and the operator who accepts the risk says so explicitly with `tagLifecycle.cutover.acceptUnfencedRollback=true`, which is rendered into the Job and logged with the decision. ## The cut-over @@ -76,21 +76,29 @@ The verification over several hundred datasets takes seconds. Run it by setting `tagLifecycle.cutover.run=true` on a `helm upgrade`. The value is off by default and the Job runs once per upgrade that sets it. + +!!! note "Helm 4 needs `--force-conflicts`" + Under Helm 4, pass `--force-conflicts` to the upgrade. + Server-side apply refuses fields owned by other field managers, which a running cluster always has, and without the flag the upgrade fails before any hook runs. + This is a pre-existing Helm 4 behaviour, not specific to the cut-over. + The Job: -1. Reads the admission policy and its binding from the Kubernetes API and checks the whole contract: that it denies rather than warns, that it fails closed, that it selects this namespace and both API deployments, and that it names the current epoch. +1. Reads the cut-over status while the cluster is up. + A database that is already canonical exits immediately, so the Job is safe to leave enabled across upgrades, and a window an earlier attempt left open is resumed at the final sweep rather than opened twice. +2. Reads the admission policy and its binding from the Kubernetes API and checks the whole contract: that it denies rather than warns, that it fails closed, that it selects this namespace and both API deployments, and that it names the current epoch. A policy with the right name but any of those wrong is not a fence. -2. Refuses to start unless the rolling upgrade has been activated, the background migration of existing tags reports done, and no tag was quarantined for failing validation. -3. Records the current replica count, scales the API deployment to zero, and waits until its pods are gone. +3. Refuses to start unless the rolling upgrade has been activated, the background migration of existing tags reports done, and no tag was quarantined for failing validation. +4. Records the current replica counts, scales both API deployments to zero, and waits until their pods are gone. At that instant every write that was in flight has either reached the file system or never will, so nothing is left to drain. -4. Sets the state to `cutting_over` while nothing is running, then restores the replica count. +5. Sets the state to `cutting_over` while nothing is running, then restores the replica counts. The API comes back refusing dataset tag writes with `CUTOVER_IN_PROGRESS`. -5. Runs a final migration pass and then a verification, and stops if the verification reports any difference between the two stores. -6. Commits. +6. Runs a final migration pass and then a verification, and stops if the verification reports any difference between the two stores. +7. Commits. Every precondition is checked again inside the transaction that publishes the new state. -If the Job stops at step 5, the cluster stays in `cutting_over` with reads unaffected. -Investigate the reported differences, then either fix them and re-run, or abort: +If the Job stops at step 6, the cluster stays in `cutting_over` with reads unaffected. +Investigate the reported differences, then either fix them and re-run the Job (it resumes the open window), or abort: ```bash curl -s -X POST -H "Authorization: ApiKey $API_KEY" \ From cd71febeef86f8eb0dab56f79fb154bd41925d6f Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 7 Aug 2026 22:56:29 +0200 Subject: [PATCH 3/9] [FSTORE-2075] Give the cut-over settings the subchart prefix helm needs https://hopsworks.atlassian.net/browse/FSTORE-2075 Every helm setting on the dataset tag storage page was written without its subchart prefix, so an operator following the page could not run any of it. tagLifecycle belongs to the hopsworks subchart, and the umbrella chart validates user-supplied values against its own schema first: helm upgrade --set tagLifecycle.cutover.run=true fails with "additional properties 'tagLifecycle' not allowed" before a single template renders. The cluster run that verified the cut-over used hopsworks.tagLifecycle.cutover.run, which is why the gap survived the end-to-end test. All four settings now carry the prefix, and the cut-over section says why it is there so the next person does not read it as a typo. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/setup_installation/admin/dataset_tag_storage.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/setup_installation/admin/dataset_tag_storage.md b/docs/setup_installation/admin/dataset_tag_storage.md index f391297855..d6e0288273 100644 --- a/docs/setup_installation/admin/dataset_tag_storage.md +++ b/docs/setup_installation/admin/dataset_tag_storage.md @@ -24,7 +24,7 @@ Two failure paths are deliberately different. A refusal by the pre-migration audit happens before any schema change, so the original replica counts are restored and the upgrade aborts with the cluster running as it was. A failure after the schema change has begun leaves the cluster scaled to zero, because starting the old nodes over a half-applied schema is worse than an outage. -Set `tagLifecycle.writeWindow.enabled=false` to skip it, only if you are taking the write window yourself. +Set `hopsworks.tagLifecycle.writeWindow.enabled=false` to skip it, only if you are taking the write window yourself. After the upgrade the cluster keeps reading dataset tags from the extended attributes and writes them to both stores. Nothing is lost while you stay in that state, and you can stay in it indefinitely. @@ -59,11 +59,11 @@ curl -s -H "Authorization: ApiKey $API_KEY" \ Once the database is the canonical store for dataset tags, a node running the previous release writes extended attributes that nothing reads any more, and those writes are lost silently. Rolling back is therefore not made safe; it is refused. -`tagLifecycle.admissionPolicy.enabled=true` installs a `ValidatingAdmissionPolicy` that refuses to admit an API pod below the current capability epoch. +`hopsworks.tagLifecycle.admissionPolicy.enabled=true` installs a `ValidatingAdmissionPolicy` that refuses to admit an API pod below the current capability epoch. It is **off by default**, because turning it on means an emergency downgrade requires restoring the pre-cut-over database and deleting the policy, in that order. A cluster that never cuts over never needs it. -The cut-over refuses to run while the policy is absent, and the operator who accepts the risk says so explicitly with `tagLifecycle.cutover.acceptUnfencedRollback=true`, which is rendered into the Job and logged with the decision. +The cut-over refuses to run while the policy is absent, and the operator who accepts the risk says so explicitly with `hopsworks.tagLifecycle.cutover.acceptUnfencedRollback=true`, which is rendered into the Job and logged with the decision. ## The cut-over @@ -74,8 +74,9 @@ Dataset tag **reads** are never interrupted, and nothing else on the cluster is The window is about five minutes on a three-node cluster, dominated by the API restart. The verification over several hundred datasets takes seconds. -Run it by setting `tagLifecycle.cutover.run=true` on a `helm upgrade`. +Run it by setting `hopsworks.tagLifecycle.cutover.run=true` on a `helm upgrade`. The value is off by default and the Job runs once per upgrade that sets it. +These settings belong to the `hopsworks` subchart, so the `hopsworks.` prefix is part of the key: the umbrella chart's schema rejects a bare `tagLifecycle` with `additional properties 'tagLifecycle' not allowed`. !!! note "Helm 4 needs `--force-conflicts`" Under Helm 4, pass `--force-conflicts` to the upgrade. From b18b15b147eef16961212bf55c35333917ba5e96 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Fri, 7 Aug 2026 23:06:40 +0200 Subject: [PATCH 4/9] [FSTORE-2075] Name the switch that turns cross-project search off https://hopsworks.atlassian.net/browse/FSTORE-2075 The search concept page states that a cluster-wide search reaches projects you are not a member of, without saying that this is a setting. The migration in this ticket seeds cross_project_global_search_enabled true, and a multi-tenant operator is expected to set it false, which restricts the global endpoint to the caller's accessible projects in-query. An administrator reading the page had no way to learn the variable existed, and a reader of the page was told as unconditional a behaviour their cluster may not have. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/concepts/projects/search.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/concepts/projects/search.md b/docs/concepts/projects/search.md index e1b30d3ab3..accf63fedb 100644 --- a/docs/concepts/projects/search.md +++ b/docs/concepts/projects/search.md @@ -24,6 +24,9 @@ This allows for easier discoverability and reusability of assets within an organ To avoid users gaining unauthorized access to data, if a search result is in a project you are **not** a member of, the information displayed is limited to: names, descriptions, tags, asset creator and create date. If the search result is within a project you are a member of, you are also able to inspect recent activities on the asset as well as statistics. +Searching across projects you are not a member of is on by default. +An administrator running a multi-tenant deployment can turn it off by setting the `cross_project_global_search_enabled` cluster variable to `false` in the [Configuration](../../setup_installation/admin/variables.md) UI, which restricts a cluster-wide search to the projects you can already access. + ## Tags A keyword is a single user-defined word attached to an ML asset. From 52c413f067a8a0d2672260327514dc190cb8743b Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sat, 8 Aug 2026 05:50:51 +0200 Subject: [PATCH 5/9] [FSTORE-2075] Say what the cut-over window actually costs https://hopsworks.atlassian.net/browse/FSTORE-2075 Review round ten called "reads are never interrupted" what it was: an availability claim the quiesce falsifies, since both API deployments are at zero for the restart and nothing answers while they are down. The page now states the short full outage plainly, and that reads serve normally only once the pods are back. Also added, matching the hardened Job in the chart PR: the recovery step for an attempt killed mid-window, the HorizontalPodAutoscaler refusal with the instruction to pause GitOps reconciliation of the replica counts, the fence re-check at commit with the exact-expression comparison, and a paragraph on what downgrading a FRESH installation without restoring its database costs, since a fresh install is database-canonical from the start and sits past the same one-way boundary without ever running a cut-over. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- .../admin/dataset_tag_storage.md | 28 +++++++++++++------ 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/docs/setup_installation/admin/dataset_tag_storage.md b/docs/setup_installation/admin/dataset_tag_storage.md index d6e0288273..d4ca18624f 100644 --- a/docs/setup_installation/admin/dataset_tag_storage.md +++ b/docs/setup_installation/admin/dataset_tag_storage.md @@ -65,15 +65,23 @@ A cluster that never cuts over never needs it. The cut-over refuses to run while the policy is absent, and the operator who accepts the risk says so explicitly with `hopsworks.tagLifecycle.cutover.acceptUnfencedRollback=true`, which is rendered into the Job and logged with the decision. +A fresh installation is database-canonical from the start, so it sits past the same boundary without ever running a cut-over. +Downgrading a fresh installation below this release without restoring its database has the same consequence a post-cut-over rollback has: dataset tags written to the database become invisible to the old code, and tags written to extended attributes during the downgrade are silently ignored when you upgrade again. +Enable the admission policy on a fresh installation if pods from an older release must be refused; the installation records `rollbackFenced=false` in its activation audit when you do not. + ## The cut-over The cut-over moves the canonical store from the extended attributes to the database. It is a scheduled maintenance action, not an administrative call to make at an arbitrary moment: dataset tag **writes** are refused with a retryable HTTP 503 from the start of the window until it commits. -Dataset tag **reads** are never interrupted, and nothing else on the cluster is affected. +The quiesce itself is a short full outage: both API deployments are scaled to zero for the restart, and nothing answers while they are down. +Once the pods are back, reads of dataset tags and everything else serve normally for the rest of the window, and no stored data is touched at any point. The window is about five minutes on a three-node cluster, dominated by the API restart. The verification over several hundred datasets takes seconds. +If a HorizontalPodAutoscaler targets either API deployment, the Job refuses to start: an autoscaler scales the deployment back up while the cut-over needs it at zero, and a pod it starts can write extended attributes after the flip. +Delete or suspend it for the window, and pause any GitOps reconciliation that owns the replica counts, for the same reason. + Run it by setting `hopsworks.tagLifecycle.cutover.run=true` on a `helm upgrade`. The value is off by default and the Job runs once per upgrade that sets it. These settings belong to the `hopsworks` subchart, so the `hopsworks.` prefix is part of the key: the umbrella chart's schema rejects a bare `tagLifecycle` with `additional properties 'tagLifecycle' not allowed`. @@ -85,18 +93,20 @@ These settings belong to the `hopsworks` subchart, so the `hopsworks.` prefix is The Job: -1. Reads the cut-over status while the cluster is up. +1. Recovers first: if a previous attempt was killed after scaling the deployments down, the replica counts it recorded in a ConfigMap are restored before anything waits on the API. + Both deployments at zero with no record is a zero somebody else owns, and the Job refuses with the manual commands printed. +2. Reads the cut-over status while the cluster is up. A database that is already canonical exits immediately, so the Job is safe to leave enabled across upgrades, and a window an earlier attempt left open is resumed at the final sweep rather than opened twice. -2. Reads the admission policy and its binding from the Kubernetes API and checks the whole contract: that it denies rather than warns, that it fails closed, that it selects this namespace and both API deployments, and that it names the current epoch. +3. Reads the admission policy and its binding from the Kubernetes API and checks the whole contract: that it denies rather than warns, that it fails closed, that it matches pod creation in this namespace for both API deployments, and that its expression is exactly the one this chart installs for the current epoch. A policy with the right name but any of those wrong is not a fence. -3. Refuses to start unless the rolling upgrade has been activated, the background migration of existing tags reports done, and no tag was quarantined for failing validation. -4. Records the current replica counts, scales both API deployments to zero, and waits until their pods are gone. +4. Refuses to start unless the rolling upgrade has been activated, the background migration of existing tags reports done, no tag was quarantined for failing validation, and no HorizontalPodAutoscaler targets the API deployments. +5. Records the current replica counts in a ConfigMap, scales both API deployments to zero, and waits until their pods are gone. At that instant every write that was in flight has either reached the file system or never will, so nothing is left to drain. -5. Sets the state to `cutting_over` while nothing is running, then restores the replica counts. +6. Sets the state to `cutting_over` in a single transaction while nothing is running, then restores the replica counts. The API comes back refusing dataset tag writes with `CUTOVER_IN_PROGRESS`. -6. Runs a final migration pass and then a verification, and stops if the verification reports any difference between the two stores. -7. Commits. - Every precondition is checked again inside the transaction that publishes the new state. +7. Runs a final migration pass and then a verification, and stops if the verification reports any difference between the two stores. +8. Checks the fence again and commits. + Every precondition is checked once more inside the transaction that publishes the new state, including that no migration pass finished after the verification started. If the Job stops at step 6, the cluster stays in `cutting_over` with reads unaffected. Investigate the reported differences, then either fix them and re-run the Job (it resumes the open window), or abort: From 9991dfde199e71686e3bd45539ac69e612cbc2c8 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sat, 8 Aug 2026 20:01:14 +0200 Subject: [PATCH 6/9] [FSTORE-2075] State the downgrade contract, hold the write-window promise https://hopsworks.atlassian.net/browse/FSTORE-2075 Review round eleven asked for the product contract the fresh-install fence decision rests on, in writing: downgrading any database-canonical cluster below this release without restoring or reinstalling its database is unsupported, and a fresh installation is such a cluster from day one. The page now says so plainly, with the admission policy as the opt-in refusal for operators who want more than a rule. The write-window escape also stops being free: disabling it renders a pre-upgrade check that refuses the migration while any API pod runs or an autoscaler targets the API deployments, and the page explains why a load-balancer window alone is not a write window. The failure-step reference in the cut-over runbook also moved from step 6 to steps 7-8, matching the renumbered Job. Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/setup_installation/admin/dataset_tag_storage.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/setup_installation/admin/dataset_tag_storage.md b/docs/setup_installation/admin/dataset_tag_storage.md index d4ca18624f..dff8a45654 100644 --- a/docs/setup_installation/admin/dataset_tag_storage.md +++ b/docs/setup_installation/admin/dataset_tag_storage.md @@ -24,7 +24,8 @@ Two failure paths are deliberately different. A refusal by the pre-migration audit happens before any schema change, so the original replica counts are restored and the upgrade aborts with the cluster running as it was. A failure after the schema change has begun leaves the cluster scaled to zero, because starting the old nodes over a half-applied schema is worse than an outage. -Set `hopsworks.tagLifecycle.writeWindow.enabled=false` to skip it, only if you are taking the write window yourself. +Set `hopsworks.tagLifecycle.writeWindow.enabled=false` only if you are taking the write window yourself, and take it for real: routing traffic away at a load balancer is not enough, because internal clients still reach the API pods directly. +The chart holds you to it: with the write window disabled, a pre-upgrade check refuses the one upgrade that applies the migration while any API pod is running or a HorizontalPodAutoscaler targets the API deployments. After the upgrade the cluster keeps reading dataset tags from the extended attributes and writes them to both stores. Nothing is lost while you stay in that state, and you can stay in it indefinitely. @@ -66,8 +67,9 @@ A cluster that never cuts over never needs it. The cut-over refuses to run while the policy is absent, and the operator who accepts the risk says so explicitly with `hopsworks.tagLifecycle.cutover.acceptUnfencedRollback=true`, which is rendered into the Job and logged with the decision. A fresh installation is database-canonical from the start, so it sits past the same boundary without ever running a cut-over. -Downgrading a fresh installation below this release without restoring its database has the same consequence a post-cut-over rollback has: dataset tags written to the database become invisible to the old code, and tags written to extended attributes during the downgrade are silently ignored when you upgrade again. -Enable the admission policy on a fresh installation if pods from an older release must be refused; the installation records `rollbackFenced=false` in its activation audit when you do not. +**Downgrading any database-canonical cluster below this release without restoring (or reinstalling) its database is unsupported**, and a fresh installation is such a cluster from day one. +Doing it anyway has the same consequence a post-cut-over rollback has: dataset tags written to the database become invisible to the old code, and tags written to extended attributes during the downgrade are silently ignored when you upgrade again. +Enable the admission policy on a fresh installation if pods from an older release must be refused rather than merely unsupported; the installation records `rollbackFenced=false` in its activation audit when you do not. ## The cut-over @@ -108,7 +110,7 @@ The Job: 8. Checks the fence again and commits. Every precondition is checked once more inside the transaction that publishes the new state, including that no migration pass finished after the verification started. -If the Job stops at step 6, the cluster stays in `cutting_over` with reads unaffected. +If the Job stops at step 7 or 8, the cluster stays in `cutting_over` with reads unaffected. Investigate the reported differences, then either fix them and re-run the Job (it resumes the open window), or abort: ```bash From 73d7e7005371ee52c7e567902a5bc764eb2f1339 Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Wed, 12 Aug 2026 17:00:38 +0200 Subject: [PATCH 7/9] [FSTORE-2075] Document what a quarantine waiver actually authorises https://hopsworks.atlassian.net/browse/FSTORE-2075 The cut-over guide told operators the Job refuses while a tag is quarantined without saying what a quarantined tag is, how to clear one, or what waiving costs. It now shows the waive call, and states the part that matters: the waiver applies to the value that was inspected and to no other, so a value that changes before the grant is refused and one that changes afterwards brings the record back for review. An operator who thinks they waived "the tag" would be surprised by that; one who reads this will not be. Reviewed-by: Claude Opus 5 (1M context) Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DLDai4Z5QRoBTJpikoToZ7 --- .../admin/dataset_tag_storage.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/setup_installation/admin/dataset_tag_storage.md b/docs/setup_installation/admin/dataset_tag_storage.md index dff8a45654..2821af9859 100644 --- a/docs/setup_installation/admin/dataset_tag_storage.md +++ b/docs/setup_installation/admin/dataset_tag_storage.md @@ -102,6 +102,21 @@ The Job: 3. Reads the admission policy and its binding from the Kubernetes API and checks the whole contract: that it denies rather than warns, that it fails closed, that it matches pod creation in this namespace for both API deployments, and that its expression is exactly the one this chart installs for the current epoch. A policy with the right name but any of those wrong is not a fence. 4. Refuses to start unless the rolling upgrade has been activated, the background migration of existing tags reports done, no tag was quarantined for failing validation, and no HorizontalPodAutoscaler targets the API deployments. + + A quarantined tag is one whose value could not be migrated: its schema is gone, it no longer + validates, or it is longer than the store accepts. The value is left where it is and recorded with + a digest, a length and a reason, so nothing is destroyed by a failed migration. Resolve each one by + correcting the value or the schema, or waive it to accept losing that value: + + ```bash + curl -X PUT "https:///hopsworks-api/api/admin/dataset-tags/quarantine//waive?expectedDigest=" \ + -H "Authorization: ApiKey " + ``` + + The digest is required and comes from reading the record first. A waiver applies to the value that + was inspected and to no other: if the value changes before the waiver is granted the request is + refused with `409`, and if it changes afterwards the waiver is dropped and the record comes back for + review. That is deliberate, because a waiver authorises destroying one specific value. 5. Records the current replica counts in a ConfigMap, scales both API deployments to zero, and waits until their pods are gone. At that instant every write that was in flight has either reached the file system or never will, so nothing is left to drain. 6. Sets the state to `cutting_over` in a single transaction while nothing is running, then restores the replica counts. From f55914b95804aa5333bf623864144f08a2da753b Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sat, 15 Aug 2026 11:26:54 +0200 Subject: [PATCH 8/9] [FSTORE-2075] Document the tag lifecycle, job and dataset tags, and keywords https://hopsworks.atlassian.net/browse/FSTORE-2075 The deprecation table gave the wrong status for the refusal it documents. Attaching a deprecated schema answers 409, not 400: the request is well formed and the schema's state is what refuses it. A client author branching on the documented status would not catch it. The per-file tag freeze named a status and no error code, while every other refusal on these pages names one, so an operator could not tell it from an ordinary validation failure. It is 370013. The cleanup that removes the extended attributes was undocumented, and with it the whole recovery path. The page said the attributes "are removed", passively, and never said by what. It now covers starting a cleanup pass, the four gates each dataset has to pass and which of them clear on their own, resuming a run whose worker pod is gone, including why that one is never taken over automatically and how the echoed pod name is checked, and reading and waiving a quarantined value against the digest it was granted for. The interval an operator has to wait before cleaning up is stated rather than implied. It is ten minutes, and a cleanup started earlier is refused. Also two sentences rewritten to the house style, and one that told the reader to take a maintenance window seriously without adding anything the surrounding text did not already say. Reviewed-by: claude Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- .../admin/dataset_tag_storage.md | 72 +++++++++++++++++-- .../fs/tags/tag_schema_lifecycle.md | 2 +- docs/user_guides/fs/tags/tags.md | 2 +- 3 files changed, 69 insertions(+), 7 deletions(-) diff --git a/docs/setup_installation/admin/dataset_tag_storage.md b/docs/setup_installation/admin/dataset_tag_storage.md index 2821af9859..e200acadd3 100644 --- a/docs/setup_installation/admin/dataset_tag_storage.md +++ b/docs/setup_installation/admin/dataset_tag_storage.md @@ -24,7 +24,7 @@ Two failure paths are deliberately different. A refusal by the pre-migration audit happens before any schema change, so the original replica counts are restored and the upgrade aborts with the cluster running as it was. A failure after the schema change has begun leaves the cluster scaled to zero, because starting the old nodes over a half-applied schema is worse than an outage. -Set `hopsworks.tagLifecycle.writeWindow.enabled=false` only if you are taking the write window yourself, and take it for real: routing traffic away at a load balancer is not enough, because internal clients still reach the API pods directly. +Set `hopsworks.tagLifecycle.writeWindow.enabled=false` only if you are taking the write window yourself. Routing traffic away at a load balancer is not enough, because internal clients still reach the API pods directly. The chart holds you to it: with the write window disabled, a pre-upgrade check refuses the one upgrade that applies the migration while any API pod is running or a HorizontalPodAutoscaler targets the API deployments. After the upgrade the cluster keeps reading dataset tags from the extended attributes and writes them to both stores. @@ -33,7 +33,7 @@ Nothing is lost while you stay in that state, and you can stay in it indefinitel ## Per-file tags are frozen Tags could previously be attached to any file or directory inside a dataset. -Attaching a new tag to a path inside a dataset is now rejected with HTTP 400. +Attaching a new tag to a path inside a dataset is now rejected with HTTP 400 and error code 370013. Per-file tags were stored outside the database, could not be searched or counted, and would have made the cut-over unbounded. Tags that were already attached to such paths remain readable and deletable, and the file browser keeps showing them. @@ -58,7 +58,7 @@ curl -s -H "Authorization: ApiKey $API_KEY" \ ## Fencing a rollback Once the database is the canonical store for dataset tags, a node running the previous release writes extended attributes that nothing reads any more, and those writes are lost silently. -Rolling back is therefore not made safe; it is refused. +Rolling back is therefore refused rather than made safe. `hopsworks.tagLifecycle.admissionPolicy.enabled=true` installs a `ValidatingAdmissionPolicy` that refuses to admit an API pod below the current capability epoch. It is **off by default**, because turning it on means an emergency downgrade requires restoring the pre-cut-over database and deleting the policy, in that order. @@ -74,7 +74,7 @@ Enable the admission policy on a fresh installation if pods from an older releas ## The cut-over The cut-over moves the canonical store from the extended attributes to the database. -It is a scheduled maintenance action, not an administrative call to make at an arbitrary moment: dataset tag **writes** are refused with a retryable HTTP 503 from the start of the window until it commits. +Schedule it as a maintenance action: dataset tag **writes** are refused with a retryable HTTP 503 from the start of the window until it commits. The quiesce itself is a short full outage: both API deployments are scaled to zero for the restart, and nothing answers while they are down. Once the pods are back, reads of dataset tags and everything else serve normally for the rest of the window, and no stored data is touched at any point. @@ -152,6 +152,68 @@ The response reports the current state, when the window opened, and what would b ## After the cut-over The extended attributes are left in place after the cut-over. -Readers holding a cached copy of the cluster state for up to one cache interval would otherwise read a store that had already been emptied, so the extended attributes are removed no earlier than one full cache interval afterwards, and only after the verification has passed. +Readers holding a cached copy of the cluster state would otherwise read a store that had already been emptied, so the extended attributes are removed no earlier than one full settings-cache interval afterwards, which is ten minutes, and only after the verification has passed. Removing them is the point of no return for a downgrade, so keep them until the release is known good. + +### Removing the extended attributes + +When you are ready, start a cleanup pass: + +```bash +curl -X POST -u \ + https:///hopsworks-api/api/admin/dataset-tags/clean +``` + +It answers with a run id. Poll it: + +```bash +curl -u \ + https:///hopsworks-api/api/admin/dataset-tags/clean/ +``` + +A dataset is cleaned only when all four of these hold, and the run reports which one refused for each dataset it skips: + +| Gate | Meaning | +| --- | --- | +| No unwaived quarantine record | Nothing in the attribute failed to migrate. Waive a record only after reading it (see below) | +| The snapshot is from the accepted verification | This dataset was covered by the verification the commit accepted | +| The attribute still matches that snapshot | Nothing has written the attribute since it was verified | +| The index has caught up | The search document reflects the projection, so nothing is lost by deleting the attribute | + +A skipped dataset is not a failure, and the pass is safe to re-run: gates two and four clear on their own once a queued search-index rewrite drains, and quarantine records are cleared by waiving them. + +### Recovering a cleanup run whose pod is gone + +A cleanup pass deletes from HopsFS, which no database transaction can fence, so a run whose worker stopped responding is never taken over automatically. +Its claim is held until you confirm that worker is gone, and the run reports which pod holds it. + +Read the run, take the `worker` field, confirm that pod no longer exists, then hand the run to a new worker by echoing that name back: + +```bash +curl -X POST -u \ + "https:///hopsworks-api/api/admin/dataset-tags/clean//resume?holderFenced=true&expectedWorker=" +``` + +The echo is checked against the recorded name in the same statement that transfers ownership, so a wrong name moves nothing and a second operator repeating a name that has already been used is refused. +The run keeps its cursor, so it continues rather than starting over. +A run recorded before this release has no worker name; omit `expectedWorker` for it, and confirm it is safe by scaling the API to zero first. + +### Reviewing a quarantined value + +A tag that could not be migrated is quarantined rather than dropped, recorded by name, reason, byte length and digest, never by value. + +```bash +curl -u \ + "https:///hopsworks-api/api/admin/dataset-tags/quarantine?limit=100" +``` + +Waiving one authorises deleting that value, so it is granted against the digest you read: + +```bash +curl -X PUT -u \ + "https:///hopsworks-api/api/admin/dataset-tags/quarantine//waive?waived=true&expectedDigest=" +``` + +If a later pass replaced the payload, the digest no longer matches and the request is refused with HTTP 409 so you can read the new value before deciding. +Pass `waived=false` to clear a waiver, which needs no digest. diff --git a/docs/user_guides/fs/tags/tag_schema_lifecycle.md b/docs/user_guides/fs/tags/tag_schema_lifecycle.md index 0918d05e94..986e9f1bd1 100644 --- a/docs/user_guides/fs/tags/tag_schema_lifecycle.md +++ b/docs/user_guides/fs/tags/tag_schema_lifecycle.md @@ -20,7 +20,7 @@ Use it when a schema is being replaced and you want existing owners to migrate o | Operation on a deprecated schema | Allowed | | --- | --- | | Read the schema, list schemas, read attached values | Yes | -| Attach it to an artifact that does not have it | No, rejected with HTTP 400 and error code 370011 | +| Attach it to an artifact that does not have it | No, rejected with HTTP 409 and error code 370011 | | Update the value already attached to an artifact | Yes, so a team can correct data while migrating off | | Detach it from an artifact | Yes | | Register it as a mandatory tag | No, because a mandatory tag that nobody may attach cannot be satisfied | diff --git a/docs/user_guides/fs/tags/tags.md b/docs/user_guides/fs/tags/tags.md index db5dc21cf8..059b4fa8e7 100644 --- a/docs/user_guides/fs/tags/tags.md +++ b/docs/user_guides/fs/tags/tags.md @@ -175,7 +175,7 @@ Tags can also be attached when the dataset is created. !!! warning "Tags on files inside a dataset are frozen" Tags could previously be attached to any file or directory inside a dataset. - Attaching a new tag to a path inside a dataset is now rejected with HTTP 400, because per-file tags were stored outside the database and could not be searched, counted, or governed. + Attaching a new tag to a path inside a dataset is now rejected with HTTP 400 and error code 370013, because per-file tags were stored outside the database and could not be searched, counted, or governed. Tags that were already attached to such paths remain readable and deletable, and the file browser keeps showing them. Attach the tag to the dataset instead. From 569f7629d842ed5d8abd5b517d4968e5ecbe85eb Mon Sep 17 00:00:00 2001 From: Jim Dowling Date: Sat, 15 Aug 2026 11:32:36 +0200 Subject: [PATCH 9/9] [FSTORE-2075] Document the tag lifecycle, job and dataset tags, and keywords https://hopsworks.atlassian.net/browse/FSTORE-2075 Link between these pages by heading id rather than by relative file path. The repo's own content guide asks for this, and gives the reason: a relative path breaks once mike publishes a versioned copy of the site. Prevailing practice in the tree is still the relative form, which is why these pages picked it up, but the documented rule is the one to follow in new writing. The tags guide needed an explicit anchor to be linkable at all. Four pages carry a "Tags" heading, so the generated id is ambiguous and autorefs refuses to resolve it; the guide now declares its own id in the style the feature-view and data-source pages already use. Verified with a strict build, which resolves every one of these links and is otherwise unchanged. Reviewed-by: claude Signed-off-by: Jim Dowling Co-Authored-By: Claude Opus 5 (1M context) --- docs/concepts/projects/search.md | 2 +- docs/user_guides/fs/tags/keywords.md | 4 ++-- docs/user_guides/fs/tags/mandatory_tags.md | 6 +++--- docs/user_guides/fs/tags/tag_schema_lifecycle.md | 4 ++-- docs/user_guides/fs/tags/tags.md | 4 ++-- 5 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/concepts/projects/search.md b/docs/concepts/projects/search.md index accf63fedb..871c863ad7 100644 --- a/docs/concepts/projects/search.md +++ b/docs/concepts/projects/search.md @@ -25,7 +25,7 @@ To avoid users gaining unauthorized access to data, if a search result is in a p If the search result is within a project you are a member of, you are also able to inspect recent activities on the asset as well as statistics. Searching across projects you are not a member of is on by default. -An administrator running a multi-tenant deployment can turn it off by setting the `cross_project_global_search_enabled` cluster variable to `false` in the [Configuration](../../setup_installation/admin/variables.md) UI, which restricts a cluster-wide search to the projects you can already access. +An administrator running a multi-tenant deployment can turn it off by setting the `cross_project_global_search_enabled` cluster variable to `false` in the [Configuration][cluster-configuration] UI, which restricts a cluster-wide search to the projects you can already access. ## Tags diff --git a/docs/user_guides/fs/tags/keywords.md b/docs/user_guides/fs/tags/keywords.md index f23a01cad8..3ae39fbaea 100644 --- a/docs/user_guides/fs/tags/keywords.md +++ b/docs/user_guides/fs/tags/keywords.md @@ -3,7 +3,7 @@ ## Introduction A keyword is a single user-defined word attached to a feature group, feature view or training dataset. -Keywords are free text with no schema behind them, which makes them the lighter alternative to [tags](tags.md): use a keyword to make an artifact easier to find, and a tag when the metadata has to be structured and validated. +Keywords are free text with no schema behind them, which makes them the lighter alternative to [tags][tags-guide]: use a keyword to make an artifact easier to find, and a tag when the metadata has to be structured and validated. Keywords are indexed for free-text search alongside names, descriptions and tags. They can also be used as a filter in the search API. @@ -91,4 +91,4 @@ The same three commands exist for feature views (`hops fv`) and training dataset !!! warning "`--value` on the keyword commands is being removed" `--value` used to make `hops fg add-keyword` write a tag rather than a keyword. It is now rejected with a message naming `hops fg add-tag` as the replacement, and it is removed in the following release. - Attach a tag with the tag commands, described in the [Tags](tags.md) guide. + Attach a tag with the tag commands, described in the [Tags][tags-guide] guide. diff --git a/docs/user_guides/fs/tags/mandatory_tags.md b/docs/user_guides/fs/tags/mandatory_tags.md index b15516e4b6..770f442ddb 100644 --- a/docs/user_guides/fs/tags/mandatory_tags.md +++ b/docs/user_guides/fs/tags/mandatory_tags.md @@ -3,7 +3,7 @@ ## Introduction Mandatory tags let a Hopsworks administrator require that specific tag schemas are populated on artifacts. -They build on top of [tags](tags.md) and are used to enforce governance rules, such as requiring every model to declare a data owner. +They build on top of [tags][tags-guide] and are used to enforce governance rules, such as requiring every model to declare a data owner. A mandatory tag is a tag schema that has been marked as required for one or more artifact types. The supported artifact types are feature groups, feature views, training datasets, models, deployments and datasets. @@ -11,7 +11,7 @@ The supported artifact types are feature groups, feature views, training dataset ## Prerequisites A mandatory tag references an existing tag schema. -Define the tag schema first, as described in the [Tags](tags.md) guide, before marking it mandatory. +Define the tag schema first, as described in the [Tags][tags-guide] guide, before marking it mandatory. Only administrators can configure mandatory tags. Attaching the tag values afterwards is done by any project member with write access to the artifact. @@ -34,7 +34,7 @@ For each mandatory tag you select the artifact types it applies to. A tag schema can be mandatory for any combination of feature groups, feature views, training datasets, models, deployments and datasets. For example, a `data_owner` schema can be marked mandatory for models and deployments only, leaving the others unaffected. -A [deprecated](tag_schema_lifecycle.md) schema cannot be registered as mandatory, because nobody is allowed to attach it. +A [deprecated][tag-schema-lifecycle] schema cannot be registered as mandatory, because nobody is allowed to attach it. Conversely, a schema that is registered as mandatory cannot be deprecated until the registrations are removed. ## Enforcement per artifact type diff --git a/docs/user_guides/fs/tags/tag_schema_lifecycle.md b/docs/user_guides/fs/tags/tag_schema_lifecycle.md index 986e9f1bd1..b662880542 100644 --- a/docs/user_guides/fs/tags/tag_schema_lifecycle.md +++ b/docs/user_guides/fs/tags/tag_schema_lifecycle.md @@ -2,7 +2,7 @@ ## Introduction -A tag schema is defined once and then attached to artifacts across every project on the cluster, as described in the [Tags](tags.md) guide. +A tag schema is defined once and then attached to artifacts across every project on the cluster, as described in the [Tags][tags-guide] guide. This guide covers what an administrator can do with a schema afterwards: retire it so that nothing new uses it, and remove it once nothing does. The JSON definition of a schema cannot be changed after it is created. @@ -27,7 +27,7 @@ Use it when a schema is being replaced and you want existing owners to migrate o | Delete the schema | Yes, under the rules below | | Restore it | Yes | -A schema that is currently registered as a [mandatory tag](mandatory_tags.md) cannot be deprecated. +A schema that is currently registered as a [mandatory tag][mandatory-tags] cannot be deprecated. The request is rejected with HTTP 409 and error code 370012. Remove the mandatory registrations first, then deprecate. diff --git a/docs/user_guides/fs/tags/tags.md b/docs/user_guides/fs/tags/tags.md index 059b4fa8e7..3ae4950215 100644 --- a/docs/user_guides/fs/tags/tags.md +++ b/docs/user_guides/fs/tags/tags.md @@ -1,4 +1,4 @@ -# Tags +# Tags { #tags-guide } ## Introduction @@ -21,7 +21,7 @@ This allows tags to be consistent no matter the project or the team generating t !!! warning "Schema definitions cannot be edited" The JSON schema of a tag schema cannot be changed after it is created, because the values already attached were validated against the original definition. A schema can be deprecated so that it accepts no new attachments, and it can be deleted once nothing references it. - Both are administrator actions and are described in the [Tag schema lifecycle](tag_schema_lifecycle.md) guide. + Both are administrator actions and are described in the [Tag schema lifecycle][tag-schema-lifecycle] guide. ## Step 1: Define a tag schema