diff --git a/docs/concepts/projects/search.md b/docs/concepts/projects/search.md index 4750bde12f..871c863ad7 100644 --- a/docs/concepts/projects/search.md +++ b/docs/concepts/projects/search.md @@ -10,15 +10,23 @@ Hopsworks supports free-text search to discover machine-learning assets: - feature groups - feature views - training data +- models +- deployments +- jobs +- datasets You can use the search bar at the top of your project to free-text search for the names or descriptions of any ML asset. You can also search using keywords or tags that are attached to an ML asset. +Keywords apply to feature groups, feature views and training data only, so a keyword filter never matches a job or a dataset. You can search for assets within a specific project or across all projects in a Hopsworks deployment, including those you are not a member of. This allows for easier discoverability and reusability of assets within an organization. To avoid users gaining unauthorized access to data, if a search result is in a project you are **not** a member of, the information displayed is limited to: names, descriptions, tags, asset creator and create date. If the search result is within a project you are a member of, you are also able to inspect recent activities on the asset as well as statistics. +Searching across projects you are not a member of is on by default. +An administrator running a multi-tenant deployment can turn it off by setting the `cross_project_global_search_enabled` cluster variable to `false` in the [Configuration][cluster-configuration] UI, which restricts a cluster-wide search to the projects you can already access. + ## Tags A keyword is a single user-defined word attached to an ML asset. diff --git a/docs/setup_installation/admin/dataset_tag_storage.md b/docs/setup_installation/admin/dataset_tag_storage.md new file mode 100644 index 0000000000..e200acadd3 --- /dev/null +++ b/docs/setup_installation/admin/dataset_tag_storage.md @@ -0,0 +1,219 @@ +# Dataset Tag Storage + +## Introduction + +Tags attached to a dataset used to be stored as HopsFS extended attributes. +They are now rows in the Hopsworks database. +The move makes dataset tags searchable, countable when an administrator asks what still references a tag schema, and deleted in the same transaction as the dataset they belong to. + +Nothing about this is visible to a project member: the same tags are read and written through the same API either way. +It is visible to an administrator, because moving the store on a running cluster is a one-way step that has to be taken deliberately, and because tags attached to individual files inside a dataset do not move. + +This page covers what an upgrade does on its own, what the cut-over is, and how to run it. + +## What the upgrade does on its own + +The upgrade that introduces this feature applies a database migration that audits the tag tables for duplicates and then adds unique keys over them. +A tag or keyword write landing between the audit and the index creation can be the row that makes the index creation fail, so the chart stops those writes for the length of the migration. + +A pre-upgrade hook scales both Payara deployments to zero, waits until their pods are gone, runs the migration, and leaves the deployments at zero for the upgrade's own apply to bring back on the new image. +The hook fires on the one upgrade that applies the migration and on no other. +It is measured at 70 seconds against a database holding a million tag values. + +Two failure paths are deliberately different. +A refusal by the pre-migration audit happens before any schema change, so the original replica counts are restored and the upgrade aborts with the cluster running as it was. +A failure after the schema change has begun leaves the cluster scaled to zero, because starting the old nodes over a half-applied schema is worse than an outage. + +Set `hopsworks.tagLifecycle.writeWindow.enabled=false` only if you are taking the write window yourself. Routing traffic away at a load balancer is not enough, because internal clients still reach the API pods directly. +The chart holds you to it: with the write window disabled, a pre-upgrade check refuses the one upgrade that applies the migration while any API pod is running or a HorizontalPodAutoscaler targets the API deployments. + +After the upgrade the cluster keeps reading dataset tags from the extended attributes and writes them to both stores. +Nothing is lost while you stay in that state, and you can stay in it indefinitely. + +## Per-file tags are frozen + +Tags could previously be attached to any file or directory inside a dataset. +Attaching a new tag to a path inside a dataset is now rejected with HTTP 400 and error code 370013. +Per-file tags were stored outside the database, could not be searched or counted, and would have made the cut-over unbounded. + +Tags that were already attached to such paths remain readable and deletable, and the file browser keeps showing them. +They are not migrated, and they are the reason the tag schema usage report states that its counts cover database-backed references only. + +## The activation gate + +Two operations are only safe once every node in the cluster runs the new code: replacing a cluster-wide mandatory tag policy, and the dataset tag cut-over. +Both are refused with HTTP 503 and `ACTIVATION_PENDING` while a rolling upgrade is in progress. + +The chart's post-upgrade hook verifies against the Kubernetes API that every pod serving the Hopsworks API belongs to the new rollout, then activates the lifecycle. +Activation is what the refusal waits for; no timer and no node decides it for itself. +A fresh install is activated by the post-install hook instead, since there is no old node to wait for. + +Check the state at any time: + +```bash +curl -s -H "Authorization: ApiKey $API_KEY" \ + https:///hopsworks-api/api/admin/tag-lifecycle +``` + +## Fencing a rollback + +Once the database is the canonical store for dataset tags, a node running the previous release writes extended attributes that nothing reads any more, and those writes are lost silently. +Rolling back is therefore refused rather than made safe. + +`hopsworks.tagLifecycle.admissionPolicy.enabled=true` installs a `ValidatingAdmissionPolicy` that refuses to admit an API pod below the current capability epoch. +It is **off by default**, because turning it on means an emergency downgrade requires restoring the pre-cut-over database and deleting the policy, in that order. +A cluster that never cuts over never needs it. + +The cut-over refuses to run while the policy is absent, and the operator who accepts the risk says so explicitly with `hopsworks.tagLifecycle.cutover.acceptUnfencedRollback=true`, which is rendered into the Job and logged with the decision. + +A fresh installation is database-canonical from the start, so it sits past the same boundary without ever running a cut-over. +**Downgrading any database-canonical cluster below this release without restoring (or reinstalling) its database is unsupported**, and a fresh installation is such a cluster from day one. +Doing it anyway has the same consequence a post-cut-over rollback has: dataset tags written to the database become invisible to the old code, and tags written to extended attributes during the downgrade are silently ignored when you upgrade again. +Enable the admission policy on a fresh installation if pods from an older release must be refused rather than merely unsupported; the installation records `rollbackFenced=false` in its activation audit when you do not. + +## The cut-over + +The cut-over moves the canonical store from the extended attributes to the database. +Schedule it as a maintenance action: dataset tag **writes** are refused with a retryable HTTP 503 from the start of the window until it commits. +The quiesce itself is a short full outage: both API deployments are scaled to zero for the restart, and nothing answers while they are down. +Once the pods are back, reads of dataset tags and everything else serve normally for the rest of the window, and no stored data is touched at any point. + +The window is about five minutes on a three-node cluster, dominated by the API restart. +The verification over several hundred datasets takes seconds. + +If a HorizontalPodAutoscaler targets either API deployment, the Job refuses to start: an autoscaler scales the deployment back up while the cut-over needs it at zero, and a pod it starts can write extended attributes after the flip. +Delete or suspend it for the window, and pause any GitOps reconciliation that owns the replica counts, for the same reason. + +Run it by setting `hopsworks.tagLifecycle.cutover.run=true` on a `helm upgrade`. +The value is off by default and the Job runs once per upgrade that sets it. +These settings belong to the `hopsworks` subchart, so the `hopsworks.` prefix is part of the key: the umbrella chart's schema rejects a bare `tagLifecycle` with `additional properties 'tagLifecycle' not allowed`. + +!!! note "Helm 4 needs `--force-conflicts`" + Under Helm 4, pass `--force-conflicts` to the upgrade. + Server-side apply refuses fields owned by other field managers, which a running cluster always has, and without the flag the upgrade fails before any hook runs. + This is a pre-existing Helm 4 behaviour, not specific to the cut-over. + +The Job: + +1. Recovers first: if a previous attempt was killed after scaling the deployments down, the replica counts it recorded in a ConfigMap are restored before anything waits on the API. + Both deployments at zero with no record is a zero somebody else owns, and the Job refuses with the manual commands printed. +2. Reads the cut-over status while the cluster is up. + A database that is already canonical exits immediately, so the Job is safe to leave enabled across upgrades, and a window an earlier attempt left open is resumed at the final sweep rather than opened twice. +3. Reads the admission policy and its binding from the Kubernetes API and checks the whole contract: that it denies rather than warns, that it fails closed, that it matches pod creation in this namespace for both API deployments, and that its expression is exactly the one this chart installs for the current epoch. + A policy with the right name but any of those wrong is not a fence. +4. Refuses to start unless the rolling upgrade has been activated, the background migration of existing tags reports done, no tag was quarantined for failing validation, and no HorizontalPodAutoscaler targets the API deployments. + + A quarantined tag is one whose value could not be migrated: its schema is gone, it no longer + validates, or it is longer than the store accepts. The value is left where it is and recorded with + a digest, a length and a reason, so nothing is destroyed by a failed migration. Resolve each one by + correcting the value or the schema, or waive it to accept losing that value: + + ```bash + curl -X PUT "https:///hopsworks-api/api/admin/dataset-tags/quarantine//waive?expectedDigest=" \ + -H "Authorization: ApiKey " + ``` + + The digest is required and comes from reading the record first. A waiver applies to the value that + was inspected and to no other: if the value changes before the waiver is granted the request is + refused with `409`, and if it changes afterwards the waiver is dropped and the record comes back for + review. That is deliberate, because a waiver authorises destroying one specific value. +5. Records the current replica counts in a ConfigMap, scales both API deployments to zero, and waits until their pods are gone. + At that instant every write that was in flight has either reached the file system or never will, so nothing is left to drain. +6. Sets the state to `cutting_over` in a single transaction while nothing is running, then restores the replica counts. + The API comes back refusing dataset tag writes with `CUTOVER_IN_PROGRESS`. +7. Runs a final migration pass and then a verification, and stops if the verification reports any difference between the two stores. +8. Checks the fence again and commits. + Every precondition is checked once more inside the transaction that publishes the new state, including that no migration pass finished after the verification started. + +If the Job stops at step 7 or 8, the cluster stays in `cutting_over` with reads unaffected. +Investigate the reported differences, then either fix them and re-run the Job (it resumes the open window), or abort: + +```bash +curl -s -X POST -H "Authorization: ApiKey $API_KEY" \ + https:///hopsworks-api/api/admin/dataset-tags/cutover/abort +``` + +Aborting returns the cluster to the extended attributes and resumes writes. +It is never blocked on anything: getting out of a paused state must not itself require the state to be healthy. + +Ask what the cluster thinks at any point: + +```bash +curl -s -H "Authorization: ApiKey $API_KEY" \ + https:///hopsworks-api/api/admin/dataset-tags/cutover +``` + +The response reports the current state, when the window opened, and what would block a commit right now. + +!!! warning "Do not edit the state directly" + Setting the canonical store through the cluster variables endpoint is refused in both directions, and the error names the cut-over endpoints. + A direct database edit bypasses every check here, as it always has. + +## After the cut-over + +The extended attributes are left in place after the cut-over. +Readers holding a cached copy of the cluster state would otherwise read a store that had already been emptied, so the extended attributes are removed no earlier than one full settings-cache interval afterwards, which is ten minutes, and only after the verification has passed. + +Removing them is the point of no return for a downgrade, so keep them until the release is known good. + +### Removing the extended attributes + +When you are ready, start a cleanup pass: + +```bash +curl -X POST -u \ + https:///hopsworks-api/api/admin/dataset-tags/clean +``` + +It answers with a run id. Poll it: + +```bash +curl -u \ + https:///hopsworks-api/api/admin/dataset-tags/clean/ +``` + +A dataset is cleaned only when all four of these hold, and the run reports which one refused for each dataset it skips: + +| Gate | Meaning | +| --- | --- | +| No unwaived quarantine record | Nothing in the attribute failed to migrate. Waive a record only after reading it (see below) | +| The snapshot is from the accepted verification | This dataset was covered by the verification the commit accepted | +| The attribute still matches that snapshot | Nothing has written the attribute since it was verified | +| The index has caught up | The search document reflects the projection, so nothing is lost by deleting the attribute | + +A skipped dataset is not a failure, and the pass is safe to re-run: gates two and four clear on their own once a queued search-index rewrite drains, and quarantine records are cleared by waiving them. + +### Recovering a cleanup run whose pod is gone + +A cleanup pass deletes from HopsFS, which no database transaction can fence, so a run whose worker stopped responding is never taken over automatically. +Its claim is held until you confirm that worker is gone, and the run reports which pod holds it. + +Read the run, take the `worker` field, confirm that pod no longer exists, then hand the run to a new worker by echoing that name back: + +```bash +curl -X POST -u \ + "https:///hopsworks-api/api/admin/dataset-tags/clean//resume?holderFenced=true&expectedWorker=" +``` + +The echo is checked against the recorded name in the same statement that transfers ownership, so a wrong name moves nothing and a second operator repeating a name that has already been used is refused. +The run keeps its cursor, so it continues rather than starting over. +A run recorded before this release has no worker name; omit `expectedWorker` for it, and confirm it is safe by scaling the API to zero first. + +### Reviewing a quarantined value + +A tag that could not be migrated is quarantined rather than dropped, recorded by name, reason, byte length and digest, never by value. + +```bash +curl -u \ + "https:///hopsworks-api/api/admin/dataset-tags/quarantine?limit=100" +``` + +Waiving one authorises deleting that value, so it is granted against the digest you read: + +```bash +curl -X PUT -u \ + "https:///hopsworks-api/api/admin/dataset-tags/quarantine//waive?waived=true&expectedDigest=" +``` + +If a later pass replaced the payload, the digest no longer matches and the request is refused with HTTP 409 so you can read the new value before deciding. +Pass `waived=false` to clear a waiver, which needs no digest. diff --git a/docs/user_guides/fs/tags/keywords.md b/docs/user_guides/fs/tags/keywords.md new file mode 100644 index 0000000000..3ae39fbaea --- /dev/null +++ b/docs/user_guides/fs/tags/keywords.md @@ -0,0 +1,94 @@ +# Keywords + +## Introduction + +A keyword is a single user-defined word attached to a feature group, feature view or training dataset. +Keywords are free text with no schema behind them, which makes them the lighter alternative to [tags][tags-guide]: use a keyword to make an artifact easier to find, and a tag when the metadata has to be structured and validated. + +Keywords are indexed for free-text search alongside names, descriptions and tags. +They can also be used as a filter in the search API. + +Keywords are not available on models, deployments, jobs or datasets. +A keyword filter passed to a job or dataset search therefore matches nothing. + +## Read the keywords of an artifact + +=== "Python" + + ```python + fg = fs.get_feature_group("transactions_4h_aggs_fraud_batch_fg", version=1) + + fg.get_keywords() # ["fraud", "eu_region"] + ``` + +Keywords record the time they were added, from the release that introduced this guide onwards. + +=== "Python" + + ```python + for keyword, added in fg.get_keywords_metadata().items(): + print(keyword, added) + ``` + +`added` is `None` for keywords attached before that release, and against an older backend that does not report the time. + +## Add, replace and delete keywords + +=== "Python" + + ```python + # Add to the existing set + fg.add_keywords("pii") + fg.add_keywords(["fraud", "eu_region"]) + + # Replace the whole set + fg.set_keywords(["fraud", "pii"]) + + # Remove one + fg.delete_keyword("pii") + ``` + +Each of these returns the resulting set of keywords. + +!!! warning "add_keywords is not atomic" + `add_keywords` reads the current set, adds to it, and writes the whole set back. + Two calls running at the same time can therefore lose one of the additions. + Use `set_keywords` when the caller already knows the full set it wants. + +The same five methods are available on `FeatureView` and `TrainingDataset`. +On a feature view, the training datasets it owns are reached through the `*_training_dataset_keywords` variants, which take the training dataset version as their first argument. + +=== "Python" + + ```python + fv = fs.get_feature_view("transactions_fv", version=1) + + fv.get_training_dataset_keywords(1) + fv.add_training_dataset_keywords(1, "backfill") + ``` + +## The cluster vocabulary + +`get_all_keywords` returns every keyword used on the cluster, not only those in the current project. +Use it to offer autocompletion, or to check whether a word is already in use before introducing a variant of it. + +=== "Python" + + ```python + fs.get_all_keywords() + ``` + +## Command line + +```bash +hops fg keywords transactions_4h_aggs_fraud_batch_fg +hops fg add-keyword transactions_4h_aggs_fraud_batch_fg fraud +hops fg remove-keyword transactions_4h_aggs_fraud_batch_fg fraud +``` + +The same three commands exist for feature views (`hops fv`) and training datasets (`hops td`). + +!!! warning "`--value` on the keyword commands is being removed" + `--value` used to make `hops fg add-keyword` write a tag rather than a keyword. + It is now rejected with a message naming `hops fg add-tag` as the replacement, and it is removed in the following release. + Attach a tag with the tag commands, described in the [Tags][tags-guide] guide. diff --git a/docs/user_guides/fs/tags/mandatory_tags.md b/docs/user_guides/fs/tags/mandatory_tags.md index e1f43000d3..770f442ddb 100644 --- a/docs/user_guides/fs/tags/mandatory_tags.md +++ b/docs/user_guides/fs/tags/mandatory_tags.md @@ -3,15 +3,15 @@ ## Introduction Mandatory tags let a Hopsworks administrator require that specific tag schemas are populated on artifacts. -They build on top of [tags](tags.md) and are used to enforce governance rules, such as requiring every model to declare a data owner. +They build on top of [tags][tags-guide] and are used to enforce governance rules, such as requiring every model to declare a data owner. A mandatory tag is a tag schema that has been marked as required for one or more artifact types. -The supported artifact types are feature groups, feature views, training datasets, models and deployments. +The supported artifact types are feature groups, feature views, training datasets, models, deployments and datasets. ## Prerequisites A mandatory tag references an existing tag schema. -Define the tag schema first, as described in the [Tags](tags.md) guide, before marking it mandatory. +Define the tag schema first, as described in the [Tags][tags-guide] guide, before marking it mandatory. Only administrators can configure mandatory tags. Attaching the tag values afterwards is done by any project member with write access to the artifact. @@ -31,12 +31,15 @@ Mandatory tags are configured in two scopes. Configure project-specific mandatory tags under Cluster settings, Projects, by editing a project For each mandatory tag you select the artifact types it applies to. -A tag schema can be mandatory for any combination of feature groups, feature views, training datasets, models and deployments. -For example, a `data_owner` schema can be marked mandatory for models and deployments only, leaving feature groups, feature views and training datasets unaffected. +A tag schema can be mandatory for any combination of feature groups, feature views, training datasets, models, deployments and datasets. +For example, a `data_owner` schema can be marked mandatory for models and deployments only, leaving the others unaffected. + +A [deprecated][tag-schema-lifecycle] schema cannot be registered as mandatory, because nobody is allowed to attach it. +Conversely, a schema that is registered as mandatory cannot be deprecated until the registrations are removed. ## Enforcement per artifact type -All five artifact types, feature groups, feature views, training datasets, models and deployments, enforce mandatory tags the same way. +All six artifact types, feature groups, feature views, training datasets, models, deployments and datasets, enforce mandatory tags the same way. The create request is validated against the configured mandatory tags. If any mandatory tag is missing from the tags provided at creation, the artifact is not created and the request is rejected with an HTTP 400 error that lists the missing tag names. @@ -129,8 +132,45 @@ The `tags` argument takes the same shape as feature group tags: a `{"name": ..., deployment.save() ``` +=== "Dataset (Python)" + + ```python + dataset_api = project.get_dataset_api() + + # data_owner is mandatory for datasets; pass it when creating the dataset + dataset_api.mkdir( + "transactions_raw", + tags=[{"name": "data_owner", "value": "email@hopsworks.ai"}], + ) + ``` + Omitting a mandatory tag from the `tags` argument rejects the create request with an HTTP 400 error that lists the missing tag names. +### Datasets + +Mandatory tags on datasets apply to the dataset itself, which is a top-level directory in the project's file system. +They are not applied per file: a policy that had to be satisfied by every uploaded file would make every upload a violation. + +The policy applies to datasets a user creates. +The datasets Hopsworks creates when it provisions a project, such as `Resources`, `Logs` and the training dataset and deployment directories, are exempt, because a cluster-wide policy would otherwise make project creation fail. + +A Hopsworks client older than the release that introduced this feature cannot send tags on the create call, so it receives the HTTP 400 listing the missing tags once a dataset policy is enabled. +This is the same contract feature groups have, and it is why enabling a dataset policy is an explicit administrator action. + +A dataset that predates the policy stays valid. +Its missing tags are reported on read through the `missingMandatoryTags` field, and shown on the dataset page in the UI. + +=== "Python" + + ```python + dataset_api = project.get_dataset_api() + + dataset = dataset_api.get("transactions_raw") + print(dataset.get("missingMandatoryTags", [])) + + dataset_api.add("transactions_raw", "data_owner", "email@hopsworks.ai") + ``` + ## Missing mandatory tags on pre-existing artifacts Marking a tag mandatory does not retroactively reject artifacts that already exist without it. diff --git a/docs/user_guides/fs/tags/tag_schema_lifecycle.md b/docs/user_guides/fs/tags/tag_schema_lifecycle.md new file mode 100644 index 0000000000..b662880542 --- /dev/null +++ b/docs/user_guides/fs/tags/tag_schema_lifecycle.md @@ -0,0 +1,134 @@ +# Tag Schema Lifecycle + +## Introduction + +A tag schema is defined once and then attached to artifacts across every project on the cluster, as described in the [Tags][tags-guide] guide. +This guide covers what an administrator can do with a schema afterwards: retire it so that nothing new uses it, and remove it once nothing does. + +The JSON definition of a schema cannot be changed after it is created. +The values already attached were validated against the original definition, and editing it would leave those values unvalidated. +Create a new schema instead, deprecate the old one, and migrate at your own pace. + +Deprecating, restoring and deleting a schema are administrator actions, performed in the `Cluster settings` > `Tag schemas` section or through the REST API. +Attaching and detaching tag values stays with the project members. + +## Deprecate a schema + +Deprecation stops a schema from being attached to anything new while leaving everything already attached untouched. +Use it when a schema is being replaced and you want existing owners to migrate off it without breaking their pipelines. + +| Operation on a deprecated schema | Allowed | +| --- | --- | +| Read the schema, list schemas, read attached values | Yes | +| Attach it to an artifact that does not have it | No, rejected with HTTP 409 and error code 370011 | +| Update the value already attached to an artifact | Yes, so a team can correct data while migrating off | +| Detach it from an artifact | Yes | +| Register it as a mandatory tag | No, because a mandatory tag that nobody may attach cannot be satisfied | +| Delete the schema | Yes, under the rules below | +| Restore it | Yes | + +A schema that is currently registered as a [mandatory tag][mandatory-tags] cannot be deprecated. +The request is rejected with HTTP 409 and error code 370012. +Remove the mandatory registrations first, then deprecate. + +In the UI, deprecated schemas carry a muted `Deprecated` badge with the date, and the schema detail drawer shows which administrator deprecated it. +The tag picker on an artifact page no longer offers them, while a deprecated tag that is already attached still renders so it can be edited or removed. + +=== "Python" + + ```python + import hopsworks + from hopsworks_common.core.tag_schemas_api import TagSchemasApi + + + hopsworks.login() + tag_schemas = TagSchemasApi() + + tag_schemas.deprecate("data_privacy_v1") + + # Existing attachments keep working; new ones are refused. + schema = tag_schemas.get("data_privacy_v1") + print(schema["deprecated"], schema["deprecatedOn"], schema["deprecatedBy"]) + ``` + +## Restore a schema + +Restoring clears the deprecation and the record of who set it and when. +Those three fields describe the current state rather than a history, so a restored schema looks like one that was never deprecated. +The audit log remains the record of the full sequence of actions. + +=== "Python" + + ```python + tag_schemas.restore("data_privacy_v1") + ``` + +## Check what uses a schema + +Before deleting a schema, ask what still references it. + +=== "Python" + + ```python + usage = tag_schemas.usage("data_privacy_v1") + + print(usage["attachmentCount"]) # values attached across all projects + print(usage["mandatoryRegistrationCount"]) # mandatory registrations + print(usage["deletable"]) # both counts are zero + for ref in usage.get("references", []): + print(ref["kind"], ref["projectName"], ref["artifactName"], ref["version"]) + ``` + +`references` lists the artifacts holding a value, up to a limit. +Above that limit the list is omitted and `truncated` is `true`, leaving `attachmentCount` as the answer. + +`complete` is always `false`. +Tags attached to individual files inside a dataset before per-file tags were frozen live in the file system rather than the database, and no query can enumerate them without walking the file system. +The counts therefore cover database-backed references only. +This is why the UI wording is "No database-backed references found" rather than "not used". + +## Delete a schema + +A schema can be deleted once nothing references it. +The request is refused with HTTP 409 and error code 370010 while any value is attached or any mandatory registration exists, and the error names what to detach. + +The check and the deletion run in one transaction against the database, and foreign keys refuse the deletion even if a value is attached between the check and the commit. +The dialog in the UI is advisory; the database is authoritative, so a delete that looked safe a moment earlier can still be refused. + +=== "Python" + + ```python + usage = tag_schemas.usage("data_privacy_v1") + if usage["deletable"]: + tag_schemas.delete("data_privacy_v1") + ``` + +### Force delete + +`force=True` deletes the schema together with every value attached to it, in a single transaction, and queues the affected artifacts for reindexing so that search stops returning the deleted tag. +It is exposed on the REST API and in the Python client, and deliberately not in the UI. + +=== "Python" + + ```python + tag_schemas.delete("data_privacy_v1", force=True) + ``` + +Above a threshold of attached values, configured cluster-wide and defaulting to 5000, a force delete is refused with HTTP 409 rather than started. +The error states the count and names the reindex endpoint. +A partial delete that leaves stale values in the search index is worse than a refusal, and reindexing that many documents is an operation an administrator should schedule. + +### Deleting against an older backend + +A plain `delete` against a Hopsworks version older than the one described here is refused by the client rather than sent. +That backend ignores the unknown `force` parameter and deletes every attached value for any delete call, so the safest-looking call would get the most destructive behaviour the old server has. +The client detects it through the usage endpoint, which shipped in the same release as the refusal, and raises a `RuntimeError` naming `force=True` as the way to proceed deliberately. + +## Command line + +The same two lifecycle actions are available in the CLI. + +```bash +hops tags deprecate data_privacy_v1 +hops tags restore data_privacy_v1 +``` diff --git a/docs/user_guides/fs/tags/tags.md b/docs/user_guides/fs/tags/tags.md index 6fc582829c..3ae4950215 100644 --- a/docs/user_guides/fs/tags/tags.md +++ b/docs/user_guides/fs/tags/tags.md @@ -1,8 +1,8 @@ -# Tags +# Tags { #tags-guide } ## Introduction -Hopsworks feature store enables users to attach tags to artifacts, such as feature groups, feature views, training datasets, models or deployments. +Hopsworks enables users to attach tags to artifacts, such as feature groups, feature views, training datasets, models, deployments, jobs or datasets. A tag is a `{key: value}` pair which provides additional information about the data managed by Hopsworks. Tags allow you to design custom metadata for your artifacts. @@ -18,9 +18,10 @@ Schemas are defined globally across all projects. When users attach tags to an artifact, the tag will be validated against a specific schema. This allows tags to be consistent no matter the project or the team generating them. -!!! warning "Immutable" - Tag schemas are immutable. - Once defined, a tag schema cannot be edited nor deleted. +!!! warning "Schema definitions cannot be edited" + The JSON schema of a tag schema cannot be changed after it is created, because the values already attached were validated against the original definition. + A schema can be deprecated so that it accepts no new attachments, and it can be deleted once nothing references it. + Both are administrator actions and are described in the [Tag schema lifecycle][tag-schema-lifecycle] guide. ## Step 1: Define a tag schema @@ -73,7 +74,7 @@ Where the type is a valid primitive type: `string`, `boolean`, `integer`, `numbe ## Step 2: Attach a tag to an artifact -Once the tag schema has been created, you can attach a tag with that schema to a feature group, feature view, training dataset, model or deployment either using the APIs, or by using the UI. +Once the tag schema has been created, you can attach a tag with that schema to a feature group, feature view, training dataset, model, deployment, job or dataset, either using the APIs or the UI. ### Using the API @@ -122,6 +123,62 @@ Finally you can remove a tag from a given artifact by calling the `delete_tag()` The same APIs work for feature views, training datasets, models and deployments alike. +#### Jobs + +Jobs carry tags through the same three methods, on the `Job` object returned by the job API. + +=== "Python" + + ```python + job_api = project.get_job_api() + job = job_api.get_job("transactions_ingestion") + + job.add_tag("data_privacy", {"business_unit": "Fraud", "pii": True}) + job.get_tags() + job.delete_tag("data_privacy") + ``` + +A job also has a free-text description, which is indexed for search alongside its tags. + +=== "Python" + + ```python + job.description = "Hourly ingestion of card transactions" + job.save() + ``` + +#### Datasets + +Tags are attached to a dataset, which is a top-level directory in the project's file system. +They are reached through the dataset API by path. + +=== "Python" + + ```python + dataset_api = project.get_dataset_api() + + dataset_api.add("Resources", "data_privacy", {"business_unit": "Fraud", "pii": True}) + dataset_api.get_tags("Resources") + dataset_api.delete("Resources", "data_privacy") + ``` + +Tags can also be attached when the dataset is created. + +=== "Python" + + ```python + dataset_api.mkdir( + "transactions_raw", + tags=[{"name": "data_privacy", "value": {"business_unit": "Fraud", "pii": True}}], + ) + ``` + +!!! warning "Tags on files inside a dataset are frozen" + Tags could previously be attached to any file or directory inside a dataset. + Attaching a new tag to a path inside a dataset is now rejected with HTTP 400 and error code 370013, because per-file tags were stored outside the database and could not be searched, counted, or governed. + Tags that were already attached to such paths remain readable and deletable, and the file browser keeps showing them. + Attach the tag to the dataset instead. + ### Using the UI You can attach tags to feature groups and feature views directly from the UI. @@ -135,11 +192,33 @@ From there you can select the tag schema of the tag you want to attach and popul

+## When a tag was attached + +Every tag attached after the upgrade that introduced this feature records the time it was attached. +`get_tags` returns values only, so the attachment time is exposed through a second pair of methods that keep the tag objects. + +=== "Python" + + ```python + fg = fs.get_feature_group("transactions_4h_aggs_fraud_batch_fg", version=1) + + tags = fg.get_tags_metadata() # dict[str, Tag] + for name, t in tags.items(): + print(name, t.value, t.created_on) + + one = fg.get_tag_metadata("data_privacy") + ``` + +`get_tag_metadata` and `get_tags_metadata` are available on feature groups, feature views, training datasets, models, deployments and jobs, and on the dataset API as `get_tag_metadata(path, name)` and `get_tags_metadata(path)`. +`created_on` is `None` for tags that were attached before the upgrade, because the attachment time was not recorded then. +The UI shows the same value under the tag name on the artifact page. + ## Step 3: Search -Hopsworks indexes the tags attached to feature groups, feature views and training datasets. +Hopsworks indexes the tags attached to feature groups, feature views, training datasets, models, deployments, jobs and datasets. The tags will then be searchable using the free text search box located at the top of the UI. -Tags attached to models and deployments are stored and retrievable through the APIs and the UI, but they are not indexed for free text search. +For jobs the search also covers the job name and description, and for datasets the dataset name and description. +Model and deployment documents are written by Hopsworks itself, so a tag change on a model or a deployment is reflected in search rather than leaving a stale answer behind.

@@ -147,3 +226,21 @@ Tags attached to models and deployments are stored and retrievable through the A
Search for tags in the feature store

+ +Jobs and datasets can also be searched from the Python client. + +=== "Python" + + ```python + search_api = project.get_search_api() + + for job_meta in search_api.jobs("ingestion"): + print(job_meta.name, job_meta.job_type) + job = job_meta.get() # the Job object, resolved in the hit's own project + + for ds_meta in search_api.datasets("transactions"): + print(ds_meta.name, ds_meta.path) + ``` + +Both accept a `tag_filter` and a `global_search` flag, like the feature group search. +Jobs and datasets carry no keywords, so a `keyword_filter` never matches them. diff --git a/mkdocs.yml b/mkdocs.yml index b53bc4becc..946ca20691 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -134,6 +134,8 @@ nav: - Sharing: user_guides/fs/sharing/sharing.md - Tags: user_guides/fs/tags/tags.md - Mandatory Tags: user_guides/fs/tags/mandatory_tags.md + - Tag Schema Lifecycle: user_guides/fs/tags/tag_schema_lifecycle.md + - Keywords: user_guides/fs/tags/keywords.md - Provenance: user_guides/fs/provenance/provenance.md - Feature Monitoring: - user_guides/fs/feature_monitoring/index.md @@ -272,6 +274,7 @@ nav: - IAM Role Chaining: setup_installation/admin/roleChaining.md - Configure Project Mapping: setup_installation/admin/configure-project-mapping.md - Airflow 3 operator notes: setup_installation/admin/airflow3.md + - Dataset Tag Storage: setup_installation/admin/dataset_tag_storage.md - Monitoring: - Services Dashboards: setup_installation/admin/monitoring/grafana.md - Export metrics: setup_installation/admin/monitoring/export-metrics.md