[FSTORE-2096] Attachment timestamps, the keyword API, and tags for jobs, apps and deployments

docs/concepts/projects/search.md

@@ -2,23 +2,39 @@
 description: "Documentation on the Hopsworks capabilities to discover machine-learning assets"
 ---
 
-## Search
+## Search { #search-concept }
 
 Hopsworks supports free-text search to discover machine-learning assets:
 
 - features
 - feature groups
 - feature views
 - training data
+- jobs, including apps
+- models
+- deployments, including agents
 
 You can use the search bar at the top of your project to free-text search for the names or descriptions of any ML asset.
 You can also search using keywords or tags that are attached to an ML asset.
 
+Tags are indexed for every asset in the list above, so a governance question can be asked once across the whole set rather than per asset type.
+Keywords apply to feature groups, feature views and training data only, so a keyword filter never matches a job, a model or a deployment.
+
+Apps and agents are reported as their own classes but are not stored as their own kind of asset.
+An app is a job whose type is PythonApp, and an agent is a deployment serving no registered model.
+Each is a narrowing of the class it belongs to, which is why they need no separate index and appear the moment the distinguishing property does.
+
 You can search for assets within a specific project or across all projects in a Hopsworks deployment, including those you are not a member of.
 This allows for easier discoverability and reusability of assets within an organization.
 To avoid users gaining unauthorized access to data, if a search result is in a project you are **not** a member of, the information displayed is limited to: names, descriptions, tags, asset creator and create date.
 If the search result is within a project you are a member of, you are also able to inspect recent activities on the asset as well as statistics.
 
+Searching across projects you are not a member of is on by default.
+An administrator running a multi-tenant deployment can turn it off, which restricts every search to the projects the caller can already access.
+See [search index administration][search-index-administration] for the setting.
+
+For how to use search, including filtering by a specific tag key and value, see the [search guide][search-guide].
+
 ## Tags
 
 A keyword is a single user-defined word attached to an ML asset.

docs/setup_installation/admin/search_index.md

@@ -0,0 +1,93 @@
+# Search Index Administration { #search-index-administration }
+
+## Introduction
+
+Hopsworks keeps a search index of the artifacts in a deployment: feature groups, feature views, training datasets, jobs, models and deployments.
+The index is updated from the database, not written to directly.
+Every change to a searchable property queues a command in the same transaction that made the change, and a background executor applies the queued commands to OpenSearch.
+
+That design is what makes the index eventually consistent rather than immediately correct, and it is the thing to understand before diagnosing a stale search result.
+A tag that was attached seconds ago and does not appear in search yet is normal.
+A tag that has not appeared after minutes means a command is stuck, which this page covers.
+
+Only administrators can reach any of this.
+
+## Commands needing attention
+
+Go to `Cluster Settings` > `Service Operations` and open the `OpenSearch Index Commands` tab.
+The page opens on the `Service Operations` tab, which carries the platform's operation logs; the index commands share the page because both answer whether the platform is digesting what it was asked to do.
+
+You do not need to check the tab on the off-chance.
+While any command needs attention, the `Service Operations` entry in the settings menu shows a pulsing orange dot and the `OpenSearch Index Commands` tab is outlined in orange.
+Both appear and clear on their own as the queue fails and drains, so no dot means there is nothing to look at.
+
+The tab lists every command that has failed at least once, oldest first.
+
+For each it shows the document, the artifact type, the operation, the number of attempts made, when the next attempt is due, the project, and the last error.
+When the list is empty, every queued update has been applied.
+
+The reason a stuck command matters more than one failure suggests is ordering.
+Commands for a single document are applied in order, so while one keeps failing, every later update to that same document waits behind it, and that artifact's search result stays as it was.
+Other documents are unaffected.
+
+Retries are automatic and unbounded, with the delay growing between attempts.
+Unbounded is deliberate: abandoning a command would not lose only its own update, it would strand every later update to that document.
+So a command in this list is usually a transient failure that will clear itself, and the list is worth acting on when an entry stops making progress.
+
+The page makes that call for you at 20 attempts.
+A command that has failed 20 times or more is marked `stuck`, with a red border and a callout explaining what to do.
+The threshold is where the retry delay has been at its cap for a while, so crossing it means roughly an hour of the same failure: whatever a retry could outwait has had its chance.
+Read the stuck command's error first, because a failure whose cause has since been fixed clears itself on the next retry and needs nothing from you.
+Otherwise cancel it, as described below.
+
+### Cancelling a command
+
+Cancel a command only to let the ones behind it through.
+
+The update the cancelled command carried is lost.
+The document keeps whatever the index already held, so cancelling an `UPDATE_TAGS` leaves the old tags visible in search until something changes that artifact again and queues a fresh command.
+That is why cancelling is an explicit operator action and never something the executor does on its own.
+
+A command can only be cancelled while it is `FAILED`, which is the state between attempts where nothing owns the row.
+A command that has just been picked up for another attempt cannot be cancelled, and the request is refused with that reason rather than interrupting the attempt.
+Wait for the attempt to finish and retry.
+
+Each cancellation is logged with the command id, the document, the attempt count, the requesting administrator and the last error, because once the row is gone the log is the only record it existed.
+
+## Rebuilding the index
+
+A reindex rebuilds the search index from the database.
+It is the recovery path after the index has been lost or has diverged, not routine maintenance.
+
+```bash
+# queue a reindex, returns the run id
+curl -X POST -H "Authorization: Bearer $JWT" \
+  "https://$HOPSWORKS_HOST/hopsworks-api/api/admin/search/featurestore/reindex"
+
+# follow that run
+curl -H "Authorization: Bearer $JWT" \
+  "https://$HOPSWORKS_HOST/hopsworks-api/api/admin/search/featurestore/reindex/$RUN_ID"
+```
+
+The `POST` returns `200` with the run id rather than an empty `204`.
+The run id is what makes progress observable: without it a caller can only watch the global command queue, which mixes the reindex with whatever ordinary use is queueing alongside it.
+
+Reindexing queues a command per document, so on a large deployment it takes a while and shows up as a long queue.
+That is expected, and ordinary updates continue to be applied while it drains.
+
+## Configuration
+
+Set these in `Cluster Settings` > `Configuration`.
+
+| Variable | Default | Effect |
+| --- | --- | --- |
+| `cross_project_global_search_enabled` | `true` | Whether a search may return artifacts from projects the caller is not a member of. Set `false` on a multi-tenant deployment to restrict every search to the projects the caller can already access. |
+| `command_search_fs_retry_backoff_base_as_ms` | `5000` | The delay before the first retry of a failed command. The delay doubles per attempt from here. |
+| `command_search_fs_retry_backoff_max_as_ms` | `300000` | The ceiling on that growing delay, so a long-failing command keeps retrying at a fixed interval rather than drifting to never. |
+
+Lowering the backoff makes a transient failure clear sooner at the cost of more load against OpenSearch while it is failing.
+Raising it does the opposite.
+
+!!! warning "Turning off cross-project search does not rewrite history"
+    `cross_project_global_search_enabled` filters searches as they are made.
+    It does not remove anything from the index, so switching it off restricts what users can find from that point on and is not a way to redact something already indexed.

docs/user_guides/fs/tags/keywords.md

@@ -0,0 +1,130 @@
+# Keywords { #keywords-guide }
+
+## Introduction
+
+A keyword is a single free-form word attached to a feature group, a feature view or a training dataset.
+Keywords need no schema and no administrator: any project member with write access can invent one and attach it.
+
+That is the difference from [tags][tags-guide], and it decides which to reach for.
+A tag is validated against a schema and is the right tool for governance, where the set of allowed keys and values has to be agreed in advance.
+A keyword is the right tool for discovery, where the point is to label something now and find it later.
+
+Keywords apply to feature groups, feature views and training datasets only.
+Jobs, apps, models and deployments take tags but not keywords, so a keyword filter never matches them.
+
+## Read the keywords of an artifact
+
+=== "Python"
+
+    ```python
+    fg = fs.get_feature_group("transactions_4h_aggs_fraud_batch_fg", version=1)
+
+    fg.get_keywords()
+    # ['fraud', 'aggregations', 'hourly']
+    ```
+
+To see when each keyword was attached, use `get_keywords_metadata()`, which returns a dict of keyword to attachment time:
+
+=== "Python"
+
+    ```python
+    for keyword, attached in fg.get_keywords_metadata().items():
+        print(keyword, attached)
+    ```
+
+The attachment time is an aware UTC `datetime`, or `None` when it is unknown.
+It is `None` for keywords attached before the cluster began recording attachment times, so `None` does not mean the keyword is new.
+
+## Add, replace and delete keywords
+
+Three methods change the keyword set, and they differ in what they do to the keywords already there.
+
+`add_keywords()` adds to the set and leaves the rest alone.
+It accepts one keyword or a list:
+
+=== "Python"
+
+    ```python
+    fg.add_keywords("fraud")
+    fg.add_keywords(["aggregations", "hourly"])
+    ```
+
+`set_keywords()` replaces the whole set.
+Anything not in the list you pass is removed, so use it when you intend the artifact to end up with exactly these keywords and nothing else:
+
+=== "Python"
+
+    ```python
+    fg.set_keywords(["fraud", "hourly"])
+    ```
+
+`delete_keyword()` removes a single keyword:
+
+=== "Python"
+
+    ```python
+    fg.delete_keyword("hourly")
+    ```
+
+All three return the resulting keyword set, so a read-back is not needed to see the effect.
+
+The same methods exist on feature views.
+A training dataset's keywords are reached through its feature view, because a training dataset is identified by the feature view it was created from plus its own version:
+
+=== "Python"
+
+    ```python
+    fv = fs.get_feature_view("fraud_detection", version=1)
+
+    fv.add_training_dataset_keywords(1, "baseline")
+    fv.get_training_dataset_keywords(1)
+    fv.delete_training_dataset_keyword(1, "baseline")
+    ```
+
+## The cluster vocabulary
+
+Keywords are free-form, which makes them prone to near-duplicates: `fraud`, `Fraud` and `fraud_detection` are three separate keywords that fragment the same idea.
+To let you reuse a word someone has already chosen, the feature store can list every keyword in use:
+
+=== "Python"
+
+    ```python
+    fs.get_all_keywords()
+    ```
+
+The vocabulary is cluster-wide rather than project-scoped, so it shows words in use in projects you are not a member of.
+Only the words are returned, never which artifact or project they came from, so this discloses no artifact you could not otherwise see.
+
+## Command line
+
+The CLI covers the same operations:
+
+```bash
+# feature group keywords
+hops fg keywords transactions_fg --version 1
+hops fg add-keyword transactions_fg fraud --version 1
+hops fg remove-keyword transactions_fg fraud --version 1
+
+# feature view keywords
+hops fv keywords fraud_detection --version 1
+hops fv add-keyword fraud_detection baseline --version 1
+hops fv remove-keyword fraud_detection baseline --version 1
+
+# training dataset keywords, addressed by feature view name and td version
+hops td keywords fraud_detection 1 --fv-version 1
+hops td add-keyword fraud_detection 1 baseline --fv-version 1
+hops td remove-keyword fraud_detection 1 baseline --fv-version 1
+```
+
+The listing commands show the attachment time next to each keyword.
+
+!!! warning "The `*-keyword` commands changed meaning"
+    They used to operate on tags, which are name and value pairs.
+    They now operate on keywords, which are plain labels, and tags moved to `hops fg tags`, `hops fg add-tag` and `hops fg remove-tag`.
+    A script that passed a tag value to `add-keyword` needs to move to `add-tag`, because a keyword has no value to pass.
+    The commands print this notice to stderr when they run.
+
+## Searching by keyword
+
+Keywords are indexed, and can be filtered on directly rather than only matched as free text.
+See the [tag and keyword search guide][search-with-tags-and-keywords].