diff --git a/docs/concepts/projects/search.md b/docs/concepts/projects/search.md index 4750bde12f..6b7b35ffa2 100644 --- a/docs/concepts/projects/search.md +++ b/docs/concepts/projects/search.md @@ -2,7 +2,7 @@ description: "Documentation on the Hopsworks capabilities to discover machine-learning assets" --- -## Search +## Search { #search-concept } Hopsworks supports free-text search to discover machine-learning assets: @@ -10,15 +10,31 @@ Hopsworks supports free-text search to discover machine-learning assets: - feature groups - feature views - training data +- jobs, including apps +- models +- deployments, including agents You can use the search bar at the top of your project to free-text search for the names or descriptions of any ML asset. You can also search using keywords or tags that are attached to an ML asset. +Tags are indexed for every asset in the list above, so a governance question can be asked once across the whole set rather than per asset type. +Keywords apply to feature groups, feature views and training data only, so a keyword filter never matches a job, a model or a deployment. + +Apps and agents are reported as their own classes but are not stored as their own kind of asset. +An app is a job whose type is PythonApp, and an agent is a deployment serving no registered model. +Each is a narrowing of the class it belongs to, which is why they need no separate index and appear the moment the distinguishing property does. + You can search for assets within a specific project or across all projects in a Hopsworks deployment, including those you are not a member of. This allows for easier discoverability and reusability of assets within an organization. To avoid users gaining unauthorized access to data, if a search result is in a project you are **not** a member of, the information displayed is limited to: names, descriptions, tags, asset creator and create date. If the search result is within a project you are a member of, you are also able to inspect recent activities on the asset as well as statistics. +Searching across projects you are not a member of is on by default. +An administrator running a multi-tenant deployment can turn it off, which restricts every search to the projects the caller can already access. +See [search index administration][search-index-administration] for the setting. + +For how to use search, including filtering by a specific tag key and value, see the [search guide][search-guide]. + ## Tags A keyword is a single user-defined word attached to an ML asset. diff --git a/docs/setup_installation/admin/search_index.md b/docs/setup_installation/admin/search_index.md new file mode 100644 index 0000000000..783d87a8ee --- /dev/null +++ b/docs/setup_installation/admin/search_index.md @@ -0,0 +1,93 @@ +# Search Index Administration { #search-index-administration } + +## Introduction + +Hopsworks keeps a search index of the artifacts in a deployment: feature groups, feature views, training datasets, jobs, models and deployments. +The index is updated from the database, not written to directly. +Every change to a searchable property queues a command in the same transaction that made the change, and a background executor applies the queued commands to OpenSearch. + +That design is what makes the index eventually consistent rather than immediately correct, and it is the thing to understand before diagnosing a stale search result. +A tag that was attached seconds ago and does not appear in search yet is normal. +A tag that has not appeared after minutes means a command is stuck, which this page covers. + +Only administrators can reach any of this. + +## Commands needing attention + +Go to `Cluster Settings` > `Service Operations` and open the `OpenSearch Index Commands` tab. +The page opens on the `Service Operations` tab, which carries the platform's operation logs; the index commands share the page because both answer whether the platform is digesting what it was asked to do. + +You do not need to check the tab on the off-chance. +While any command needs attention, the `Service Operations` entry in the settings menu shows a pulsing orange dot and the `OpenSearch Index Commands` tab is outlined in orange. +Both appear and clear on their own as the queue fails and drains, so no dot means there is nothing to look at. + +The tab lists every command that has failed at least once, oldest first. + +For each it shows the document, the artifact type, the operation, the number of attempts made, when the next attempt is due, the project, and the last error. +When the list is empty, every queued update has been applied. + +The reason a stuck command matters more than one failure suggests is ordering. +Commands for a single document are applied in order, so while one keeps failing, every later update to that same document waits behind it, and that artifact's search result stays as it was. +Other documents are unaffected. + +Retries are automatic and unbounded, with the delay growing between attempts. +Unbounded is deliberate: abandoning a command would not lose only its own update, it would strand every later update to that document. +So a command in this list is usually a transient failure that will clear itself, and the list is worth acting on when an entry stops making progress. + +The page makes that call for you at 20 attempts. +A command that has failed 20 times or more is marked `stuck`, with a red border and a callout explaining what to do. +The threshold is where the retry delay has been at its cap for a while, so crossing it means roughly an hour of the same failure: whatever a retry could outwait has had its chance. +Read the stuck command's error first, because a failure whose cause has since been fixed clears itself on the next retry and needs nothing from you. +Otherwise cancel it, as described below. + +### Cancelling a command + +Cancel a command only to let the ones behind it through. + +The update the cancelled command carried is lost. +The document keeps whatever the index already held, so cancelling an `UPDATE_TAGS` leaves the old tags visible in search until something changes that artifact again and queues a fresh command. +That is why cancelling is an explicit operator action and never something the executor does on its own. + +A command can only be cancelled while it is `FAILED`, which is the state between attempts where nothing owns the row. +A command that has just been picked up for another attempt cannot be cancelled, and the request is refused with that reason rather than interrupting the attempt. +Wait for the attempt to finish and retry. + +Each cancellation is logged with the command id, the document, the attempt count, the requesting administrator and the last error, because once the row is gone the log is the only record it existed. + +## Rebuilding the index + +A reindex rebuilds the search index from the database. +It is the recovery path after the index has been lost or has diverged, not routine maintenance. + +```bash +# queue a reindex, returns the run id +curl -X POST -H "Authorization: Bearer $JWT" \ + "https://$HOPSWORKS_HOST/hopsworks-api/api/admin/search/featurestore/reindex" + +# follow that run +curl -H "Authorization: Bearer $JWT" \ + "https://$HOPSWORKS_HOST/hopsworks-api/api/admin/search/featurestore/reindex/$RUN_ID" +``` + +The `POST` returns `200` with the run id rather than an empty `204`. +The run id is what makes progress observable: without it a caller can only watch the global command queue, which mixes the reindex with whatever ordinary use is queueing alongside it. + +Reindexing queues a command per document, so on a large deployment it takes a while and shows up as a long queue. +That is expected, and ordinary updates continue to be applied while it drains. + +## Configuration + +Set these in `Cluster Settings` > `Configuration`. + +| Variable | Default | Effect | +| --- | --- | --- | +| `cross_project_global_search_enabled` | `true` | Whether a search may return artifacts from projects the caller is not a member of. Set `false` on a multi-tenant deployment to restrict every search to the projects the caller can already access. | +| `command_search_fs_retry_backoff_base_as_ms` | `5000` | The delay before the first retry of a failed command. The delay doubles per attempt from here. | +| `command_search_fs_retry_backoff_max_as_ms` | `300000` | The ceiling on that growing delay, so a long-failing command keeps retrying at a fixed interval rather than drifting to never. | + +Lowering the backoff makes a transient failure clear sooner at the cost of more load against OpenSearch while it is failing. +Raising it does the opposite. + +!!! warning "Turning off cross-project search does not rewrite history" + `cross_project_global_search_enabled` filters searches as they are made. + It does not remove anything from the index, so switching it off restricts what users can find from that point on and is not a way to redact something already indexed. diff --git a/docs/user_guides/fs/tags/keywords.md b/docs/user_guides/fs/tags/keywords.md new file mode 100644 index 0000000000..9c1f067e39 --- /dev/null +++ b/docs/user_guides/fs/tags/keywords.md @@ -0,0 +1,130 @@ +# Keywords { #keywords-guide } + +## Introduction + +A keyword is a single free-form word attached to a feature group, a feature view or a training dataset. +Keywords need no schema and no administrator: any project member with write access can invent one and attach it. + +That is the difference from [tags][tags-guide], and it decides which to reach for. +A tag is validated against a schema and is the right tool for governance, where the set of allowed keys and values has to be agreed in advance. +A keyword is the right tool for discovery, where the point is to label something now and find it later. + +Keywords apply to feature groups, feature views and training datasets only. +Jobs, apps, models and deployments take tags but not keywords, so a keyword filter never matches them. + +## Read the keywords of an artifact + +=== "Python" + + ```python + fg = fs.get_feature_group("transactions_4h_aggs_fraud_batch_fg", version=1) + + fg.get_keywords() + # ['fraud', 'aggregations', 'hourly'] + ``` + +To see when each keyword was attached, use `get_keywords_metadata()`, which returns a dict of keyword to attachment time: + +=== "Python" + + ```python + for keyword, attached in fg.get_keywords_metadata().items(): + print(keyword, attached) + ``` + +The attachment time is an aware UTC `datetime`, or `None` when it is unknown. +It is `None` for keywords attached before the cluster began recording attachment times, so `None` does not mean the keyword is new. + +## Add, replace and delete keywords + +Three methods change the keyword set, and they differ in what they do to the keywords already there. + +`add_keywords()` adds to the set and leaves the rest alone. +It accepts one keyword or a list: + +=== "Python" + + ```python + fg.add_keywords("fraud") + fg.add_keywords(["aggregations", "hourly"]) + ``` + +`set_keywords()` replaces the whole set. +Anything not in the list you pass is removed, so use it when you intend the artifact to end up with exactly these keywords and nothing else: + +=== "Python" + + ```python + fg.set_keywords(["fraud", "hourly"]) + ``` + +`delete_keyword()` removes a single keyword: + +=== "Python" + + ```python + fg.delete_keyword("hourly") + ``` + +All three return the resulting keyword set, so a read-back is not needed to see the effect. + +The same methods exist on feature views. +A training dataset's keywords are reached through its feature view, because a training dataset is identified by the feature view it was created from plus its own version: + +=== "Python" + + ```python + fv = fs.get_feature_view("fraud_detection", version=1) + + fv.add_training_dataset_keywords(1, "baseline") + fv.get_training_dataset_keywords(1) + fv.delete_training_dataset_keyword(1, "baseline") + ``` + +## The cluster vocabulary + +Keywords are free-form, which makes them prone to near-duplicates: `fraud`, `Fraud` and `fraud_detection` are three separate keywords that fragment the same idea. +To let you reuse a word someone has already chosen, the feature store can list every keyword in use: + +=== "Python" + + ```python + fs.get_all_keywords() + ``` + +The vocabulary is cluster-wide rather than project-scoped, so it shows words in use in projects you are not a member of. +Only the words are returned, never which artifact or project they came from, so this discloses no artifact you could not otherwise see. + +## Command line + +The CLI covers the same operations: + +```bash +# feature group keywords +hops fg keywords transactions_fg --version 1 +hops fg add-keyword transactions_fg fraud --version 1 +hops fg remove-keyword transactions_fg fraud --version 1 + +# feature view keywords +hops fv keywords fraud_detection --version 1 +hops fv add-keyword fraud_detection baseline --version 1 +hops fv remove-keyword fraud_detection baseline --version 1 + +# training dataset keywords, addressed by feature view name and td version +hops td keywords fraud_detection 1 --fv-version 1 +hops td add-keyword fraud_detection 1 baseline --fv-version 1 +hops td remove-keyword fraud_detection 1 baseline --fv-version 1 +``` + +The listing commands show the attachment time next to each keyword. + +!!! warning "The `*-keyword` commands changed meaning" + They used to operate on tags, which are name and value pairs. + They now operate on keywords, which are plain labels, and tags moved to `hops fg tags`, `hops fg add-tag` and `hops fg remove-tag`. + A script that passed a tag value to `add-keyword` needs to move to `add-tag`, because a keyword has no value to pass. + The commands print this notice to stderr when they run. + +## Searching by keyword + +Keywords are indexed, and can be filtered on directly rather than only matched as free text. +See the [tag and keyword search guide][search-with-tags-and-keywords]. diff --git a/docs/user_guides/fs/tags/tags.md b/docs/user_guides/fs/tags/tags.md index 6fc582829c..c45474f0cd 100644 --- a/docs/user_guides/fs/tags/tags.md +++ b/docs/user_guides/fs/tags/tags.md @@ -1,8 +1,8 @@ -# Tags +# Tags { #tags-guide } ## Introduction -Hopsworks feature store enables users to attach tags to artifacts, such as feature groups, feature views, training datasets, models or deployments. +Hopsworks feature store enables users to attach tags to artifacts, such as feature groups, feature views, training datasets, jobs, apps, models or deployments. A tag is a `{key: value}` pair which provides additional information about the data managed by Hopsworks. Tags allow you to design custom metadata for your artifacts. @@ -71,9 +71,61 @@ You can achieve this by defining a JSON schema like the following: Where the type is a valid primitive type: `string`, `boolean`, `integer`, `number`. +### Archiving a tag for analytics + +Most tags are only ever read as they are now: who owns this feature group, whether it holds PII. +Some are interesting over time, and for those the current value is the least useful part. +Marking a schema as archived says that attachments of this tag are worth keeping once they stop being current, so the tag's history can be analysed and not just its present state. + +Tick `Archive deleted tags` when defining the schema, or pass `archive=True` through the API: + +=== "Python" + + ```python + from hopsworks.core.tag_schemas_api import TagSchemasApi + + + schema = { + "type": "object", + "properties": {"state": {"type": "string"}}, + "required": ["state"], + "additionalProperties": False, + } + + TagSchemasApi().create("asset_lifecycle", schema, archive=True) + ``` + +The flag is a property of the schema rather than of any one attachment, which is why it is set where the schema is defined and applies to every tag attached with it afterwards. +It defaults to `False`, which discards an attachment once it stops being current. +Registering a schema requires administrator privileges, as it does without the flag. + +#### What it is for + +Take an `asset_lifecycle` tag whose `state` is `dev`, `qa` or `prod`. +Every artifact carries it, and the value moves forward as the artifact is promoted. + +Read as an ordinary tag it answers one question, which is where an artifact is now. +The questions worth asking are about the pipeline rather than the artifact: how long does something sit in `qa` before it reaches `prod`, is that getting slower, whose artifacts stall. + +Those become answerable once the tag's history is kept as one record per value, each with the time that value became current. +The analysis is then a group-by over the artifact: order its `dev`, `qa` and `prod` records by time, and the gaps between them are how long it spent in each stage. +Aggregated across every artifact, that gives the promotion times for the deployment as a whole and how they are moving. + +Without archiving, promoting an artifact to `prod` discards the record that it was ever in `qa`, and the question stops being answerable at all. +So the flag is worth setting on a tag whose values are states an artifact passes through, rather than facts about it. + +This history is not the same thing as the [attachment time][when-a-tag-was-attached] on the live tag. +That timestamp deliberately stays at the first attachment when a value is corrected, so it records when an artifact was first classified and not when it entered its current state. +The per-value history is what the archive is for. + +!!! note "Records intent, no behaviour yet" + Setting `archive` today only records the decision on the schema. + Nothing reads it: copying the retained attachments into an offline feature group, where they can be queried as above, is a later change. + Set it now on the schemas whose history you expect to want, because the flag cannot recover attachments that were already discarded while it was off. + ## Step 2: Attach a tag to an artifact -Once the tag schema has been created, you can attach a tag with that schema to a feature group, feature view, training dataset, model or deployment either using the APIs, or by using the UI. +Once the tag schema has been created, you can attach a tag with that schema to a feature group, feature view, training dataset, job, app, model or deployment either using the APIs, or by using the UI. ### Using the API @@ -122,6 +174,39 @@ Finally you can remove a tag from a given artifact by calling the `delete_tag()` The same APIs work for feature views, training datasets, models and deployments alike. +#### Jobs and apps + +Jobs and apps carry tags through the same three methods, reached from the job handle rather than the feature store: + +=== "Python" + + ```python + jobs_api = project.get_jobs_api() + job = jobs_api.get_job("transactions_ingest") + + job.add_tag("data_privacy", {"business_unit": "Fraud", "pii": True}) + job.get_tags() + job.delete_tag("data_privacy") + ``` + +An app is a job whose type is `PYTHON_APP`, so an app is tagged exactly the same way, through the handle its name resolves to. + +The CLI covers the same three operations: + +```bash +hops job tags transactions_ingest +hops job add-tag transactions_ingest data_privacy --value '{"business_unit": "Fraud", "pii": true}' +hops job remove-tag transactions_ingest data_privacy +``` + +`--value` takes JSON for a schema with properties, or a plain string for a single-property schema. + +Tags on a job are also editable in the UI, on the job's `Tags` section, and when creating or editing the job. +For an app, the equivalent section is on the app overview page. + +Deleting a job deletes its tags with it. +They are not restored by creating a new job under the same name, because the tags belong to the job that was deleted and not to its name. + ### Using the UI You can attach tags to feature groups and feature views directly from the UI. @@ -135,11 +220,39 @@ From there you can select the tag schema of the tag you want to attach and popul

+## When a tag was attached + +Hopsworks records the time each tag was attached and reports it alongside the value. +`get_tags()` returns values only, so read the attachment time through the `_metadata` variants, which return `Tag` objects instead of bare values: + +=== "Python" + + ```python + fg = fs.get_feature_group("transactions_4h_aggs_fraud_batch_fg", version=1) + + tag = fg.get_tag_metadata("data_privacy") + print(tag.value, tag.created_on) + + # every tag on the artifact, keyed by name + for name, attached in fg.get_tags_metadata().items(): + print(name, attached.created_on) + ``` + +`created_on` is an aware UTC `datetime`, and the same methods exist on feature views, training datasets and jobs. + +The timestamp records when the tag was **attached**, not when its value last changed. +Re-attaching a tag to change its value keeps the original attachment time, so the value can be corrected without losing the record of when the artifact was first classified. + +`created_on` is `None` when the attachment time is unknown rather than recent. +That happens for tags attached before the cluster recorded attachment times, and for legacy per-file dataset tags, which are stored as HopsFS extended attributes and carry no timestamp. + ## Step 3: Search -Hopsworks indexes the tags attached to feature groups, feature views and training datasets. -The tags will then be searchable using the free text search box located at the top of the UI. -Tags attached to models and deployments are stored and retrievable through the APIs and the UI, but they are not indexed for free text search. +Hopsworks indexes the tags attached to feature groups, feature views, training datasets, jobs, models and deployments. +The tags are then searchable using the free text search box located at the top of the UI, and can be filtered on directly. +See the [tag and keyword search guide][search-with-tags-and-keywords] for filtering by a specific tag key and value rather than by free text. + +Tags on artifacts of every indexed class are searchable, so a governance question such as "which artifacts are missing a data owner" can be answered across feature groups, jobs and deployments in one query.

diff --git a/docs/user_guides/projects/search.md b/docs/user_guides/projects/search.md new file mode 100644 index 0000000000..7d76fc2f9e --- /dev/null +++ b/docs/user_guides/projects/search.md @@ -0,0 +1,112 @@ +# Search { #search-guide } + +## Introduction + +Hopsworks indexes your artifacts so you can find them by name, description, [tag][tags-guide] or [keyword][keywords-guide]. +This guide covers the search UI and the equivalent REST call. +For what search is and how its scope relates to project membership, see the [search concept page][search-concept]. + +## What you can search + +Search returns ten classes of artifact, each on its own tab: + +| Tab | What it contains | +| --- | --- | +| All | Every class below, in one view. The default. | +| Feature Groups | Feature groups. | +| Feature Views | Feature views. | +| Training Datasets | Training datasets. | +| Features | Individual features, matched by feature name. | +| Jobs | Jobs, excluding those that are apps. | +| Apps | Jobs of type PythonApp. | +| Models | Models in a model registry. | +| Deployments | Deployments that serve a registered model. | +| Agents | Deployments that serve no registered model. | + +Apps and agents are not separate kinds of artifact, which is why they are not separate tabs in the sense the others are. +An app is a job, and an agent is a deployment. +They appear as their own tabs because the question "which agents are tagged for production" is worth asking on its own, and each tab excludes the other: a job that is an app is reported under Apps and not under Jobs. + +Each tab shows the number of matches next to its name, so you can see where the results are without visiting every tab. + +## Free-text search + +The search box at the top of the UI matches names and descriptions, and the content of tags and keywords. +Matches are highlighted in the results, including the tag key and value that matched, so it is clear why a result was returned. + +## Search with tags and keywords + +Free text cannot express "the tag `data_privacy` has `pii` set to `true`", because it matches text anywhere in the document. +For that, turn on `Search with Tags & Keywords` above the results. + +The panel opens beside the results, and has four rows: + +- `Selected:` shows every filter currently applied, each removable on its own, with a `Clear search` action for all of them. +- `Tag:` is a three-column browser: pick a tag schema, then a key within it, then a value for that key. +- `Keyword:` takes a keyword and adds it with `Add keyword filter`. +- `Free-text search:` adds words to the free-text part of the query, with `Add` or by pressing Enter. + +Filters combine, so a tag filter and a keyword filter together return only artifacts matching both. + +### Only what exists is offered + +The three tag columns each have a `filter` box, which matters because a cluster can hold more values than are worth scrolling. +Entries that no artifact you can see actually uses are greyed out and cannot be selected, with the hint `No available matching assets in Hopsworks that use this tag/key/value`. + +This is drawn from the tags in use on indexed artifacts, not from the schema definitions. +A schema permitting a value nobody has ever attached will show that value as unavailable, which is deliberate: selecting it could only ever return nothing. + +The offered vocabulary respects the scope you are searching, so it never reveals a tag value used only in a project you cannot see. + +### Clearing a search + +`Clear search results` above the results removes every filter and the free-text term, and resets the per-tab counts. +The same action is in the filter panel as `Clear search`, but the panel is closed by default, so the button above the results is the one to reach for after searching from the text box. + +## Search from the API + +The REST endpoint is per project and takes the class as `docType`: + +```bash +curl -H "Authorization: ApiKey $API_KEY" \ + "https://$HOPSWORKS_HOST/hopsworks-api/api/project/$PROJECT_ID/elastic/featurestore?searchTerm=fraud&docType=ALL" +``` + +`docType` accepts `ALL`, `FEATURE`, `FEATUREGROUP`, `FEATUREVIEW`, `TRAININGDATASET`, `JOB`, `APP`, `MODEL`, `DEPLOYMENT` and `AGENT`, and defaults to `ALL`. +`from` and `size` page the results, with `size` capped at 10000. + +Tag and keyword filters are JSON arrays in the `tags` and `keywords` query parameters. +A tag filter names the schema, and optionally a key and a value within it: + +```json +[{"name": "data_privacy", "key": "pii", "value": "true"}] +``` + +A request must carry at least one of `searchTerm`, `tags` or `keywords`. +Without any of them it is rejected with a `422`, because there is no "match everything" search: the result would be every artifact on the cluster. + +The response carries one bucket per class, each with its own total, for example `featuregroups` with `featuregroupsTotal` and `apps` with `appsTotal`. + +### API key scopes + +Search results are filtered to the scopes of the API key you use, so a key cannot discover a class it was not minted for. +A `FEATURESTORE` key sees feature groups, feature views, training datasets and features, `JOB` sees jobs and apps, `MODELREGISTRY` sees models, and `SERVING` sees deployments and agents. + +Naming a `docType` the key does not carry the scope for is rejected, rather than returned empty, so a missing scope is distinguishable from a genuinely empty result. +A `docType=ALL` request is instead narrowed to the classes the key does carry, which is what makes `ALL` usable from a single-scope key. + +A search made with a JWT, as the UI and the Python client do after logging in, is not scope-restricted; it is limited by project membership alone. + +### The tag vocabulary in use + +The endpoint behind the greyed-out entries is available directly: + +```bash +curl -H "Authorization: ApiKey $API_KEY" \ + "https://$HOPSWORKS_HOST/hopsworks-api/api/project/$PROJECT_ID/elastic/featurestore/tagfacets" +``` + +It returns the tags, keys and values attached to artifacts within your search scope. + +The answer is read from a bounded number of documents, so on a large cluster it can be incomplete. +When it is, the response sets `partial` to `true`, which means the vocabulary shown is a subset and a value missing from it may still exist. diff --git a/mkdocs.yml b/mkdocs.yml index e7a7223d1d..2fe6db28ed 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -134,6 +134,7 @@ nav: - Sharing: user_guides/fs/sharing/sharing.md - Tags: user_guides/fs/tags/tags.md - Mandatory Tags: user_guides/fs/tags/mandatory_tags.md + - Keywords: user_guides/fs/tags/keywords.md - Provenance: user_guides/fs/provenance/provenance.md - Feature Monitoring: - user_guides/fs/feature_monitoring/index.md @@ -154,6 +155,7 @@ nav: - Projects: - Create Project: user_guides/projects/project/create_project.md - Manage Members: user_guides/projects/project/manage_members.md + - Search: user_guides/projects/search.md - Python: - Environments Overview: user_guides/projects/python/python_env_overview.md - Clone Environment: user_guides/projects/python/python_env_clone.md @@ -278,6 +280,7 @@ nav: - IAM Role Chaining: setup_installation/admin/roleChaining.md - Configure Project Mapping: setup_installation/admin/configure-project-mapping.md - Airflow 3 operator notes: setup_installation/admin/airflow3.md + - Search Index: setup_installation/admin/search_index.md - Monitoring: - Services Dashboards: setup_installation/admin/monitoring/grafana.md - Export metrics: setup_installation/admin/monitoring/export-metrics.md