diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 8bbf2dbc3a..4a38675045 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -3,10 +3,10 @@ ## Commands ```bash -uv venv && uv pip install -r requirements-docs.txt # setup -uv pip install "git+https://github.com/logicalclocks/hopsworks-api.git@main#subdirectory=python" # install Python API (needed for API docs section) +uv sync # setup (dependencies live in pyproject.toml) +uv pip install "hopsworks[python] @ git+https://github.com/logicalclocks/hopsworks-api.git@main#subdirectory=python" # install Python API (needed for API docs section) touch docs/javadoc; uv run mkdocs build -s; rm docs/javadoc # build (strict) -uv run mkdocs serve # preview with live reload +touch docs/javadoc; uv run mkdocs serve; rm docs/javadoc # preview with live reload npx markdownlint-cli2 "**/*.md" # lint Markdown (requires Node.js) uv tool install md-snakeoil && snakeoil --line-length 88 --rules "E,F,B,C4,ISC,PIE,PYI,Q,RSE,RET,SIM,TC,I,W,D2,D3,D4,INP,UP,FA" docs # lint Python code blocks ``` diff --git a/.claude/docs/README.md b/.claude/docs/README.md index 64e4a9f278..98f573d18c 100644 --- a/.claude/docs/README.md +++ b/.claude/docs/README.md @@ -6,8 +6,8 @@ There is no application code — all work is writing Markdown under `docs/` and ## Commands ```bash -uv venv && uv pip install -r requirements-docs.txt # setup -uv pip install "git+https://github.com/logicalclocks/hopsworks-api.git@main#subdirectory=python" # needed for Python API section +uv sync # setup (dependencies live in pyproject.toml) +uv pip install "hopsworks[python] @ git+https://github.com/logicalclocks/hopsworks-api.git@main#subdirectory=python" # needed for Python API section touch docs/javadoc; uv run mkdocs serve; rm docs/javadoc # preview with live reload touch docs/javadoc; uv run mkdocs build -s; rm docs/javadoc # build in strict mode npx markdownlint-cli2 "**/*.md" # lint Markdown (requires Node.js) diff --git a/docs/assets/images/admin/trino/catalogs-pending-restart.png b/docs/assets/images/admin/trino/catalogs-pending-restart.png deleted file mode 100644 index 6d6c39f9ee..0000000000 Binary files a/docs/assets/images/admin/trino/catalogs-pending-restart.png and /dev/null differ diff --git a/docs/assets/images/admin/trino/catalogs-pending.png b/docs/assets/images/admin/trino/catalogs-pending.png index 56e7436802..01d2c9e32d 100644 Binary files a/docs/assets/images/admin/trino/catalogs-pending.png and b/docs/assets/images/admin/trino/catalogs-pending.png differ diff --git a/docs/assets/images/admin/trino/catalogs-recover.png b/docs/assets/images/admin/trino/catalogs-recover.png index 79f577a113..6c824d47a0 100644 Binary files a/docs/assets/images/admin/trino/catalogs-recover.png and b/docs/assets/images/admin/trino/catalogs-recover.png differ diff --git a/docs/assets/images/admin/trino/project-catalog-limit.png b/docs/assets/images/admin/trino/project-catalog-limit.png new file mode 100644 index 0000000000..2297a2370d Binary files /dev/null and b/docs/assets/images/admin/trino/project-catalog-limit.png differ diff --git a/docs/assets/images/admin/trino/restart-confirm.png b/docs/assets/images/admin/trino/restart-confirm.png index 10e01e63c6..53cf72e704 100644 Binary files a/docs/assets/images/admin/trino/restart-confirm.png and b/docs/assets/images/admin/trino/restart-confirm.png differ diff --git a/docs/assets/images/guides/mountable_secrets/mountable-secrets-files.png b/docs/assets/images/guides/mountable_secrets/mountable-secrets-files.png new file mode 100644 index 0000000000..d44f7b52d9 Binary files /dev/null and b/docs/assets/images/guides/mountable_secrets/mountable-secrets-files.png differ diff --git a/docs/assets/images/guides/mountable_secrets/mountable-secrets-list.png b/docs/assets/images/guides/mountable_secrets/mountable-secrets-list.png new file mode 100644 index 0000000000..d59ecef951 Binary files /dev/null and b/docs/assets/images/guides/mountable_secrets/mountable-secrets-list.png differ diff --git a/docs/assets/images/guides/mountable_secrets/mountable-secrets-new.png b/docs/assets/images/guides/mountable_secrets/mountable-secrets-new.png new file mode 100644 index 0000000000..fd352001e0 Binary files /dev/null and b/docs/assets/images/guides/mountable_secrets/mountable-secrets-new.png differ diff --git a/docs/assets/images/guides/mountable_secrets/mountable-secrets-reference.png b/docs/assets/images/guides/mountable_secrets/mountable-secrets-reference.png new file mode 100644 index 0000000000..dee0245f4d Binary files /dev/null and b/docs/assets/images/guides/mountable_secrets/mountable-secrets-reference.png differ diff --git a/docs/assets/images/guides/trino/catalog-created-admin.png b/docs/assets/images/guides/trino/catalog-created-admin.png new file mode 100644 index 0000000000..cfad41780e Binary files /dev/null and b/docs/assets/images/guides/trino/catalog-created-admin.png differ diff --git a/docs/assets/images/guides/trino/catalog-failed.png b/docs/assets/images/guides/trino/catalog-failed.png index cac35cdf08..809bfef305 100644 Binary files a/docs/assets/images/guides/trino/catalog-failed.png and b/docs/assets/images/guides/trino/catalog-failed.png differ diff --git a/docs/assets/images/guides/trino/catalog-pending-approval.png b/docs/assets/images/guides/trino/catalog-pending-approval.png new file mode 100644 index 0000000000..e799af6591 Binary files /dev/null and b/docs/assets/images/guides/trino/catalog-pending-approval.png differ diff --git a/docs/assets/images/guides/trino/catalog-pending-sync.png b/docs/assets/images/guides/trino/catalog-pending-sync.png deleted file mode 100644 index 9da146be09..0000000000 Binary files a/docs/assets/images/guides/trino/catalog-pending-sync.png and /dev/null differ diff --git a/docs/assets/images/guides/trino/catalogs-list.png b/docs/assets/images/guides/trino/catalogs-list.png index 77d37507ce..3a4ca3c3e6 100644 Binary files a/docs/assets/images/guides/trino/catalogs-list.png and b/docs/assets/images/guides/trino/catalogs-list.png differ diff --git a/docs/assets/images/guides/trino/cluster-overview.png b/docs/assets/images/guides/trino/cluster-overview.png index a7f32f8156..c6fd7f7ba1 100644 Binary files a/docs/assets/images/guides/trino/cluster-overview.png and b/docs/assets/images/guides/trino/cluster-overview.png differ diff --git a/docs/assets/images/guides/trino/create-catalog.png b/docs/assets/images/guides/trino/create-catalog.png index 7ffd21fc8a..21c8091f13 100644 Binary files a/docs/assets/images/guides/trino/create-catalog.png and b/docs/assets/images/guides/trino/create-catalog.png differ diff --git a/docs/assets/images/guides/trino/queries.png b/docs/assets/images/guides/trino/queries.png index 4239096403..18234301f5 100644 Binary files a/docs/assets/images/guides/trino/queries.png and b/docs/assets/images/guides/trino/queries.png differ diff --git a/docs/assets/images/guides/trino/query-details-json.png b/docs/assets/images/guides/trino/query-details-json.png index a8b5b6866d..cfc8476690 100644 Binary files a/docs/assets/images/guides/trino/query-details-json.png and b/docs/assets/images/guides/trino/query-details-json.png differ diff --git a/docs/assets/images/guides/trino/query-details-live-plan.png b/docs/assets/images/guides/trino/query-details-live-plan.png index a77144d64e..db803e31f5 100644 Binary files a/docs/assets/images/guides/trino/query-details-live-plan.png and b/docs/assets/images/guides/trino/query-details-live-plan.png differ diff --git a/docs/assets/images/guides/trino/query-details-ref.png b/docs/assets/images/guides/trino/query-details-ref.png index d2b44d5c53..11b4335582 100644 Binary files a/docs/assets/images/guides/trino/query-details-ref.png and b/docs/assets/images/guides/trino/query-details-ref.png differ diff --git a/docs/assets/images/guides/trino/query-details-split.png b/docs/assets/images/guides/trino/query-details-split.png index 162b948af9..571fa76f7f 100644 Binary files a/docs/assets/images/guides/trino/query-details-split.png and b/docs/assets/images/guides/trino/query-details-split.png differ diff --git a/docs/assets/images/guides/trino/query-details-stage.png b/docs/assets/images/guides/trino/query-details-stage.png index 1ebaae5cd7..6f2afaa465 100644 Binary files a/docs/assets/images/guides/trino/query-details-stage.png and b/docs/assets/images/guides/trino/query-details-stage.png differ diff --git a/docs/assets/images/guides/trino/query-details.png b/docs/assets/images/guides/trino/query-details.png index 97d970fb00..ce22e7a510 100644 Binary files a/docs/assets/images/guides/trino/query-details.png and b/docs/assets/images/guides/trino/query-details.png differ diff --git a/docs/assets/images/guides/trino/query-engine.png b/docs/assets/images/guides/trino/query-engine.png index 9965fe0830..1a3842ad10 100644 Binary files a/docs/assets/images/guides/trino/query-engine.png and b/docs/assets/images/guides/trino/query-engine.png differ diff --git a/docs/assets/images/guides/trino/sql-runner.png b/docs/assets/images/guides/trino/sql-runner.png index 8dd09f6502..efa04a9426 100644 Binary files a/docs/assets/images/guides/trino/sql-runner.png and b/docs/assets/images/guides/trino/sql-runner.png differ diff --git a/docs/assets/images/guides/trino/sql-statement-syntax.png b/docs/assets/images/guides/trino/sql-statement-syntax.png index cdc8cfef72..22b7f5879a 100644 Binary files a/docs/assets/images/guides/trino/sql-statement-syntax.png and b/docs/assets/images/guides/trino/sql-statement-syntax.png differ diff --git a/docs/assets/images/guides/trino/test-connection-failed.png b/docs/assets/images/guides/trino/test-connection-failed.png index 829cd7fdbe..aff7f4c8f0 100644 Binary files a/docs/assets/images/guides/trino/test-connection-failed.png and b/docs/assets/images/guides/trino/test-connection-failed.png differ diff --git a/docs/assets/images/guides/trino/test-connection-success.png b/docs/assets/images/guides/trino/test-connection-success.png index b528a53e3f..9942a76d0d 100644 Binary files a/docs/assets/images/guides/trino/test-connection-success.png and b/docs/assets/images/guides/trino/test-connection-success.png differ diff --git a/docs/setup_installation/admin/trino.md b/docs/setup_installation/admin/trino.md index 5b5c7ec352..b214aad5df 100644 --- a/docs/setup_installation/admin/trino.md +++ b/docs/setup_installation/admin/trino.md @@ -92,7 +92,7 @@ With the schedule on, a pending catalog goes live at the next restart that finds
Pending catalogs -
Catalogs waiting to be applied
+
The lifecycle settings, the catalogs waiting to be applied, and the one action that applies them
### Applying pending requests @@ -109,16 +109,13 @@ The restart interrupts queries running anywhere on the cluster, so check the rep A connector's credentials end up in the places below. Anyone who can read those places can read the credentials, so plan access to them accordingly. -- A `${HOPSWORKS_SECRET:}` reference is stored verbatim in the `trino_catalog` database row and is resolved to its value only at sync time. The database row never holds the value. -- A literal value typed straight into the properties editor is stored as-is in the `trino_catalog` database row, in cleartext, and is captured by database backups. Use a secret reference for any credential you do not want in the database. -- Either way, the synced file holds the resolved plaintext, because Trino reads the credential from the catalog file itself. +- A `${HOPSWORKS_SECRET:}` reference is stored verbatim in the `trino_catalog` database row and is resolved to its value only at approval time. + The database row never holds the value. +- A literal value typed straight into the properties editor is stored as-is in the `trino_catalog` database row, in cleartext, and is captured by database backups. + Use a secret reference for any credential you do not want in the database. +- Either way, the written file holds the resolved plaintext, because Trino reads the credential from the catalog file itself. That file lives in a Kubernetes Secret rather than a ConfigMap, so it is covered by the RBAC that applies to Secrets in the Hopsworks namespace and by etcd encryption-at-rest on clusters that enable it. -
- Catalogs pending restart -
An applied catalog waits in Pending restart until the query engine reloads
-
- ### Lifecycle settings The same tab carries the **Catalog lifecycle** card, where the whole schedule is configured and saved as one group: @@ -139,7 +136,12 @@ Saving needs no redeploy: every Hopsworks instance derives its schedule from the ### A single project's allowance The cluster-wide maximum is a ceiling on the deployment; how many catalogs any one project may create is set per project. -Open the project under Cluster Settings, Projects, and edit **Query Engine**, **Trino catalogs**, which shows the project's current count beside its limit. +Open **Cluster Settings** → **Projects**, click **Edit configuration** on the project's row, and scroll to **Query engine**, **Trino catalogs**, just after the Kafka topic quota. + +
+ A project's Trino catalog allowance +
The project's catalog count, its own limit, and the unlimited checkbox
+
A new project starts on the cluster default (`trino_catalog_max_per_project`, 10), so raising one project here raises that project only. Checking **unlimited** removes the project's own bound, leaving only the cluster-wide ceiling; a limit of 0 blocks new catalogs in the project. @@ -157,16 +159,16 @@ The bounded wait is what makes that safe: a coordinator that is genuinely down n Trino reads catalogs only at startup, so a catalog change takes effect on the next restart, whether the schedule performs it or an administrator does. Clicking "Restart Trino" applies the selected pending requests and rolls out the coordinator and workers. -The confirmation dialog reports how many queries are currently running or queued, so you can choose a low-traffic window before confirming. +The confirmation dialog reports how many queries are currently running or queued, so you can choose a low-traffic window, and asks you to type `confirm` before it will proceed. The restart cancels those queries for **every project on the cluster**, not only the project whose catalog is being applied, and in-flight results are lost. Trino keeps recent query detail in the coordinator's memory, so after a restart the live query views show only what the new coordinator has seen; older queries remain in the query history, which is stored separately.
Restart confirmation -
The restart confirmation reports the running queries the restart will interrupt
+
The confirmation names what the restart applies and what it interrupts
-A restart is refused while another sync or restart is already running, so concurrent actions by different administrators cannot collide or trigger redundant restarts. +A restart is refused while another one is already running, so concurrent actions by different administrators cannot collide or trigger redundant restarts. If nothing is waiting to load or unload, the restart is skipped and reported as such rather than interrupting queries for no reason. ### Recovering a catalog Trino cannot load @@ -180,7 +182,7 @@ The button stays available when nothing is waiting to be applied, because this s
Recover restart -
With nothing pending, the restart action is still available to recover a failed rollout
+
A query engine that will not start is reported on the tab, and the restart action recovers it
Hopsworks reads the coordinator log, identifies the catalog Trino rejected, removes it from the mount, marks it **Failed**, and restarts so the cluster comes back without it. @@ -192,7 +194,7 @@ Only user-created catalogs are removed this way. A default catalog that fails to load is left in place, because that is a cluster configuration problem rather than something an administrator should resolve by deleting data. A removed catalog keeps its row, so its owner can see what happened on the project's Catalogs page along with the error Trino reported. -Editing the definition returns it to Pending sync and it re-enters the normal flow. +Editing the definition returns it to Pending approval and it re-enters the normal flow. Failed catalogs are not listed under pending, because they no longer block anything and no administrator action can fix them. If Hopsworks cannot attribute the failure to a user catalog, it reports the connection error instead of removing anything, and the coordinator log is the place to look. @@ -215,13 +217,93 @@ The consequence is that the scheduled pass does not notice a file whose name is Two things neither form does. Neither restarts Trino, so a restored catalog is in the mount but not loaded until the next restart, like any other catalog change. -Neither touches a catalog that is pending sync, because that catalog's stored definition is the change an administrator has not applied yet, and applying it here would bypass that decision. -Those catalogs are reported as still needing a sync. +Neither touches a catalog that is pending approval, because that catalog's stored definition is the change an administrator has not approved yet, and applying it here would bypass that decision. +Those catalogs are reported as still needing approval. A catalog whose `${HOPSWORKS_SECRET:}` reference no longer resolves cannot be rebuilt, since the file Trino reads has to hold the resolved value. -The repair reports it, leaves any file it already has in place, because that copy resolved when it was synced and still works, and carries on with every other catalog. +The repair reports it, leaves any file it already has in place, because that copy resolved when it was approved and still works, and carries on with every other catalog. Its owner has to repoint the reference at an existing secret. +## Credential files a project supplies + +A connector that authenticates with a file, such as an Oracle wallet or a Java keystore, cannot be served by a catalog property alone. +Projects supply those files as [mountable secrets][mountable-secrets], and this section covers what that adds to a cluster. + +A bundle is a directory of files in HopsFS under `mountable_secrets_path`, which defaults to `/apps/mountable-secrets`. +It is keyed by **project id** rather than by project name, so a deleted project and a later project of the same name can never share a directory. +The `charts/hopsfs` preset Job creates the root as `payara:hdfs` with mode `0750`. +The backend checks that owner, group and mode against the filesystem on a project's first bundle, and again before it deletes a project's tree, and refuses if any of the three differs, so a root created by hand with the wrong mode is rejected rather than quietly widening access. + +Project members never reach those files directly. +The path is outside any project's dataset, and a catalog can only ever name a bundle in its own project. +A reference resolves to a path built from the project id, and a property that tries to extend a reference with a path, or to walk out of it with `..`, is refused when the catalog is created and again when it is resolved. + +### How the files reach the query engine + +Each Trino pod, the coordinator, every worker and the test coordinator, runs a sidecar container named `mountable-secrets` from the `hopsfs-mount` image. +It mounts the whole store read-only at `trino_mountable_secrets_root`, which defaults to `/opt/hopsworks/mounts`, with `ro`, `nosuid` and `nodev`. +Entries appear as uid 0 with modes that let any user read them, which is what lets the unprivileged Trino process open a wallet. + +Two consequences of that sidecar are worth knowing before an upgrade. + +It is **privileged**, because FUSE requires it. +On a cluster running the Kyverno restricted policies the chart ships a `PolicyException` for these pods, gated on Kyverno being enabled. +The same privileged FUSE sidecar already runs on the three Airflow deployments in the release namespace, so this is not a new class of workload for the cluster. + +The Trino pods have their **own ServiceAccounts**, `hopsworks-trino` and `hopsworks-trino-test`, rather than the namespace default. +An SCC or a cloud identity can therefore be granted to Trino narrowly. +An upgrade from a release before this feature moves those pods off the `default` ServiceAccount, so any binding that named `default` to reach Trino has to be repointed. + +!!! warning "OpenShift is not supported" + The sidecar has to run privileged and as root, so the default restricted SCC rejects it. + `values.openshift.yaml` therefore turns the store off, and a project on such a cluster cannot supply credential files. + Note that Trino was already rejected by the restricted SCC before this feature, because the subchart pins `runAsUser: 1000` regardless of `securityContextEnabled`, so the sidecar adds a second reason rather than a new break. + +### Turning the store off + +Set `global._hopsworks.trino.mountableSecrets.enabled` to `false`, which seeds the `mountable_secrets_enabled` variable and stops the store being offered. +Turning it off is not a single value. +The sidecar entries live in untemplated subchart values, so the `initContainers` lists have to be restated without them, which is what `values.openshift.yaml` does and is the worked example to copy. +The chart fails the render when the flag and the mount disagree, so a half-done change stops the upgrade instead of producing pods that mount nothing. + +An **already approved catalog keeps working only as far as its definition**. +Its reference still resolves to a path, but nothing populates that path any more. +For a connector that opens its files when a connection is made, such as Oracle, the coordinator starts cleanly and queries fail. +Writing a catalog out does not consult the flag, by design, so switching the store off does not quarantine catalogs that already use it. + +### Backup + +Bundles are HopsFS files. +They are covered by the HopsFS backup, and **not** by the Kubernetes object backup that captures the catalog Secrets and the database. +A restore that brings back the database and the Secrets without the HopsFS path leaves catalogs that reference bundles which no longer exist, and those catalogs fail to authenticate at the next restart. +Recreating the bundle under the same name with the same filenames repairs it without editing any catalog. + +### Diagnosing a bundle + +There is no admin API for the store, so the checks are on the cluster. + +```bash +# What the query engine can actually see for project +kubectl exec -n hopsworks -c -- ls -l /opt/hopsworks/mounts// + +# The mount itself, including its options +kubectl exec -n hopsworks -c -- grep /opt/hopsworks/mounts /proc/mounts + +# The source side +kubectl exec -n hopsworks -- /srv/hops/hadoop/bin/hdfs dfs -ls /apps/mountable-secrets/ +``` + +Check the mount on a **worker** and not only on the coordinator, since a query reads the source from the workers. +A missing mount is otherwise invisible: the sidecar mounts into its own filesystem, both containers report ready, and only a `${HOPSWORKS_MOUNT:...}` reference resolving to an empty directory gives it away. + +The outbound addresses a data source must admit are reported in the project's Catalogs tab only when `global._hopsworks.trino.mountableSecrets.egressProbe.echoUrl` is set. +It is empty by default, because the probe otherwise calls a third-party service from every Trino pod on every start, and with it unset the UI reports that the addresses could not be determined. +Without it: + +```bash +kubectl exec -n hopsworks -c -- curl -s https://ifconfig.me +``` + ## Configuration Trino behavior can be customized through cluster configuration variables. To modify these settings, navigate to **Cluster Settings** → **Configuration** and search for the variable name. @@ -253,10 +335,40 @@ Trino behavior can be customized through cluster configuration variables. To mod These settings control the availability and default behavior of the Trino query engine across your Hopsworks cluster. +### Mountable secret settings + +These are not all editable the same way, so they are listed apart from the variables above. + +Three are seeded by the chart and belong to Helm, not to the variables table. + +| Setting | Helm value | Seeded default | +| --- | --- | --- | +| `mountable_secrets_enabled` | `global._hopsworks.trino.mountableSecrets.enabled` | `true` | +| `mountable_secrets_path` | `global._hopsworks.trino.mountableSecrets.storeRoot` | `/apps/mountable-secrets` | +| `trino_mountable_secrets_root` | `global._hopsworks.trino.mountableSecrets.mountPath` | `/opt/hopsworks/mounts` | + +Change these through your Helm values and an upgrade. +Editing the row instead moves only one end of the arrangement: the store root also presets the HopsFS directory and is passed to the mount sidecar as its source, and the mount root is what the Trino containers actually mount, so a row edited on its own points the backend at a path nothing is mounted from. +The chart keeps the two ends together, and refuses to render when the flag and the mount disagree. +Note also that the code's own fallback for the flag is `false`, which is what a cluster whose chart predates the row gets; the chart seeds `true`. + +The five per-project limits have no seeded row at all. +The code's defaults apply until an administrator creates one, so searching for them in Cluster Settings finds nothing on a fresh cluster, which is expected rather than a fault. + +| Setting | Default | What it caps | +| --- | --- | --- | +| `mountable_secret_max_per_project` | `10` | bundles one project may hold | +| `mountable_secret_max_files` | `32` | files in one bundle | +| `mountable_secret_max_file_bytes` | `1048576` | largest single file, in bytes | +| `mountable_secret_max_project_bytes` | `16777216` | a project's total across all its bundles, in bytes | +| `max_mountable_secret_upload_bytes` | `33554432` | largest upload request, refused before the body is read | + +Turning the store off is described in [Turning the store off][turning-the-store-off]. + ### Test coordinator resource cost `trino_test_coordinator_enabled` is on by default, and enabling it runs **an additional single-node Trino coordinator pod** for the lifetime of the cluster. -It exists only to connection-test user catalogs before they are synced, so on a small or cost-sensitive cluster it is reasonable to turn it off. +It exists only to connection-test user catalogs before they are approved, so on a small or cost-sensitive cluster it is reasonable to turn it off. When it is off, "Test connection" reports that testing is unavailable and every other part of the catalog workflow is unaffected. ### Supported connectors @@ -275,7 +387,7 @@ Removing a connector from the list does not affect catalogs already created on i User-created catalogs are stored across a fixed number of Kubernetes Secrets, set by the Helm value `global._hopsworks.trino.userCatalogShards` (default: `2`). Each Secret holds up to roughly 800 KiB of catalog definitions, so the default gives about 1.6 MiB in total, which is a large number of catalogs. -When they are full, a sync fails with an error naming the limit. +When they are full, an approval fails with an error naming the limit. Raise the value in your Helm values to add capacity. The chart mounts one source per shard and refuses to render if the two disagree, so a mismatch fails the upgrade rather than silently dropping catalogs. @@ -283,7 +395,7 @@ The chart mounts one source per shard and refuses to render if the two disagree, Two per-catalog limits keep one project from consuming that shared budget. `trino_catalog_max_per_project` caps how many catalogs a project may create, and `trino_catalog_max_bytes` caps how large a single definition may be. The size is measured after `${HOPSWORKS_SECRET:}` references are resolved, because the resolved form is what occupies a Secret: a stored definition is bounded by its database column, but a reference costs a couple of dozen characters and expands to a secret of up to about 10 KiB, and the same secret may be referenced repeatedly, so a row that fits its column can resolve to megabytes. -The check therefore runs both when a catalog is created, so its owner hears about it, and again at sync, because a secret can be rotated to a larger value in between. +The check therefore runs both when a catalog is created, so its owner hears about it, and again at approval, because a secret can be rotated to a larger value in between. Both defaults are generous against real catalogs, which are a few hundred bytes; the largest legitimate ones inline a service account JSON or a certificate pair and stay a few KiB. Raise them for a project with an unusual number of external sources, and remember that the product of the two bounds a single project's share of the shard budget. diff --git a/docs/user_guides/projects/mountable_secrets/mountable_secrets.md b/docs/user_guides/projects/mountable_secrets/mountable_secrets.md new file mode 100644 index 0000000000..61d39f753e --- /dev/null +++ b/docs/user_guides/projects/mountable_secrets/mountable_secrets.md @@ -0,0 +1,145 @@ +--- +description: Supply the credential files a Trino connector needs, such as an Oracle wallet or a Java keystore, and reference them from a catalog. +--- + +# Mountable Secrets + +Some connectors authenticate with a file rather than with a password. +An Oracle Autonomous Database over `tcps` needs a wallet directory, Elasticsearch, MongoDB and Cassandra can need a keystore, and BigQuery and GCS need a key file. +A catalog property cannot name a path on the query engine's machines, so a project needs a way to put its own files where the connector will look for them. + +A **mountable secret** is a named bundle of files that belongs to your project. +You upload the files once, then refer to the bundle by name from a catalog property, and Hopsworks substitutes the real location when the catalog is written for Trino. +The files are stored where project members cannot read or write them directly, and a catalog can only ever reach its own project's bundles. + +Only a project Data Owner can list, create or delete mountable secrets. +Through the API the same endpoints need an API key with the `MOUNTABLE_SECRET` scope. + +## Creating a bundle + +Open **Project Settings**, then **Mountable Secrets**, and click **New**. +Give the bundle a name and add its files, either by selecting the files individually or by uploading a single zip archive, which is the form a downloaded wallet usually arrives in. + +
+ Mountable secrets +
The project's bundles, what each one holds, which catalogs use it, and how much of the budget is used
+
+ +
+ New mountable secret +
Creating a bundle from individual files or from a zip
+
+ +The whole bundle is created in one step. +There is no way to add a file to a bundle afterwards, or to replace one, which is what makes a bundle safe to reference: it is either complete or absent, and it cannot change under a catalog that is using it. + +A name starts with a letter or a digit and continues with letters, digits, dots, underscores or hyphens, up to 63 characters. +Names that could not be written into a catalog property are rejected, so a leading dot, a space or `..` will not be accepted. + +A zip must hold its files flat. +An archive whose entries sit inside a directory is rejected, because the bundle is the directory, and a wallet nested one level down would not be found by a driver pointed at it. +Empty files are rejected too, as is the same filename twice in one request. + +These limits apply per project. +All of them are cluster settings an administrator can raise. + +| Limit | Default | +| --- | --- | +| Bundles per project | 10 | +| Files per bundle | 32 | +| Bytes per file | 1 MiB | +| Bytes per project | 16 MiB | +| Bytes per upload request | 32 MiB | + +The listing shows what a bundle holds, with a SHA-256 for every file, when it was uploaded, and which catalogs use it. +File contents are never returned: once uploaded, a file can be referenced and deleted, but not read back. +The hash is there so you can tell which file is present without reading it. + +
+ Files in a mountable secret +
Names, sizes and hashes are visible; contents are not
+
+ +!!! warning "Treat a bundle as readable by the cluster, not by your project alone" + Names, sizes, hashes and timestamps are visible to every Data Owner in the project. + More importantly, where the cluster runs a Trino test coordinator, every mountable secret on it is readable from that coordinator, because the store is mounted whole and the test coordinator connection-tests catalogs before they go live. + Prefer a credential scoped to the data the project needs over an administrative one. + +## Referencing a bundle from a catalog + +Two forms are available, and which one a connector wants depends on whether it reads a directory or a single named file. + +```text +${HOPSWORKS_MOUNT:my_bundle} # the bundle directory +${HOPSWORKS_MOUNT:my_bundle/keystore.jks} # one file inside it +``` + +Type `${` in the catalog properties editor to pick a bundle, or a file inside one, from a list. + +
+ Referencing a bundle from a catalog property +
Typing ${HOPSWORKS_MOUNT: offers the bundle directory and the files in it
+
+ +A reference stands on its own and cannot be extended with a path. +Writing `${HOPSWORKS_MOUNT:my_bundle}/keystore.jks` is rejected, because the file form above already expresses it, and allowing a path after a reference would let a property address something outside the bundle. +For the same reason a reference cannot contain `..`. + +References are checked when you create or edit the catalog, and again when you test the connection. +A bundle or a file that does not exist is reported at that point rather than at the next restart. + +## Changing or removing a bundle + +To change a wallet, delete the bundle and create it again under the same name. +The listing names the catalogs that reference a bundle, so check there before removing one. + +A deletion takes effect immediately and is never refused for being in use, and it does not wait for a restart to bite. +The query engine sees the store through a live mount, and a connector that reads its files when it opens a connection, Oracle among them, will fail on its next connection or query. +Recreating the bundle under the same name with the same filenames restores it, and no catalog has to be edited, because a catalog refers to the bundle by name. +Queries can fail in the gap between the two. + +Where an interruption is unacceptable, do not replace a bundle in place. +Create the new one under a new name, edit the catalog to reference it, and delete the old bundle once the query engine has restarted and the catalog is working. + +## Worked example: an Oracle Autonomous Database + +Download the wallet from the OCI console, upload the zip as a bundle called `oracle_wallet`, then create an `oracle` catalog whose connection URL points `TNS_ADMIN` at the bundle directory. + +```properties +connection-url=jdbc:oracle:thin:@dbname_low?TNS_ADMIN=${HOPSWORKS_MOUNT:oracle_wallet} +connection-user=TRINO +connection-password=${HOPSWORKS_SECRET:oracle_password} +``` + +Three things about this URL cause most of the failures. + +**The name before `?` is a TNS alias, not a service name.** +It has to be one of the aliases in the wallet's own `tnsnames.ora`, such as `dbname_low` or `dbname_high`, and not the service name shown in the OCI console. +A name that is not in the file produces `Could not find alias in tnsnames.ora`, which is a wallet-contents problem rather than a connectivity one. + +**The database's access control list has to admit the cluster.** +`ORA-12506` has two causes that look identical from the outside: the connection came from an address the Autonomous Database does not accept, or the client never loaded the wallet at all. +Add the outbound addresses of every query engine pod, coordinator and workers, since a query runs on the workers. +The Catalogs tab reports those addresses when an administrator has enabled the check. + +**A downloaded wallet retries by default.** +Each alias in `tnsnames.ora` carries `(retry_count=20)(retry_delay=3)` inside its connect descriptor, so a refused connection waits about a minute before any error appears and a rejected address looks like a hang. +The driver accepts a connect descriptor in place of an alias, so to get the real error at once, paste the descriptor from the alias you were using into `connection-url` and set `retry_count=0` there. +The wallet still authenticates, through `TNS_ADMIN`. + +```properties +connection-url=jdbc:oracle:thin:@(description=(retry_count=0)(address=(protocol=tcps)(port=1522)(host=))(connect_data=(service_name=))(security=(ssl_server_dn_match=yes)))?TNS_ADMIN=${HOPSWORKS_MOUNT:oracle_wallet} +connection-user= +connection-password=${HOPSWORKS_SECRET:oracle_password} +``` + +Take the host, port and `service_name` from the alias's entry in the wallet's `tnsnames.ora`, and drop `retry_delay`, which means nothing once `retry_count` is zero. +A catalog can keep this form, and doing so records which consumer group it connects to instead of leaving it to an alias name. + +## When the feature is unavailable + +An administrator can turn the store off for a whole cluster. +While it is off, the Mountable Secrets page reports that it is not available, and creating or editing a catalog that references a bundle is refused. + +A catalog that already went live keeps its stored definition, and its reference still resolves to a location, but nothing populates that location any more. +For a connector that reads its files when a connection is opened, such as Oracle, the query engine starts normally and queries fail. diff --git a/docs/user_guides/projects/trino/catalogs.md b/docs/user_guides/projects/trino/catalogs.md index 8059241899..9af0e019f0 100644 --- a/docs/user_guides/projects/trino/catalogs.md +++ b/docs/user_guides/projects/trino/catalogs.md @@ -8,8 +8,17 @@ A Trino catalog makes an external data source queryable from the query engine. Each catalog names a Trino connector and the properties that connector needs to reach the source, such as a connection URL and credentials. Once a catalog is live, its databases and tables can be queried from the SQL runner alongside your feature groups. -Creating, updating, or deleting a catalog records the change immediately, but the change only takes effect once the query engine restarts and reloads its catalogs. -Every screen that changes a catalog therefore tells you when that will happen: at the next scheduled restart, or immediately if a cluster administrator restarts the query engine. +Navigate to **Query Engine** → **Catalogs** in your project to see the project's catalogs together with the cluster's shared default catalogs. +A catalog you create is named `__` and is queryable only inside your own project. +Only a project Data Owner can create, edit or delete one. + +
+ Catalogs list +
The project's catalogs alongside the cluster's shared default catalogs
+
+ +A catalog change is recorded immediately, but it reaches the query engine only when the engine restarts, because Trino reads catalogs at startup. +[When the catalog goes live][when-the-catalog-goes-live] covers when that happens and who can bring it forward. ## Creating a catalog from a data source @@ -24,58 +33,147 @@ When you browse a data source's databases and tables to create an external featu Both are disabled with the reason when the data source cannot be mapped to a Trino connector, and show an **Already added** state when a catalog for this source already exists. The catalog is created first, so you can review its properties before the feature groups and their ingestion are set up. -The **Create Trino Catalog** dialog opens pre-filled with: - -- A suggested catalog name, prefixed with your project name in lowercase. - The prefix is required, so catalogs from different projects cannot collide on one query engine. -- The Trino connector type derived from the data source. -- The connector properties derived from the data source's settings. - You can edit them and add any property the connector supports that the data source does not carry. +The **Create Trino Catalog** dialog opens pre-filled with a suggested name, the connector type derived from the data source, and the connector properties derived from its settings. +You can edit them and add any property the connector supports that the data source does not carry. ### Credentials are references, not copies -Credential properties are pre-filled as references rather than values: +Credential properties are pre-filled as [references][referencing-a-credential-instead-of-typing-it] rather than values. +The credential is read from the data source server-side and stored as a secret owned by you, or as a bundle of files where the credential is a file, and the catalog keeps only the reference. +An Oracle data source that authenticates with a wallet is delivered the second way: the bundle is built from the data source's own wallet, and the catalog's `TNS_ADMIN` property points at it. -- `${HOPSWORKS_SECRET:}` references a Hopsworks secret. - When the catalog is created, the credential is read from the data source and stored as a secret owned by you, and the catalog keeps only the reference. -- `${HOPSWORKS_MOUNT:}` references a credential-file bundle, used when the credential is a file rather than a value. - An Oracle data source that authenticates with a wallet is delivered this way: the bundle is built from the data source's own wallet, and the catalog's `TNS_ADMIN` property points at it. +No credential is ever sent to the browser, and rotating one stays a single operation on the secret rather than an edit of every catalog that uses it. -No credential is ever sent to the browser, and rotating a credential stays a single operation on the secret rather than an edit of every catalog that uses it. +## Creating a catalog by hand -### Test connection +A source that has no Hopsworks data source is added by hand. +Click **Create catalog**, give the catalog a short name, choose the connector, and write the properties one `key=value` per line. +The `__` prefix is added for you. -The **Create** button is enabled only after **Test connection** succeeds, where the cluster supports it, so a catalog that cannot reach its source is caught now rather than after a restart. -A failed test shows the query engine's own error message inline. +
+ Create catalog +
Creating a catalog by hand, with the name prefix added automatically
+
-### When the catalog goes live +The picker offers every connector installed in the query engine, so a connector it does not list cannot be used: -After creation, the **Catalog created** dialog shows the current query engine activity and when the catalog becomes queryable: +`bigquery`, `cassandra`, `clickhouse`, `delta_lake`, `druid`, `duckdb`, `elasticsearch`, `exasol`, `faker`, `gsheets`, `hive`, `hudi`, `iceberg`, `ignite`, `kafka`, `lakehouse`, `loki`, `mariadb`, `mongodb`, `mysql`, `opensearch`, `oracle`, `pinot`, `postgresql`, `prometheus`, `redis`, `redshift`, `singlestore`, `snowflake`, `sqlserver`, `trino_thrift`. -- The next scheduled restart, in your own timezone, when the cluster has one configured. -- A **Restart now** action, if you are a cluster administrator, together with the number of queries currently running and queued. - Restarting cancels those queries everywhere on the cluster, so the dialog shows the activity to let you judge whether restarting now is safe. -- Otherwise, a note to ask an administrator if the catalog is needed sooner. +Connectors that expose no external data source are rejected, because a catalog on one would either read the query engine's own internals or hold nothing: `system`, `jmx`, `memory`, `blackhole`, `datasketches`, `ai`, and the `tpch` and `tpcds` sample generators. +The last two are already available to every project as shared catalogs. -## Managing catalogs +Properties must address the data source over the network, for example `jdbc:`, `thrift:`, `https:` or `s3:`. +A property that points at a file path on the query engine's own machines is rejected, because you cannot place files there and the only files such a path could reach belong to the cluster itself. -Navigate to **Query Engine** → **Catalogs** in your project to see the project's catalogs together with the cluster's shared default catalogs. -Each catalog shows its status: pending until the next restart, live, or quarantined. +Two limits apply, because every project's catalogs share a fixed amount of storage in the cluster. +A project may create a set number of catalogs, ten by default, and a single definition may not exceed a set size, 16 KiB by default, measured after any references are resolved. +Both are cluster settings an administrator can raise, and neither is close to what an ordinary catalog needs: a few hundred bytes is typical. +Property values must also be latin1 text, which is what the definition is stored as, so a credential containing characters outside it has to come from a Hopsworks secret rather than being typed into a property. + +### Referencing a credential instead of typing it + +Two reference forms keep a credential out of the stored catalog definition. +Type `${` in the properties editor to pick from either. + +- `${HOPSWORKS_SECRET:}` for a value, such as a password. +- `${HOPSWORKS_MOUNT:}` or `${HOPSWORKS_MOUNT:/}` for a file, such as an Oracle wallet or a Java keystore. + See [Mountable Secrets][mountable-secrets]. + +A secret reference resolves against the secrets of the person who created the catalog, so you can only reference your own. +Naming a colleague's secret does not work, even if you can both see the catalog. +Two consequences follow. +A referenced secret cannot be deleted while a catalog still uses it, and the catalog keeps working after you leave the project only if the secret still exists. +If a catalog needs to outlive your account, have someone recreate it under theirs. + +!!! warning "Only secrets created from typed text can be referenced" + A secret created by uploading a file holds the base64 encoding of that file's contents, not the contents themselves. + A reference to such a secret puts that base64 text into the catalog, and the connector then fails, because it receives an encoded string where it expects a password, a key, or a JSON document. + Nothing records how a secret was created, so Hopsworks cannot detect the case and decode it for you, and the resulting error comes from the connector rather than from Hopsworks. + + Create the secret by typing or pasting the value as text when you intend to reference it from a catalog. + For a credential that is naturally a file, use a mountable secret instead. + +## Testing the connection + +**Create** stays disabled until **Test connection** succeeds, so a catalog that cannot reach its source is caught now rather than after a restart. +The test creates a temporary catalog on the cluster's test coordinator, lists its schemas, and reports the result. + +A failed test shows the query engine's own error, which is what says how to fix the properties. + +
+ Test connection failed +
A failed test reports the error the query engine saw, and Create stays disabled
+
+ +
+ Test connection succeeded +
Create is enabled once the connection test passes
+
-From this page you can also: +On a cluster where the administrator has turned the test coordinator off, the test reports that testing is unavailable and **Create** is enabled without it. -- **Create a catalog by hand**, choosing the connector type and writing the properties yourself. - This is the path for a source that has no Hopsworks data source, and the same validation, connection test, and restart handling apply. -- **Edit a catalog.** - Secret-bearing property values come back masked, and a masked value left unchanged keeps the stored secret, so you do not retype credentials to change an unrelated property. -- **Delete a catalog.** - The catalog stays queryable until the query engine restarts and unloads it, and the dialog tells you when that is, with the same restart-now option for administrators. +## When the catalog goes live -### Quarantined catalogs +A new catalog is saved with the status **Pending approval**. +It does not appear as a target in the SQL runner until the query engine restarts and loads it. +The **Catalog created** dialog says when that happens, and what it says depends on whether you can restart the query engine yourself. -The query engine refuses to start when a catalog cannot be loaded. -If that happens during a restart, the offending catalog is quarantined so the query engine can recover, and the catalog's page carries the load error. -A quarantined catalog is not applied; fix its properties and save it to try again. +A project user is told the next scheduled restart, in their own timezone. + +
+ Catalog pending approval +
A project user is told when the scheduled restart will make the catalog queryable
+
+ +A cluster administrator is offered **Restart query engine now** instead, together with the number of queries currently running and queued. +A restart cancels those queries everywhere on the cluster, so the dialog reports the activity to let the administrator judge whether restarting now is safe. + +
+ Catalog created, as a cluster administrator +
A cluster administrator can apply the change immediately
+
+ +Two cluster settings change this. +Where the administrator has enabled an eager restart, the dialog also says the catalog may go live earlier than the scheduled time, because the query engine restarts as soon as it is idle while changes are pending. +Where the administrator has instead required approval for all catalog changes, there is no schedule at all, and the dialog says to ask an administrator. +Both are described in the [administrator guide][lifecycle-settings]. + +Deleting a catalog follows the same path in reverse. +The catalog is marked **Pending removal** immediately, and stops being queryable at the next restart, because a running query engine keeps the catalogs it started with. +A catalog you have deleted can therefore still answer queries for a while. + +Editing one works the same way: the stored definition changes at once, and the loaded catalog changes at the next restart. +Secret-bearing property values come back masked, and a masked value left unchanged keeps the stored secret, so you do not retype credentials to change an unrelated property. + +The Catalogs tab shows where each catalog stands. + +| Status | Meaning | +| --- | --- | +| Approved | Loaded by the query engine and queryable. | +| Pending approval | Saved, not yet written for the query engine. | +| Pending restart | Written, waiting for the restart that loads it. | +| Pending removal | Deleted, still loaded until the next restart. | +| Failed | The query engine could not load it. See below. | + +## When a catalog fails to load + +A catalog can be valid to save and still be rejected by the query engine, for example when a property name is not one the connector accepts. +The query engine reads catalogs only at startup and refuses to start if it cannot load one, so such a catalog is removed from the engine automatically and marked **Failed**, which keeps the query engine available for everyone. + +The status carries the error the query engine reported, which says what to correct. +Edit the catalog to fix the definition, and it returns to Pending approval and follows the normal flow again. +Testing the connection before saving catches most of these earlier. + +
+ Failed catalog +
A catalog the query engine could not load, with the reason it reported
+
+ +## Who can query a catalog + +Access to a user-created catalog is granted at the catalog level per project: a project's Data Owners can read and write, and Data Scientists can read. +There is no per-schema or per-table configuration for these catalogs. +To limit what a catalog exposes, scope the database user in the connection credentials at the source, since the query engine reads the external system as that user and can only ever see what those credentials allow. ## Creating a catalog from the Python client @@ -131,11 +229,3 @@ print(result["restarted"], result["quarantined"]) ``` The restart interrupts queries running anywhere on the cluster, so prefer waiting for the scheduled restart unless the change is needed sooner. - -## When changes take effect - -A cluster administrator configures a scheduled restart that applies pending catalog changes on its own, so a catalog goes live without anyone holding cluster rights. -The schedule does nothing at all unless a catalog change is actually pending, so a cluster with no changes is never interrupted, and a due restart waits for a quiet moment (no query running, queued, or blocked) for up to an hour before going ahead. -Administrators can also enable an eager restart, which applies pending changes the moment the query engine goes idle, so a catalog may go live earlier than the scheduled time; the dialogs tell you when the cluster does this. -Administrators can instead require approval for all catalog changes, in which case nothing goes live until they apply the pending requests, and the dialogs say to ask an administrator. -The schedule's configuration is described in the [administrator guide][lifecycle-settings]. diff --git a/docs/user_guides/projects/trino/query_engine.md b/docs/user_guides/projects/trino/query_engine.md index fe5b96356a..20b5169078 100644 --- a/docs/user_guides/projects/trino/query_engine.md +++ b/docs/user_guides/projects/trino/query_engine.md @@ -8,11 +8,12 @@ The Query Engine in Hopsworks is powered by Trino, a distributed SQL query engin ## Accessing the Query Engine -Navigate to the Query Engine from your project's left sidebar. The Query Engine interface provides access to the SQL runner, cluster information, and query history. +Open **Queries** under **Analytics** in your project's left sidebar. +The page carries four tabs: **SQL Runner**, **Cluster Overview**, **Catalogs** and **Queries**.
Query Engine -
Query Engine
+
The Query Engine, open on the SQL Runner tab
## SQL Runner @@ -21,12 +22,14 @@ The SQL runner is where you write and execute SQL queries against your data. **To run a query:** -1. Write your SQL query in the editor -2. Select the database/catalog you want to query -3. Click "Run" to execute the query -4. View results in the table below the editor +1. Pick a **Catalog** and a **Schema**. + The tables in that schema are then listed below, and you can write the query against bare table names instead of qualifying every one. +2. Write the query in the editor, or click a table to start from `SELECT * FROM `. +3. Choose a row limit, which is appended to the query as a `LIMIT`. +4. Click **Run**. -The SQL runner supports standard SQL syntax and provides auto-completion for databases, tables, and columns. +Results appear below the editor on two tabs: **Results** holds the rows, and **Table** holds the column names and types. +The editor auto-completes catalogs, schemas, tables and columns, and **Add query** opens a second tab so several queries can be kept side by side.
SQL runner @@ -44,14 +47,14 @@ Need help with SQL syntax? Click the help icon in the SQL runner to access the c ## Cluster Overview -The cluster overview shows the health and status of your Trino cluster. Here you can monitor: +The cluster overview reports the query engine's version, environment and uptime, then a tile per metric, each with a sparkline of its recent history: -- **Active workers**: Number of workers currently processing queries -- **Running queries**: Queries currently being executed -- **Resource utilization**: CPU and memory usage across the cluster -- **Worker status**: Health status of individual worker nodes +- **Running queries**, **Queued queries** and **Blocked Queries** +- **Active workers** and **Worker Parallelism** +- **Runnable drivers**, **Input Rows/s** and **Input Bytes/s** +- **Reserved Memory** -This information helps you understand cluster performance and capacity. +Together they say whether the cluster is busy and whether a slow query is competing for capacity.
cluster overview @@ -60,133 +63,20 @@ This information helps you understand cluster performance and capacity. ## Managing Catalogs -Catalogs connect the Query Engine to external data sources such as PostgreSQL, MySQL, BigQuery, or MongoDB. -Project Data Owners can create, edit, and delete catalogs for their project from the Catalogs tab. -A catalog you create is named `__` and is only queryable within your own project. - -
- Catalogs list -
The Catalogs tab lists the shared default catalogs and your project's catalogs
-
- -### Creating a Catalog - -Click "Create catalog", then provide the catalog details: - -1. Enter a name. - The `__` prefix is added automatically, so you type only the short name. -2. Choose the connector type from the list. -3. Enter the connector properties, one `key=value` per line. - -The picker offers every connector installed in the Query Engine, so a connector it does not list cannot be used: - -`bigquery`, `cassandra`, `clickhouse`, `delta_lake`, `druid`, `duckdb`, `elasticsearch`, `exasol`, `faker`, `gsheets`, `hive`, `hudi`, `iceberg`, `ignite`, `kafka`, `lakehouse`, `loki`, `mariadb`, `mongodb`, `mysql`, `opensearch`, `oracle`, `pinot`, `postgresql`, `prometheus`, `redis`, `redshift`, `singlestore`, `snowflake`, `sqlserver`, `trino_thrift`. - -Connectors that expose no external data source are rejected, because a catalog on one would either read the Query Engine's own internals or hold nothing: `system`, `jmx`, `memory`, `blackhole`, `datasketches`, `ai`, and the `tpch` and `tpcds` sample generators. -The last two are already available to every project as shared catalogs, so there is no reason to create your own. - -Reference a Hopsworks secret with `${HOPSWORKS_SECRET:}` instead of typing the value inline to keep the secret out of the stored catalog definition. -Type `${HOPSWORKS_SECRET:` in the properties editor to pick from your own secrets. - -A reference resolves against the secrets of the person who created the catalog, so you can only reference your own: naming a colleague's secret does not work, even if you can both see the catalog. -Two consequences follow. -A referenced secret cannot be deleted while a catalog still uses it, and the catalog keeps working after you leave the project only if the secret still exists. -If a catalog needs to outlive your account, have someone recreate it under theirs, or use a literal value instead of a reference. - -!!! warning "Only secrets created from typed text can be referenced" - A secret created by uploading a file holds the base64 encoding of that file's contents, not the contents themselves. - A reference to such a secret puts that base64 text into the catalog, and the connector then fails, because it receives an encoded string where it expects a password, a key, or a JSON document. - Nothing records how a secret was created, so Hopsworks cannot detect the case and decode it for you, and the resulting error comes from the connector rather than from Hopsworks: an authentication failure, or a complaint that a value is malformed. - - Create the secret by typing or pasting the value as text when you intend to reference it from a catalog. - For a credential that is naturally a file, such as a service account JSON, paste the file's contents rather than uploading the file. - -Properties must address the data source over the network, for example `jdbc:`, `thrift:`, `https:` or `s3:`. -A property that points at a file path on the query engine's own machines is rejected, because you cannot place files there and the only files such a path could reach belong to the cluster itself. -If a connector you need requires a local file, ask an administrator to provide it. - -Two limits apply, because every project's catalogs share a fixed amount of storage in the cluster. -A project may create a set number of catalogs, five by default, and a single definition may not exceed a set size, 16 KiB by default, measured after any secret references are resolved. -Both are cluster settings an administrator can raise, and neither is close to what an ordinary catalog needs: a few hundred bytes is typical, and even one carrying a service account JSON stays a few KiB. -Property values must also be latin1 text, which is what the definition is stored as, so a credential containing characters outside it has to come from a Hopsworks secret rather than being typed into a property. - -
- Create catalog -
Creating a catalog with the auto-prefixed name and connector properties
-
- -### Testing the Connection - -Click "Test connection" to validate the configuration against the backing system before you save. -The test creates a temporary catalog, lists its schemas, and reports the result, so you catch a wrong host, port, or credential immediately. - -When the connection cannot be established, the connector's own error is shown. - -
- Test connection failed -
A failed test reports the underlying connector error
-
- -When the configuration is correct, the test confirms that the catalog connects. - -
- Test connection succeeded -
A successful connection test
-
- -### Availability After Creation - -A newly created catalog has the status Pending sync, meaning it is saved but not yet loaded by the running Query Engine. -It becomes queryable only after an administrator syncs it and restarts Trino, because Trino reads catalogs only at startup. -Until then the catalog is listed with its pending status and does not appear as a target in the SQL runner. - -Administrators are not notified when you create a catalog, and you are not notified when they apply it. -There is no service level on this step, so contact your administrator if a catalog has been pending longer than you expect, and watch the status on this page to see when it becomes Synced. - -Deleting a catalog follows the same path in reverse. -The catalog is marked for removal immediately and disappears from your Catalogs tab, an administrator's next sync removes it from the Query Engine's configuration, and it stops being queryable at the next restart, because a running Trino keeps the catalogs it started with. -A catalog you have deleted can therefore still answer queries for a while. - -
- Catalog pending sync -
A created catalog waits in Pending sync until an administrator applies it
-
- -### When a Catalog Fails to Load - -A catalog can be valid to save and still be rejected by the Query Engine, for example when a connector requires a property the definition does not set. -Trino reads catalogs only at startup and refuses to start if it cannot load one, so such a catalog is removed from the engine automatically and marked Failed to keep the Query Engine available for everyone. - -The status shows the error the Query Engine reported, which says what to correct. -Edit the catalog to fix the definition: it returns to Pending sync and follows the normal flow again. -Use "Test connection" before saving to catch most of these earlier. - -
- Failed catalog -
A catalog the Query Engine could not load, with the reason it reported
-
- -### Access to Catalog Tables - -Access to a user-created catalog is granted at the catalog level per project: a project's Data Owners can read and write, and Data Scientists can read. -The project's roles are granted the whole catalog; there is no per-schema or per-table configuration for these catalogs. -To limit what a catalog exposes, scope the database user in the connection credentials at the source database, since Trino reads the external system as that user and can only ever see what those credentials allow. +The Catalogs tab is where a project makes an external data source queryable from the SQL runner. +Creating a catalog from a data source or by hand, referencing credentials, testing the connection, and when a change reaches the query engine are all covered in [Trino Catalogs][trino-catalogs]. ## Queries -The Queries tab displays a history of all executed queries. For each query, you can see: - -- **Query ID**: Unique identifier for the query -- **Status**: Completed, failed, or running -- **Duration**: How long the query took to execute -- **User**: Who submitted the query -- **Timestamp**: When the query was run +The Queries tab lists the queries the project has run, filterable by state and sortable, with the query text alongside each entry. +A card carries its id and state with a progress bar, the user and source that submitted it, the resource group, its split counts, wall, total and CPU time, and its reserved, peak and cumulative memory. +A query that is still running is listed the same way and updates in place. -Click on any query to view detailed execution information. +Click a query id to open its details.
queries -
Queries
+
The project’s query history
## Query Details @@ -210,27 +100,22 @@ The overview tab shows query metadata, execution timeline, and performance metri ### Live Plan -The live plan visualizes the query execution plan in real-time, showing how Trino processes your query across different stages and operators. +The live plan draws the query's stages and the operators inside them, with each stage's state and its CPU time, memory, drivers and tasks, updating while the query runs. +The graph is usually taller than the page, so it can be laid out **Vertical** or **Horizontal**, panned by dragging, and zoomed by scrolling.
Query details live plan
Query details: live plan
-### Stages - -The stages view breaks down query execution into individual stages, showing: - -- Stage dependencies -- Data flow between stages -- Resource usage per stage -- Execution time for each stage +### Stage performance -This helps identify performance bottlenecks in complex queries. +This view takes one stage at a time, chosen with the **Stage** selector, and draws its pipelines operator by operator. +Each operator reports its throughput, output rows and bytes, driver count, and CPU, wall and blocked time, which is what locates a bottleneck inside a stage rather than merely between stages.
Query details stages -
Query details: stages
+
Query details: stage performance
### Splits @@ -244,7 +129,7 @@ Splits show how Trino parallelizes query execution. Each split represents a port ### References -The references tab lists all tables and data sources accessed by the query, helping you understand data dependencies. +The references tab lists the tables the query read, each with the user it was authorized as and whether the query named it directly, and the routines it called.
Query details references diff --git a/mkdocs.yml b/mkdocs.yml index 2fe6db28ed..7142179172 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -203,6 +203,7 @@ nav: - Sharing: user_guides/projects/datasets/sharing.md - Secrets: - Create Secret: user_guides/projects/secrets/create_secret.md + - Mountable Secrets: user_guides/projects/mountable_secrets/mountable_secrets.md - Environment variables: - Account-level Environment Variables: user_guides/projects/env_vars/create.md - Api Keys: