[HWORKS-3017] File-based credentials for user-created Trino catalogs (wallets, keystores)

[ErmiasG]

HWORKS-3017

Documents mountable secrets, the project-scoped bundles of credential files that let a connector authenticate with a wallet or a keystore instead of a password, and rewrites the Trino documentation around them for the lifecycle that #634 shipped.

Why the other pages are rewritten and not merely extended

HWORKS-3045 landed while this branch was open and changed the model these pages described. Applying a pending catalog is now one Restart Trino action instead of a sync followed by a restart, a scheduled restart applies pending changes with no administrator involved, and the dialog shown after a catalog is created depends on the viewer: a cluster administrator is offered Restart query engine now alongside the live query count, while a project user is told the next scheduled restart in their own timezone.

#634 added catalogs.md without touching query_engine.md, so main currently carries two accounts of the same feature and the older one is wrong. That is the largest change here.

query_engine.md

Loses its Managing Catalogs section, which duplicated and contradicted the catalogs page. The content that existed only there moves rather than being dropped: the installed connector list and the rejected internal connectors, the rule that a secret reference resolves against its creator's own secrets, the warning that a secret created from an uploaded file holds base64 and cannot be referenced, the latin1 constraint on property values, and the per-role grants on a user-created catalog. What is left on the page is a pointer.

Retaking its figures corrected the surrounding prose in five places, all of which described an older UI:

• the sidebar entry is Queries under Analytics, not "Query Engine"
• the SQL runner steps had catalog selection after writing the query; catalog and schema come first, and picking them lists the schema's tables so a query can use bare table names
• Results holds the rows and Table holds the column names and types, which was not documented
• Cluster Overview listed "CPU and memory usage" and per-worker health; neither is on that tab, and worker health is the administrator's Workers tab
• the query detail tab is Stage performance, takes one stage at a time, and draws its pipelines operator by operator

catalogs.md

Becomes the single user-facing catalog page and gains the screenshots it never had. Creating a catalog by hand is its own section instead of a bullet, the five statuses the list actually shows are named (Approved, Pending approval, Pending restart, Pending removal, Failed), and When changes take effect is folded into When the catalog goes live so the lifecycle is explained once rather than twice.

mountable_secrets.md

New page, at project level next to Secrets rather than under Query Engine. A bundle belongs to the project, is managed from Project Settings, and the store is deliberately not Trino-owned: Trino asks the store for a mount rather than owning it, so any service could be given read-only access to one. It covers creating a bundle from files or a flat zip, why a bundle cannot be edited and what to do instead, the two reference forms and the rule that a reference cannot be extended with a path, the per-project limits with their defaults, and the Data Owner role and MOUNTABLE_SECRET scope it takes.

It ends on an Oracle Autonomous Database worked example, because three things there cause most of the failures and none of them name themselves:

• the alias before the ? comes from the wallet's own tnsnames.ora, not the service name in the OCI console
• ORA-12506 means either a source address the database does not accept or a wallet the client never loaded, and the two look identical from outside
• each alias in tnsnames.ora carries (retry_count=20)(retry_delay=3) inside its connect descriptor, so a rejected address presents as a minute-long hang rather than an error

setup_installation/admin/trino.md

A new section covers what the store adds to a cluster: where bundles live and why the tree is keyed by project id, the privileged FUSE sidecar that mounts them read-only on the coordinator, the workers and the test coordinator, the Kyverno exception and the two ServiceAccounts an upgrade introduces (a binding that named default to reach Trino has to be repointed), why OpenShift is unsupported, and what turning the store off does to a catalog that already went live.

Backup is called out separately because bundles are HopsFS files, so the Kubernetes object backup that captures the catalog Secrets does not cover them. A short runbook follows, since there is no admin endpoint for the store: the checks are kubectl exec, and the mount is worth checking on a worker rather than only the coordinator, because a missing mount is otherwise invisible (the sidecar mounts into its own filesystem and both containers report ready).

Eight configuration variables are documented with their defaults. Three are seeded rows; five fall back to coded defaults until an administrator adds them, which is why they do not appear in Cluster Settings on a fresh install.

Three existing passages are also corrected. Two still described select-then-sync-then-restart. The restart confirmation now asks for a typed confirm. And the sentence about the store root claimed the backend checks it "before it writes a bundle or deletes a project's tree ... so a root created by hand with the wrong mode fails every upload": the check sits inside if (!dfso.exists(projectPath)), so it runs on a project's first bundle and again before a project's tree is deleted, not on every upload, and it compares owner, group and mode rather than two of the three.

Screenshots

26 figures added or replaced, 2 removed, every one captured today against a cluster running this build. The two removed showed states that no longer exist: the two-step apply, and a catalog "Pending sync".

The role split is demonstrated with two accounts rather than described. A project Data Owner who is not a cluster administrator creates an Oracle catalog over a wallet bundle and is told it goes live at the next scheduled restart; the administrator sees that request on the cluster Catalogs tab and applies it immediately, behind the confirmation that names what it interrupts.

A catalog carrying a property the connector does not accept was created through the API, because the dialog will not enable Create until the connection test passes. It quarantined the query engine on the next restart, which is where the failed-catalog and recovery figures come from, and the engine was recovered with the same Restart Trino action the page documents.

A figure of a project's own catalog allowance is added, because the previous wording (edit Query Engine, Trino catalogs) read as though there were something named Query Engine to click. It is a section of the project configuration form, after the Kafka topic quota, reached through Edit configuration on the project's row.

Verification

Strict mkdocs build -s passes locally with no warnings, markdownlint-cli2 reports 0 errors on all four pages, snakeoil is clean on the Python blocks, and the cross-page heading links were confirmed to resolve in the build output rather than assumed.

Every fact was read out of the merged code or checked against a cluster running it: the error codes and caps from RESTCodes.java and Settings.java, the name pattern [A-Za-z0-9][A-Za-z0-9._-]{0,62}, the DATA_OWNER and @ProjectObserversNotAllowed annotations, the ownership gate in MountableSecretsController, the payara:hdfs mode 750 preset in charts/hopsfs/templates/preset_job.yaml, and the variables confirmed live (mountable_secrets_path = /apps/mountable-secrets, trino_mountable_secrets_root = /opt/hopsworks/mounts). The Oracle examples are configurations that connected on that cluster.

Prose follows the repo conventions: one sentence per line, links by heading ID rather than relative path, and every new page is in mkdocs.yml.

Corrections an earlier adversarial review found

Kept here because each was checked against the source or the running cluster rather than argued about.

The mountable secret settings were listed under "to modify these settings, navigate to Cluster Settings". For two of them that advice breaks a cluster: the store root and the mount root are seeded by the chart from one Helm value that also presets the HopsFS directory and feeds the mount sidecar, so editing either row alone points the backend at a path nothing is mounted from. They now have their own section naming the Helm value, together with the fact that the five per-project limits have no seeded row. The flag's coded fallback is false while the chart seeds true, which is now stated.

The Oracle retry parameters were attributed to sqlnet.ora. They are in tnsnames.ora, once per alias, inside the connect descriptor. Checked on the wallet mounted in a running coordinator: retry_count appears five times in tnsnames.ora and not once in sqlnet.ora, which holds only the wallet location and the DN check. Editing the file the page named would have left the minute-long delay in place, which is the only reason that paragraph exists.

Deleting a bundle was described as breaking a catalog "the next time Trino loads it". The query engine sees the store through a live mount and a driver-time connector reads its files when it opens a connection, so the failure arrives on the next query with no restart involved. The page now says so, and says what to do when an interruption is unacceptable. The admin guide already had this right, so the two pages had disagreed.

The review upheld two things it was asked to attack: the backup statement, and the retaken recovery figure, whose banner, caption and surrounding prose it found consistent.

Also in this branch

The repository's own agent instructions told the reader to install from a requirements-docs.txt that no longer exists (it is uv sync now) and omitted that a local mkdocs serve needs the docs/javadoc stub. Fixed in passing, having hit both while building these pages.

Not covered

The Add Trino Catalog action on a data source is documented but has no figure: neither project on the verification cluster has a data source that maps to a Trino connector, so there was nothing to photograph, and that section arrived with #634 already accurate.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds documentation for the new “mountable secrets” feature used by Trino catalogs that require file-based credentials (wallets/keystores), and updates existing Trino docs to reflect the current approval workflow and credential handling.

Changes:

• Adds a new user guide page describing mountable secrets, creation rules/limits, reference syntax, and an Oracle wallet example.
• Updates the Trino Query Engine user guide to point users to mountable secrets for file-based connector auth and to use “Pending approval” terminology.
• Expands the Trino admin guide with mountable secret internals, operational considerations (mount sidecar, SA changes, backup/diagnostics), and updates “sync” → “approve” terminology; adds a nav entry for the new page.

Reviewed changes

Copilot reviewed 4 out of 16 changed files in this pull request and generated 3 comments.

File	Description
mkdocs.yml	Adds navigation entry for the new Mountable Secrets user guide page.
docs/user_guides/projects/trino/query_engine.md	Updates user-facing Trino catalog workflow terminology and links to mountable secrets for file-based credentials.
docs/user_guides/projects/trino/mountable_secrets.md	New page documenting mountable secret bundles, usage, limits, and an Oracle worked example.
docs/setup_installation/admin/trino.md	Updates admin workflow terminology and documents cluster-level mechanics/operations for mountable secrets (mounting, config, backup, diagnostics).

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ErmiasG]

All three taken, in 8cd5121. Two of them caught a word this branch itself made wrong.

"an administrator's next approval removes it" was incorrect. A deletion is applied, not approved, and this branch's own admin guide already said so, so the two pages contradicted each other. The cluster agrees with the admin guide: with only removals waiting, the alert reads "Nothing needs your approval" and the action button relabels itself to "Apply removals (N)". The user guide now says an administrator applies the removal.

"Sync does not consult the flag" kept the old name for the step this branch renamed to approval, in the paragraph about turning the store off. Now reads "Approval does not consult the flag". Note the other sync in this file, "a GitOps sync that prunes resources it does not manage" in the recovery section, is a different concept and stays.

The list items now match their neighbour. The third bullet under "Where catalog credentials are stored" already used a continuation line for its second sentence; the first two did not. They do now.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (2)

docs/user_guides/projects/trino/query_engine.md:149

• The admin guide distinguishes between approving catalogs and applying removals (deletions are Pending removal and are applied, not approved). This sentence says an administrator’s next approval removes a deleted catalog, which conflicts with the documented workflow and could confuse users about which action they’re waiting for.

Deleting a catalog follows the same path in reverse.
The catalog is marked for removal immediately and disappears from your Catalogs tab, an administrator then applies the removal rather than approving it, and it stops being queryable at the next restart, because a running Trino keeps the catalogs it started with.

docs/setup_installation/admin/trino.md:245

• This section consistently uses the new “approval” terminology, but this sentence still refers to “Sync”, which can be read as the old workflow name. Align it with the rest of the page to avoid ambiguity.

Its reference still resolves to a path, but nothing populates that path any more.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 35 changed files in this pull request and generated no new comments.

Suppressed comments (1)

docs/user_guides/projects/trino/catalogs.md:11

• The navigation path here contradicts query_engine.md, which now says the UI entry is Queries under Analytics (with Catalogs as a tab). To avoid confusing readers, align this page’s first instruction with the updated UI naming/path.