cluster-controller: reconfiguration observability + SHOW CLUSTERS by aljoscha · Pull Request #37628 · MaterializeInc/materialize

aljoscha · 2026-07-14T09:33:29Z

NOTE: This is PR4 in a stack of PRs, the full branch is at #36738. The first commit is #37615 (a small controller panic fix), review just the top commit here.

Surface cluster reconfigurations and autoscaling state in SQL so a background ALTER CLUSTER is observable. All dark: the durable reconfiguration and burst records only ever move under the enable_cluster_controller master gate (default off), so an ordinary cluster has nothing in flight and the new relations are empty for it.

Two introspection relations

Both are builtin materialized views in mz_catalog_server, computed from the raw catalog (mz_catalog_raw) so they are pure functions of durable state, and both are indexed on cluster_id. Each carries its structural payload as JSON so the schema is stable as strategies grow.

mz_internal.mz_cluster_reconfigurations — one row per cluster that carries a graceful reconfiguration record: cluster_id, a status that is in-progress while the controller converges and then a terminal finalized / timed-out / cancelled / resource-exhausted, the deadline, the on_timeout action, and the target config shape as JSON. The record is retained after it settles, so the latest outcome stays inspectable until a later reconfiguration overwrites it. The realized (current) shape stays in mz_clusters. Status values are kebab-case, matching the audit log's transition values and the catalog's other multi-word values (materialized-view, on-refresh). The CASE mappings pass unmapped variants through verbatim rather than falling to NULL, where the ASSERT NOT NULL would error every read of the relation (and of SHOW CLUSTERS, which joins it).
mz_internal.mz_cluster_auto_scaling_strategies — one row per cluster that has an AUTO SCALING STRATEGY configured or an autoscaling action running: the configured strategy as JSON and the in-flight state (the active burst, when one is up) as JSON or NULL. Nothing can configure a strategy until the burst PR lands its SQL surface, so its user-facing docs ship with that PR and this PR marks the relation RELATION_SPEC_UNDOCUMENTED.

SHOW CLUSTERS

SHOW CLUSTERS gains one nullable activity column (between replicas and comment): a human-readable one-line summary of any in-flight reconfiguration or burst, NULL when the cluster is steady. A single SHOW CLUSTERS answers "is something in progress" without the user knowing about the two relations. Settled reconfiguration records are retained, so the join matches only status = 'in-progress'; the auto-scaling relation's state is set only while a burst runs. Neither needs mz_now(), so mz_show_clusters stays indexable.

Plumbing

The two new builtin indexes change the SQL fingerprint of mz_indexes (it inlines the builtin-index set as VALUES), which requires an explicit builtin-schema-migration replacement step. NOTE for the final rebase: the step version must track the workspace's current dev version until this ships in a release, upstream version bumps silently re-break upgrade tests otherwise.
The new builtin MVs cannot advance their write frontier in read-only (0dt) mode, so the caught-up check's exclusion walk in coord.rs now also seeds from new builtin MVs, excluding their transitive dependents (here: the indexed mz_show_clusters).

Tests

An observability section in cluster-controller.td asserts the two relations, the activity column, and the reconfiguration audit-log trail end to end (finalized, cancel-race, rollback-at-deadline, forced COMMIT cut-over, and a deterministically frozen in-progress record). Gate-off column assertions in show_clusters.slt. Catalog-snapshot slt/testdrive expectations and the mz_internal system-catalog docs updated.

Motivation

Implements the two-relation observability surface and the SHOW CLUSTERS activity column from doc/developer/design/20260522_cluster_autoscaling.md.

User-visible change: SHOW CLUSTERS output gains an activity column, and mz_internal gains the mz_cluster_reconfigurations relation (plus the yet-undocumented mz_cluster_auto_scaling_strategies).

A DropReplica decision can outlive its replica: a user DDL that removes the replica can land between the tick's catalog read and the apply. The in-transaction witness check would reject such a stale batch, but resource-limit validation runs before the transaction and panics on a missing replica ("unknown cluster replica"), so it never gets the chance. Reject the batch when building the ops instead, mirroring the existing vanished-cluster handling. The controller re-derives its decisions on the next tick. Co-Authored-By: Claude <noreply@anthropic.com>

aljoscha · 2026-07-14T09:40:30Z


-                // Migrated MVs can't make progress in read-only mode. Exclude them and all their
-                // transitive dependents.
+                // A collection that can't advance its write frontier in read-only mode also


we need to boil down this large wall of text to be way more concise please

Boiled down to 7 lines (from 18), keeping the two collection kinds and the accepted-blip tradeoff.

aljoscha · 2026-07-14T09:41:50Z

+                "The config shape the cluster is reconfiguring to, as JSON: `size`, `replication_factor`, `availability_zones`, and `logging`. The realized (current) shape is in `mz_clusters`.",
+            ),
+        ]),
+        // One row per managed cluster that carries a graceful reconfiguration


I think we can also boil down these comments on the new builtins, big time please

Boiled both down: the reconfigurations block is now 8 lines (retention, JSON-null presence check, kebab values, and the ELSE-passthrough rationale since that one is load-bearing), the auto-scaling one is 5.

aljoscha · 2026-07-14T09:42:17Z

        .with_column("comment", SqlScalarType::String.nullable(false))
        .finish(),
    column_comments: BTreeMap::new(),
+    // `activity` is a one-line, human-readable summary built from the two base


also boil down this one please

Done, 5 lines: the in-progress-only match, no-mz_now indexability, and the NULL-concat gotcha.

aljoscha · 2026-07-14T09:53:28Z

+> SELECT recon.status FROM mz_internal.mz_cluster_reconfigurations recon JOIN mz_clusters ON mz_clusters.id = recon.cluster_id WHERE mz_clusters.name = 'cc_rollback'
+finalized
+
+# An in-flight reconfiguration observed deterministically: freeze the controller by


didn't we move away from this pattern in earlier changes of our STACK, we can't be 100% sure that nothing interferes here, or at least we have to change this to use mz_wait as some other tests already do, no?

Yes, good catch, that was a leftover racy pattern: cut-over could in principle land before the gate flip. Reworked to the established mz_sleep pattern from the deadline scenarios above: a sleeping MV in the cluster keeps the target set from hydrating, so the record deterministically stays in-progress (deadline at 600s, out of reach), and dropping the view lets it hydrate and settle. No gate flipping anymore.

Correction to my last reply: the unblock-and-converge step did not survive contact with CI. The sleep runs on the replica's worker thread, so dropping the view cannot unwedge the replicas, and the cut-over never comes (Testdrive 9 caught it). The scenario now ends by dropping the cluster with the record in flight instead, which must succeed and leaves the audit lifecycle at started. The deterministic in-progress observation is unchanged.

**NOTE: This is PR4 in a stack of PRs, the full branch is at MaterializeInc#36738 Surface cluster reconfigurations and autoscaling state in SQL so a background ALTER CLUSTER is observable. All dark: the durable reconfiguration and burst records only ever move under the `enable_cluster_controller` master gate, so an ordinary cluster has nothing in flight and the new relations are empty for it. Two introspection relations. Both are builtin materialized views in `mz_catalog_server`, computed from the raw catalog (`mz_catalog_raw`) so they are pure functions of durable state, and both are indexed on `cluster_id`. Each carries its structural payload as JSON so the schema is stable as strategies grow. * `mz_internal.mz_cluster_reconfigurations`, one row per cluster that carries a graceful reconfiguration record: `cluster_id`, a `status` that is `in-progress` while the controller converges and then a terminal `finalized` / `timed-out` / `cancelled` / `resource-exhausted`, the `deadline`, the `on_timeout` action, and the `target` config shape as JSON. The record is retained after it settles, so the latest outcome stays inspectable until a later reconfiguration overwrites it. The realized (current) shape stays in `mz_clusters`. Status values are kebab-case, matching the audit log's transition values and the catalog's other multi-word values, and the CASE mappings pass unmapped variants through verbatim rather than falling to NULL, where the ASSERT NOT NULL would error every read of the relation. * `mz_internal.mz_cluster_auto_scaling_strategies`, one row per cluster that has an AUTO SCALING STRATEGY configured or an autoscaling action running: `cluster_id`, the configured `strategy` as JSON, and the in-flight `state` (the active burst, when one is up) as JSON or NULL. Nothing can configure a strategy until the burst PR lands its SQL surface, so its user-facing docs ship with that PR and this PR marks the relation RELATION_SPEC_UNDOCUMENTED. SHOW CLUSTERS. `mz_show_clusters` LEFT JOINs both relations to add one `activity` column, a human-readable one-line summary of any in-flight reconfiguration or burst, so a single SHOW CLUSTERS answers "is something in progress" without the user knowing about the two relations. Settled reconfiguration records are retained, so the join matches only `status = 'in-progress'`; the auto-scaling relation's `state` is set only while a burst runs. Neither needs `mz_now()`, so the view stays indexable. The two new builtin indexes change the SQL fingerprint of `mz_indexes` (it inlines the builtin-index set as VALUES), which requires an explicit builtin-schema-migration replacement step. The step version must track the workspace's current dev version until this ships in a release. The new builtin MVs also need the read-only caught-up check to account for their dependents, so `coord.rs` seeds new-builtin-MV dependents into the exclusion walk. Tests: an observability section in `cluster-controller.td` asserting the two relations, the SHOW CLUSTERS `activity` column, and the reconfiguration audit-log trail end to end, gate-off column assertions in `show_clusters.slt`, and the catalog-snapshot slt/testdrive expectations plus the `mz_internal` system-catalog doc updated for the new relation, its index, and the SHOW CLUSTERS `activity` column. Implements the two-relation observability surface and the SHOW CLUSTERS activity column from `doc/developer/design/20260522_cluster_autoscaling.md`. Co-Authored-By: Claude <noreply@anthropic.com>

mtabebe

Clean change, and scope to just the introspection. One question around how we could have null records (the comment wasn't particularly clear).

mtabebe · 2026-07-14T23:18:47Z

            MZ_CATALOG_SCHEMA,
            "mz_audit_events",
        ),
+        // Required because we added the `mz_cluster_reconfigurations_ind` and


mtabebe · 2026-07-14T23:22:38Z

+        FROM mz_internal.mz_catalog_raw
+        WHERE
+            data->>'kind' = 'Cluster' AND
+            data->'value'->'config'->'variant'->'Managed' IS NOT NULL


When are the cases when it is null? The comment above wasn't very informative

mtabebe · 2026-07-14T23:24:35Z

-                // transitive dependents.
+                    new_builtin_collections.iter().copied().collect();
+
+                // A collection that can't advance its write frontier in read-only mode


Thanks for improving the comments

aljoscha requested review from a team as code owners July 14, 2026 09:33

aljoscha commented Jul 14, 2026

View reviewed changes

aljoscha force-pushed the adapter-cluster-controller-observability branch from 1031cae to b41a71a Compare July 14, 2026 10:00

aljoscha force-pushed the adapter-cluster-controller-observability branch from b41a71a to 5e1d9b7 Compare July 14, 2026 10:29

aljoscha added the ci-nightly PR CI control: also trigger Nightly label Jul 14, 2026

mtabebe self-requested a review July 14, 2026 15:52

mtabebe approved these changes Jul 14, 2026

View reviewed changes

Uh oh!

Conversation

aljoscha commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Two introspection relations

SHOW CLUSTERS

Plumbing

Tests

Motivation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtabebe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aljoscha commented Jul 14, 2026 •

edited

Loading