feat: ddl statements + catalog

[clintropolis]

Description

This PR adds DDL statement support to Druid so that CREATE and ALTER statements can be used to manage the catalog contents. This functionality requires WRITE permissions to the datasource (same as the backing catalog APIs), and is gated behind a new runtime property druid.sql.planner.enableCatalogDdl which is false by default. This PR includes support for defining the logical schema (columns list in catalog), aggregate projections (projections property in catalog), base table projection (baseTable property in catalog), clustering (clusterKeys in catalog), time partitioning (segmentGranularity in catalog), and any other properties (sealed, targetSegmentRows) with generic property setting syntax.

These operations are purely metadata operations, and so DROP TABLE has been omitted since in my mind it has the most room for confusion (a DROP TABLE only cleared metadata and didn't do anything else sounds kind of confusing to me). Dropping columns and projections from a table felt more easily explainable that these do not modify any existing data and instead stage the schema for subsequent ingestion (and future work should wire this catalog stuff better into compaction/reindexing so that we can also frame it such that compaction/reindexing will begin to eventually converge on the updated schema).

These all go through the regular query path, and return 0 rows on successful operation. Some follow-up work is needed to improve web-console syntax highlighting and behaviors, and since there is no drop table statement it would still probably be nice to eventually add a catalog management ui, but the statements at least already work as-is through the current query interface.

Some examples:

create table:

CREATE TABLE "wikipedia" (
  "channel" VARCHAR,
  "__time" TIMESTAMP,
  "page" VARCHAR,
  "namespace" VARCHAR,
  "user" VARCHAR,
  "comment" VARCHAR,
  "added" BIGINT,
  "delta" BIGINT
)
PARTITIONED BY DAY
CLUSTERED BY "channel"

alter table to add column:

ALTER TABLE "wikipedia" ADD COLUMN "deleted" BIGINT

alter table to add projection:

ALTER TABLE "wikipedia" ADD PROJECTION "channel_sums" AS (
  SELECT 
    TIME_FLOOR("__time", 'PT1H'),
    "channel",
    SUM("added") as "sum_added",
    SUM("delta") as "sum_delta",
    SUM("deleted") as "sum_deleted"
  GROUP BY 1,2
)

alter table set property:

ALTER TABLE "wikipedia" SET PROPERTIES (sealed = true)

alter table set 'base table' projection to create clustered segments:

ALTER TABLE "wikipedia" ADD PROJECTION __base AS (
  SELECT
    "channel",
    "__time",
    "page",
    "namespace,"
    "user",
    "comment",
    "added",
    "delta",
    "deleted"
  CLUSTERED BY "channel"
)

create table with base table and projection definitions:

CREATE TABLE "wikipedia" (
  "channel" VARCHAR,
  "__time" TIMESTAMP,
  "page" VARCHAR,
  "namespace" VARCHAR,
  "user" VARCHAR,
  "comment" VARCHAR,
  "added" BIGINT,
  "delta" BIGINT,
  "deleted" BIGINT,
  PROJECTION __base AS (
    SELECT
      "channel",
      "__time",
      "page",
      "namespace",
      "user",
      "comment",
      "added",
      "delta",
      "deleted"
    CLUSTERED BY "channel"
  ),
  PROJECTION "channel_sums" AS (
    SELECT 
      TIME_FLOOR("__time", 'PT1H'),
      "channel",
      SUM("added") as "sum_added",
      SUM("delta") as "sum_delta",
      SUM("deleted") as "sum_deleted"
    GROUP BY 1,2
  ),
  PROJECTION "channel_page_max" AS (
    SELECT 
      TIME_FLOOR("__time", 'PT1H'),
      "channel",
      "page",
      MAX("added") as "sum_added",
      MAX("delta") as "sum_delta",
      MAX("deleted") as "sum_deleted"
    GROUP BY 1,2,3
  )
)
PARTITIONED BY DAY
CLUSTERED BY channel
SEALED

[FrankChen021]

Severity	Findings
P0	0
P1	2
P2	3
P3	0
Total	5

Reviewed 42 of 42 changed files.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

[P1] Revalidate the complete table after property edits

This uses the property-only edit endpoint, whose transaction loads and validates properties without columns, so DatasourceDefn.validate(ResolvedTable) and its cross-field checks never run. For example, after defining a DAY projection, SET PROPERTIES can change segmentGranularity to PT1H even though full validation rejects a projection coarser than its segments; it can likewise clear sealed while __base remains. The catalog then contains an invalid specification and subsequent ingestion fails. Load and validate the complete revised TableSpec inside the Coordinator transaction.

[FrankChen021]

[P1] Enforce column operation predicates atomically

UpdateColumns appends a column when its name is absent, so ALTER TABLE t ALTER COLUMN typo SET DATA TYPE BIGINT silently adds typo instead of rejecting the nonexistent target. ADD COLUMN has the inverse predicate checked by a separate Broker read, allowing concurrent ADDs to both pass and the later merge to overwrite the first type. Add/alter existence semantics need dedicated checks inside the Coordinator's column-update transaction.

[FrankChen021]

[P2] Preserve statement context when planning projections

The nested planner receives only the hard-coded CONTEXT and discards the enclosing statement context, even though SET clauses are explicitly supported before DDL. For example, SET sqlTimeZone = 'America/Los_Angeles' followed by a projection using TIME_FLOOR stores a UTC expression, so the equivalent query under the same context plans differently and cannot match the projection. Merge relevant outer PlannerContext values before applying the deterministic overrides.

[FrankChen021]

[P2] Handle direct aliases in base projections

ScanQuery deduplicates its column list, while outputNames retains every SELECT item. A valid declared layout [id, copy] with SELECT id, id AS copy therefore has two outputs but only one selected entry, and the second iteration throws IndexOutOfBoundsException; a direct alias also has no virtual column to materialize copy. Preserve the select-to-source mapping or reject this form with a user-facing validation error.

[FrankChen021]

[P2] Do not unparse omitted columns as empty parentheses

The grammar permits CREATE TABLE tbl PARTITIONED BY DAY with no parenthesized element list, but unparse always emits CREATE TABLE tbl () PARTITIONED BY DAY. Empty parentheses cannot be parsed because AddDruidTableElement is mandatory once '(' is present, so a valid AST does not round-trip. Omit the frame when both lists are empty or teach the grammar to accept ().