From b9cfe127ed5d4efdd37fe48a03ada4165acae888 Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 10 Jul 2026 14:28:33 +0200 Subject: [PATCH 1/2] Document ALTER COLUMN TYPE behaviour on parquet partitions --- .../sql/alter-table-change-column-type.md | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/documentation/query/sql/alter-table-change-column-type.md b/documentation/query/sql/alter-table-change-column-type.md index 2d53e366e..9919b8ad7 100644 --- a/documentation/query/sql/alter-table-change-column-type.md +++ b/documentation/query/sql/alter-table-change-column-type.md @@ -130,6 +130,40 @@ result in `NULL` values for inconvertable string values. When column type change results into range overflow or precision loss, the same rules as explicit [CAST](/docs/query/sql/cast/) apply. +## Parquet partitions + +`ALTER TABLE ... ALTER COLUMN ... TYPE` works on tables that contain +[Parquet partitions](/docs/query/export-parquet/#in-place-conversion). Reads +return the converted values on every query, matching the behaviour of native +partitions exactly, including `NULL` sentinels, `BYTE`/`SHORT`/`CHAR` column +tops, date and timestamp scaling, and UTF-16/UTF-8 text transcoding. + +By default the conversion is **lazy**: the Parquet partitions are not rewritten +by the `ALTER`. The type change is recorded in the table metadata and applied on +the query path, where each Parquet row group is decoded once and buffered for +the rows that follow. This keeps the `ALTER` cheap regardless of how much data +is stored as Parquet. + +An [out-of-order (O3)](/docs/concepts/out-of-order-data/) write that later lands +on a converted Parquet partition rewrites that partition with the new type, +materialising the conversion permanently for that partition. + +:::note Eager conversions + +Some conversions cannot be applied lazily. In these cases QuestDB first converts +the affected Parquet partitions back to native format, one rewrite per affected +partition, before applying the type change: + +- Conversions where the source or target type is `SYMBOL`. +- Chained conversions where the on-disk Parquet type no longer matches the + current column metadata, for example a second type change on a column that was + already converted lazily. + +Unlike the metadata-only lazy path, these partition rewrites carry an I/O cost +proportional to the amount of Parquet data in the affected partitions. + +::: + ## Unsupported Conversions Converting from the type to itself is not supported. From cc8a167edb31b48ff423092babc95a94f15f5945 Mon Sep 17 00:00:00 2001 From: javier Date: Fri, 10 Jul 2026 17:58:21 +0200 Subject: [PATCH 2/2] Update Parquet in-place conversion limitations for production readiness --- documentation/query/export-parquet.md | 35 +++++++++++---------------- 1 file changed, 14 insertions(+), 21 deletions(-) diff --git a/documentation/query/export-parquet.md b/documentation/query/export-parquet.md index fc5c7c3d2..537495bc5 100644 --- a/documentation/query/export-parquet.md +++ b/documentation/query/export-parquet.md @@ -147,33 +147,26 @@ COPY market_data TO 'market_data_parquet_table' WITH FORMAT PARQUET COMPRESSION_ ## In-place conversion -:::warning -At the moment, converting to Parquet in-place is work-in-progress and -not recommended for production. We recommend caution and taking a snapshot before starting any -in-place data conversion. -::: - When using in-place conversion, the partition(s) remain under QuestDB's control, and data can still be queried as if it were in native format. -### Limitations +Parquet partitions behave like native partitions for most operations: -At its current state, in-place conversion of native partitions into Parquet has the following limitations: +* Inserts are supported, with and without dedup. When new or out-of-order rows + land on a Parquet partition, the partition is rewritten to include them. +* Column type changes via + [`ALTER TABLE ALTER COLUMN TYPE`](/docs/query/sql/alter-table-change-column-type/) + are supported. On Parquet partitions the change is applied lazily; see that + page for details. +* TTL applies to Parquet partitions, so expired partitions are dropped as usual. -* We have been testing Parquet support for months, and we haven't experienced data corruption or data loss, but this is -not guaranteed. It is strongly advised to back up first. -* We have seen cases in which querying Parquet partitions leads to a database crash. This can happen if metadata in the -table is different from metadata in the Parquet partitions, but it could also happen in other cases. -* While converting data, writes to the partitions remain blocked. -* After a partition has been converted to Parquet, it will not register any changes you send to that partition, -including respecting any applicable TTL, unless you convert back to native. -* Schema changes are not supported. -* Some parallel queries are still not optimized for Parquet. - -For the reasons above, we recommend not using in-place conversion in production yet, unless you test extensively with -the shape of the data and queries you will be running, and take frequent snapshots. +### Limitations -All those caveats should disappear in the next few months, when we will announce it is ready for production. +* `UPDATE` statements are not supported on Parquet partitions. Overwriting rows + through a dedup upsert is supported, since it is an insert. +* While a partition is being converted, writes to that partition are briefly + blocked. +* Some parallel queries are not yet optimized for Parquet. ### Converting to Parquet