Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 14 additions & 21 deletions documentation/query/export-parquet.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,33 +147,26 @@ COPY market_data TO 'market_data_parquet_table' WITH FORMAT PARQUET COMPRESSION_

## In-place conversion

:::warning
At the moment, converting to Parquet in-place is work-in-progress and
not recommended for production. We recommend caution and taking a snapshot before starting any
in-place data conversion.
:::

When using in-place conversion, the partition(s) remain under QuestDB's control, and data can still be queried as if it
were in native format.

### Limitations
Parquet partitions behave like native partitions for most operations:

At its current state, in-place conversion of native partitions into Parquet has the following limitations:
* Inserts are supported, with and without dedup. When new or out-of-order rows
land on a Parquet partition, the partition is rewritten to include them.
* Column type changes via
[`ALTER TABLE ALTER COLUMN TYPE`](/docs/query/sql/alter-table-change-column-type/)
are supported. On Parquet partitions the change is applied lazily; see that
page for details.
* TTL applies to Parquet partitions, so expired partitions are dropped as usual.

* We have been testing Parquet support for months, and we haven't experienced data corruption or data loss, but this is
not guaranteed. It is strongly advised to back up first.
* We have seen cases in which querying Parquet partitions leads to a database crash. This can happen if metadata in the
table is different from metadata in the Parquet partitions, but it could also happen in other cases.
* While converting data, writes to the partitions remain blocked.
* After a partition has been converted to Parquet, it will not register any changes you send to that partition,
including respecting any applicable TTL, unless you convert back to native.
* Schema changes are not supported.
* Some parallel queries are still not optimized for Parquet.

For the reasons above, we recommend not using in-place conversion in production yet, unless you test extensively with
the shape of the data and queries you will be running, and take frequent snapshots.
### Limitations

All those caveats should disappear in the next few months, when we will announce it is ready for production.
* `UPDATE` statements are not supported on Parquet partitions. Overwriting rows
through a dedup upsert is supported, since it is an insert.
* While a partition is being converted, writes to that partition are briefly
blocked.
* Some parallel queries are not yet optimized for Parquet.

### Converting to Parquet

Expand Down
34 changes: 34 additions & 0 deletions documentation/query/sql/alter-table-change-column-type.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,40 @@ result in `NULL` values for inconvertable string values.
When column type change results into range overflow or precision loss, the same
rules as explicit [CAST](/docs/query/sql/cast/) apply.

## Parquet partitions

`ALTER TABLE ... ALTER COLUMN ... TYPE` works on tables that contain
[Parquet partitions](/docs/query/export-parquet/#in-place-conversion). Reads
return the converted values on every query, matching the behaviour of native
partitions exactly, including `NULL` sentinels, `BYTE`/`SHORT`/`CHAR` column
tops, date and timestamp scaling, and UTF-16/UTF-8 text transcoding.

By default the conversion is **lazy**: the Parquet partitions are not rewritten
by the `ALTER`. The type change is recorded in the table metadata and applied on
the query path, where each Parquet row group is decoded once and buffered for
the rows that follow. This keeps the `ALTER` cheap regardless of how much data
is stored as Parquet.

An [out-of-order (O3)](/docs/concepts/out-of-order-data/) write that later lands
on a converted Parquet partition rewrites that partition with the new type,
materialising the conversion permanently for that partition.

:::note Eager conversions

Some conversions cannot be applied lazily. In these cases QuestDB first converts
the affected Parquet partitions back to native format, one rewrite per affected
partition, before applying the type change:

- Conversions where the source or target type is `SYMBOL`.
- Chained conversions where the on-disk Parquet type no longer matches the
current column metadata, for example a second type change on a column that was
already converted lazily.

Unlike the metadata-only lazy path, these partition rewrites carry an I/O cost
proportional to the amount of Parquet data in the affected partitions.

:::

## Unsupported Conversions

Converting from the type to itself is not supported.
Expand Down
Loading