Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 34 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,40 @@ pre-release; the version marker is `1.0-alpha`, recorded in `VERSION`. New table
are written in the native on-disk format, PGCN v1. For the forward-looking plan see
[design/ROADMAP.md](design/ROADMAP.md); for full history see the git log.

The extension's own `default_version` is still `1.0-dev`. That is deliberate: it
governs `ALTER EXTENSION UPDATE`, and moving it needs an upgrade script that does
not exist yet. `SELECT extversion FROM pg_extension` therefore reports `1.0-dev`
on a 1.0-alpha build.
The extension's `default_version` is `1.0-alpha`, and an upgrade script ships with
it. Older notes in this file describe `default_version` as pinned at `1.0-dev`,
which was true until that script existed.

## [Unreleased]

### Changed

- The extension's exported C symbols are namespaced under `pgcolumnar` (#382).
Two extensions that both call themselves `columnar` could define the same
symbol. `columnar_handler` and `columnar_relation_storageid` collided with
Citus columnar. Four settings variables such as `columnar_stripe_row_limit`
also shared names with the same settings there. That case binds one library's
setting to the other's storage.
- `default_version` moves from `1.0-dev` to `1.0-alpha`, so
`SELECT extversion FROM pg_extension` now agrees with `VERSION`.

### Upgrading

**Run `ALTER EXTENSION pgcolumnar UPDATE;` in every database that has the
extension, after installing this build.**

The rename moves the C symbol names that each installed function recorded when it
was created. Replace the shared library without this step and those records
point at symbols the new library does not export. The extension then stops
working until the catalog is updated. Reading an existing columnar table fails
with `could not find function "columnar_handler"`.

Nothing happens to your data, and no conversion runs. The upgrade replaces
catalog entries only, and keeps each function's identity, so the access method
binding and every dependency survive. The SQL you write does not change.

See [Upgrade](docs/installation.md#upgrade) for the commands, including how to
list the databases that need it.

## [1.0-alpha] - 2026-08-04

Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ OBJS = \
src/columnar_parallel_export.o

EXTENSION = pgcolumnar
DATA = pgcolumnar--1.0-dev.sql
DATA = pgcolumnar--1.0-alpha.sql pgcolumnar--1.0-dev--1.0-alpha.sql
PGFILEDESC = "pgColumnar - column-oriented table access method"

# make installcheck. Not the project's gate -- that is test/run_all_versions.sh,
Expand Down
43 changes: 43 additions & 0 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,49 @@ To install a new build of the extension:

1. Run `make install` with the same `PG_CONFIG`.
2. Start the server again, so that it loads the new library.
3. Run `ALTER EXTENSION pgcolumnar UPDATE;` **in every database that has the
extension**.

Step 3 is not optional, and it is easy to miss because nothing prompts for it.
The first two steps replace the shared library. The third updates the catalog to
match it.

```sql
-- in each database that ran CREATE EXTENSION pgcolumnar
ALTER EXTENSION pgcolumnar UPDATE;
SELECT extversion FROM pg_extension WHERE extname = 'pgcolumnar';
```

To find the databases that need it:

```sql
SELECT datname FROM pg_database WHERE datallowconn
AND EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pgcolumnar');
```

### If you skipped step 3

Every function the extension installs records the name of a C symbol. When those
names change between builds, the recorded names no longer resolve, and the
extension stops working until the catalog is updated. Reading an existing
columnar table then fails:

```
ERROR: could not find function "columnar_handler" in file ".../pgcolumnar.so"
```

The fix is step 3. Run `ALTER EXTENSION pgcolumnar UPDATE;` in that database and
the error goes away. **Your data is not affected.** The tables are intact and no
conversion happens. Only the catalog entry is stale.

Do not run `DROP EXTENSION`. It removes your columnar tables with it.

### Upgrading from 1.0-dev to 1.0-alpha

This release renames the extension's C symbols into the `pgcolumnar` namespace,
so that two extensions named `columnar` can be loaded without colliding. That is
the change step 3 applies. The SQL you write does not change. Function names,
settings and table syntax are all the same.

The source records the on-disk format version. The specification also records it,
in [../design/NATIVE_FORMAT_AND_INTERFACE_SPEC.md](https://github.com/commandprompt/pgcolumnar/blob/main/design/NATIVE_FORMAT_AND_INTERFACE_SPEC.md).
Expand Down
26 changes: 19 additions & 7 deletions docs/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,16 +89,28 @@ A physical copy does not replace the source across a version change.

The same posture covers the extension's own catalog, not only the on-disk data
format. The install script of a build defines the `pgcolumnar` catalog tables for a fresh
`CREATE EXTENSION`. The pre-release ships no `ALTER EXTENSION UPDATE` scripts, so
there is no in-place catalog migration either.
`CREATE EXTENSION`. An `ALTER EXTENSION UPDATE` script ships when a build needs one.
The 1.0-dev to 1.0-alpha script is the first.

You can replace the shared library and the SQL script, and then restart, without
a new `CREATE EXTENSION`. This can leave a catalog table without a column that a
newer build needs. For example, `sort_by` was added to `pgcolumnar.options`. A function that
**Replacing the shared library is not sufficient on its own.** After installing a new
build, run `ALTER EXTENSION pgcolumnar UPDATE;` in every database that has the extension.
The library and the catalog have to agree, and only that command updates the catalog.

Skipping it can leave the catalog describing the previous build. Each installed function
records the name of a C symbol. 1.0-alpha moved those names, so an un-updated catalog
names symbols the new library does not export. Reading an existing columnar table then
fails with `could not find function "columnar_handler"`. The data is untouched, and the
command above fixes it. See [Upgrade](installation.md#upgrade).

A catalog can also lack a column that a newer build needs, where no upgrade script covers
the gap. For example, `sort_by` was added to `pgcolumnar.options`. A function that
uses that column fails against an `options` table that an older build created.

Across an incompatible build, recreate the extension with `DROP EXTENSION` and
`CREATE EXTENSION`, and load the data again. Do not replace the files in place. A
Across an incompatible build, meaning one where no upgrade script covers the change,
recreate the extension with `DROP EXTENSION` and `CREATE EXTENSION`, and load the data
again. This is not the remedy for the un-updated catalog described above, where
`ALTER EXTENSION pgcolumnar UPDATE` is enough. `DROP EXTENSION` removes your columnar
tables with it. Do not replace the files in place. A
dump that exists still restores into a newer build. `pg_dump` writes an explicit
column list for the configuration tables of the extension, and a new column takes
its default of NULL.
Expand Down
50 changes: 25 additions & 25 deletions pgcolumnar--1.0-dev.sql → pgcolumnar--1.0-alpha.sql
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,7 @@ CREATE INDEX free_space_fit
CREATE FUNCTION pgcolumnar.columnar_handler(internal)
RETURNS table_am_handler
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_handler';
AS 'MODULE_PATHNAME', 'pgcolumnar_handler';

CREATE ACCESS METHOD pgcolumnar
TYPE TABLE
Expand Down Expand Up @@ -492,7 +492,7 @@ COMMENT ON FUNCTION pgcolumnar.reset_options(regclass, bool, bool, bool, bool, b
CREATE FUNCTION pgcolumnar.get_storage_id(rel regclass)
RETURNS bigint
LANGUAGE C STABLE STRICT
AS 'MODULE_PATHNAME', 'columnar_relation_storageid';
AS 'MODULE_PATHNAME', 'pgcolumnar_relation_storageid';

COMMENT ON FUNCTION pgcolumnar.get_storage_id(regclass)
IS 'storage id linking a columnar table to its metadata rows';
Expand All @@ -504,15 +504,15 @@ CREATE FUNCTION pgcolumnar.add_projection(
sort_key text[] DEFAULT '{}')
RETURNS void
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_add_projection';
AS 'MODULE_PATHNAME', 'pgcolumnar_add_projection';

COMMENT ON FUNCTION pgcolumnar.add_projection(regclass, text, text[], text[])
IS 'declare a physical projection: a named column subset sorted on sort_key (gap 26)';

CREATE FUNCTION pgcolumnar.drop_projection(rel regclass, name text)
RETURNS void
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_drop_projection';
AS 'MODULE_PATHNAME', 'pgcolumnar_drop_projection';

COMMENT ON FUNCTION pgcolumnar.drop_projection(regclass, text)
IS 'drop a declared projection and free its storage (gap 26)';
Expand Down Expand Up @@ -572,15 +572,15 @@ COMMENT ON FUNCTION pgcolumnar.rebuild_projections(regclass)
CREATE FUNCTION pgcolumnar.read_projection(rel regclass, name text)
RETURNS SETOF text
LANGUAGE C STABLE
AS 'MODULE_PATHNAME', 'columnar_read_projection';
AS 'MODULE_PATHNAME', 'pgcolumnar_read_projection';

COMMENT ON FUNCTION pgcolumnar.read_projection(regclass, text)
IS 'read a projection''s stored columns (live rows), joined by | -- verification/debug (gap 26)';

CREATE FUNCTION pgcolumnar.reconstruct_via_projection(rel regclass, name text)
RETURNS SETOF text
LANGUAGE C STABLE
AS 'MODULE_PATHNAME', 'columnar_reconstruct_via_projection';
AS 'MODULE_PATHNAME', 'pgcolumnar_reconstruct_via_projection';

COMMENT ON FUNCTION pgcolumnar.reconstruct_via_projection(regclass, text)
IS 'read all live rows via a projection, reconstructing non-covered columns from the base by row number (gap 26)';
Expand Down Expand Up @@ -707,7 +707,7 @@ COMMENT ON FUNCTION pgcolumnar.sort_status(regclass)
CREATE FUNCTION pgcolumnar.vacuum(tablename regclass, stripe_count int DEFAULT 0)
RETURNS void
LANGUAGE C STRICT
AS 'MODULE_PATHNAME', 'columnar_vacuum';
AS 'MODULE_PATHNAME', 'pgcolumnar_vacuum';

COMMENT ON FUNCTION pgcolumnar.vacuum(regclass, int)
IS 'compact a columnar table by combining stripes and reclaiming deleted rows';
Expand All @@ -717,7 +717,7 @@ CREATE FUNCTION pgcolumnar.vacuum_sorted(
VARIADIC sort_columns name[])
RETURNS void
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_vacuum_sorted';
AS 'MODULE_PATHNAME', 'pgcolumnar_vacuum_sorted';

COMMENT ON FUNCTION pgcolumnar.vacuum_sorted(regclass, name[])
IS 'compact a columnar table, storing rows sorted ascending (NULLS LAST) on the given columns. With no columns, applies the table''s declared sort_by key from set_options (#288), like a bare CLUSTER re-applying a remembered index; errors if none is declared. Supports any btree-orderable column including text (unlike the numeric-only Z-order cluster()). One-shot: not auto-maintained.';
Expand All @@ -732,7 +732,7 @@ COMMENT ON FUNCTION pgcolumnar.vacuum_sorted(regclass, name[])
CREATE FUNCTION pgcolumnar.vacuum_sorted(tablename regclass)
RETURNS void
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_vacuum_sorted';
AS 'MODULE_PATHNAME', 'pgcolumnar_vacuum_sorted';

COMMENT ON FUNCTION pgcolumnar.vacuum_sorted(regclass)
IS 'apply the table''s declared sort_by key from set_options (#288); errors if none is declared. Equivalent to a bare CLUSTER re-applying a remembered index.';
Expand All @@ -742,23 +742,23 @@ CREATE FUNCTION pgcolumnar.cluster(
VARIADIC columns name[])
RETURNS void
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_cluster';
AS 'MODULE_PATHNAME', 'pgcolumnar_cluster';

COMMENT ON FUNCTION pgcolumnar.cluster(regclass, name[])
IS 'eager reorg: rewrite a columnar table with rows ordered by the Z-order space-filling curve over the given columns. Holds AccessExclusiveLock like CLUSTER/VACUUM FULL; the online incremental path is Phase F3';

CREATE FUNCTION pgcolumnar.compact(tablename regclass)
RETURNS bigint
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_compact';
AS 'MODULE_PATHNAME', 'pgcolumnar_compact';

COMMENT ON FUNCTION pgcolumnar.compact(regclass)
IS 'lazy online compaction: retire row groups that are fully deleted, dropping their metadata so scans skip them. Holds only ShareUpdateExclusiveLock (concurrent reads and writes). Returns the number of groups retired (Phase F3a)';

CREATE FUNCTION pgcolumnar.truncate(tablename regclass)
RETURNS bigint
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_truncate';
AS 'MODULE_PATHNAME', 'pgcolumnar_truncate';

COMMENT ON FUNCTION pgcolumnar.truncate(regclass)
IS 'physical end-truncation: return trailing reclaimed blocks to the OS. Best-effort -- takes AccessExclusiveLock conditionally for the brief physical step and returns 0 without waiting if the table is busy. Only removes space freed before the oldest-xmin horizon. Gated by pgcolumnar.enable_end_truncation. Returns the number of blocks truncated (Phase F)';
Expand All @@ -769,7 +769,7 @@ CREATE FUNCTION pgcolumnar.compact_rewrite(
max_groups int DEFAULT 0)
RETURNS bigint
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_compact_rewrite';
AS 'MODULE_PATHNAME', 'pgcolumnar_compact_rewrite';

COMMENT ON FUNCTION pgcolumnar.compact_rewrite(regclass, float8, int)
IS 'lazy online space reclaim: rewrite partially-deleted row groups (deleted fraction >= min_deleted_fraction) to drop their dead rows, under ShareUpdateExclusiveLock (concurrent reads and writes). Returns the number of groups rewritten (Phase F3b)';
Expand All @@ -779,23 +779,23 @@ CREATE FUNCTION pgcolumnar.recluster(
VARIADIC columns name[])
RETURNS bigint
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_recluster';
AS 'MODULE_PATHNAME', 'pgcolumnar_recluster';

COMMENT ON FUNCTION pgcolumnar.recluster(regclass, name[])
IS 'lazy online reclustering: re-establish global Z-order clustering over the given columns under ShareUpdateExclusiveLock (concurrent reads and writes), unlike the eager cluster() which holds AccessExclusiveLock. Returns the number of groups reclustered (Phase F3c)';

CREATE FUNCTION pgcolumnar.export_arrow(rel regclass, path text)
RETURNS bigint
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_export_arrow';
AS 'MODULE_PATHNAME', 'pgcolumnar_export_arrow';

COMMENT ON FUNCTION pgcolumnar.export_arrow(regclass, text)
IS 'export a columnar table to an Arrow IPC stream file; returns rows written';

CREATE FUNCTION pgcolumnar.export_parquet(rel regclass, path text)
RETURNS bigint
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_export_parquet';
AS 'MODULE_PATHNAME', 'pgcolumnar_export_parquet';

COMMENT ON FUNCTION pgcolumnar.export_parquet(regclass, text)
IS 'export a columnar table to a Parquet file; returns rows written';
Expand All @@ -804,39 +804,39 @@ CREATE FUNCTION pgcolumnar.parallel_export_parquet(target regclass, path text,
workers int DEFAULT NULL)
RETURNS bigint
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_parallel_export_parquet';
AS 'MODULE_PATHNAME', 'pgcolumnar_parallel_export_parquet';

COMMENT ON FUNCTION pgcolumnar.parallel_export_parquet(regclass, text, int)
IS 'parallel Parquet export using read-only background workers into a directory readable by pgcolumnar.read_parquet: a single columnar table split by row-group ranges, or a partitioned columnar table one file per partition; returns rows written (#300)';

CREATE FUNCTION pgcolumnar.import_arrow(rel regclass, path text)
RETURNS bigint
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_import_arrow';
AS 'MODULE_PATHNAME', 'pgcolumnar_import_arrow';

COMMENT ON FUNCTION pgcolumnar.import_arrow(regclass, text)
IS 'insert rows from an Arrow IPC stream file into a columnar table; returns rows inserted';

CREATE FUNCTION pgcolumnar.import_parquet(rel regclass, path text)
RETURNS bigint
LANGUAGE C STRICT
AS 'MODULE_PATHNAME', 'columnar_import_parquet';
AS 'MODULE_PATHNAME', 'pgcolumnar_import_parquet';

COMMENT ON FUNCTION pgcolumnar.import_parquet(regclass, text)
IS 'insert rows from a Parquet file, directory, or glob into a table; returns rows inserted (gap 27)';

CREATE FUNCTION pgcolumnar.parquet_schema(path text)
RETURNS TABLE(column_name text, data_type text, nullable boolean)
LANGUAGE C STRICT
AS 'MODULE_PATHNAME', 'columnar_parquet_schema';
AS 'MODULE_PATHNAME', 'pgcolumnar_parquet_schema';

COMMENT ON FUNCTION pgcolumnar.parquet_schema(text)
IS 'report the leaf columns of a Parquet file and the PostgreSQL type each maps to; for a directory or glob, of its first file (Phase G scan core)';

CREATE FUNCTION pgcolumnar.read_parquet(path text)
RETURNS SETOF record
LANGUAGE C STRICT
AS 'MODULE_PATHNAME', 'columnar_read_parquet';
AS 'MODULE_PATHNAME', 'pgcolumnar_read_parquet';

COMMENT ON FUNCTION pgcolumnar.read_parquet(text)
IS 'read a Parquet file, directory, or glob in place as a set of rows; requires a column definition list covering every leaf column, e.g. SELECT * FROM pgcolumnar.read_parquet(path) AS t(id int, name text) (Phase G)';
Expand Down Expand Up @@ -872,15 +872,15 @@ COMMENT ON FOREIGN DATA WRAPPER pgcolumnar_parquet
CREATE FUNCTION pgcolumnar.vm_selftest(rel regclass, blk int)
RETURNS boolean
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_vm_selftest';
AS 'MODULE_PATHNAME', 'pgcolumnar_vm_selftest';

COMMENT ON FUNCTION pgcolumnar.vm_selftest(regclass, int)
IS 'gap 28 phase-1 self-test: set a VM-fork all-visible bit and read it back';

CREATE FUNCTION pgcolumnar.vm_is_visible(rel regclass, blk int)
RETURNS boolean
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_vm_is_visible';
AS 'MODULE_PATHNAME', 'pgcolumnar_vm_is_visible';

COMMENT ON FUNCTION pgcolumnar.vm_is_visible(regclass, int)
IS 'gap 28: is the synthetic block marked all-visible in the VM fork?';
Expand Down Expand Up @@ -927,7 +927,7 @@ COMMENT ON FUNCTION pgcolumnar.vacuum_full(name, real, int)
CREATE FUNCTION pgcolumnar.file_split_offsets(path text, workers int)
RETURNS bigint[]
LANGUAGE C STRICT
AS 'MODULE_PATHNAME', 'columnar_file_split_offsets';
AS 'MODULE_PATHNAME', 'pgcolumnar_file_split_offsets';

COMMENT ON FUNCTION pgcolumnar.file_split_offsets(text, int)
IS 'byte offsets that split a COPY text-format file into N record-aligned ranges (#300)';
Expand Down Expand Up @@ -956,7 +956,7 @@ CREATE FUNCTION pgcolumnar.parallel_copy(target regclass, filename text,
workers int DEFAULT NULL)
RETURNS bigint
LANGUAGE C
AS 'MODULE_PATHNAME', 'columnar_parallel_copy';
AS 'MODULE_PATHNAME', 'pgcolumnar_parallel_copy';

COMMENT ON FUNCTION pgcolumnar.parallel_copy(regclass, text, int)
IS 'atomic parallel bulk load of a COPY text file into a columnar table using background workers: a single columnar table (any row order), or a RANGE-partitioned columnar table sorted by the partition key with one distinct partition set per worker (#300)';
Loading
Loading