Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,15 @@ which was true until that script existed.

### Added

- `pgcolumnar.maintenance_due(rel, compact_due_fraction, recluster_due_fraction)`
reports whether an online maintenance verb is worth running on a table, from its
statistics alone (#415). It takes no lock and rewrites nothing. It returns the
deleted and appended fractions, whether `compact_rewrite` or `recluster` has
crossed its threshold, and a `recommendation`. This is the policy the
`pgcolumnar.autovacuum` daemon consults, and a monitoring role can call it
directly. It is `SECURITY DEFINER` and checks that the caller may `SELECT` the
table, so the owner can run it without superuser rights.

- `pgcolumnar.parallel_flush` dispatches a stripe flush across background workers
(#445). Default off. When on, a flush of two or more columns fans the per-column
encode and compress work out to a worker pool. Any column a worker does not
Expand Down Expand Up @@ -85,6 +94,13 @@ which was true until that script existed.

### Fixed

- `pgcolumnar.sort_status` works for a non-superuser who owns the table (#608).
The function reads pgColumnar's internal catalogs, which carry no `GRANT`. As an
invoker-rights function it therefore failed for any caller who was not a
superuser. It is now `SECURITY DEFINER` and checks that the caller may `SELECT`
the table. The owner can read the sort status of their own table, and no caller
gains access to a table they could not already read.

- `pgcolumnar.analyze()` counts `null_frac` over live rows (#485). It came from
the zone maps, which record what was written, so a deleted row kept counting
toward the denominator until the table was rewritten. `VACUUM` did not correct
Expand Down Expand Up @@ -180,6 +196,15 @@ which was true until that script existed.

### Changed

- `pgcolumnar.recluster` no longer rewrites a table that is already clustered by
the requested key (#415). The function records the clustering key and kind it
establishes. A later call with the same key returns 0 and touches nothing when
the existing sorted run still covers every row group. Before this, it re-sorted
on every call, so a scheduled recluster rewrote the whole table each time, which
is why the maintenance daemon could not have run it safely. `pgcolumnar.sort_status`
now reports this recorded key as `sort_key`, and falls back to the declared
`sort_by` when there is no recorded key.

- A columnar scan whose filter cannot be pushed down now skips decoding the
projected columns of a 1024-row vector that holds no matching row (#452). The
scan decodes the filter columns first, rules out the vectors with no match, and
Expand Down
36 changes: 35 additions & 1 deletion docs/sql-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,12 @@ reclustered.
SELECT pgcolumnar.recluster('events', 'customer_id', 'ts');
```

Re-running `recluster` with the same key on a table whose clustering is intact is
a fast no-op. The function records the key it last established. It returns 0
without rewriting anything when the recorded key matches, the kind is Z-order,
and the existing sorted run already covers every row group. This is what lets the
maintenance daemon call it on a schedule without churning storage.

### pgcolumnar.compact(tablename regclass) returns bigint

Retires row groups that are fully deleted, dropping their metadata so scans skip
Expand Down Expand Up @@ -263,7 +269,7 @@ Reports how much of a table's sorted order is still in place. Returns one row:

| Column | Type | Meaning |
| --- | --- | --- |
| `sort_key` | name[] | The `sort_by` key declared by `set_options`, or NULL. |
| `sort_key` | name[] | The clustering key in effect. It is the key the last `recluster` recorded, or the `sort_by` declared by `set_options`, or NULL. |
| `total_groups` | bigint | Row groups in the table. |
| `sorted_groups` | bigint | Row groups written by the last ordering rewrite. |
| `appended_groups` | bigint | Row groups written after it. |
Expand Down Expand Up @@ -304,6 +310,34 @@ which counts as appended. The record is internal storage metadata and `pg_dump`
does not carry it, so a restored table reports no sorted groups until you sort it
again.

### pgcolumnar.maintenance_due(rel regclass, compact_due_fraction float8 DEFAULT 0.2, recluster_due_fraction float8 DEFAULT 0.05)

Reports whether an online maintenance verb is worth running on a table, from its
statistics alone. It takes no lock and rewrites nothing. This is the policy the
`pgcolumnar.autovacuum` daemon consults on each sweep. You can also call it from a
monitoring query. Returns one row:

| Column | Type | Meaning |
| --- | --- | --- |
| `total_rows` | bigint | Stored rows, including deleted rows not yet reclaimed. |
| `deleted_rows` | bigint | Rows deleted but still stored. |
| `deleted_fraction` | float8 | `deleted_rows` over `total_rows`. |
| `sort_key` | name[] | The recorded clustering key, or NULL. |
| `appended_groups` | bigint | Row groups written after the last ordering. |
| `appended_rows` | bigint | Rows in those appended groups. |
| `appended_fraction` | float8 | Appended rows over the sorted plus appended rows. |
| `compact_rewrite_due` | boolean | True when `deleted_fraction` reaches `compact_due_fraction`. |
| `recluster_due` | boolean | True when a sorted run exists and `appended_fraction` reaches `recluster_due_fraction`. |
| `recommendation` | text | The verbs to run, comma-separated, or NULL when nothing is due. |

The two thresholds default to the values the daemon uses. The function is
`SECURITY DEFINER` and checks that the caller may `SELECT` the table. A monitoring
role that owns the table can therefore call it without superuser rights.

```sql
SELECT recommendation FROM pgcolumnar.maintenance_due('events');
```

## Projections

A projection is a named subset of a table's columns stored a second time,
Expand Down
Loading