From bd031adaa3f00f5f532c3635ae4d4d32a7460d2a Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" Date: Tue, 4 Aug 2026 18:30:18 -0600 Subject: [PATCH 1/2] Make the roadmap reachable, and stop a phase plan claiming shipped work is proposed Three parts, following the plan on #395. A user-facing docs/roadmap.md, in the nav under Development. jd went looking for the file listing upcoming features and could not find it, because the only routes to design/ROADMAP.md were a raw GitHub link in docs/index.md and one in the CHANGELOG header. This publishes a new entry-point page rather than moving design/ROADMAP.md into docs/. That document is a working record: it fails the user-doc style rules in 26 places, and it carries internal notes such as an account of four previous rewrites of itself being wrong. Publishing it verbatim would put working notes on the documentation site, which is the class of problem this issue is about. The new page states status, done, planned and under-investigation, and links to the working record for detail. PHASE_G_EXTERNAL_PARQUET_PLAN.md said "Status: proposed, not implemented" for work that has shipped: the reader, the FDW surface, pushdown, multi-file and partition pruning are all in the tree with suites behind them. It now says shipped and points at ROADMAP.md for anything outstanding. The object-storage item moves out of that file's "Open decisions for review" and into ROADMAP.md's Remaining, pointing at #393 and #394. A breadcrumb stays behind so a reader does not conclude it was dropped. That heading was the trap: a decided item that was also outstanding work, filed under open decisions. Also carries a correction. The original wording said the scan core is unchanged because both paths just hand it bytes. True of the core, and it understates ranged GETs, concurrency, retries and credentials. docs_style gains the narrow check from the plan: fail if the roadmap is not in the nav. A page outside the nav is not published, and nothing else would notice it had gone. Closes #395. --- design/PHASE_G_EXTERNAL_PARQUET_PLAN.md | 25 +++++++---- design/ROADMAP.md | 6 +++ docs/roadmap.md | 56 +++++++++++++++++++++++++ mkdocs.yml | 1 + test/docs_style.sh | 6 +++ 5 files changed, 85 insertions(+), 9 deletions(-) create mode 100644 docs/roadmap.md diff --git a/design/PHASE_G_EXTERNAL_PARQUET_PLAN.md b/design/PHASE_G_EXTERNAL_PARQUET_PLAN.md index ae27db0..c07e8ba 100644 --- a/design/PHASE_G_EXTERNAL_PARQUET_PLAN.md +++ b/design/PHASE_G_EXTERNAL_PARQUET_PLAN.md @@ -1,9 +1,14 @@ # Phase G: read external Parquet in place (design, for review) -Status: proposed, not implemented. This is a large net-new capability, not a -correctness-critical change to existing storage, so it is written up for review -and a surface decision before code. Parquet first; ORC and the table formats -(Iceberg, Delta) are follow-on phases that build on the same scan core. +Status: **shipped** (external Parquet read, the FDW surface, projection and predicate +pushdown, multi-file and partition pruning). Kept as the design record. + +**Superseded by `design/ROADMAP.md` for anything still outstanding.** This document +describes the state at the time it was written and its open questions have been +answered or moved. Do not plan from it. + +Parquet first; ORC and the table formats (Iceberg, Delta) are follow-on phases that +build on the same scan core. Iceberg is #388. ## Goal @@ -165,11 +170,13 @@ partitions from predicates on those columns. ## Open decisions for review - Confirm building **both** surfaces (this document assumes yes). -- File access scope: DECIDED (2026-07-23) -- local filesystem first. Object storage - (S3 and S3-compatible such as MinIO, plus GCS/Azure) is a future todo behind the - same path/URL option: an `s3://bucket/key` path resolves through an object-store - reader while `/path/file.parquet` reads the local FS. The scan core is unchanged - either way because both just hand it bytes. +- File access scope: DECIDED (2026-07-23) -- local filesystem first. **Object storage + is now tracked as outstanding work in `design/ROADMAP.md`, and as #393 (reads) and + #394 (writes).** It is not recorded here any more, because a decided item that is + also outstanding work does not belong under a heading called "open decisions". + Note for anyone reading the original wording: it said the scan core is unchanged + because both paths just hand it bytes. That is true of the core and understates the + work. See #393. - Whether the function's column list is always caller-supplied (`AS (...)`) or we also provide a fixed-shape `read_parquet(path, columns jsonb)` convenience. Recommend caller-supplied `AS`, matching every other record-returning function, diff --git a/design/ROADMAP.md b/design/ROADMAP.md index 05d72c1..608d65a 100644 --- a/design/ROADMAP.md +++ b/design/ROADMAP.md @@ -43,6 +43,12 @@ matrix. Gap specifications are in [gaps/](gaps/). ## Remaining +**Object storage, read and write.** Parquet and Iceberg data normally live on S3, GCS +or ADLS, and we read local files only. This is a prerequisite for #388 rather than a +parallel feature. Reads are #393, writes are #394. It was previously recorded only in +`PHASE_G_EXTERNAL_PARQUET_PLAN.md` under a heading reading "open decisions", which is +why nobody found it. + Ordered by value-to-effort. **Gap 27 (Arrow/Parquet interop) is fully complete**: export and import, flat and nested, for both Arrow and Parquet, all self-contained (no libarrow/libparquet dependency) and matrix-gated. See diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..88b8038 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,56 @@ +# Roadmap + +What is done, what is planned, and where the detail lives. + +This page is the entry point. The working record is +[design/ROADMAP.md](https://github.com/commandprompt/pgcolumnar/blob/main/design/ROADMAP.md), +which carries the full list with per-item history. Issues are the authority on anything +being worked now. + +## Status + +pgColumnar is [pre-release](limitations.md#release-status). The version marker is +`1.0-alpha`. A table `USING pgcolumnar` is stored in the native on-disk format, PGCN v1. + +## Done + +The large pieces that have shipped: + +- **Storage and scan.** Native PGCN v1 format, zone maps and bloom filters for skipping, + column projection, vectorized execution, delete vectors. +- **Interoperability.** Arrow and Parquet, import and export, flat and nested, with no + libarrow or libparquet dependency. External Parquet read in place, with an FDW surface, + projection and predicate pushdown, multi-file reads and partition pruning. +- **Maintenance.** Vacuum, compaction, clustering and reclustering, projections. +- **PostgreSQL integration.** Read stream and asynchronous IO, virtual generated columns, + temporal constraints, statistics collection for the planner. + +## Planned + +Nothing here is committed to a release. Each links to the issue that owns it. + +| area | item | issue | +| --- | --- | --- | +| Storage | Object storage reads for external Parquet | [#393](https://github.com/commandprompt/pgcolumnar/issues/393) | +| Storage | Object storage writes for the export functions | [#394](https://github.com/commandprompt/pgcolumnar/issues/394) | +| Formats | Apache Iceberg support | [#388](https://github.com/commandprompt/pgcolumnar/issues/388) | +| Planner | Grouped parallel aggregate arm, cost model | [#369](https://github.com/commandprompt/pgcolumnar/issues/369) | +| Benchmarks | Join-heavy analytical measurement | [#401](https://github.com/commandprompt/pgcolumnar/issues/401) | + +Object storage is a prerequisite for Iceberg rather than a parallel feature. Parquet and +Iceberg data normally live on S3, GCS or ADLS, and today we read local files only. + +## Under investigation + +Recorded so the work is visible, without implying it will be built: + +- Techniques from published columnar systems, ranked against what we already implement. + See [#403](https://github.com/commandprompt/pgcolumnar/issues/403) and + [#405](https://github.com/commandprompt/pgcolumnar/issues/405). +- PostgreSQL 19 features we can adopt, and what the 20 branch may bring. + See [#390](https://github.com/commandprompt/pgcolumnar/issues/390). + +## What this page is not + +It is not a commitment, and it is not a schedule. An item here means the work is +recorded and reasoned about. It does not mean anyone is working on it. diff --git a/mkdocs.yml b/mkdocs.yml index 4a38893..afdda3f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -110,4 +110,5 @@ nav: - SQL reference: sql-reference.md - Limitations & compatibility: limitations.md - Development: + - Roadmap: roadmap.md - Testing: testing.md diff --git a/test/docs_style.sh b/test/docs_style.sh index f7597d2..0bf24e7 100755 --- a/test/docs_style.sh +++ b/test/docs_style.sh @@ -66,6 +66,12 @@ check "and it actually examined the documents" \ "$([ "$n" -ge 10 ] && echo yes || echo "no (examined $n)")" "yes" # CHANGELOG.md: dash characters only. See the scope note above. +# The roadmap has to stay reachable. It went unfound once because the only routes to it +# were a raw GitHub link and a line in the changelog (#395). A page that is not in the nav +# is not published, and nothing else would notice. +nav_roadmap=$(grep -c "roadmap.md" "$SRCDIR/mkdocs.yml" || true) +check "the roadmap is in the documentation nav" "$([ "$nav_roadmap" -ge 1 ] && echo yes || echo no)" "yes" + dashes=$(grep -c '—\|–' "$SRCDIR/CHANGELOG.md" || true) check "CHANGELOG.md carries no em or en dash" "$dashes" "0" From e14662d622941d9c07429bb25f676944503c466c Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" <136637981+ChronicallyJD@users.noreply.github.com> Date: Wed, 5 Aug 2026 08:30:32 -0600 Subject: [PATCH 2/2] docs: stop the adoption list contradicting the Done table it sits under (#395) The roadmap's Done table said PostgreSQL integration was covered while its adoption list four screens down still said go and confirm generated columns and temporal constraints, and go and investigate REPACK. Whoever marked read stream shipped left the three below it stale. The Done table was the one telling the truth. test/generated_columns.sh and test/temporal.sh both exist and run in the matrix, and REPACK was investigated and does not work on a columnar table (#399, #409). The list now says so. Verified rather than assumed, because the first draft of this commit got it wrong in the same direction: it recorded the all-null chunk for a virtual generated column as an open write-path finding. columnar_write_state.c skips attgenerated == 'v' entirely and generated_columns.sh pins it, so that shipped too, and the bullet now says what the code does. Also, the reachability check owned half its property. It grepped mkdocs.yml and never asked whether docs/roadmap.md existed, so deleting the page while keeping the nav entry passed. mkdocs build --strict catches that in docs.yml, but a property asserted half here and half in a workflow is how the missing half goes unnoticed. Both halves are now stated where the property is. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UqprqkCXuH8SegiZejE1Tw --- design/ROADMAP.md | 16 +++++++++------- test/docs_style.sh | 10 +++++++++- 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/design/ROADMAP.md b/design/ROADMAP.md index 608d65a..b012df7 100644 --- a/design/ROADMAP.md +++ b/design/ROADMAP.md @@ -250,14 +250,16 @@ preserve the 15-19 matrix. Detail and sources in [PG18_19_OPPORTUNITIES.md](PG18_19_OPPORTUNITIES.md): - Read stream / AIO in the scan — shipped, see the Done table. -- Virtual generated columns (PostgreSQL 18): confirm read-time generation on a - columnar table and add differential coverage. -- Temporal constraints (`WITHOUT OVERLAPS` in 18, `FOR PORTION OF` in 19): verify - enforcement and add coverage. -- REPACK (PostgreSQL 19): investigate whether concurrent, lower-lock compaction is - reachable through the table AM. +- Virtual generated columns (PostgreSQL 18) — done, covered by + `test/generated_columns.sh`. Stored and virtual columns both read correctly on a + columnar table, and a virtual column has no chunk written for it at all + (`columnar_write_state.c` skips `attgenerated == 'v'`), which the same suite pins. +- Temporal constraints (`WITHOUT OVERLAPS` in 18, `FOR PORTION OF` in 19) — done, + covered by `test/temporal.sh` against the heap oracle, `FOR PORTION OF` gated to 19. +- REPACK (PostgreSQL 19) — investigated, and the answer was no. It does not work on a + columnar table, and the error now says so (#399, #409). Not a remaining item. - Optimizer statistics injection (PostgreSQL 18) and a btree skip-scan benchmark - line: smaller follow-ups. + line: smaller follow-ups, still open with no measurement. ## Test-harness follow-up diff --git a/test/docs_style.sh b/test/docs_style.sh index 0bf24e7..9e57b42 100755 --- a/test/docs_style.sh +++ b/test/docs_style.sh @@ -65,13 +65,21 @@ n=$(echo "$out" | grep -c '^ ok' || true) check "and it actually examined the documents" \ "$([ "$n" -ge 10 ] && echo yes || echo "no (examined $n)")" "yes" -# CHANGELOG.md: dash characters only. See the scope note above. # The roadmap has to stay reachable. It went unfound once because the only routes to it # were a raw GitHub link and a line in the changelog (#395). A page that is not in the nav # is not published, and nothing else would notice. +# +# Reachability is two facts, so both are asserted. The nav check alone passes when the +# PAGE is deleted and the entry is kept, which is a broken link rather than reachability. +# That case is also caught by "mkdocs build --strict" in docs.yml, which fails on a nav +# entry pointing at nothing. Half a property here and half in a workflow is how the +# missing half goes unnoticed, so both halves are stated here. nav_roadmap=$(grep -c "roadmap.md" "$SRCDIR/mkdocs.yml" || true) check "the roadmap is in the documentation nav" "$([ "$nav_roadmap" -ge 1 ] && echo yes || echo no)" "yes" +check "and the page that nav entry points at exists" \ + "$([ -f "$SRCDIR/docs/roadmap.md" ] && echo yes || echo no)" "yes" +# CHANGELOG.md: dash characters only. See the scope note above. dashes=$(grep -c '—\|–' "$SRCDIR/CHANGELOG.md" || true) check "CHANGELOG.md carries no em or en dash" "$dashes" "0"