diff --git a/design/PHASE_G_EXTERNAL_PARQUET_PLAN.md b/design/PHASE_G_EXTERNAL_PARQUET_PLAN.md index ae27db0..c07e8ba 100644 --- a/design/PHASE_G_EXTERNAL_PARQUET_PLAN.md +++ b/design/PHASE_G_EXTERNAL_PARQUET_PLAN.md @@ -1,9 +1,14 @@ # Phase G: read external Parquet in place (design, for review) -Status: proposed, not implemented. This is a large net-new capability, not a -correctness-critical change to existing storage, so it is written up for review -and a surface decision before code. Parquet first; ORC and the table formats -(Iceberg, Delta) are follow-on phases that build on the same scan core. +Status: **shipped** (external Parquet read, the FDW surface, projection and predicate +pushdown, multi-file and partition pruning). Kept as the design record. + +**Superseded by `design/ROADMAP.md` for anything still outstanding.** This document +describes the state at the time it was written and its open questions have been +answered or moved. Do not plan from it. + +Parquet first; ORC and the table formats (Iceberg, Delta) are follow-on phases that +build on the same scan core. Iceberg is #388. ## Goal @@ -165,11 +170,13 @@ partitions from predicates on those columns. ## Open decisions for review - Confirm building **both** surfaces (this document assumes yes). -- File access scope: DECIDED (2026-07-23) -- local filesystem first. Object storage - (S3 and S3-compatible such as MinIO, plus GCS/Azure) is a future todo behind the - same path/URL option: an `s3://bucket/key` path resolves through an object-store - reader while `/path/file.parquet` reads the local FS. The scan core is unchanged - either way because both just hand it bytes. +- File access scope: DECIDED (2026-07-23) -- local filesystem first. **Object storage + is now tracked as outstanding work in `design/ROADMAP.md`, and as #393 (reads) and + #394 (writes).** It is not recorded here any more, because a decided item that is + also outstanding work does not belong under a heading called "open decisions". + Note for anyone reading the original wording: it said the scan core is unchanged + because both paths just hand it bytes. That is true of the core and understates the + work. See #393. - Whether the function's column list is always caller-supplied (`AS (...)`) or we also provide a fixed-shape `read_parquet(path, columns jsonb)` convenience. Recommend caller-supplied `AS`, matching every other record-returning function, diff --git a/design/ROADMAP.md b/design/ROADMAP.md index 05d72c1..b012df7 100644 --- a/design/ROADMAP.md +++ b/design/ROADMAP.md @@ -43,6 +43,12 @@ matrix. Gap specifications are in [gaps/](gaps/). ## Remaining +**Object storage, read and write.** Parquet and Iceberg data normally live on S3, GCS +or ADLS, and we read local files only. This is a prerequisite for #388 rather than a +parallel feature. Reads are #393, writes are #394. It was previously recorded only in +`PHASE_G_EXTERNAL_PARQUET_PLAN.md` under a heading reading "open decisions", which is +why nobody found it. + Ordered by value-to-effort. **Gap 27 (Arrow/Parquet interop) is fully complete**: export and import, flat and nested, for both Arrow and Parquet, all self-contained (no libarrow/libparquet dependency) and matrix-gated. See @@ -244,14 +250,16 @@ preserve the 15-19 matrix. Detail and sources in [PG18_19_OPPORTUNITIES.md](PG18_19_OPPORTUNITIES.md): - Read stream / AIO in the scan — shipped, see the Done table. -- Virtual generated columns (PostgreSQL 18): confirm read-time generation on a - columnar table and add differential coverage. -- Temporal constraints (`WITHOUT OVERLAPS` in 18, `FOR PORTION OF` in 19): verify - enforcement and add coverage. -- REPACK (PostgreSQL 19): investigate whether concurrent, lower-lock compaction is - reachable through the table AM. +- Virtual generated columns (PostgreSQL 18) — done, covered by + `test/generated_columns.sh`. Stored and virtual columns both read correctly on a + columnar table, and a virtual column has no chunk written for it at all + (`columnar_write_state.c` skips `attgenerated == 'v'`), which the same suite pins. +- Temporal constraints (`WITHOUT OVERLAPS` in 18, `FOR PORTION OF` in 19) — done, + covered by `test/temporal.sh` against the heap oracle, `FOR PORTION OF` gated to 19. +- REPACK (PostgreSQL 19) — investigated, and the answer was no. It does not work on a + columnar table, and the error now says so (#399, #409). Not a remaining item. - Optimizer statistics injection (PostgreSQL 18) and a btree skip-scan benchmark - line: smaller follow-ups. + line: smaller follow-ups, still open with no measurement. ## Test-harness follow-up diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..88b8038 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,56 @@ +# Roadmap + +What is done, what is planned, and where the detail lives. + +This page is the entry point. The working record is +[design/ROADMAP.md](https://github.com/commandprompt/pgcolumnar/blob/main/design/ROADMAP.md), +which carries the full list with per-item history. Issues are the authority on anything +being worked now. + +## Status + +pgColumnar is [pre-release](limitations.md#release-status). The version marker is +`1.0-alpha`. A table `USING pgcolumnar` is stored in the native on-disk format, PGCN v1. + +## Done + +The large pieces that have shipped: + +- **Storage and scan.** Native PGCN v1 format, zone maps and bloom filters for skipping, + column projection, vectorized execution, delete vectors. +- **Interoperability.** Arrow and Parquet, import and export, flat and nested, with no + libarrow or libparquet dependency. External Parquet read in place, with an FDW surface, + projection and predicate pushdown, multi-file reads and partition pruning. +- **Maintenance.** Vacuum, compaction, clustering and reclustering, projections. +- **PostgreSQL integration.** Read stream and asynchronous IO, virtual generated columns, + temporal constraints, statistics collection for the planner. + +## Planned + +Nothing here is committed to a release. Each links to the issue that owns it. + +| area | item | issue | +| --- | --- | --- | +| Storage | Object storage reads for external Parquet | [#393](https://github.com/commandprompt/pgcolumnar/issues/393) | +| Storage | Object storage writes for the export functions | [#394](https://github.com/commandprompt/pgcolumnar/issues/394) | +| Formats | Apache Iceberg support | [#388](https://github.com/commandprompt/pgcolumnar/issues/388) | +| Planner | Grouped parallel aggregate arm, cost model | [#369](https://github.com/commandprompt/pgcolumnar/issues/369) | +| Benchmarks | Join-heavy analytical measurement | [#401](https://github.com/commandprompt/pgcolumnar/issues/401) | + +Object storage is a prerequisite for Iceberg rather than a parallel feature. Parquet and +Iceberg data normally live on S3, GCS or ADLS, and today we read local files only. + +## Under investigation + +Recorded so the work is visible, without implying it will be built: + +- Techniques from published columnar systems, ranked against what we already implement. + See [#403](https://github.com/commandprompt/pgcolumnar/issues/403) and + [#405](https://github.com/commandprompt/pgcolumnar/issues/405). +- PostgreSQL 19 features we can adopt, and what the 20 branch may bring. + See [#390](https://github.com/commandprompt/pgcolumnar/issues/390). + +## What this page is not + +It is not a commitment, and it is not a schedule. An item here means the work is +recorded and reasoned about. It does not mean anyone is working on it. diff --git a/mkdocs.yml b/mkdocs.yml index 4a38893..afdda3f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -110,4 +110,5 @@ nav: - SQL reference: sql-reference.md - Limitations & compatibility: limitations.md - Development: + - Roadmap: roadmap.md - Testing: testing.md diff --git a/test/docs_style.sh b/test/docs_style.sh index f7597d2..9e57b42 100755 --- a/test/docs_style.sh +++ b/test/docs_style.sh @@ -65,6 +65,20 @@ n=$(echo "$out" | grep -c '^ ok' || true) check "and it actually examined the documents" \ "$([ "$n" -ge 10 ] && echo yes || echo "no (examined $n)")" "yes" +# The roadmap has to stay reachable. It went unfound once because the only routes to it +# were a raw GitHub link and a line in the changelog (#395). A page that is not in the nav +# is not published, and nothing else would notice. +# +# Reachability is two facts, so both are asserted. The nav check alone passes when the +# PAGE is deleted and the entry is kept, which is a broken link rather than reachability. +# That case is also caught by "mkdocs build --strict" in docs.yml, which fails on a nav +# entry pointing at nothing. Half a property here and half in a workflow is how the +# missing half goes unnoticed, so both halves are stated here. +nav_roadmap=$(grep -c "roadmap.md" "$SRCDIR/mkdocs.yml" || true) +check "the roadmap is in the documentation nav" "$([ "$nav_roadmap" -ge 1 ] && echo yes || echo no)" "yes" +check "and the page that nav entry points at exists" \ + "$([ -f "$SRCDIR/docs/roadmap.md" ] && echo yes || echo no)" "yes" + # CHANGELOG.md: dash characters only. See the scope note above. dashes=$(grep -c '—\|–' "$SRCDIR/CHANGELOG.md" || true) check "CHANGELOG.md carries no em or en dash" "$dashes" "0"