Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 16 additions & 9 deletions design/PHASE_G_EXTERNAL_PARQUET_PLAN.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,14 @@
# Phase G: read external Parquet in place (design, for review)

Status: proposed, not implemented. This is a large net-new capability, not a
correctness-critical change to existing storage, so it is written up for review
and a surface decision before code. Parquet first; ORC and the table formats
(Iceberg, Delta) are follow-on phases that build on the same scan core.
Status: **shipped** (external Parquet read, the FDW surface, projection and predicate
pushdown, multi-file and partition pruning). Kept as the design record.

**Superseded by `design/ROADMAP.md` for anything still outstanding.** This document
describes the state at the time it was written and its open questions have been
answered or moved. Do not plan from it.

Parquet first; ORC and the table formats (Iceberg, Delta) are follow-on phases that
build on the same scan core. Iceberg is #388.

## Goal

Expand Down Expand Up @@ -165,11 +170,13 @@ partitions from predicates on those columns.
## Open decisions for review

- Confirm building **both** surfaces (this document assumes yes).
- File access scope: DECIDED (2026-07-23) -- local filesystem first. Object storage
(S3 and S3-compatible such as MinIO, plus GCS/Azure) is a future todo behind the
same path/URL option: an `s3://bucket/key` path resolves through an object-store
reader while `/path/file.parquet` reads the local FS. The scan core is unchanged
either way because both just hand it bytes.
- File access scope: DECIDED (2026-07-23) -- local filesystem first. **Object storage
is now tracked as outstanding work in `design/ROADMAP.md`, and as #393 (reads) and
#394 (writes).** It is not recorded here any more, because a decided item that is
also outstanding work does not belong under a heading called "open decisions".
Note for anyone reading the original wording: it said the scan core is unchanged
because both paths just hand it bytes. That is true of the core and understates the
work. See #393.
- Whether the function's column list is always caller-supplied (`AS (...)`) or we
also provide a fixed-shape `read_parquet(path, columns jsonb)` convenience.
Recommend caller-supplied `AS`, matching every other record-returning function,
Expand Down
6 changes: 6 additions & 0 deletions design/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,12 @@ matrix. Gap specifications are in [gaps/](gaps/).

## Remaining

**Object storage, read and write.** Parquet and Iceberg data normally live on S3, GCS
or ADLS, and we read local files only. This is a prerequisite for #388 rather than a
parallel feature. Reads are #393, writes are #394. It was previously recorded only in
`PHASE_G_EXTERNAL_PARQUET_PLAN.md` under a heading reading "open decisions", which is
why nobody found it.

Ordered by value-to-effort. **Gap 27 (Arrow/Parquet interop) is fully complete**:
export and import, flat and nested, for both Arrow and Parquet, all self-contained
(no libarrow/libparquet dependency) and matrix-gated. See
Expand Down
56 changes: 56 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Roadmap

What is done, what is planned, and where the detail lives.

This page is the entry point. The working record is
[design/ROADMAP.md](https://github.com/commandprompt/pgcolumnar/blob/main/design/ROADMAP.md),
which carries the full list with per-item history. Issues are the authority on anything
being worked now.

## Status

pgColumnar is [pre-release](limitations.md#release-status). The version marker is
`1.0-alpha`. A table `USING pgcolumnar` is stored in the native on-disk format, PGCN v1.

## Done

The large pieces that have shipped:

- **Storage and scan.** Native PGCN v1 format, zone maps and bloom filters for skipping,
column projection, vectorized execution, delete vectors.
- **Interoperability.** Arrow and Parquet, import and export, flat and nested, with no
libarrow or libparquet dependency. External Parquet read in place, with an FDW surface,
projection and predicate pushdown, multi-file reads and partition pruning.
- **Maintenance.** Vacuum, compaction, clustering and reclustering, projections.
- **PostgreSQL integration.** Read stream and asynchronous IO, virtual generated columns,
temporal constraints, statistics collection for the planner.

## Planned

Nothing here is committed to a release. Each links to the issue that owns it.

| area | item | issue |
| --- | --- | --- |
| Storage | Object storage reads for external Parquet | [#393](https://github.com/commandprompt/pgcolumnar/issues/393) |
| Storage | Object storage writes for the export functions | [#394](https://github.com/commandprompt/pgcolumnar/issues/394) |
| Formats | Apache Iceberg support | [#388](https://github.com/commandprompt/pgcolumnar/issues/388) |
| Planner | Grouped parallel aggregate arm, cost model | [#369](https://github.com/commandprompt/pgcolumnar/issues/369) |
| Benchmarks | Join-heavy analytical measurement | [#401](https://github.com/commandprompt/pgcolumnar/issues/401) |

Object storage is a prerequisite for Iceberg rather than a parallel feature. Parquet and
Iceberg data normally live on S3, GCS or ADLS, and today we read local files only.

## Under investigation

Recorded so the work is visible, without implying it will be built:

- Techniques from published columnar systems, ranked against what we already implement.
See [#403](https://github.com/commandprompt/pgcolumnar/issues/403) and
[#405](https://github.com/commandprompt/pgcolumnar/issues/405).
- PostgreSQL 19 features we can adopt, and what the 20 branch may bring.
See [#390](https://github.com/commandprompt/pgcolumnar/issues/390).

## What this page is not

It is not a commitment, and it is not a schedule. An item here means the work is
recorded and reasoned about. It does not mean anyone is working on it.
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -110,4 +110,5 @@ nav:
- SQL reference: sql-reference.md
- Limitations & compatibility: limitations.md
- Development:
- Roadmap: roadmap.md
- Testing: testing.md
6 changes: 6 additions & 0 deletions test/docs_style.sh
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,12 @@ check "and it actually examined the documents" \
"$([ "$n" -ge 10 ] && echo yes || echo "no (examined $n)")" "yes"

# CHANGELOG.md: dash characters only. See the scope note above.
# The roadmap has to stay reachable. It went unfound once because the only routes to it
# were a raw GitHub link and a line in the changelog (#395). A page that is not in the nav
# is not published, and nothing else would notice.
nav_roadmap=$(grep -c "roadmap.md" "$SRCDIR/mkdocs.yml" || true)
check "the roadmap is in the documentation nav" "$([ "$nav_roadmap" -ge 1 ] && echo yes || echo no)" "yes"

dashes=$(grep -c '—\|–' "$SRCDIR/CHANGELOG.md" || true)
check "CHANGELOG.md carries no em or en dash" "$dashes" "0"

Expand Down
Loading