Conversation
github-actions
Bot
force-pushed
the
master
branch
5 times, most recently
from
3b6f413 to
5c6cce4
Compare
- Hourly upstream sync from postgres/postgres (24x daily) - AI-powered PR reviews using AWS Bedrock Claude Sonnet 4.5 - Multi-platform CI via existing Cirrus CI configuration - Cost tracking and comprehensive documentation Features: - Automatic issue creation on sync conflicts - PostgreSQL-specific code review prompts (C, SQL, docs, build) - Cost limits: $15/PR, $200/month - Inline PR comments with security/performance labels - Skip draft PRs to save costs Documentation: - .github/SETUP_SUMMARY.md - Quick setup overview - .github/QUICKSTART.md - 15-minute setup guide - .github/PRE_COMMIT_CHECKLIST.md - Verification checklist - .github/docs/ - Detailed guides for sync, AI review, Bedrock See .github/README.md for complete overview Complete Phase 3: Windows builds + fix sync for CI/CD commits Phase 3: Windows Dependency Build System - Implement full build workflow (OpenSSL, zlib, libxml2) - Smart caching by version hash (80% cost reduction) - Dependency bundling with manifest generation - Weekly auto-refresh + manual triggers - PowerShell download helper script - Comprehensive usage documentation Sync Workflow Fix: - Allow .github/ commits (CI/CD config) on master - Detect and reject code commits outside .github/ - Merge upstream while preserving .github/ changes - Create issues only for actual pristine violations Documentation: - Complete Windows build usage guide - Update all status docs to 100% complete - Phase 3 completion summary All three CI/CD phases complete (100%): ✅ Hourly upstream sync with .github/ preservation ✅ AI-powered PR reviews via Bedrock Claude 4.5 ✅ Windows dependency builds with smart caching Cost: $40-60/month total See .github/PHASE3_COMPLETE.md for details Fix sync to allow 'dev setup' commits on master The sync workflow was failing because the 'dev setup v19' commit modifies files outside .github/. Updated workflows to recognize commits with messages starting with 'dev setup' as allowed on master. Changes: - Detect 'dev setup' commits by message pattern (case-insensitive) - Allow merge if commits are .github/ OR dev setup OR both - Update merge messages to reflect preserved changes - Document pristine master policy with examples This allows personal development environment commits (IDE configs, debugging tools, shell aliases, Nix configs, etc.) on master without violating the pristine mirror policy. Future dev environment updates should start with 'dev setup' in the commit message to be automatically recognized and preserved. See .github/docs/pristine-master-policy.md for complete policy See .github/DEV_SETUP_FIX.md for fix summary Optimize CI/CD costs by skipping builds for pristine commits Add cost optimization to Windows dependency builds to avoid expensive builds when only pristine commits are pushed (dev setup commits or .github/ configuration changes). Changes: - Add check-changes job to detect pristine-only pushes - Skip Windows builds when all commits are dev setup or .github/ only - Add comprehensive cost optimization documentation - Update README with cost savings (~40% reduction) Expected savings: ~$3-5/month on Windows builds, ~$40-47/month total through combined optimizations. Manual dispatch and scheduled builds always run regardless.
Review every PR (including drafts) with two jobs that authenticate to AWS Bedrock (Claude Opus 4.8) via GitHub OIDC (vars.AWS_ROLE_ARN); no static AWS credentials are stored in the repo. - ocr-review: runs Alibaba Open Code Review through an ephemeral LiteLLM proxy bridging OCR's OpenAI protocol to Bedrock, and posts inline review comments. Uses output_config.effort=xhigh (Opus 4.8 adaptive thinking). Path-scoped rules (.github/ocr/rule.json) encode PostgreSQL community review standards plus reviewer discipline (verify against the diff, don't hallucinate, state confidence, be blunt, accuracy over approval). - pg-history: OCR cannot call MCP, so a separate Bedrock tool-use agent (.github/ocr/pg-history.py) queries the Agora MCP server (pg.ddx.io) to tie the change to git + pgsql-hackers history, and upserts a comment linking threads as https://pg.ddx.io/m/pgsql-hackers/<message-id>.
The pg-history workflow job has been failing every run with 'Bedrock call failed: The read operation timed out' -- botocore's default 60s read timeout on bedrock-runtime is too short for a multi-round (MAX_ROUNDS=14) tool-use loop against a large PR diff on a reasoning-capable model; a single converse() call alone can take several minutes under load (the sibling ocr-review job's own LLM pass over a similarly large diff took 30-40 minutes). Confirmed via two consecutive live runs against PR #26. Set read_timeout=900s (15 min) explicitly via botocore.config.Config; leave connect_timeout short since a stuck TCP handshake is a different, cheaper-to-detect failure mode that shouldn't wait as long.
github-actions
Bot
force-pushed
the
master
branch
from
ce2678e to
82cc694
Compare
This is the first stage of a native full-text search subsystem intended to
provide true BM25/BM25F relevance ranking with index-only scoring and a
richer query language, addressing long-standing limitations of the
tsvector/tsquery + GIN stack: no corpus statistics (N, avgdl, df) are stored
anywhere, ts_rank is cover-density rather than BM25, and GIN posting lists
carry only TIDs so ranked queries must always recheck the heap.
Rather than land that as one large patch, the work is structured as a
reviewable series (see FTS_NEXTGEN_PLAN.md). This first commit introduces
only the SQL surface, evaluated by sequential scan, with no index access
method -- the same way tsvector/tsquery were originally introduced.
Adds two types:
ftsdoc an analyzed document (sorted, de-duplicated terms with term
frequencies, plus the document length that BM25 will need)
ftsquery a parsed boolean query (AND/OR/NOT and grouping)
Both are varlena and TOAST-able with version-tagged binary send/recv formats.
A hand-written recursive-descent parser produces the query; the grammar is
small enough not to warrant a generator. Matching is a boolean stack machine
over the postfix item list, mirroring TS_execute.
The stage-1 tokenizer is deliberately minimal (ASCII case-fold, split on
non-alphanumerics). It is isolated behind fts_analyze_text() so that a later
stage can reuse PostgreSQL's existing text-search parser and dictionary
pipeline (snowball, ispell, synonyms, thesaurus, stopwords) without changing
the types, the operator, or the on-disk format.
Includes a regression test exercising analysis, query parsing and canonical
output, all boolean match cases, sequential-scan use in a WHERE clause, and
error handling for malformed queries.
Add to_ftsdoc(regconfig, text), which runs an installed text search configuration's parser and dictionary chain via parsetext() and folds the normalized lexemes into an ftsdoc. This reuses the existing snowball/ispell/ synonym/thesaurus/stopword pipeline rather than reimplementing tokenization. Shipped as extension upgrade 1.0 -> 1.1.
Add fts_bm25(doc, query, n_docs, avgdl, dfs), computing the Okapi BM25 score with Lucene-style IDF and standard k1/b defaults. Corpus statistics are caller-supplied for now (the bm25 index AM will maintain them later), which is enough to validate the scoring math by sequential scan. Shipped as 1.1 -> 1.2.
Add a real index access method (USING bm25) over an ftsdoc column that answers the @@@ operator via a bitmap scan. The build scans the heap, collects per-term postings (tid, tf), and writes a metapage (N, sum(doclen), nterms), a sorted dictionary, and chained posting pages -- all through GenericXLog, so the index is crash-safe and replicated without a custom resource manager. The scan evaluates the boolean ftsquery by set algebra over posting lists (AND=intersect, OR=union, NOT=complement against the indexed universe), matching @@@ semantics exactly with no heap access. Corpus statistics are maintained for the coming index-only BM25 scoring. The skeleton is build-once: aminsert raises an error directing REINDEX, and incremental maintenance (pending list + background merge) is a later stage. Shipped as extension upgrade 1.2 -> 1.3.
Add fts_bm25_opts(doc, query, n_docs, avgdl, k1, b, variant, dfs) supporting lucene, robertson (classic), atire, and bm25+ IDF/scoring variants with explicit k1/b, for reproducing reference implementations (Lucene/bm25s) in conformance tests. Shipped as 1.3 -> 1.4.
Add fts_highlight(text, query, pre, post) and fts_snippet(text, query, pre, post, ellipsis, max_tokens), giving FTS5-parity result presentation. Both tokenize the source with the same folding as the analyzer and mark query-term matches; snippet slides a token window and returns the densest match region. Shipped as 1.4 -> 1.5.
Add tsquery_to_ftsquery() and an ASSIGNMENT cast so existing tsquery values and queries port to the @@@ operator with minimal churn: &/|/! map to AND/OR/NOT. The phrase operator <-> degrades to AND with a NOTICE (phrase support is a later stage), preserving recall. Shipped as 1.5 -> 1.6.
Add prefix matching to the query language: a trailing '*' on a term (e.g. quick*) matches any document term with that prefix. Implemented in the parser (a per-item FTS_QF_PREFIX flag, carried through send/recv), the sequential matcher (binary-search lower bound on the sorted term set), and the bm25 index scan (union the posting lists of all dictionary terms sharing the prefix). Phrase and NEAR need per-term positions, which the stage-1 ftsdoc format omits; they follow as an ftsdoc v2 format addition.
Add fts_index_stats(regclass) -> (ndocs, avgdl, nterms) and fts_index_df( regclass, ftsquery) -> float8[], reading N, avgdl and per-term document frequency from the bm25 index metapage and dictionary. BM25 can now be scored from statistics the index maintains rather than caller guesses, closing the loop between the AM and the scorer. (Streaming index-only WAND top-K is a further optimization.) Shipped as 1.6 -> 1.7.
…partial) Update the README to reflect the nine qualified stages (versions 1.0-1.7 plus prefix queries) and to state honestly what remains: phrase/NEAR, WAND top-K, incremental maintenance, contentless indexes, the parity gate, and the fuzzy/regex stages.
The bm25 access method's aminsert no longer errors: it appends the new document verbatim to an in-index chain of pending pages and bumps the metapage N and sum(doclen). The scan searches pending documents directly with the per-document matcher, so newly inserted rows are immediately visible to @@@ without a REINDEX. Per-term df in the dictionary stays stale until a merge (REINDEX), matching GIN fastupdate's documented behavior. All page writes go through GenericXLog. Shipped as 1.7 -> 1.8 (bm25 metapage format changed; REINDEX required for pre-1.8 bm25 indexes).
Extend the ftsdoc format to v2, storing per-term token positions, and add
quoted-phrase query syntax ("a b c"): the parser emits an FTS_OP_PHRASE chain
(distance 1), and the matcher verifies adjacency by intersecting term position
lists. The bm25 index treats a phrase as AND for candidate generation and now
requests a bitmap recheck, so @@@ re-evaluates adjacency exactly against the
heap ftsdoc. Position-free v1 docs remain valid (phrase degrades to AND).
NEAR(a b, k) reuses the same distance-aware phrase_step and is a small parser
addition (comma + integer) on top of this. Shipped as 1.8 -> 1.9 (ftsdoc
format v2).
The bm25 index stores only postings, never document text, so an expression index on to_ftsdoc(text_column) is exactly FTS5's external-content model: the text lives in the base table, the index is derived from it, and @@@ queries (including phrases, via recheck) work against the expression. Shipped as 1.9 -> 1.10 (documentation marker; no new SQL objects).
Add two ftsquery term forms:
term~k matches document terms within Levenshtein distance k (default 2),
using core varstr_levenshtein_less_equal (bounded, no new dependency)
/re/ matches document terms against a POSIX regex via core's cached
regex engine
Both are evaluated per-document in the matcher. The bm25 index returns all
indexed tuples as candidates for fuzzy/regex queries and the bitmap heap
recheck applies the exact test, so results are correct through the index.
This follows the plan's 'no new dependency for the common case' path; the
pg_tre trigram-formula pre-filter (with its Lime grammar converted to
Bison+Flex, and sparsemap v5.1.1 for posting compression) to narrow candidates
at scale is future work. Shipped as 1.10 -> 1.11.
Add bench/bench.sql and bench/README: a reproducible A/B harness comparing the bm25 stack against tsvector + GIN + ts_rank on a user-supplied corpus (index size, ranked top-10 EXPLAIN ANALYZE). The full parity gate (latency percentiles, NDCG vs qrels, concurrent-ingest throughput, Lucene/bm25s score conformance) is documented as a manual, reported measurement rather than a make-check regression, since it needs an external corpus. Update README.pg_fts to describe all implemented stages (1.0-1.11), the full query language, a worked BM25 ranking example, and the remaining future work (WAND top-K, trigram pre-filter for fuzzy/regex, NEAR, background merge, BM25F).
Add fts_bm25f(docs ftsdoc[], query, weights[], n_docs, avgdls[], dfs[]): the Robertson/Zaragoza BM25F, where per-field term frequencies are length- normalized per field and combined by weight before the tf-saturation step (not a naive sum of per-field BM25 scores). This lets a term in a heavily weighted field (e.g. title) outrank the same term in the body. Shipped as 1.11 -> 1.12.
Add bm25_merge_pending(): read the existing dictionary + posting chains and all pending documents back into a build state, rewrite the merged structure into fresh blocks, and repoint the metapage -- no heap access. Wired into amvacuumcleanup (VACUUM now folds pending docs automatically) and exposed as fts_merge(regclass) for on-demand merge. Merging resolves the df staleness that incremental inserts introduce (formerly-pending terms gain dictionary df). Old blocks are left unreferenced and reclaimed by REINDEX; an FSM-based page recycler is future work. Shipped as 1.12 -> 1.13.
Add fts_search(index, query, k) -> setof(ctid, score): BM25 top-k computed entirely from the index -- postings supply per-doc tf, the dictionary supplies df and a per-term max-tf impact bound (now stored), the metapage supplies N and avgdl -- with no heap access. Per-document scores accumulate across query terms and the top-k are returned by descending score; join on ctid to fetch rows. This is the index-only-scoring path (no heap fetch to rank), the core performance win for ranked search. Stored per-term max_tf in the dictionary provides the WAND upper bound for document skipping. Full executor integration via amcanorderbyop (an ORDER BY score LIMIT k ordering scan with block-max WAND early termination) and exact per-document |D| in postings are the remaining optimizations. Shipped as 1.13 -> 1.14.
Add pg_fts_trgm.c: reduce a fuzzy term to its trigrams and test only document terms sharing a trigram with it (Levenshtein is the expensive step). This is the pg_tre-style pruning that makes fuzzy matching viable on a large vocabulary, applied at the term level. Results are unchanged: the filter only skips candidates that provably cannot match (pigeonhole: a match within k edits shares a trigram when the term has more than k trigrams) and falls back to a full scan for short terms. A persistent on-disk trigram posting index in the bm25 AM (the full three-tier funnel) is the remaining scale work. Shipped as 1.14 -> 1.15.
fts_search() returned candidate ctids straight from the postings, which can reference dead or updated tuples that the index has not yet merged out. It now opens the base table and checks each candidate against the active snapshot via table_index_fetch_tuple, returning only visible tuples in score order and stopping once k visible rows are found. This makes the SRF correct under all isolation levels, matching the visibility contract the @@@ bitmap path already gets from the executor's bitmap heap scan + recheck. (The @@@ operator path was already MVCC-correct: amgetbitmap sets recheck=true and the bitmap heap scan applies snapshot visibility. All page I/O uses the buffer manager, and every writer is WAL-logged via GenericXLog, so the index is correct on physical standbys and after crash recovery.)
github-actions
Bot
force-pushed
the
master
branch
from
73281bd to
c3ee91f
Compare
Iterating the parallel merge to one segment was measured at 2M Wikipedia to be WORSE (32min vs 27min single-pass): each pass rewrites data (write amplification) and the final reduction is still the write of one multi-GB output segment by a single backend -- which no group-partition scheme parallelizes. Revert to a single parallel pass + serial finish, and record the finding. The merge tail is a single-output-write cost; cutting it needs a streamed/columnar write path (DEFERRED.md), not more merge parallelism. qualify PASS; regression + isolation green.
…ter decode (2M) fts_vacuum: 15 GB -> 3.77 GB (4x) in 1.7s, counts identical (on par with pg_search 4.1 GB). Word-oriented decode: common-term fts_count 305 -> 101 ms (now below pg_search 123 ms). Iterated parallel merge measured worse (32 vs 27 min) and reverted -- the merge tail is a single-output-segment write, a codec matter not a parallelism one.
…not the win) AIO for parallel-merge writes: rejected with evidence -- no buffer-manager AIO write path exists in this tree (reads only), using it would break the GenericXLog invariant, and the merge tail is CPU-bound re-encode (not I/O wait) so AIO would not help. Recorded in CAPABILITIES.md. Format-v3 codec: profiled the common-term ranked query (perf --no-children). It is ~30% decode+block-load and ~70% scoring/heap/executor, and block-max WAND cannot skip blocks on common English terms -- so a columnar-codec rewrite is capped at ~30% and cannot enable pruning (the NOTE_IMPACT_ORDERING result, now confirmed by profiling). A reusable per-cursor block buffer was tried and measured SLOWER (per-block palloc is only 1.2%), reverted. Evidence-supported levers are SIMD docid unpack (~5-8%, decode micro-opt) and a PARALLEL ranked scan (the real lever, an executor/AM change). bench/NOTE_FORMAT_V3_PROFILE.md; DEFERRED.md item 4 updated. No speculative codec rewrite shipped.
Add pg_fts_customscan.c with a _PG_init that installs create_upper_paths_hook. A bare 'SELECT count(*) ... WHERE col @@@ q' over a single base rel with a bm25 index on col is now answered by a Custom Scan (FtsCount) that calls the index bulk-count (bm25_count_visible_oid, VM-based) instead of a bitmap heap scan -- ~3x faster on a common term (transparent count 240ms -> the fts_count 75ms path, without the user having to call fts_count()). Strictly additive and precisely gated: fires only for count(*) with no GROUP BY/HAVING/DISTINCT/window/set-op, exactly one @@@ qual, and an FtsQuery Const; any extra qual or grouping falls back to the ordinary plan (verified in the regress test). MVCC-correct (uses the active snapshot's visibility, same as fts_count). Establishes the CustomScan plumbing for the ranked stages. qualify PASS; regression + isolation green.
github-actions
Bot
force-pushed
the
master
branch
from
c3ee91f to
c7e7712
Compare
ORDER BY col <=> q [LIMIT k] on a bm25-indexed rel is now handled by a Custom Scan (FtsRankedScan) installed via set_rel_pathlist_hook. Under the hood the WAND/MaxScore candidate generation is fanned across parallel workers by docid range: bm25_topk_candidates_range(index, q, wantk, docid_lo, docid_hi) scores a disjoint docid slice (each WandCursor seeks to docid_lo and reports exhausted at docid_hi, so the existing WAND loops need no range awareness), the leader merges the per-slice candidate lists, sorts by score, and applies MVCC visibility once. Gated on max_parallel_workers_per_gather and a >=50k-hit estimate, so selective queries stay serial (no parallel setup overhead). The docid ranges partition the corpus disjointly, so the parallel result is byte-identical to the serial one -- verified in the regress test (parallel vs serial top-k equal) and locally on 200k rows for single- and multi-term queries. Refactor: bm25_topk_visible now calls the new range function with [0, MAX) then does visibility (behavior unchanged for the SRF/amgettuple paths). qualify PASS; regression + isolation green.
bm25_vacuum_compact only truncated after merging, so a freshly built index that already merged to one segment (common: the build compacts to nseg=1) kept its dead build/merge pages interleaved -- nothing was truncatable and fts_vacuum was a no-op (12 GB stayed 12 GB at 2M). Now always REWRITE the live segments through the low-page allocator first (even a single segment), relocating live pages to the front so the stale pages form a contiguous free tail that RelationTruncate removes. Verified: a parallel-built nseg=1 index shrinks 206 MB -> 70 MB (one pass) -> 35 MB (second pass, the compacted floor), counts identical. qualify PASS; regression + isolation green.
github-actions
Bot
force-pushed
the
master
branch
from
c7e7712 to
ab6faa8
Compare
… COUNT pushdown The ranked CustomScan (Option B stages 2+3) was built, verified byte-identical to serial, and measured at 2M: no win (top-100 common 73 vs 66 ms; others within noise). Two reasons (bench/NOTE_PARALLEL_RANKED.md): the query is only ~30% decode+WAND with a ~70% scoring/heap/visibility tail the leader runs serially (Amdahl ceiling ~30%), and launching workers from inside ExecCustomScan fell back to serial at scale. The serial ranked CustomScan is also redundant with the existing bm25 AM ordering scan. Reverted per the project rule against shipping non-paying optimizations. Kept: the COUNT pushdown CustomScan (a real transparent ~3x win) and the behavior-preserving bm25_topk_candidates_range refactor. qualify PASS; regression + isolation green.
SGML + README: add fts_vacuum() (compact + truncate to reclaim physical bloat without REINDEX; auto during VACUUM), correct the stale fts_merge note that said REINDEX was required, and note the transparent count(*) WHERE @@@ CustomScan pushdown. qualify PASS.
github-actions
Bot
force-pushed
the
master
branch
19 times, most recently
from
331d425 to
45c1114
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.