pgvector-filterbench

Audit filtered pgvector HNSW recall, latency, and query-plan correctness against exact PostgreSQL search.

pgvector's filtered HNSW search applies filters after the approximate index scan, which can silently reduce recall. This tool measures how badly filtering hurts your results — and proves whether PostgreSQL actually used your HNSW index.

Quick Start (Docker Compose)

cd docker
docker compose up --build --abort-on-container-exit --exit-code-from benchmark

This starts PostgreSQL 16 with pgvector, seeds 10,000 synthetic documents with 128-dimensional vectors, runs the benchmark, writes reports to the repository's reports/ directory, and stops the services when the benchmark exits. You should see a recall table in under a minute. The demo database is ephemeral.

To select another supported PostgreSQL major or avoid a local port conflict:

POSTGRES_MAJOR=17 POSTGRES_PORT=55432 docker compose up --build --abort-on-container-exit --exit-code-from benchmark

Tested PostgreSQL Matrix

The complete Docker benchmark gate was run against these image versions on 2026-07-16. The image tags can move; the versions below are what those tags resolved to during verification.

Image tag	PostgreSQL	pgvector	Matrix cells	Query errors	Sequential scans	HNSW plans	Observed worst recall	Result
pg16	16.14	0.8.5	9/9	0	0	810/810	0.650	Pass
pg17	17.10	0.8.5	9/9	0	0	810/810	0.630	Pass
pg18	18.4	0.8.5	9/9	0	0	810/810	0.647	Pass

Recall values are observations from the deterministic demo corpus, not promises for other datasets or HNSW builds. PostgreSQL 18 uses a different container data-directory layout and emits decimal Actual Rows values in JSON plans; both forms are handled by the included Compose setup and validator.

Installation

Binary

Download a static binary from the releases page, or build from source:

go build -o pgvector-filterbench .

Go install

go install github.com/lame13/pgvector-filterbench@latest

Docker

docker build -f docker/Dockerfile -t pgvector-filterbench .
docker run --rm pgvector-filterbench version

Usage

pgvector-filterbench run -c config.yaml

Commands

Command	Description
run	Run the benchmark pipeline
version	Print the version

Exit Codes

Code	Meaning
0	Benchmark passed all gate checks
1	Runtime error
2	Gate check failed

Configuration

See testdata/sample-config.yaml for a fully documented example.

Key Sections

connection — PostgreSQL DSN plus transaction-local statement and lock timeouts. Read-only credentials are recommended; every benchmark transaction is also explicitly set read-only.

table — Schema, table name, ID column, vector column, and optional vector expression override.

index — HNSW index name, type (vector or halfvec), and distance metric (cosine, l2, ip).

filters — Column/value pairs forming the WHERE clause. Adjust to test different selectivity levels.

sampling — Seeded random sampling from the database, or external JSONL query vectors. Set self_exclude: true for corpus-derived queries to remove the guaranteed self-neighbor.

strategies — Which query strategies to benchmark:

• exact — Ground truth with index scans disabled (read-only transaction)
• bounded_ann — Default HNSW scan (no iterative scan)
• strict_iterative — hnsw.iterative_scan = strict_order
• relaxed_iterative — hnsw.iterative_scan = relaxed_order

ef_search — Sweep values for the HNSW search parameter.

max_scan_tuples — Sweep values for hnsw.max_scan_tuples (iterative strategies only).

gate — CI thresholds: minimum mean recall across every measured matrix cell, maximum sequential scans, and required HNSW index use. A deliberately weak baseline will fail the gate unless its recall is above the configured threshold.

report — Output formats (JSON, Markdown), directory, a neutral target label, and privacy controls. Schema, index, filter values, vectors, row IDs, and full plans stay out of reports unless explicitly enabled. When row IDs are enabled, they remain hashed by default.

Configuration parsing is strict: unknown keys, duplicate strategies/parameter values, unsupported formats, and inconsistent k values fail before the benchmark connects.

How It Works

1. Sample query vectors from your table (seeded random or external file).
2. For each sample, run exact ground truth with index scans disabled inside a read-only transaction. Exact-first and ANN-first samples are counterbalanced when randomized ordering is enabled, avoiding systematic cache warming by the exact scan.
3. Sweep every (strategy × ef_search × max_scan_tuples) combination with configurable warmup and measured runs.
4. For each measured ANN execution, run EXPLAIN (ANALYZE, VERBOSE, FORMAT JSON) on the same query shape in the same transaction.
5. Validate the configured index is a ready/valid/live HNSW index with the expected opclass, then prove every measured plan used it on the expected schema and table.
6. Compute recall@k, same-rank, and latency statistics.
7. Evaluate gate thresholds and write privacy-safe reports.

What Makes This Different

Other pgvector benchmarks compare latency. This tool catches the case where recall looks good but PostgreSQL silently fell back to a sequential scan — a common and confusing failure mode with filtered HNSW.

Privacy-Safe Defaults

Reports are safe to commit or share by default:

• Raw vectors are omitted unless explicitly included
• Row IDs are omitted (and hashed with SHA-256 when explicitly included)
• Connection strings are redacted
• Schema/table/index names and filter values are replaced by a neutral target label
• Full EXPLAIN output is opt-in
• Report files are written with 0600 permissions

The aggregate JSON summary always includes eligible/ineligible sample counts and exact-result cardinality statistics without exposing row-level data.

Operational Safety

Exact scans can put meaningful load on PostgreSQL even though the tool performs no writes. Prefer a replica or staging database for large datasets, keep the sample/matrix size bounded, and set connection.statement_timeout and connection.lock_timeout for your environment. Reports do not include the DSN by default.

Synthetic Dataset

The Docker Compose demo seeds a documents table with:

Rows	Filter	Filter selectivity
10,000	status = active	100%
1,000	tier = standard	10%
100	tier = premium	1%
10	tier = enterprise	0.1%

All rows have status = 'active' for 100% selectivity filtering.

JSONL Query Vectors

To use your own query vectors instead of sampling from the database:

{"id": "q1", "vector": [0.1, 0.2, 0.3]}
{"id": "q2", "vector": [0.4, 0.5, 0.6]}

Set sampling.query_vectors_file in your config to the path of this file.

Requirements

• PostgreSQL 16, 17, or 18 with pgvector 0.8+ installed (all three majors integration-tested)
• Read-only database credentials (recommended)
• Go 1.21+ (for building from source)

License

Apache-2.0. See LICENSE.

Disclaimer

This is an unofficial tool for benchmarking pgvector. It is not affiliated with or endorsed by the pgvector project. pgvector is developed by Andrew Kane and contributors.