paimon-vector-index/docs/development.html at main · apache/paimon-vector-index · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements. See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to you under the Apache License, Version 2.0.
-->
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <meta name="description" content="Paimon Vector Index repository layout, development checks, cross-language tests, and ANN benchmarks.">
  <title>Development and Benchmarks · Paimon Vector Index</title>
  <link rel="stylesheet" href="styles.css">
  <script src="docs.js" defer></script>
</head>
<body>
  <a class="skip-link" href="#main">Skip to content</a>
  <header class="site-header"><div class="header-inner">
    <a class="brand" href="index.html" aria-label="Paimon Vector Index documentation home"><span class="brand-mark">VI</span><span>Paimon Vector Index</span></a>
    <nav class="site-nav" data-site-nav aria-label="Documentation"><a href="index.html">Overview</a><a href="api.html">API</a><a href="development.html" aria-current="page">Development</a><a href="ivf-flat.html">IVF-FLAT</a><a href="ivf-pq.html">IVF-PQ</a><a href="ivf-rq.html">IVF-RQ</a><a href="ivf-sq.html">IVF-SQ</a><a href="diskann.html">DiskANN</a><a href="releases.html">Releases</a></nav>
    <div class="header-actions"><button class="icon-button" type="button" data-theme-toggle aria-label="Switch color theme">◐</button><button class="nav-toggle" type="button" data-nav-toggle aria-expanded="false" aria-label="Open navigation">☰</button></div>
  </div></header>

  <main id="main"><div class="page-shell">
    <section class="hero detail-hero"><p class="eyebrow">Build · test · measure</p><h1>Development and benchmarks</h1><p class="hero-lead">Run the standard Rust checks, exercise the C/C++, Java, and Python integrations, and measure ANN and filtered-query behavior with reproducible workloads.</p><div class="badge-row"><span class="badge strong">Cargo workspace</span><span class="badge">CMake smoke tests</span><span class="badge">Maven / pytest</span><span class="badge">Criterion benchmarks</span></div></section>

    <div class="doc-layout">
      <aside class="toc" aria-label="On this page"><strong>On this page</strong><a href="#workspace">Repository layout</a><a href="#rust">Rust checks</a><a href="#ann">ANN benchmark</a><a href="#diskann-bench">DiskANN benchmark</a><a href="#c">C FFI</a><a href="#cpp">C++</a><a href="#java">Java/JNI</a><a href="#python">Python</a><a href="#format">Format compatibility</a></aside>
      <article class="article">
        <section class="article-section" id="workspace">
          <h2>Repository layout</h2>
          <div class="table-wrap"><table><thead><tr><th>Directory</th><th>Contents</th><th>Primary verification</th></tr></thead><tbody>
            <tr><td><code>core/</code></td><td>Indexes, quantizers, I/O, format fixtures, and benchmarks</td><td><code>cargo test -p paimon-vindex-core</code></td></tr>
            <tr><td><code>ffi/</code></td><td>C ABI and shared library</td><td>C smoke test</td></tr>
            <tr><td><code>include/</code></td><td>C++ RAII header</td><td>C++ smoke test</td></tr>
            <tr><td><code>jni/</code></td><td>Rust JNI bridge</td><td>Java Maven tests</td></tr>
            <tr><td><code>java/</code></td><td>Java public API and tests</td><td><code>mvn test</code></td></tr>
            <tr><td><code>python/</code></td><td>ctypes package and pytest suite</td><td><code>pytest</code></td></tr>
            <tr><td><code>c/</code> / <code>cpp/</code></td><td>Integration smoke-test projects</td><td>CMake</td></tr>
            <tr><td><code>docs/</code></td><td>Index, API, and development documentation</td><td>Static link and structure checks</td></tr>
          </tbody></table></div>
        </section>

        <section class="article-section" id="rust">
          <h2>Standard Rust checks</h2>
          <div class="code-block"><span class="code-label">Shell</span><pre><code>cargo fmt --all
cargo test --workspace
cargo clippy --workspace --all-targets</code></pre></div>
          <p>Before submitting a change, keep formatting, workspace tests, and Clippy across all targets clean. Changes to binary layouts should also run the core storage-format fixtures.</p>
        </section>

        <section class="article-section" id="ann">
          <h2>ANN benchmark</h2>
          <p><code>ann_bench</code> compares IVF-FLAT, IVF-SQ, IVF-PQ, IVF-RQ, and DiskANN under one workload. By default it generates independent clustered queries and computes exact Top-10 ground truth. It can instead read standard <code>fvecs</code>/<code>ivecs</code> files and use their published ground truth. Every CSV row starts with a dataset label and reports vector and training counts, raw-dataset bytes, total build time plus train/add/write stages, process peak RSS, warm-up time, serialized bytes, Recall@10, first/P50/P95 latency, sequential and batch throughput, positional-read rounds, range count, and bytes.</p>
          <div class="code-block"><span class="code-label">Shell · defaults</span><pre><code>cargo bench -p paimon-vindex-core --bench ann_bench -- --nocapture</code></pre></div>
          <div class="code-block"><span class="code-label">Shell · documented 100k comparison</span><pre><code>ANN_N=100000 ANN_NQ=64 ANN_D=64 ANN_K=10 \
ANN_NLIST=256 ANN_NPROBE=32 ANN_PQ_CODE_RATIO=0.0625 \
ANN_DISKANN_L_SEARCH=200 \
ANN_RQ_BITS=4 \
cargo bench -p paimon-vindex-core --bench ann_bench -- --nocapture</code></pre></div>
          <div class="code-block"><span class="code-label">Shell · 1024 physical dimensions, exactly 10 GiB of raw vectors</span><pre><code>ANN_DATA_GIB=10 ANN_D=1024 ANN_NOISE_DIMENSIONS=64 \
ANN_TRAIN_N=65536 ANN_NQ=8 ANN_K=10 \
ANN_NLIST=1 ANN_NPROBE=1 ANN_PQ_CODE_RATIO=0.0625 ANN_CLUSTERS=64 \
ANN_DISKANN_L_SEARCHES=200,1000 \
ANN_DISKANN_MEMORY_BUDGET_BYTES=34359738368 \
ANN_INDEXES=DISKANN ANN_KEEP_INDEXES=1 \
ANN_OUTPUT_DIR=/path/on/target/ssd \
cargo bench -p paimon-vindex-core --bench ann_bench</code></pre></div>
          <div class="code-block"><span class="code-label">Shell · complete five-index 10 GiB comparison, isolated processes</span><pre><code>for index in IVF_FLAT IVF_SQ IVF_PQ IVF_RQ DISKANN; do
  ANN_DATA_GIB=10 ANN_D=1024 ANN_NOISE_DIMENSIONS=64 \
  ANN_TRAIN_N=65536 ANN_NQ=8 ANN_K=10 \
  ANN_NLIST=1024 ANN_NPROBE=64 ANN_PQ_CODE_RATIO=0.0625 ANN_CLUSTERS=64 \
    ANN_DISKANN_L_SEARCHES=200,1000 \
  ANN_DISKANN_MEMORY_BUDGET_BYTES=34359738368 \
  ANN_INDEXES="$index" ANN_OUTPUT_DIR=/path/on/target/ssd \
  cargo bench -p paimon-vindex-core --bench ann_bench
done</code></pre></div>
          <h3>Public SIFT1M, GIST1M, and GloVe-100 data</h3>
          <p><a href="https://github.com/erikbern/ann-benchmarks">ANN-Benchmarks</a> publishes dense HDF5 files with independent test queries and exact Top-100 neighbors. Convert them once to the streaming-friendly <code>fvecs</code>/<code>ivecs</code> format. <code>--query-limit=1000</code> keeps all recorded datasets at the same query count while retaining enough queries for stable Recall@10.</p>
          <div class="code-block"><span class="code-label">Shell · download and convert a public dataset</span><pre><code>curl -LO https://ann-benchmarks.com/sift-128-euclidean.hdf5
python3 -m pip install h5py numpy
python3 tools/convert_ann_benchmarks.py \
  sift-128-euclidean.hdf5 /data/sift1m \
  --prefix sift1m --query-limit 1000</code></pre></div>
          <div class="code-block"><span class="code-label">Shell · convert GloVe angular data for the common L2 benchmark</span><pre><code>curl -LO https://ann-benchmarks.com/glove-100-angular.hdf5
python3 tools/convert_ann_benchmarks.py \
  glove-100-angular.hdf5 /data/glove100 \
  --prefix glove100 --query-limit 1000 --normalize-l2</code></pre></div>
          <div class="code-block"><span class="code-label">Shell · run all public-data indexes in isolated processes</span><pre><code>RAYON_NUM_THREADS=12 ANN_DATASET_NAME=sift1m \
ANN_BASE_FVECS=/data/sift1m/sift1m_base.fvecs \
ANN_QUERY_FVECS=/data/sift1m/sift1m_query.fvecs \
ANN_GROUND_TRUTH_IVECS=/data/sift1m/sift1m_ground_truth.ivecs \
ANN_K=10 ANN_NLIST=1024 ANN_NPROBE=64 ANN_PQ_CODE_RATIO=0.0625 \
ANN_DISKANN_L_SEARCH=100 \
ANN_DISKANN_RAW_VECTOR_ENCODING=f16 \
ANN_DISKANN_MEMORY_BUDGET_BYTES=8589934592 \
ANN_STORAGE_CASES=local_ssd_warm_cache,remote_cache_2ms,object_store_20ms \
ANN_OUTPUT_DIR=/path/on/target/ssd \
cargo bench -p paimon-vindex-core --bench ann_bench</code></pre></div>
          <p>The benchmark infers <code>ANN_N</code>, <code>ANN_NQ</code>, and <code>ANN_D</code> from the public files, chooses <code>ANN_TRAIN_N=min(N, max(65,536, 64 × nlist))</code>, and automatically starts one child process per selected index so each peak-RSS value is independent. Explicit shape variables remain available as assertions. For GIST1M, download and convert <code>gist-960-euclidean.hdf5</code>; for GloVe-100, use the command above. Unit-normalized L2 has the same neighbor ordering as cosine for non-zero vectors; that equivalence was used by the recorded GloVe run. DiskANN now also accepts native cosine and inner-product metrics, which should be benchmarked against ground truth generated with the same production metric. Keep the other recorded settings unchanged. The default <code>ANN_PQ_CODE_RATIO=0.0625</code> resolves automatically to <code>m=32</code>, <code>m=240</code>, and <code>m=25</code> for SIFT, GIST, and GloVe respectively. Set <code>ANN_PQ_M</code> only to benchmark an explicit expert override.</p>
          <p>At 1024 dimensions, <code>ANN_DATA_GIB=10</code> derives exactly 2,621,440 vectors. Run large comparisons one index per process by changing <code>ANN_INDEXES</code>; this isolates process peak RSS and, unless <code>ANN_KEEP_INDEXES=1</code>, deletes that index after all selected serving cases. A comma-separated subset or <code>all</code> is convenient for smaller workloads. The benchmark retains only the selected training prefix, then adds the entire data set. The clustered generator is round-robin by cluster, so a prefix still covers every cluster when <code>ANN_TRAIN_N ≥ ANN_CLUSTERS</code>.</p>
          <div class="callout"><strong>Physical versus intrinsic dimension</strong><code>ANN_NOISE_DIMENSIONS=64</code> spreads independent per-vector noise evenly over 64 of the 1024 stored dimensions; the other dimensions retain cluster-center signal. This creates a repeatable storage/compute scale workload without pretending that 1024 independent uniform-noise dimensions resemble production embeddings. With all 1024 dimensions independently noisy, same-cluster distances concentrate so strongly that exact Top-10 membership is nearly arbitrary: a 65k diagnostic produced local Recall@10 of 0.0125 at <code>l_search=200</code> and only 0.178 at <code>l_search=5000</code>. Use a public or production corpus for an algorithm-quality claim.</div>
          <div class="table-wrap"><table><thead><tr><th>Variable</th><th>Meaning</th></tr></thead><tbody><tr><td><code>ANN_DATASET_NAME</code></td><td>Comma-free label written to every CSV row</td></tr><tr><td><code>ANN_BASE_FVECS / ANN_QUERY_FVECS / ANN_GROUND_TRUTH_IVECS</code></td><td>Optional public-data files; set all three together. Their vector count, query count, dimension, and ground-truth width are inspected before the full payload is loaded.</td></tr><tr><td><code>ANN_N / ANN_NQ / ANN_D</code></td><td>Inferred for public files; an explicit value is validated as a shape assertion. Generated workloads retain defaults of 20,000 / 64 / 64.</td></tr><tr><td><code>ANN_DATA_GIB</code></td><td>Generated-workload raw float-vector GiB; derives <code>n = floor(bytes / (4d))</code> and cannot be combined with public file paths.</td></tr><tr><td><code>ANN_TRAIN_N</code></td><td>Optional training-sample override; omitted values resolve to <code>min(N, max(65,536, 64 × nlist))</code>.</td></tr><tr><td><code>ANN_K</code></td><td>Top K</td></tr><tr><td><code>ANN_INDEXES</code></td><td><code>all</code> or a comma-separated subset of the five index names. Public multi-index runs automatically isolate each index in a child process.</td></tr><tr><td><code>ANN_STORAGE_CASES</code></td><td><code>all</code> or a comma-separated subset of <code>local_ssd_warm_cache</code>, <code>remote_cache_2ms</code>, and <code>object_store_20ms</code>. The default runs all three.</td></tr><tr><td><code>ANN_NLIST / ANN_NPROBE</code></td><td>IVF list count / probed lists</td></tr><tr><td><code>ANN_PQ_CODE_RATIO</code></td><td>IVF-PQ and DiskANN PQ-code/raw-vector byte ratio; default 0.0625 and automatically resolves a valid <code>m</code></td></tr><tr><td><code>ANN_PQ_M</code></td><td>Optional explicit subquantizer-count override; takes precedence over automatic sizing</td></tr><tr><td><code>ANN_RQ_BITS</code></td><td>Persisted IVF-RQ bit width, 1–8; default 4</td></tr><tr><td><code>ANN_DISKANN_L_SEARCH / ANN_DISKANN_L_SEARCHES</code></td><td>One DiskANN search-list size, or a comma-separated sweep over one built index</td></tr><tr><td><code>ANN_DISKANN_BUILD_DISTANCE</code></td><td><code>product_quantized</code> by default; use <code>full_precision</code> as a graph-quality control</td></tr><tr><td><code>ANN_DISKANN_RAW_VECTOR_ENCODING</code></td><td><code>f16</code> by default; use <code>f32</code> as an exact-distance and numeric-range control. The resolved value is written to every CSV row.</td></tr><tr><td><code>ANN_DISKANN_MEMORY_BUDGET_BYTES</code></td><td>DiskANN's internal preflight build budget; default 8 GiB and excludes the benchmark's source-vector allocation</td></tr><tr><td><code>ANN_NOISE_DIMENSIONS / ANN_CLUSTERS / ANN_SEED</code></td><td>Generated-workload distribution and reproducibility controls; ignored by public file loading.</td></tr><tr><td><code>ANN_OUTPUT_DIR</code></td><td>Directory whose filesystem supplies the local-storage path</td></tr><tr><td><code>ANN_KEEP_INDEXES</code></td><td><code>1</code> retains generated index files; the default deletes each index after all selected serving cases</td></tr><tr><td><code>ANN_REUSE_INDEX_PATH</code></td><td>Skips training/build and queries one retained index; requires a single <code>ANN_INDEXES</code> value</td></tr></tbody></table></div>
          <div class="callout warning"><strong>Large-run memory boundary</strong>The current public writer API accepts an in-memory vector slice and each index writer retains its encoded or raw build state. A raw 10 GiB DiskANN run therefore needs roughly the 10 GiB source allocation plus DiskANN's separately budgeted build state. On a 48 GiB machine, use a 32 GiB DiskANN budget and run one index per process. This benchmark validates large immutable builds; it is not a streaming-ingest benchmark.</div>
          <p>The benchmark executes three serving cases by default. <code>local_ssd_warm_cache</code> uses the real index file with no artificial delay. <code>remote_cache_2ms</code> adds a fixed 2 ms delay per <code>pread</code> call while reading all ranges in that call concurrently. <code>object_store_20ms</code> uses the <code>ObjectStore</code> read plan: open/optimization and sequential-query latency are the measured CPU/I/O time plus 20 ms for every observed dependent read round, avoiding hours of idle wall-clock sleep while retaining the exact fixed-latency model; batch QPS still injects the real 20 ms sleep so concurrent-client overlap is measured rather than inferred. These models do not include bandwidth or operating-system cache misses. DiskANN uses a dedicated range-I/O pool separate from Rayon query workers, preventing a saturated batch from starving nested positional reads; IVF retains the shared compute pool because its batch read scheduling is not nested inside per-query sessions. The unified IVF Reader reuses the 64-byte type-dispatch header and loads resident metadata in one further contiguous read, reducing initialization from three rounds to two without changing any format. DiskANN is opened explicitly as <code>LocalStorage</code>, <code>RemoteStorage</code>, or <code>ObjectStore</code> with the default 4 GiB Reader budget, which is automatically partitioned into resident state and caches. The sequential sweep and batch call use separate opened-and-optimized Readers, preventing all benchmark queries from prewarming their own batch measurement. CSV output records optimization rounds, ranges, and bytes separately from query I/O. Use <code>ANN_STORAGE_CASES</code> for a focused rerun against a retained index. See the <a href="index.html#public-corpus-check">recorded public-corpus results and interpretation</a>.</p>
        </section>

        <section class="article-section" id="diskann-bench">
          <h2>DiskANN access-pattern benchmark</h2>
          <p><code>diskann_bench</code> measures the page-oriented search path separately from the IVF benchmark. With no dataset paths it generates a small deterministic correctness workload. SIFT1M mode reads standard <code>.fvecs</code> and <code>.ivecs</code> files.</p>
          <div class="code-block"><span class="code-label">Shell · generated smoke workload</span><pre><code>RAYON_NUM_THREADS=4 \
DISKANN_BENCH_ACCEPTANCE=1 DISKANN_BENCH_SIMULATED_RTT_MS=1 \
DISKANN_BENCH_CONCURRENCY=4 \
DISKANN_BENCH_STORAGE_LAYOUT=compact \
DISKANN_BENCH_RAW_VECTOR_ENCODING=f32 \
DISKANN_BENCH_BUILD_DISTANCE=product_quantized \
cargo bench -p paimon-vindex-core --bench diskann_bench</code></pre></div>
          <div class="code-block"><span class="code-label">Shell · SIFT1M acceptance</span><pre><code>DISKANN_BASE_FVECS=/data/sift_base.fvecs \
DISKANN_QUERY_FVECS=/data/sift_query.fvecs \
DISKANN_GROUND_TRUTH_IVECS=/data/sift_groundtruth.ivecs \
DISKANN_BENCH_INDEX_DIR=/mnt/local-nvme \
RAYON_NUM_THREADS=16 \
DISKANN_BENCH_CONCURRENCY=16 \
DISKANN_BENCH_READER_MEMORY_BUDGET_BYTES=4294967296 \
DISKANN_BENCH_ACCEPTANCE=1 \
DISKANN_BENCH_FILTERED_MATRIX=1 \
DISKANN_BENCH_SIMULATED_RTT_MS=5 \
cargo bench -p paimon-vindex-core --bench diskann_bench</code></pre></div>
          <p>The benchmark writes a temporary <code>.dann</code> file under <code>DISKANN_BENCH_INDEX_DIR</code> (or the system temporary directory), reopens it through a benchmark-private <code>SeekRead</code> adapter, and deletes it on exit. This adapter exercises native positional I/O without adding a local-file type to the public core API. Use <code>DISKANN_BENCH_STORAGE_LAYOUT=compact|interleaved</code>, <code>DISKANN_BENCH_RAW_VECTOR_ENCODING=f32|f16</code>, and <code>DISKANN_BENCH_BUILD_DISTANCE=full_precision|product_quantized</code> for controlled comparisons. PQ defaults to <code>DISKANN_BENCH_PQ_CODE_RATIO=0.0625</code>; <code>DISKANN_BENCH_PQ_M</code> is an optional explicit override in <code>1..=dimension</code> and <code>DISKANN_BENCH_PQ_BITS</code> selects 4 or 8 bits. Generated workloads can be scaled with <code>DISKANN_BENCH_SMOKE_N</code>, <code>DISKANN_BENCH_SMOKE_NQ</code>, and <code>DISKANN_BENCH_SMOKE_DIMENSION</code>; exact top-10 truth is generated automatically. CSV rows inject representative latency hints for all four internal read tiers at <code>l_search</code> 50, 100, and 200 so timing noise does not make planner comparisons ambiguous. <code>DISKANN_BENCH_SIMULATED_RTT_MS</code> adds the same fixed delay to every <code>pread</code> round for every tier. Set <code>DISKANN_BENCH_WARMUP_QUERIES=0</code> to measure resident-only initialization; its default 8 replays representative queries before latency sampling. <code>DISKANN_BENCH_READER_MEMORY_BUDGET_BYTES</code> controls the one public Reader budget; the benchmark records automatically resolved cache behavior through search statistics. Rows additionally identify layout, raw-vector encoding, build distance, selected graph-shard count, and complete parallel-session query counts.</p>
          <p>Set <code>DISKANN_BENCH_FILTERED_MATRIX=1</code> to add random and contiguous/cluster-correlated filters at 0.01%, 0.1%, 1%, 10%, 50%, and 100% selectivity. The matrix compares adaptive results with a forced exhaustive matching-node baseline. It reports the actual executed strategy and graph fallback count from <code>diskann_search_stats</code>, Recall@10, elapsed time, I/O, PQ distance evaluations versus PQ code loads, query chunks, rerank chunks, candidate references, unique raw windows, batch LRU hits/misses/evictions, and parallel exact-rerank work. Under <code>DISKANN_BENCH_ACCEPTANCE=1</code>, every graph-enabled cell must stay within one absolute Recall@10 percentage point of the exhaustive baseline.</p>
          <p>The benchmark warms one Reader. <code>DISKANN_BENCH_CONCURRENCY=1</code> repeatedly invokes the single-query API; larger values invoke the native batch path at that width. Retained sessions run complete queries concurrently for small batches and all interleaved-layout batches. Large compact batches retain shared vector-window rerank. Graph sessions clone only the storage handle and share resident PQ/row-ID/code allocations, validation state, hot prefix, and cold-adjacency LRU. Local results use the operating-system page cache after index creation; set <code>DISKANN_BENCH_INDEX_DIR</code> to the production SSD mount and run cold-cache testing with an external, platform-appropriate cache-control procedure.</p>
          <div class="callout warning"><strong>Acceptance scope</strong>When enabled, the gate requires unfiltered Recall@10 ≥ 0.90, filtered graph Recall@10 within one point of exhaustive matching-node scan, build time below two hours, peak RSS below 4 GiB, and at most seven graph rounds plus one exact-rerank round per <code>ObjectStore</code> query. The generated workload validates mechanics only. SIFT1M or a representative production dataset remains a required non-CI release check.</div>
        </section>

        <section class="article-section" id="c">
          <h2>C FFI smoke test</h2>
          <p>Build the release shared library, then compile the C test against the generated header.</p>
          <div class="code-block"><span class="code-label">Shell</span><pre><code>cargo build --release -p paimon-vindex-ffi
cmake -S c -B c/build
cmake --build c/build
LD_LIBRARY_PATH=target/release c/build/test_vindex</code></pre></div>
          <div class="callout"><strong>Platform note</strong><code>LD_LIBRARY_PATH</code> is the Linux loader variable. Use the equivalent loader configuration on other platforms.</div>
        </section>

        <section class="article-section" id="cpp">
          <h2>C++ smoke test</h2>
          <p>The C++ test reuses the same FFI shared library and validates the RAII header.</p>
          <div class="code-block"><span class="code-label">Shell</span><pre><code>cargo build --release -p paimon-vindex-ffi
cmake -S cpp -B cpp/build
cmake --build cpp/build
LD_LIBRARY_PATH=target/release cpp/build/test_vindex_cpp</code></pre></div>
        </section>

        <section class="article-section" id="java">
          <h2>Java / JNI tests</h2>
          <p>The Maven test phase compiles the Java sources and runs <code>VectorIndexJavaApiTest</code>. This test covers result and metadata objects, native resource mapping, closed-handle behavior, and API compilation without loading the native library:</p>
          <div class="code-block"><span class="code-label">Shell</span><pre><code>mvn -f java/pom.xml test</code></pre></div>
          <p>Native validation, panic-boundary, and handle-safety tests are standalone <code>main</code> programs. The following Linux commands mirror the additional JNI checks run by CI:</p>
          <div class="code-block"><span class="code-label">Shell · Linux JNI checks</span><pre><code>cargo build -p paimon-vindex-jni --release

java -cp java/target/test-classes:java/target/classes \
  org.apache.paimon.index.vector.VectorIndexNativeValidationTest \
  "$(pwd)/target/release/libpaimon_vindex_jni.so"
java -cp java/target/test-classes:java/target/classes \
  org.apache.paimon.index.vector.VectorIndexNativePanicBoundaryTest \
  "$(pwd)/target/release/libpaimon_vindex_jni.so"
java -cp java/target/test-classes:java/target/classes \
  org.apache.paimon.index.vector.VectorIndexNativeHandleSafetyTest \
  "$(pwd)/target/release/libpaimon_vindex_jni.so"</code></pre></div>
          <div class="callout"><strong>Platform note</strong>The shared-library filename above is for Linux. Use the native library filename produced in <code>target/release</code> on macOS or Windows.</div>
          <p>Release CI additionally builds the multi-platform Maven JAR and runs <code>VectorIndexNativeLoaderSmokeTest</code> from the final <code>java-package</code> artifact with no native path argument on Linux x86-64, Linux aarch64, macOS arm64, and Windows x86-64. This ensures consumers can use every bundled JNI library directly.</p>
        </section>

        <section class="article-section" id="python">
          <h2>Python ctypes tests</h2>
          <p>The Python package must be able to locate the newly built FFI library:</p>
          <div class="code-block"><span class="code-label">Shell</span><pre><code>cargo build --release -p paimon-vindex-ffi
cd python
PAIMON_VINDEX_LIB_PATH=../target/release pip install -e ".[test]"
PAIMON_VINDEX_LIB_PATH=../target/release pytest -v</code></pre></div>
          <p>The suite uses NumPy inputs and covers all index types, persisted RQ bit metadata, single/batch shapes, and error paths.</p>
        </section>

        <section class="article-section" id="format">
          <h2>Storage format and compatibility</h2>
          <p>Changes to headers, flags, sections, sort order, or graph encoding must update both format fixtures and documentation. The normative v1 field definitions live in <a href="../core/STORAGE_FORMAT.md">core/STORAGE_FORMAT.md</a>.</p>
          <div class="card-grid"><article class="card"><h3>Reader compatibility</h3><p>Unknown versions, required flags, non-zero reserved bytes, negative counts, and out-of-bounds sections must fail rather than be guessed.</p></article><article class="card"><h3>Stable fixtures</h3><p><code>core/tests/fixtures</code> stores byte-exact hexadecimal fixtures for every v1 index family. DiskANN additionally freezes compact multi-page, 4-bit interleaved, FOR row-ID, and raw row-ID variants.</p></article><article class="card"><h3>Outer integrity</h3><p>Vector-index files do not embed a checksum. Length and checksum validation belong to the outer Paimon file and manifest layer.</p></article></div>
          <nav class="pager" aria-label="Related documentation"><a href="api.html"><small>Previous</small>← API and language bindings</a><a href="releases.html"><small>Next</small>Releases →</a></nav>
        </section>
      </article>
    </div>
  </div></main>
  <footer class="site-footer"><div class="footer-inner"><span>Apache Paimon Vector Index</span><span>Development and benchmarks</span></div></footer>
</body>
</html>