diff --git a/.github/workflows/generate-release-notes.yml b/.github/workflows/generate-release-notes.yml index 95afe7701..8230a5907 100644 --- a/.github/workflows/generate-release-notes.yml +++ b/.github/workflows/generate-release-notes.yml @@ -18,6 +18,7 @@ jobs: runs-on: ubuntu-latest permissions: contents: write + pull-requests: write steps: - uses: actions/checkout@v4 @@ -132,7 +133,7 @@ jobs: for f in "${all_files_for_section[@]}"; do fname=$(basename "$f") [[ "$fname" =~ ^([0-9]+)\. ]] && echo "${BASH_REMATCH[1]} $f" - done | sort -n | awk '{print $2}' + done | sort -n | cut -d' ' -f2- )) unset IFS @@ -170,14 +171,21 @@ jobs: echo "Generated ${OUTPUT}:" cat "${OUTPUT}" - - name: Commit and push - run: | - git config user.name "github-actions[bot]" - git config user.email "github-actions[bot]@users.noreply.github.com" - git add "docs/release notes/${{ inputs.version }}/${{ inputs.version }}.md" - if git diff --cached --quiet; then - echo "Output file is unchanged — nothing to commit." - else - git commit -m "Generate release notes for ${{ inputs.version }}" - git push - fi + - name: Create Pull Request + uses: peter-evans/create-pull-request@v6 + with: + token: ${{ secrets.GITHUB_TOKEN }} + branch: release-notes/${{ inputs.version }} + base: main + author: "github-actions[bot] " + add-paths: | + docs/release notes/${{ inputs.version }}/${{ inputs.version }}.md + commit-message: "Generate release notes for ${{ inputs.version }}" + title: "docs: release notes for ${{ inputs.version }}" + body: | + Generated by the **Generate Release Notes** workflow. + + Assembled release notes for **${{ inputs.version }}** from source files in `docs/release notes/${{ inputs.version }}/`. + + Review `docs/release notes/${{ inputs.version }}/${{ inputs.version }}.md` before merging. + labels: documentation diff --git a/docs/release notes/4.0.0-RC.9/4.0.0-RC.9.md b/docs/release notes/4.0.0-RC.9/4.0.0-RC.9.md new file mode 100644 index 000000000..1de283f45 --- /dev/null +++ b/docs/release notes/4.0.0-RC.9/4.0.0-RC.9.md @@ -0,0 +1,640 @@ +# JVector Release Notes + +## Version: 4.0.0-RC.8 - 4.0.0-RC.9 + +--- + +## New Features + +### Streaming N:1 Graph Index Compaction + +**PR:** [#659](https://github.com/datastax/jvector/pull/659) + +**Description** +`OnDiskGraphIndexCompactor` merges multiple on-disk HNSW graph indexes into a single compacted index. This targets write-heavy workloads where data is continuously ingested into small segment indexes that accumulate over time; periodically compacting those segments into one larger index improves search throughput and recall without rebuilding from scratch. + +Each source is an `OnDiskGraphIndex` paired with a `FixedBitSet` marking its live (non-deleted) nodes. The compactor merges all live nodes into a single graph, remaps each source's ordinals into a contiguous global ordinal space via user-supplied `OrdinalMapper`s, and rebuilds each node's neighbor list using Vamana-style diversity selection over candidates gathered from the node's own source and from graph searches in the other sources. Sources are processed in parallel, batch-by-batch, with a bounded backpressure window so memory stays flat regardless of dataset size. + +**Purpose / Impact** +- Consolidates many small segment indexes into one larger index, improving search throughput and recall without a full rebuild +- Excludes deleted nodes from the output, reclaiming space as part of the merge +- Bounded heap footprint — compaction streams per-source batches in parallel rather than materializing the full merged graph in memory. All benchmark datasets compact under `-Xmx5g`, including a 10M-vector, 2560-dimension dataset +- Measured recall is on par with or better than building a single index from scratch across a range of datasets + +**How to Enable** +Construct an `OnDiskGraphIndexCompactor` from the source indexes, their live-node bitsets, and an ordinal remapper per source, then call `compact()`: + +```java +List sources = List.of(index0, index1, index2); + +// Mark all nodes live (no deletions) +List liveNodes = sources.stream() + .map(s -> { var bs = new FixedBitSet(s.size()); bs.set(0, s.size()); return bs; }) + .collect(toList()); + +// Sequential ordinal remapping: source[s] node i -> global offset[s] + i +int offset = 0; +List remappers = new ArrayList<>(); +for (var src : sources) { + remappers.add(new OrdinalMapper.OffsetMapper(offset, src.size())); + offset += src.size(); +} + +var compactor = new OnDiskGraphIndexCompactor( + sources, liveNodes, remappers, + VectorSimilarityFunction.COSINE, + /* executor= */ null // null = use the core JVector thread pool +); + +compactor.compact(Path.of("compacted.index")); +``` + +Deleted nodes are dropped by clearing their bit in the corresponding `FixedBitSet` and remapping only the surviving ordinals (for example with `OrdinalMapper.MapMapper`). + +**Notes** +- The public compactor API is marked `@Experimental` and may change in a future release. +- Quantization is preserved across the merge. For `FUSED_PQ` sources the PQ codes are stored inline and the codebook is retrained on the combined dataset; non-fused sources can supply a sidecar `CompressedVectors` that is retrained and rewritten as a single merged file. +- PQ retraining uses balanced proportional sampling across all sources (up to `ProductQuantization.MAX_PQ_TRAINING_SET_SIZE` vectors total, at least 1000 per source). +- A new `CompactorBenchmark` (in `benchmarks-jmh`) measures compaction time, heap footprint, recall, and search latency. See `docs/compaction.md` for the full algorithm description and benchmarking instructions. + + +--- + +## Performance Improvements + +Fixes https://github.com/datastax/jvector/issues/629. + +This pull request simplifies AVX-512 feature detection logic in the jvector_simd_check.c file by replacing manual CPUID bit checks with built-in compiler functions (available only on gcc and llvm compilers). The earlier code was missing XSAVE support at runtime. On VMs where 512‑bit register state saving is disabled, this can result in #UD faults when executing AVX‑512 instructions. + +CPU feature detection improvements: + + +1. Replaced manual CPUID register checks for AVX-512 features with calls to __builtin_cpu_supports for each required feature, improving code readability and portability. + Added a call to __builtin_cpu_init() to ensure proper initialization before feature checks. +2. . Rename `check_compatibility `to `check_avx512_compatibility`. + + +--- + +### Faster Index Construction via Bulk Vector Serialization + +**PR:** [#681](https://github.com/datastax/jvector/pull/681) + +**Description** +Optimized vector serialization in `MemorySegmentVectorProvider` by replacing +per-element scalar writes with bulk operations. At high core counts, +per-element scalar writes become a serialization bottleneck: multiple threads +contend on a single lock, stalling index construction and negating the benefit +of additional cores. Bulk writes eliminate that contention, restoring scaling +at large core counts. + +- **`writeFloatVector`** — uses backing `float[]` with `writeFloats()` instead + of per-element `writeFloat()` +- **`writeByteSequence`** — uses backing `byte[]` with bulk `write()` instead + of per-element `writeByte()` + +**How to enable** +No configuration required. The improvement is applied automatically when using +the native SIMD backend (`jvector-native`). + +**Performance (AWS `x8i.24xlarge`)** +- Bulk serialization throughput: **~1.3× to 1.9×** improvement across tested vector sizes +- **openai-1536-1m** index build: ~57% faster (105.59s → 45.22s) +- **openai-3072-1m** index build: ~38% faster (164.63s → 102.75s) + + +--- + +## Bug Fixes and Issue Resolutions + +This PR makes dataset loading more robust in the face of file corruption errors. +It also indirects dataset access through DataSetInfo, caching the attached DataSet locally, and enabling access via getDataSet(). This will vastly speed up testing flows which are uneccesssarily processing datasets into memory right when the actual data is not yet accessed. + +[UPDATE] + +Due to a request for fixing some longer-standing issues, the scope of this PR has increased moderately: + +* DataSet loader formats which can not provide metadata (VSF, ...) on their own now pull details from dataset_metadata.yml for VSF. If none such is available, an error should be thrown. + +* Addtionally, the contract type (DataSetProperties) is now a layer of requirements added onto the DataSetInfo type, including an indicator for dataset preprocessing aspects (zero vectors, dupes) and normalization. + +* surefire tests were disabled in examples. enabled, and fixed up a unit test + +* DataSetProperties is a carrier type in the new DataSetInfo contract + + + +--- + +### Disable Graph Search Pruning + +**PR:** [#686](https://github.com/datastax/jvector/pull/686) + +**Description** +Resolves [Issue #678](https://github.com/datastax/jvector/issues/678). Deprecated and disabled `GraphSearcher.usePruning(boolean)` for TopK and filtered searches. +The existing TopK/filter pruning heuristic could reduce recall and did not show reliable production value. +The method remains available for source and binary compatibility, but enabling it no longer activates pruning for `threshold == 0` searches. +Legacy threshold-search behavior is preserved: searches with `threshold > 0` continue to use the existing threshold early-termination path. + +**How to Enable** +No API change. The `usePruning` setting will be ignored and set to `false` by the library. + +**Notes** +The recall regression for `usePruning = true` is present in release 4.0.0-rc.4 onward (possibly earlier). +[PR #501](https://github.com/datastax/jvector/pull/501) 4.0.0-rc.3 inadvertently partially addressed it. + + +--- + +## Documentation and Tutorials + +This includes: + +- A set of tutorials covering basic JVector concepts + + +--- + +This PR adds links to the introductory tutorials in the main README. The existing step-by-step guide is moved out to it's own file, considering that it's rather cryptic for new users to decipher. I elected not to remove it entirely since it still contains useful information that isn't (yet) documented elsewhere. + + + +--- + + + +--- + +## Testing Enhancements + +## Summary + +This PR introduces YAML-driven benchmark compute + reporting selection, and adds run-scoped logging artifacts (`sys_info.json`, `dataset_info.csv`, `experiments.csv`) so Grid benchmark runs are reproducible and easier to analyze offline. Reporting/benchmark policy is now run-level (via `yaml-configs/run.yml`), while dataset YAML files are simplified to dataset-tuned construction/search settings only. + +--- + +## Key changes + +### Run-level policy file: `yaml-configs/run.yml` + +**PR:** [#625](https://github.com/datastax/jvector/pull/625) + +- Added a new run-level config file that defines: + - **benchmark computation policy** (`benchmarks`: what to compute) + - **console projection** (`console`: subset + named metrics) + - **experiments logging policy** (`logging`: selection + sink type + run metadata) +- Dataset YAMLs are now focused on **dataset-tuned parameters only** (construction/search grids). Reporting/benchmark controls are no longer expected in dataset YAMLs. + +### Config versioning clarity + +- Renamed the old vague YAML `version` field to **`onDiskIndexVersion`** (still validated against `OnDiskGraphIndex.CURRENT_VERSION`). +- Introduced **`yamlSchemaVersion`** (starting at **1**) to version YAML schema independently of on-disk index header format. +- Added deprecated back-compat for legacy files that still use `version: 6`. + +### Compute vs display vs logging separation + +- Benchmarks specify **what is computed**; console/logging are **projections** only. +- Added stable `Metric.key` identifiers for all benchmark outputs and telemetry so selection and CSV columns are robust (no header-string matching). +- Enforced: + - **strict subset validation** for benchmark-stat selections (must be computed) + - **best-effort warnings** for runtime-unavailable telemetry/metrics (omitted from output/CSV, with warnings) + +### Run-scoped artifacts and logging + +- When logging is enabled, each BenchYAML invocation creates a run directory under: + - `logging//` +- The run directory contains (see examples below): + - `sys_info.json` — host/OS/JVM/SIMD/threading/memory + stable `system_id` + - `dataset_info.csv` — one row per dataset used in the run + - `experiments.csv` — flat, append-only table of fixed params + selected output-key columns +- `experiments.csv` schema stability: + - fixed columns are defined centrally in `ExperimentsSchemaV1` + - output-key columns are an ordered union across all topK scenarios encountered in the run + - non-applicable / missing values are written as **empty CSV fields** (easy for pandas/matplotlib) + +### Plumbing / ergonomics + +- Added `RunArtifacts` to centralize: + - run setup + validation + - dataset registration (`dataset_info.csv`) + - row logging (`experiments.csv`) +- Updated BenchYAML and HelloVectorWorld to use `run.yml` + `RunArtifacts` (no manual wiring of compute/display/logging maps). +- Refactored Grid signatures to take a single `RunArtifacts` object instead of many reporting parameters. +- Legacy callers continue to work via the legacy Grid overload + `RunArtifacts.disabled()`. + +### Legacy compatibility + +- Legacy YAML files **without** `yamlSchemaVersion` are treated as **schema 0**: + - parsed leniently (unknown fields ignored) + - emit a **deprecation warning** + - optional extraction of legacy `search.benchmarks` is supported for legacy behavior when `run.yml` is absent +- AutoBenchYAML is not migrated to run.yml yet; it continues to work with legacy configs. + +--- + +## Behavior + +- **Logging enabled** (`run.yml` has `logging.type`): + - BenchYAML writes run artifacts under `logging//` + - prints a startup message pointing to the run directory and how to delete it to reclaim space + - appends experiment rows as configurations execute +- **Logging disabled** (no `logging` section or blank `type` in `run.yml`): + - run artifacts are not created + - `RunArtifacts` becomes a no-op via `RunArtifacts.disabled()` +- Console output continues to work via run-level selection; dataset configs tune only construction/search parameters. + +--- + +## Notes / limitations + +- `experiments.csv` uses `Metric.key` strings as column names for benchmark outputs/telemetry; missing values are empty CSV fields. +- `sys_info.json` SIMD reports the configured vector width via `io.github.jbellis.jvector.vector_bit_size` (with `simd_config_present`), avoiding brittle runtime probing. +- Threading in `sys_info.json` distinguishes build executor parallelism vs parallel streams (common FJP), explaining why build/query values can differ. +- This PR intentionally does not overhaul per-index build time attribution (e.g., cache hits have no build time by design); deeper build-timing correctness is left for a future pass. +- Dataset YAML breaking changes are intentional: `version` → `onDiskIndexVersion`, plus removal of dataset-level benchmarks/console/logging fields in favor of `run.yml`. + +## Example of run.yml and yaml-config for dataset +ada002-100k yml +run yml + +## Examples of artifacts (sys_info, dataset_info, experiments) + +experiments csv +dataset_info csv +sys_info json + + +--- + +This PR fixes a resource leak where BenchmarkDiagnostics (and its DiskUsageMonitor/NIO WatchService) was created without being closed in several benchmark execution paths. Over repeated runs/config sweeps this leaked OS watch resources/file handles and could fail with “User limit of inotify instances reached.” + +Changes: +- Wrap BenchmarkDiagnostics usage in try-with-resources in QueryTester and Grid run paths so it is always closed. +- Refactor ThroughputBenchmark to create BenchmarkDiagnostics within runBenchmark using try-with-resources (instead of storing it as a field), ensuring proper cleanup. + + +--- + +This PR expands the capability of the DiskUsageMonitor in the BenchYAML and autoBenchYAML testing applications to be able to monitor more than a single directory and to provide detailed information on disk used and number of new files for each directory monitored, as well as a summary of the total bytes used and number of new files across all monitored directories. + +Although this allows DiskUsageMonitor to monitor an arbitrary number of directories in practice as of this PR it is used to monitor the work directory and the cache directory which contains indexes that may be reused for testing purposes. + +example rollup summary at index construction time: +``` +Disk Usage Summary Graph Index Build: + [testDirectory]: + Total Disk Used: 1.61 GB + Total Files: 1 + Net Change: 1.61 GB, +1 files + [indexCache]: + Total Disk Used: 0 B + Total Files: 0 + Net Change: 0 B, +0 files + [Overall Total]: + Total Disk Used: 1.61 GB + Total Files: 1 + Net Change: 1.61 GB, +1 files +Index build time: 956.781921 seconds + + +--- + +## Summary + +This PR adds quantization timing telemetry to BenchYAML reporting (console + experiments.csv), and makes it selectable from `run.yml`. The new metrics capture quantizer **compute** and **encoding** time for both **index construction** and **search-time vector encoding**, with stable `Metric.key` identifiers. + +## Key changes + +### Quantization timing metrics (new) + +**PR:** [#636](https://github.com/datastax/jvector/pull/636) +- Emit quant timing metrics as `Metric` outputs: + - **Index construction:** `construction.index_quant_time_s..(compute_time_s|encoding_time_s)` + - **Search-time:** `search.search_quant_time_s..(compute_time_s|encoding_time_s)` +- Per-quantizer keys are emitted for PQ/BQ; NVQ reports **compute only** (NVQ encoding is intentionally not measured). +- Human-readable headers are phase-tagged to disambiguate: + - `Idx PQ compute (s)`, `Idx PQ encoding (s)`, `Search PQ compute (s)`, etc. + +### Run-level selection support (prefix expansion) +- Added prefix-style named metric selection in `run.yml`: + - `metrics.construction.index_quant_time_s` → expands to all `construction.index_quant_time_s.*` metrics + - `metrics.construction.search_quant_time_s` → expands to all `search.search_quant_time_s.*` metrics +- Selection + application now supports “prefix families” of metrics (similar to `recall_at_{topK}` expansion). + +### Logging fixes (experiments.csv) +- Updated logging schema planning to include prefix-selected metrics so quant timing columns appear in `experiments.csv` when selected via run.yml. +- CSV column keys remain `Metric.key` strings; missing/non-applicable values remain empty fields. + +### Console ergonomics +- Added 2-line header rendering with a max column width cap so wide metric names wrap cleanly. +- Manual header breaks are supported via `\n` in the `Metric.header` string (useful for specific columns like Avg QPS). + +### Runtime warnings (availability) +- Preserved “warn once” behavior for missing exact-key metrics (e.g., index build time on cache hit). +- Quant timing warnings were adjusted so we do not warn about NVQ encoding (not measured by design). + +## Notes / limitations +- PQ compute time is recorded when the PQ compressor is computed (cache miss). On cache hits, compute time is intentionally blank (no compute performed). +- NVQ encoding time is intentionally not measured; NVQ reports compute only. +- Prefix selection is used for quant timing because concrete per-quantizer keys are only known once the run produces metrics. + + +--- + +This extends the v1 experiments.csv schema with build_compressor and search_compressor columns so logged runs record the resolved compressor configurations used during index build and search. Values use the compressor’s printable toString() (e.g., ProductQuantization[...]) and are CSV-escaped to preserve commas. Search compressor logging is null-safe when compression is disabled (cv == null). +build-and-search-compressor-info + + +--- + +## Investigation Complete: Root Cause Found and Fixed + +### Key Findings + +**PR:** [#645](https://github.com/datastax/jvector/pull/645) + +#### 1. The "Recall Degradation" Was Actually a Test Harness Bug +Test results showed: +- **BenchYAML + fusedGraph: No** → recall: 0.65 +- **BenchYAML + fusedGraph: Yes** → recall: 0.65 +- **AutoBenchYAML + fusedGraph: No** → recall: 0.77 ← **ANOMALY** +- **AutoBenchYAML + fusedGraph: Yes** → recall: 0.65 + +The anomaly was that AutoBenchYAML with fusedGraph:No produced artificially **high** recall (0.77), not that FusedPQ was producing low recall. All configurations using FusedPQ correctly produced consistent recall of 0.65. + +#### 2. Root Cause: Missing Vector Encoding in `runAllAndCollectResults` +**Location:** `jvector-examples/src/main/java/io/github/jbellis/jvector/example/Grid.java`, line 833 + +**The Bug:** +```java +CompressedVectors cvArg = (searchCompressorObj instanceof CompressedVectors) ? (CompressedVectors) searchCompressorObj : null; +``` + +This line attempted to cast a `VectorCompressor` (ProductQuantization) to `CompressedVectors`, which always failed, resulting in `cvArg = null`. When `cvArg` is null and fusedGraph is disabled, the `ConfiguredSystem.scoreProviderFor` method falls back to **exact scoring using uncompressed vectors** (line 1084 / 1094), which artificially inflates recall. + +**The Fix:** +```java +// Encode vectors for reranking if a compressor is provided +CompressedVectors cvArg; +if (searchCompressorObj == null) { + cvArg = null; +} else { + cvArg = searchCompressorObj.encodeAll(ds.getBaseRavv()); +} +``` + +This fix properly encodes the vectors using the compressor, matching the behavior of `runOneGraph` (the correct implementation used by BenchYAML). + +### Impact +- **Before fix:** AutoBenchYAML with fusedGraph:No was using exact (uncompressed) scoring, giving misleadingly high recall +- **After fix:** AutoBenchYAML will now correctly use PQ-compressed vectors for approximate scoring during graph traversal, with NVQ reranking, producing consistent recall across all test harnesses +### Verification +After applying this fix: +- AutoBenchYAML + fusedGraph:No → recall: ~0.65 (matching other configurations) +- All test harnesses will produce consistent recall measurements +- The perceived "recall degradation with high dimensionality" will disappear, as it was never a real issue with FusedPQ + +### Additional Notes +The investigation also examined the FusedPQ implementation thoroughly and confirmed that: +- The storage and retrieval of fused PQ data is correct +- The scoring mathematics are correct +- The SIMD implementations are correct +- FusedPQ works correctly for all dimensionalities when properly configured + + +--- + +## Summary + +This PR optimizes `AccuracyMetrics` to improve the performance and readability of recall and precision measurements. The primary focus is shifting from $O(N^2)$ list-scanning to $O(N)$ Set-based lookups and reducing object allocation overhead by interacting directly with primitive `NodeScore` arrays. + +## Key changes + +### Logic cleanup and "dead code" removal + +**PR:** [#647](https://github.com/datastax/jvector/pull/647) +- **Unused Signature Removal:** Removed the redundant `topKCorrect(List, List, ...)` signature, consolidating logic into a single private method that operates directly on `SearchResult`. +- **Branch Elimination:** Removed the unreachable `if (gtView.size() > retrieved.size())` branch. Due to existing guard clauses (`kGT <= kRetrieved` and `kRetrieved <= retrieved.size()`), this condition was mathematically impossible. +- **Parity Preservation:** Preserved specific, argument-focused exception messaging to ensure clear feedback for invalid `k` parameters. + +### Performance optimizations +- **Algorithmic Shift:** Replaced $O(N)$ `List.contains()` calls with $O(1)$ `HashSet.contains()`. This moves the core intersection logic from quadratic to linear time complexity. +- **AP Nested Scan Removal:** In `averagePrecisionAtK`, replaced the $O(i)$ `subList(0, i).contains(p)` duplicate check with a `HashSet` lookup. This transforms the AP calculation from an $O(K^2)$ operation to $O(K)$. +- **Allocation Reduction:** Eliminated intermediate boxing and `ArrayList` creation in the `SearchResult` path. The logic now iterates directly over the `NodeScore[]` array, significantly reducing GC pressure during large benchmark runs. +- **Loop Efficiency:** Replaced Stream API calls with manual `for` loops in high-frequency paths to avoid the object overhead and "setup tax" of the Stream API. +- **HashSet Pre-Sizing:** Sets are now initialized with explicit capacities ($K / 0.75$) to prevent expensive internal rehashing. + +### Test coverage +- **Functional Validation:** Verified Recall, AP, and MAP calculations against hand-calculated results to ensure mathematical parity with standard IR definitions. +- **Duplicate Handling:** Confirmed that duplicate IDs in search results are handled via the "seen" Set, preventing artificial score inflation while maintaining $O(1)$ lookup speed. +- **Exception Parity:** Explicitly verified that `IllegalArgumentException` strings remain 1:1 identical to the original implementation to prevent breaking downstream error-parsing. + +## Performance results + +Benchmarks were conducted using 10,000 queries on the `gecko-100k` dataset (768 dimensions). + +| Metric | Original Time | Optimized Time | Delta | +| :--- | :--- | :--- | :--- | +| **recall@10** | **~8ms**| ~same | - | +| **recall@100** | **~58ms** | **~25ms** | **~57% reduction** | +| **AP@100 / MAP@100** | **~62ms** | **~26ms** | **~58% reduction** | + +While the benefit is negligible for $K=10$, the optimization becomes critical as $K$ increases. The reduction in AP measurement time is particularly significant as it eliminates the $O(K^2)$ complexity previously caused by nested sublist scans. + +## Notes / limitations + + +--- + +# Summary + +This PR restructures benchmark dataset loading so scrubbing behavior is explicit, metadata-controlled, and uniform across HDF5 and MFD loaders. The primary goal is to stop hard-coding legacy load-time scrubbing behavior into loader-specific paths and prepare a safe transition toward prescrubbed datasets whose offline ground truth matches the stored vectors exactly. + +# Key changes + +## Scrubbing behavior becomes explicit + +- Added `DataSetProperties.LoadBehavior` with: + - `LEGACY_SCRUB` + - `NO_SCRUB` +- Added `load_behavior` support to `dataset_metadata.yml`. +- Added `DataSetUtils.processDataSet(...)` as the new metadata-aware entry point for benchmark dataset processing. +- Preserved the prior load-time scrubbing implementation behind `legacyScrubDataSet(...)`. +- Kept `getScrubbedDataSet(...)` temporarily as a deprecated compatibility shim. + +## Unified metadata-driven loader flow + +- Updated both `DataSetLoaderHDF5` and `DataSetLoaderMFD` to carry full `DataSetProperties` through the load path instead of collapsing metadata down to only `similarity_function`. +- Routed both loaders through `DataSetUtils.processDataSet(...)` so load behavior is applied in one place. +- Removed the HDF5-specific filename inference path and now require explicit metadata for curated HDF5 datasets. + +## Metadata coverage updates + +- Added `load_behavior` to existing curated dataset entries. +- Added explicit metadata entries for curated ann-benchmarks HDF5 datasets that previously relied on filename inference. +- Added metadata entries for remaining MFD datasets in the static registry so known datasets fail clearly if metadata is missing. + +## Visibility / debugging + +- Added runtime printing of the dataset similarity function in the graph run path so the effective metadata-supplied similarity can be seen during execution. + +# Behavior changes + +## What changes now + +- Curated benchmark datasets no longer rely on HDF5 filename suffix inference for similarity. +- Known datasets are now expected to have an explicit `dataset_metadata.yml` entry. +- Benchmark dataset load behavior is controlled centrally through metadata rather than implicitly by format-specific code paths. +- `NO_SCRUB` now loads vectors and ground truth exactly as stored. +- `LEGACY_SCRUB` preserves the existing load-time behavior: + - zero-vector removal + - duplicate base-vector removal + - query filtering against invalid / overlapping vectors + - ground-truth remapping + - conditional normalization + +## What does not change yet + +- Existing deployed datasets can remain on `LEGACY_SCRUB` during the transition. +- The previous scrubbing behavior is still available and intentionally preserved for compatibility while prescrubbed datasets are rolled out. + +# Why this change + +- Benchmark datasets with precomputed offline ground truth should not be silently mutated by loader-specific cleanup logic unless that behavior is explicitly intended. +- The old behavior broke the correspondence between stored vectors and offline ground truth for unscrubbed datasets, even though runs still completed. +- Scrubbing policy is transitional benchmark-loader behavior, not an intrinsic property of the raw dataset format. +- Unifying HDF5 and MFD under the same metadata architecture makes the code easier to reason about, easier to maintain, and safer to evolve as prescrubbed datasets become available. + +# Notes / limitations + +- `LEGACY_SCRUB` remains the default during the transition to avoid breaking currently deployed datasets. +- `getScrubbedDataSet(...)` is still present as a deprecated compatibility API and should be removed after downstream callers are fully migrated. +- Locally added datasets in `DataSetLoaderMFD` must also be added to `dataset_metadata.yml` to participate in the new configuration model. +- This PR does not move scrubbing into the core library yet; it only prepares the benchmark loader path for a clean transition. + + + +--- + +This dataset loader is _like_ the previous MDF loader but with very targeted changes to make things easier and more robust. +* It uses HTTP, not requiring S3 to fetch data. It's still plenty fast enough and less complicated. + * Ted improved the transport to support both. It will do HTTP* and S3 as specified. +* It indirects dataset name to data facets through a datasets.yaml file. This is just a name to facet mapping file. + * It uses the catalog entries file `catalog_entries.yaml` within the source directory for clarity. +* It supports local and remote, and simply caches the remote dataset.yaml file locally and uses it by default. +* If will still inform the user if the remote datasets.yaml file has changed if they choose. +* It does the same kind of loading as the previous MFD loader once initialized. All other dataset and run settings are unaffected. + + + +--- + +Updating regression tests (AutoBenchYAML) to use the datasets.yml configuration rather than hard-coding its datasets. + +- AutoBenchYAML.java updated to dynamically load datasets for testing from yaml +- DataSets.java updated to return a specific subsection of datasets.yml based on section key passed +- new subsection ("regression-tests") added to datasets.yml +- new yml files added for the datasets used by regression (primarily the only difference is that we don't want to run as many permutations of overquery or high k value in regression due to time concerns) + + +--- + +`buildInMemory `(the no-build-compressor path) never recorded the index build time into indexBuildTimes, so getIndexBuildTimeSeconds() always returned null for it. This caused construction.index_build_time_s to be omitted from benchmark results, triggering a warning whenever the metric was selected in the logging config: + +`WARNING: selected logging output not available; skipping metrics.construction.index_build_time_s (construction.index_build_time_s)` + +Fix: capture the duration of builder.build() in buildInMemory and store it in indexBuildTimes, consistent with how buildOnDisk already does it. Write time is intentionally excluded — only graph construction is measured. + +Before: +``` +Index build time: null seconds +``` + +After: +``` +Index build time: 32.192336 seconds + + +--- + +This adds the name entries for the openai datasets in our shared config. + + +--- + + + +--- + +This PR updates the tracking of offheap memory usage by calculating max usage comparing the pre and post phase snapshots instead of just using the final snapshot, and adds offheap memory usage to the tabular results generated by BenchYAML. + +Note that the reported offheap usage will appear static across configurations for a given dataset in this test. This is because off-heap memory is allocated during index loading (once per dataset). It includes memory-mapped index files and Direct ByteBuffers for vector storage. All query configurations reuse the same loaded index, so off-heap usage remains constant but different datasets have different index sizes, hence different off-heap usage. + +Example output: +``` +Disk Usage Summary Graph Index Build: + [testDirectory]: + Total Disk Used: 115.99 MB + Total Files: 1 + Net Change: 115.99 MB, +1 files + [indexCache]: + Total Disk Used: 0 B + Total Files: 0 + Net Change: 0 B, +0 files + [Overall Total]: + Total Disk Used: 115.99 MB + Total Files: 1 + Net Change: 115.99 MB, +1 files +Index build time: 26.442781 seconds + +cohere-english-v3-100k: ProductQuantization(M=128, clusters=256, centered=false) codebooks loaded from PQ_cohere-english-v3-100k_128_256_false_-1.0 +cohere-english-v3-100k: ProductQuantization(M=128, clusters=256, centered=false) encoded 99685 vectors [13.17 MB] in 1.75s +cohere-english-v3-100k: Using OnDiskGraphIndex(layers=[LayerInfo{size=99685, degree=32}, LayerInfo{size=3224, degree=32}, LayerInfo{size=95, degree=32}, LayerInfo{size=2, degree=32}], entryPoint=NodeAtLevel(level=3, node=75059), features=NVQ_VECTORS): + +Index configuration: + featureSetForIndex [NVQ_VECTORS] + M 32 + efConstruction 100 + neighborOverflow 1.2 + addHierarchy true + refineFinalGraph true + +Query configuration: + usePruning true + +Overquery Avg QPS ± Std Dev CV % Mean Latency Avg Visited Recall@10 Max heap Max offheap + (of 3) (ms) usage (MB) usage (MB) +--------------------------------------------------------------------------------------------------------------------------------- +1.00 52315.1 378.7 0.7 0.163 355.3 0.78 755.2 116.4 +2.00 44155.5 619.7 1.4 0.201 515.2 0.92 834.3 116.4 +5.00 28832.6 264.1 0.9 0.306 951.1 0.98 834.3 116.4 +10.00 17835.6 1134.1 6.4 0.504 1536.7 0.99 786.6 116.4 + + +Overquery Avg QPS ± Std Dev CV % Mean Latency Avg Visited Recall@100 Max heap Max offheap + (of 3) (ms) usage (MB) usage (MB) +---------------------------------------------------------------------------------------------------------------------------------- +1.00 18204.6 119.4 0.7 0.522 1536.7 0.85 803.5 116.4 +2.00 11269.1 83.9 0.7 0.846 2495.2 0.97 803.5 116.4 +``` + + +--- + +Allows all CI workflows to run correctly on PRs from forks. This mainly applies to the workflow which runs AutoBenchYAML. + +- The workflow replaces `github.head.ref` (`fork-workflow-2`) with `github.ref` (`/refs/pull/688/merge`). The checkout action fails for the former if the branch is from a fork, but the latter works in both cases. + + +--- + + +### Provide a possibility to store and load an empty graph + +**Description** +Provide a possibility to store and load an empty graph from/to disk + +**Purpose / Impact** +- +- Helps to preserve graph metadata (similarity function, features, dimensions, even if the graph is empty) + diff --git a/docs/release notes/4.0.0-RC.9/pr685.feature.md b/docs/release notes/4.0.0-RC.9/pr685.feature.md index 806e19c9a..bcb0306eb 100644 --- a/docs/release notes/4.0.0-RC.9/pr685.feature.md +++ b/docs/release notes/4.0.0-RC.9/pr685.feature.md @@ -4,5 +4,6 @@ Provide a possibility to store and load an empty graph from/to disk **Purpose / Impact** +- - Helps to preserve graph metadata (similarity function, features, dimensions, even if the graph is empty)