Implement data-blind scalar quantization by mccullocht · Pull Request #16030 · apache/lucene

mccullocht · 2026-05-04T05:02:31Z

Add an option to the quantization format to enable or disable centering (enabled by default). When centering is disabled we also stop writing the float vectors which can lead to significant storage savings. Special handling is included during merges -- we check that all of the input is in the same encoding, and handle transcoding if some of the input is float vectors.

Large portions of this change were generated using claude code. I reviewed, tweaked, and tested the code before putting it up for review.

This change appears as a new parameter on the existing Lucene104ScalarQuantizedVectorsFormat to enable centering (default is true). Disabling centering causes an internal format version bump triggering the new behavior -- disabling computation and storage of the centroid vector and related parameters.

luceneutil results -- 1M cohere vectors, 8 bit quantization.
before:

recall  latency(ms)  netCPU  avgCpuCount     nDoc  searchType  topK  fanout  resultSimilarity  decay  resultCount  maxConn  beamWidth  quantized  visited  index(s)  index_docs/s  force_merge(s)  num_segments  index_size(MB)  filterStrategy  filterSelectivity  overSample  vec_disk(MB)  vec_RAM(MB)  bp-reorder  indexType
 0.974        2.304   2.297        0.997  1000000         KNN   100     100               N/A    N/A      100.000       64        250     8 bits     8619    132.85       7527.40          235.00             1         5047.27            null                N/A       1.000      4898.071      991.821       false       HNSW

after

recall  latency(ms)  netCPU  avgCpuCount     nDoc  searchType  topK  fanout  resultSimilarity  decay  resultCount  maxConn  beamWidth  quantized  visited  index(s)  index_docs/s  force_merge(s)  num_segments  index_size(MB)  filterStrategy  filterSelectivity  overSample  vec_disk(MB)  vec_RAM(MB)  bp-reorder  indexType
 0.972        2.281   2.274        0.997  1000000         KNN   100     100               N/A    N/A      100.000       64        250     8 bits     8612    143.06       6990.07          160.33             1         1140.98            null                N/A       1.000      4898.071      991.821       false       HNSW

The harness extrapolates vector size from the input size so believe the on-disk index_size number -- this is about 4x smaller. Force merge is faster since we don't have to re-quantize vectors on merge. Recall is very similar but YMMV.

See also #16029

Allow callers to disable centering at the format level, which also disables writing of float vectors since they are no longer needed. Includes a path to handle of mix of centered and uncentered segments as input. In this case the uncentered/no float vectors will be dequantized and requantized but this case should be relatively uncommon. Includes OSQ changes to allow a zero vector for COSINE if the vector is not a unit vector. Maybe fix this in upstream callers?

github-actions · 2026-05-19T00:57:55Z

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

github-actions · 2026-07-12T00:43:28Z

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

kaivalnp · 2026-08-12T01:46:50Z

Sorry for getting to this PR so late, but I realized that data-blind quantization would be super helpful for another use-case too: in an attempt to support multiple HNSW graphs backed by the same vector storage (#14758), we need to be able to effectively de-duplicate vectors across fields.

Lucene currently supports a centroid-centered quantization, which might cause the same raw vector to be quantized differently across different fields. If it is data-blind (i.e. centered on the zero vector), quantized bytes can be sanely de-duplicated.

See #16506.

mikemccand · 2026-08-13T15:10:32Z

+1, I love that we are making progress on data-blind quantization. This should work well when incoming vectors are isotropic (all dimensions behave the same i.e. the histograms of their per-dimension values approximate little baby gaussians with mean 0) and variance as required to be on unit sphere at that dimensionality.

But I think your average trained model won't just produce isotropic embeddings? For such cases (probably the common case? not sure), we have pre-conditioning / random Hadamard rotation (another PR in flight for this? -- yes #16092!) which should (usually? there are adversaries (intentional or otherwise) for any rotation matrix right?) scrub anisotropic vectors. Sort of like the record and record players in GEB -- thank you Kurt Gödel!).

These are all experimental vector codecs but I hope they eventually become default. If we always pre-condition then we almost always can do data blind quantization that is just as good as non-data-blind quantization.

Does this PR make any effort / at least javadocs to explain that you should ensure your incoming vectors are isotropic? Or to spot check if they really seem to be isotropic? luceneutil has all sorts of smell detection ("smelling pipeline" a recent genai model called it!) now to detect all sorts of problems your otherwise very-opaque-to-humans vectors might have.

mccullocht · 2026-08-13T16:26:33Z

I'll try to revive this over the next week or two.

This should work well when incoming vectors are isotropic (all dimensions behave the same i.e. the histograms of their per-dimension values approximate little baby gaussians with mean 0) and variance as required to be on unit sphere at that dimensionality.

IIRC the base quantizer should behave pretty well if the distribution is uncentered as well (this is what the lower/upper interval are for and why the dot product is unsigned), but if it's not ~Gaussian I would expect the error rate to be high.

RE: rotation -- I think we might want this to provide this but it should be layered just above Lucene. My concern is that if rotation is delegated to the segments during search costs will be very high -- I would expect O(1-2usecs) for a heavily optimized FWHT on good hardware, and you would have to multiply this cost by the number of segments. It'll look fine in a benchmark that's merged to one segment but poor in practice.

Does this PR make any effort / at least javadocs to explain that you should ensure your incoming vectors are isotropic? Or to spot check if they really seem to be isotropic? luceneutil has all sorts of smell detection ("smelling pipeline" a recent genai model called it!) now to detect all sorts of problems your otherwise very-opaque-to-humans vectors might have.

This is not documented, but is generally a constraint for most quantizers. It would be easy to document but I'm not sure what we would do if we detected your vectors don't quantize well. If we had an auto-quantization setting of some kind at that point you would just fall back to float32 or float16.

In general I'm annoyed by the amount of code I have to copy here, especially since all of the flat formats on the read side are just a fixed stride index read. Maybe the layer of abstraction should be closer to the vector layer (float[] -> byte[]) which would also be usable for unquantized float/byte inputs.

github-actions · 2026-08-28T03:05:58Z

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

mccullocht · 2026-09-01T05:01:39Z

Ok, I've revived this branch, renaming to Lucene106 and merging in main and all of the fixes that entails. PTAL.

kaivalnp

Thanks @mccullocht, this enables some great improvements in indexing vectors! (faster merges / smaller indexes)

I didn't get through the whole PR, but leaving some initial thoughts..

kaivalnp · 2026-09-01T15:18:06Z

+              - 2 * score;
+      // Ensure that 'score' (the squared euclidean distance) is non-negative. The computed value
+      // may be negative as a result of quantization loss.
+      return 1 / (1f + Math.max(score, 0f));


Can we use existing utility functions for this? Same for dot product.

This file has been deleted.

kaivalnp · 2026-09-01T15:27:03Z

+
+  @Override
+  public FlatFieldVectorsWriter<?> addField(FieldInfo fieldInfo) throws IOException {
+    if (fieldInfo.getVectorEncoding().equals(VectorEncoding.FLOAT32)) {


#16473 recently added scalar-quantization support for FP16 vectors, should we include that too?

Moving everything back into Lucene104 means we get this feature "for free"

kaivalnp · 2026-09-01T15:37:19Z

+ *   <li><b>int</b> the field number
+ *   <li><b>int</b> the vector encoding ordinal
+ *   <li><b>int</b> the vector similarity ordinal
+ *   <li><b>vint</b> the vector dimensions
+ *   <li><b>vlong</b> the offset to the vector data in the .veq file
+ *   <li><b>vlong</b> the length of the vector data in the .veq file
+ *   <li><b>vint</b> the number of vectors
+ *   <li><b>vint</b> the wire number for ScalarEncoding
+ *   <li><b>[float]</b> the centroid
+ *   <li><b>float</b> the centroid square magnitude
+ *   <li>The sparse vector information, if required, mapping vector ordinal to doc ID


This appears to be the same as the Lucene104 codec?

Is there no change to the on-disk representation? If so, could / should we add the data-blind option to the Lucene104 class instead of creating a new format?

This is largely the same. There's no guidance on this and I probably could merge it back into the Lucene104 codec as a version bump. I have mixed feelings about new version vs codec internal version bump because it's less obvious with the codec internal version bump that I am breaking you on upgrade.

Pushed everything back into the Lucene104 codec. Disabling centering results in an internal VERSION bump that omits the centroid and derived data and replaces it with zeros if needed. Writers with centering enable (default) won't see anything different at all, writers with centering disabled will experience the usually one way door.

kaivalnp · 2026-09-01T15:39:18Z

+ */
+
+/**
+ * Lucene 10.5 scalar quantized vector format, extending 10.4 with data-blind mode ({@code


10.5 -> 10.6

kaivalnp · 2026-09-01T15:44:57Z

+        fi.isDataBlind() ? null : rawVectorsReader.getFloatVectorValues(field);
+
+    if (rawFloatVectorValues == null) {
+      // Data-blind mode: full-precision float vectors were never stored. Reconstruct floats from


[DISCUSS] I think Lucene should disallow operations on data it dropped (i.e. throw an error when trying to retrieve floats / rescore using floats that were not stored in the index, instead of using a lossy value).

Does Lucene have other use-cases that support indexing some data (and specific operations to search it), but not retain / support returning the original data?

One example: if a user wrote some segments using enableCentering = false, then re-opens the index and attempts to merge with enableCentering = true, the dequantized vectors would be used instead of the original ones, leading to further loss of information?

There might be some use for this in some very high dimensional data sets -- I may want to quantize the original vectors at a high bit rate (4 or 8) bits and discard them, and also provide 1 or 2 bit quantization with centering that are used for graph navigation.

In your hypothetical situation yes you would absolutely lose precision, how much depends on how aggressive the initial quantization is.

It's not part of the automated indexing flow but IIUC there is already support for truncating the float vectors that works in a comparable (but not identical) way -- if the float vector files mysteriously end up empty then we reconstruct the values from the quantized data. At 4 or 8 bits this is almost always pretty reasonable.

kaivalnp · 2026-09-01T15:48:14Z

+      if (centroid == null) return false;
+      for (float v : centroid) {
+        if (v != 0f) return false;
+      }


Q: what if the centroid happens to be zero, even if enableCentering = true was used?

Would this cause the returned FloatVectorValues to be built from dequantized bytes, even when the full floats were available?

Should enableCentering be persisted in the index, instead of inferring from the centroid?

If the center is already a zero vector that is geometrically equivalent to our data blind configuration. Centering wouldn't subtract anything from the vector, and for angular similarity center_dot would be zero.

mccullocht

Hey @kaivalnp thanks for the review, I know this is very large. I will address the rest of your comments over the next couple of days.

mccullocht · 2026-09-02T03:23:35Z

+ *   <li><b>int</b> the field number
+ *   <li><b>int</b> the vector encoding ordinal
+ *   <li><b>int</b> the vector similarity ordinal
+ *   <li><b>vint</b> the vector dimensions
+ *   <li><b>vlong</b> the offset to the vector data in the .veq file
+ *   <li><b>vlong</b> the length of the vector data in the .veq file
+ *   <li><b>vint</b> the number of vectors
+ *   <li><b>vint</b> the wire number for ScalarEncoding
+ *   <li><b>[float]</b> the centroid
+ *   <li><b>float</b> the centroid square magnitude
+ *   <li>The sparse vector information, if required, mapping vector ordinal to doc ID


This is largely the same. There's no guidance on this and I probably could merge it back into the Lucene104 codec as a version bump. I have mixed feelings about new version vs codec internal version bump because it's less obvious with the codec internal version bump that I am breaking you on upgrade.

mccullocht · 2026-09-02T03:30:44Z

+        fi.isDataBlind() ? null : rawVectorsReader.getFloatVectorValues(field);
+
+    if (rawFloatVectorValues == null) {
+      // Data-blind mode: full-precision float vectors were never stored. Reconstruct floats from


There might be some use for this in some very high dimensional data sets -- I may want to quantize the original vectors at a high bit rate (4 or 8) bits and discard them, and also provide 1 or 2 bit quantization with centering that are used for graph navigation.

In your hypothetical situation yes you would absolutely lose precision, how much depends on how aggressive the initial quantization is.

mccullocht · 2026-09-02T03:31:53Z

+      if (centroid == null) return false;
+      for (float v : centroid) {
+        if (v != 0f) return false;
+      }


If the center is already a zero vector that is geometrically equivalent to our data blind configuration. Centering wouldn't subtract anything from the vector, and for angular similarity center_dot would be zero.

mccullocht added 7 commits May 2, 2026 21:51

audit of format + reader

af00705

fix check

fc84a81

fix field writer to better match lucene99 raw writer

f37bdab

move to lucene105

818fe85

change wire format to omit center on uncentered data sets

153c28c

propagate parameter to hnsw codec

27c5861

mccullocht added this to the 10.5.0 milestone May 4, 2026

github-actions Bot added the module:core/codecs label May 4, 2026

ch-ch-ch-changes

b9c9cc6

github-actions Bot added the Stale label May 19, 2026

romseygeek modified the milestones: 10.5.0, 10.6.0 Jun 26, 2026

github-actions Bot removed the Stale label Jun 27, 2026

github-actions Bot added the Stale label Jul 12, 2026

github-actions Bot removed the Stale label Aug 14, 2026

Pranshu-S mentioned this pull request Aug 18, 2026

Data-blind scalar quantization #16029

Open

3 tasks

github-actions Bot added the Stale label Aug 28, 2026

mccullocht added 3 commits August 31, 2026 20:42

Merge remote-tracking branch 'origin/main' into sq-data-blind

1dce44d

Lucene106

bfb30ff

backported test fixes

6dcfb50

kaivalnp reviewed Sep 1, 2026

View reviewed changes

mccullocht commented Sep 2, 2026

View reviewed changes

github-actions Bot removed the Stale label Sep 3, 2026

mccullocht added 4 commits September 8, 2026 11:18

merge back in to Lucene104SQ

b2b5655

fix test

566da19

improve class comment for format

0d55dad

more incremental approach to conversions

6ab6b9f

mccullocht requested a review from kaivalnp September 8, 2026 21:39

Uh oh!

Conversation

mccullocht commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 19, 2026

Uh oh!

github-actions Bot commented Jul 12, 2026

Uh oh!

kaivalnp commented Aug 12, 2026

Uh oh!

mikemccand commented Aug 13, 2026

Uh oh!

mccullocht commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 28, 2026

Uh oh!

mccullocht commented Sep 1, 2026

Uh oh!

kaivalnp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mccullocht left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mccullocht commented May 4, 2026 •

edited

Loading

mccullocht commented Aug 13, 2026 •

edited

Loading