Skip to content

[SVS] Refactor SVS element size calculations and block size handling#980

Open
rfsaliev wants to merge 5 commits into
mainfrom
rfsaliev/svs-blocksize-elements
Open

[SVS] Refactor SVS element size calculations and block size handling#980
rfsaliev wants to merge 5 commits into
mainfrom
rfsaliev/svs-blocksize-elements

Conversation

@rfsaliev

@rfsaliev rfsaliev commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

Describe the changes in the pull request

This pull request refactors and improves the way element sizes and block sizes are calculated and used in the SVS storage traits, enhancing clarity and modularity in the code. The changes introduce clearer separation between primary, residual, and secondary element sizes, update the block size calculation to use a new structure, and simplify related test logic.

Which issues this PR fixes

This PR fixes index size estimation issues for SVSIndex by using new blocksize_elements parameter introduced in intel/ScalableVectorSearch#344.

TODO:

  • Update SVS shared library binary URLs

Main objects this PR modified

  1. SVSIndex
  2. SVS tests

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

Note

Medium Risk
Changes SVS blocked allocation and memory estimation for quantized indexes and bumps the fetched shared library; incorrect blocking could affect memory use or stability, but scope is limited to SVS build/runtime paths.

Overview
Aligns SVS integration with ScalableVectorSearch v0.3.1 (prebuilt tarball URLs in cmake/svs.cmake, including a simpler Clang path that only selects the gcc11 binary when libstdc++ ≥ 11).

LVQ / LeanVec storage now splits byte layout into primary_element_size, residual_element_size, and secondary_element_size, and uses the max of the relevant parts when sizing blocks. SVSBlockSize returns svs::data::BlockingParameters (power-of-two byte block plus blocksize_elements from the VecSim block count), matching upstream blocking behavior and fixing index size estimation.

Tests drop LVQ-specific dimension workarounds in testSizeEstimation (tighter 1% tolerance) and set an explicit blockSize in tiered runGCAPI so GC exercises at least two vector blocks.

Reviewed by Cursor Bugbot for commit fcc801d. Bugbot is set up for automated code reviews on this repo. Configure here.

@rfsaliev rfsaliev requested a review from meiravgri June 17, 2026 09:55
@jit-ci

jit-ci Bot commented Jun 17, 2026

Copy link
Copy Markdown

🛡️ Jit Security Scan Results

CRITICAL HIGH MEDIUM

✅ No security findings were detected in this PR


Security scan by Jit

Comment thread cmake/svs.cmake

cmake_dependent_option(SVS_SHARED_LIB "Use SVS pre-compiled shared library" ON "USE_SVS AND GLIBC_FOUND AND SVS_LVQ_SUPPORTED" OFF)
if (CMAKE_CXX_COMPILER_ID MATCHES "Clang")
if(GLIBC_VERSION VERSION_GREATER_EQUAL "2.31" AND CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL "21.0")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if(GLIBCXX_VERSION VERSION_GREATER_EQUAL "11")
set(SVS_URL "https://github.com/intel/ScalableVectorSearch/releases/download/nightly/svs-shared-library-reduced-clang21-gcc11-2026-06-17-1412.tar.gz" CACHE STRING "SVS URL")

@rfsaliev rfsaliev marked this pull request as ready for review June 18, 2026 11:55
@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.10%. Comparing base (6ca08ae) to head (92c2189).

⚠️ Current head 92c2189 differs from pull request most recent head fcc801d

Please upload reports for the commit fcc801d to get more accurate results.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #980   +/-   ##
=======================================
  Coverage   97.10%   97.10%           
=======================================
  Files         141      141           
  Lines        8179     8179           
=======================================
  Hits         7942     7942           
  Misses        237      237           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rfsaliev rfsaliev requested review from alonre24 and removed request for meiravgri June 25, 2026 16:00
alonre24
alonre24 previously approved these changes Jul 1, 2026

@alonre24 alonre24 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - did we confirm that there is no regression in terms of memory consumption in benchmarks?

Comment thread cmake/svs.cmake Outdated
if(GLIBC_VERSION VERSION_GREATER_EQUAL "2.28")
if(CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL "14.0")
set(SVS_URL "https://github.com/intel/ScalableVectorSearch/releases/download/v0.3.0/svs-shared-library-0.3.0-reduced-gcc14.tar.gz" CACHE STRING "SVS URL")
set(SVS_URL "https://github.com/intel/ScalableVectorSearch/releases/download/nightly/svs-shared-library-reduced-gcc14-2026-06-18-1414.tar.gz" CACHE STRING "SVS URL")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to merge it as it is (using the nightly version) or bump to an official svs release?

@rfsaliev rfsaliev force-pushed the rfsaliev/svs-blocksize-elements branch from 465995d to fcc801d Compare July 10, 2026 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants