Indexing failed on a big codebase leaving the cache empty

[kafumanto]

Hello. I was trying semble v0.4.0 on a big C++ codebase (~135k files, ~2.5GB of source code), but indexing crashed (I think for memory exhaustion) and no index cache was written at all. It looks like cache is not built incrementally, so this makes impossible to use semble on such code base.

Command run to warm up the index:

semble search "test" /workspace --top-k 1

After 90 minutes it crashed with a simple "Killed" message (searching around it could be caused by an OOM of Python).
During the run, a few times the warning "Recursion depth exceeded in chunk." appeared (but looking at the source code, it seems an handled case).

The critical problem is that no partial index cache was written. No other output about the caching progress was available (so it's impossible to understand how much RAM was used and needed for the task to complete).

Ideally an incremental build of the cache should solve this problem, since it should be able to recover from a crash. But I think that this requires also another improvement to make everything work properly: to avoid loading in memory all the index to process a search request (otherwise the OOM error would return).

[stephantul]

Hey @kafumanto,

Thanks for reporting this issue. I'm gonna go on a limb here and state that Indexing codebases this large is out of scope for Semble. In my opinion, codebases this large need specialized tools (i.e., tools that do partial indexing, offload indexes to disk etc.). Semble is an almost ephemeral semantic search tool: we don't pay a lot of attention to saving indexes on disk, doing partial rebuilds, etc. We can afford to do this because it is so fast, and fast enough for almost all codebases.

I agree that we should be more transparent about what is happening under the hood though, this is something we'll add.

[kafumanto]

Hi! I completely understand.

What do you think about a simpler architectural approach instead: defining a small, clean API that allows external implementations to extend Semble with more advanced indexing and search logic, while keeping the current in‑memory implementation as the default?

In practice, this would mean introducing a lightweight interface that abstracts index construction and querying, along with a small refactor to adapt the existing code.

Here are a few initial ideas for methods this interface could expose:

• is_file_indexed(path, timestamp) — supports incremental index builds by allowing Semble to skip chunk generation when a file is already indexed and hasn’t changed.
• add_file_to_index(path, timestamp, chunks, embeddings, …) — stores the chunks for a given file, replacing any existing entry. If I’m not mistaken, this would integrate naturally in create_index_from_path() in src/semble/index/create.py.
• search(...) (and any other query-related methods) — performs lookups on the index.

This would likely require a refactor of SembleIndex in src/semble/index/index.py: part of the current logic would move into a new concrete class implementing the interface (preserving the current in‑memory behavior), and the file‑parsing flow would call is_file_indexed() / add_file_to_index() directly instead of collecting all chunks first and saving them in one batch later.

Of course, you know the Semble codebase far better than I do, so you probably have a clearer sense of what such an interface would need to support incremental indexing and more advanced search backends (e.g., using vector database), as well as the actual amount of work involved.