Cut idle memory of the embedding service

[TorstenDittmann]

Summary

The embedding container holds high RSS while idle. Appwrite self-hosted starts appwrite/embedding with EMBEDDING_MODELS=nomic-embed-text and does not set EMBEDDING_POOL_SIZE, so the process defaults to available_parallelism() copies of full-precision Nomic.

This issue tracks measuring idle RSS and cutting it without changing embedding quality (stay on NomicEmbedTextV15).

Constraints

• Keep POST /embed request/response shape
• Keep full-precision nomic (NomicEmbedTextV15, 768-d)
• No quantized / smaller-model default

Planned levers

1. Default EMBEDDING_POOL_SIZE to 1 (not CPU count)
2. Lazy-load models so /health does not pin weights
3. Idle-unload after EMBEDDING_IDLE_UNLOAD_SECS (reload same checkpoint from disk)
4. Cap ORT intra-op threads
5. jemalloc so dropped sessions return RSS to the OS
6. Cache nomic only in the image (not nomic,bge-small)

Measurements

Baseline vs each lever will be posted as comments.

[TorstenDittmann]

Baseline RSS (this host: 4 CPUs, ~16 GiB RAM)

Measured with the current master release binary. /health is only served after models are loaded, so health RSS == idle-with-model-resident.

Config	Effective pool	RSS after /health
nomic-embed-text, pool unset	capped to 1 (3× arena estimate vs ~6.7 GiB free)	868 MiB
nomic-embed-text, EMBEDDING_POOL_SIZE=1	1	885 MiB
minilm, pool unset	4 (nproc; estimate fit the budget)	435 MiB
minilm, EMBEDDING_POOL_SIZE=1	1	138 MiB
nomic,bge-small, pool=1	1+1	918 MiB

Notes:

• Appwrite compose does not set EMBEDDING_POOL_SIZE. On this box the 3× arena guard already forces nomic to pool=1, so idle is still ~870 MiB of one unused full-precision Nomic session.
• On a host with more free RAM the same unset default would keep desired=nproc copies. MiniLM shows the multiplier: 138 MiB → 435 MiB at pool=4.
• Unload-after-idle cannot be measured on master (no eviction). That row comes after the implementation.

Quality: all nomic runs are NomicEmbedTextV15, 768-d. No quantized checkpoint.

[TorstenDittmann]

After the changes (PR #8)

Same host (4 CPUs, ~16 GiB). Model is still full-precision NomicEmbedTextV15 (768-d); reload after unload confirmed the same model and dimension.

State	master	PR
/health (never embedded)	868–885 MiB	13 MiB
After one /embed	885 MiB	587 MiB
After idle unload (EMBEDDING_IDLE_UNLOAD_SECS=5 in the test; default is 300)	always resident	83 MiB

What moved the needle:

• Lazy-load: /health no longer constructs ONNX sessions.
• Default EMBEDDING_POOL_SIZE=1 (MiniLM on master was 138 MiB at pool=1 vs 435 MiB at nproc=4).
• Idle unload + jemalloc overriding libc malloc so ORT's C++ heap is actually returned.

Watch after deploy: first /embed after idle pays a ~1s reload from EMBEDDING_CACHE_DIR. Set EMBEDDING_IDLE_UNLOAD_SECS=0 to keep sessions hot.

[TorstenDittmann]

CI note: docker build failed because tikv-jemalloc-sys compiles jemalloc from source and the slim builder image had g++ but not make (failed to execute command: No such file or directory after configure). Added make to the Dockerfile builder apt list.

[TorstenDittmann]

Follow-up: concurrency while serving.

EMBEDDING_POOL_SIZE defaults to CPU count again (still RAM-capped at load), so parallel /embed calls are not stuck on one mutex. Idle unload still drops all sessions, and in-flight embeds block unload so a long request cannot reload a second full pool.

On this 4-CPU box nomic remains capped to pool=1 by the 3× arena estimate; MiniLM-class models get nproc sessions while hot. After 300s idle they all go away.