fix: gate multimodal preprocessing concurrency

[CUHKSZzxy]

Summary

Add an optional multimodal preprocessing concurrency gate for serving requests.

This limits how many multimodal requests can be in the media preprocessing / API-to-engine handoff stage at the same time. It is intended to reduce CPU RSS spikes from very large pixel_values tensors under high request concurrency, where max_batch_size does not help because the pressure happens before engine batching.

Main changes:

• add --max-mm-preprocess-concurrency to VisionConfig / serve CLI
• add a MultimodalPreprocessGate and lease helper for request-scoped slot ownership
• acquire the gate only for requests that contain multimodal input
• for PyTorch MP engine, release the slot after the request payload is accepted by the engine process, rather than waiting for decode completion
• pass the gate through OpenAI chat/completions, Responses API, and Anthropic messages serving paths
• reuse the existing multimodal input detector in the multimodal processor
• add focused tests for gate behavior and MP callback propagation

Benchmark

Single-run image payload pressure test, PyTorch backend, TP=2, MP engine, client concurrency=128, total requests=128, max_tokens=1.

Variant	Gate	Throughput req/s	Delta vs main	Peak CPU RSS	RSS reduction vs main	Success
main	off	0.213	0.00%	222.5 GiB	0.00%	128/128
gate=1	1	0.211	-1.21%	68.1 GiB	69.37%	128/128
gate=2	2	0.221	+3.58%	158.3 GiB	28.84%	128/128
gate=4	4	0.222	+3.98%	179.2 GiB	19.43%	128/128
gate=8	8	0.217	+1.61%	191.1 GiB	14.11%	128/128

Takeaway: gate=1 gave the strongest CPU memory reduction with only noise-level throughput change for this workload. gate=2 is a more balanced setting if the machine can tolerate higher CPU RSS.

Notes:

• RSS is summed process-tree CPU RSS, so shared mappings may be overcounted.
• Throughput deltas under roughly 3-5% should be treated as noise without repeated runs.

Tests

• Focused multimodal preprocess gate tests
• Anthropic messages direct gate propagation test
• multimodal input detector coverage
• ZMQ RPC callback propagation test
• diff whitespace check

Assistance

Assisted with Codex + GPT-5.5 xHigh Fast, reviewed manually

[Copilot]

Pull request overview

This PR introduces an optional concurrency gate to limit how many multimodal requests can be in the media preprocessing / API→engine handoff stage concurrently, aiming to reduce CPU RSS spikes from large pixel_values under high request concurrency.

Changes:

• Add VisionConfig.max_mm_preprocess_concurrency and a serve CLI flag --max-mm-preprocess-concurrency.
• Introduce MultimodalPreprocessGate/lease and acquire it only when multimodal input is detected; release timing differs for PyTorch MP vs non-PyTorch.
• Propagate the gate through OpenAI (chat/generate/responses) and Anthropic messages paths, plus add focused tests (gate behavior + MP callback propagation).

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/test_lmdeploy/serve/test_mm_preprocess_gate.py	Adds unit tests for gate blocking/release and MP callback locality.
tests/test_lmdeploy/serve/anthropic/test_endpoints.py	Verifies Anthropic endpoint passes mm_preprocess_gate (including direct endpoint call).
tests/pytorch/engine/test_zmq_rpc.py	Updates MP streaming RPC signature to accept a local “request accepted” callback.
lmdeploy/serve/utils/mm_preprocess.py	New gate + lease implementation (asyncio semaphore-based).
lmdeploy/serve/processors/multimodal.py	Reuses a centralized multimodal-input detector and makes it more robust to input types.
lmdeploy/serve/openai/responses/serving.py	Passes the gate into the Responses API generation call.
lmdeploy/serve/openai/api_server.py	Creates/configures the gate from VisionConfig and forwards it into chat/generate.
lmdeploy/serve/core/async_engine.py	Acquires/releases the gate around prompt preprocessing + adds PyTorch MP handoff callback release.
lmdeploy/serve/anthropic/endpoints/messages.py	Forwards the gate into Anthropic message generation.
lmdeploy/pytorch/engine/mp_engine/zmq_rpc.py	Triggers the local “request accepted” callback once the stream is established.
lmdeploy/pytorch/engine/mp_engine/zmq_engine.py	Threads the local callback through to the RPC client streaming call.
lmdeploy/pytorch/engine/mp_engine/ray_engine.py	Triggers the local “request accepted” callback once the stream is established.
lmdeploy/pytorch/engine/mp_engine/base.py	Ensures the callback stays local (not pickled/sent to engine process).
lmdeploy/messages.py	Extends VisionConfig with max_mm_preprocess_concurrency.
lmdeploy/cli/utils.py	Adds the CLI argument helper for --max-mm-preprocess-concurrency.
lmdeploy/cli/serve.py	Wires the new CLI flag into VisionConfig creation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.