Skip to content

fix(template): support MiMo audio inputs - #9891

Open
RerankerGuo wants to merge 1 commit into
modelscope:mainfrom
RerankerGuo:fix/mimo-audio-input
Open

fix(template): support MiMo audio inputs#9891
RerankerGuo wants to merge 1 commit into
modelscope:mainfrom
RerankerGuo:fix/mimo-audio-input

Conversation

@RerankerGuo

Copy link
Copy Markdown
Contributor

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

MiMo-V2.5 is registered as an audio-capable model, and its official chat
template represents audio with:

<|mimo_audio_start|><|audio_pad|><|mimo_audio_end|>

However, MiMoV2Template.replace_tag() accepted only image and video tags.
Any SWIFT request containing <audio> therefore failed before reaching the
inference backend. In environments without qwen_vl_utils, the audio path
also attempted an unrelated vision import first.

This change:

  • renders the official MiMo audio placeholder for vLLM and SGLang;
  • leaves the raw audio input unchanged for backend multimodal processing;
  • protects <|audio_pad|> when prompt truncation is enabled;
  • reports a clear error for Transformers and LMDeploy, which are outside the
    MiMo support scope introduced by support Xiaomi-MiMo-V2.5 inference( sglang / vllm ) #9880;
  • keeps existing image and video processing unchanged.

Experiment results

Before:

.venv/bin/python -m unittest tests.general.test_mimo_template
ModuleNotFoundError: No module named 'qwen_vl_utils'
AssertionError: '<|audio_pad|>' not found in [...]
FAILED (failures=1, errors=4)

After:

.venv/bin/python tests/run.py \
  --test_dir tests/general \
  --pattern test_mimo_template.py
SUCCESS (Runs=3, success=3)

.venv/bin/python -m unittest \
  tests.general.test_optional_template_dependencies \
  tests.general.test_model
Ran 2 tests
OK

.venv/bin/pre-commit run --all-files
All hooks passed

Render the official MiMo audio placeholder sequence for vLLM and SGLang while preserving raw audio for backend processing. Reject unsupported inference backends explicitly and protect the audio placeholder during truncation.

Test: python tests/run.py --test_dir tests/general --pattern test_mimo_template.py
@RerankerGuo

RerankerGuo commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

CI note: all three MiMo audio regression tests pass in the CUDA run:

  • test_audio_placeholder_for_supported_backends
  • test_audio_placeholder_is_protected_from_truncation
  • test_audio_rejects_unsupported_backends

The run's only error is the repository-wide
ModuleNotFoundError: transformers.models.gemma3 in
test_gemma3_template.py. The same shared-environment dependency mutation was
traced on #9862:
#9862 (comment).

Lint and the NPU unittest pass.

@RerankerGuo

Copy link
Copy Markdown
Contributor Author

@z0o0ey, could you take a look when convenient? This is a focused follow-up
to the MiMo-V2.5 vLLM/SGLang support in #9880.

The change only adds the official MiMo audio placeholder sequence, preserves
raw audio for backend processing, and rejects backends outside #9880's support
scope. Existing image/video paths are unchanged. All three targeted CUDA tests,
lint, and the full NPU suite pass; the unrelated CUDA Gemma3 environment
failure is documented above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant