ci: Update transformers to latest version 5.15.1 - #3628
Open
svcnvidia-nemo-ci wants to merge 85 commits into
Open
ci: Update transformers to latest version 5.15.1#3628svcnvidia-nemo-ci wants to merge 85 commits into
svcnvidia-nemo-ci wants to merge 85 commits into
Conversation
…r0.6.0` (#3304) fix(docs): hide Kimi tokenizer regex from autodoc (#3288) * fix(docs): hide Kimi tokenizer regex from autodoc * test(models): cover Kimi tokenizer pattern equivalence --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
…` into `r0.6.0` (#3306) docs(distributed): review frozen multimodal FSDP guidance (#3272) * docs(distributed): mark frozen multimodal sections for review * docs(distributed): apply technical publications review --------- Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Yuhe Zhang <yuhez@nvidia.com>
…3) (3273)` into `r0.6.0` (#3307) fix(vlm): fused linear CE in gemma4 31B FFPA 8k recipe (AMINT-203) (#3273) fix(vlm): expandable segments for gemma4 31B FFPA 8k recipe (AMINT-203) gemma4_31b_ffpa_mock_8k OOM'd in the release sweep during backward, on an 8.00 GiB allocation with 7.65 GiB free and 10.56 GiB reserved but unallocated. Backward asks for two large contiguous buffers on an 80 GiB card. The 8.00 GiB one is the fp32 logits gradient: Gemma-4-31B has vocab_size 262144 and the recipe runs local_batch_size 1 at seqlen 8192, so the logits are [1, 8192, 262144] and 8192 * 262144 * 4 = 8.00 GiB exactly. The other is a 10.50 GiB FSDP root post-backward reduce-scatter. Peak demand is ~66 GiB of 79.11 GiB, so both do fit. The failure is that the default allocator strands enough memory in reserved-but-unallocated blocks that one of the two contiguous requests cannot be served. That also explains why the recipe passed for a while and then started failing: it was never OOM-proof, only narrowly fitting whenever the allocator happened to stay unfragmented. Measured on 8xH100-80GB in the CI container, at the recipe's real CI shape (8k, MAX_STEPS=10): 10/10 steps at 65.67 GiB peak allocated with expandable_segments set, including the validation pass, exit 0. env_vars is the existing mechanism -- generate_ci_tests forwards it verbatim into the job variables, and 12 other recipes already set this key. The two gemma4 eagle3 recipes also document needing it for a 31B-class target with the 262k-vocab embedding on a single 80 GB GPU. Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…6.0` (#3305) fix(pp): use static metadata with PyTorch 2.13 (#3290) * fix(pp): use static metadata with PyTorch 2.13 * test(pp): document tensor metadata helper * ci(glm): extend pipeline test timeout --------- Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Yuhe Zhang <yuhez@nvidia.com>
) ci: add Tulu-3 convergence + eval flow (#3171) * ci: add Tulu-3 weekly convergence eval flow (moonlight, qwen3) Weekly convergence flow that trains a recipe to completion (1000 steps) then gates on a downstream IFEval score staying within k*stderr of a recorded baseline. - convergence_tests_launcher.sh: train -> setup_lm_eval.sh -> eval -> gate - convergence_eval.py: runs run_eval.sh on the consolidated checkpoint; passes iff |prompt_level_strict_acc - baseline| < k*stderr - ci.downstream_eval blocks on the moonlight and qwen3 recipes carry the per-model eval args + baseline/stderr (k=2) - configs/convergence/: recipe list + overrides for the convergence scope - generate_ci_tests.py: convergence stage gemma4-31b recipe + README land in a follow-up commit. * ci: add gemma4-31b to Tulu-3 convergence flow (recipe + README) The recipe referenced by convergence_recipes.yml plus its README. - gemma4_31b_tulu3_packed2k_cp1_gbs32_4node.yaml: the validated convergence recipe (2k packing, cp1/dp32 GBS=32, TE FusedAdam fp32-master lr 5e-6 cosine warmup 100 wd 0), with the ci.downstream_eval baseline (allow_failure until the CI env stages the base chat template). - README.md: arch, HP rationale, data/train/eval steps, loss + IFEval results. Multi-node; launch with the SLURM .sub referenced in the README. * ci(convergence): pin generated child-pipeline template include to commit SHA A dynamic child pipeline built from a generated artifact resolves a bare local `include:` against the project's DEFAULT branch, so an unmerged nemo-ci template change (a new `convergence` stage + `.convergence_test`) is invisible to the generated pipeline and downstream creation fails. Resolve the template at $CI_PROJECT_PATH@$CI_COMMIT_SHA so the child uses the same template revision as the parent pipeline: reproducible on main, testable on a branch pre-merge. * ci(convergence): resolve template include ref at generation time The previous attempt used include:project/ref with $CI_* variables, but `include:` variable expansion is unavailable for dynamic downstream (artifact) child pipelines -- GitLab expanded them to "" and rejected the include. Resolve CI_PROJECT_PATH/CI_COMMIT_SHA from the generator job's environment (real env vars there) and bake the literal project+ref into the generated pipeline, so the child fetches the same template revision as the parent. Falls back to a bare local include if the env vars are absent. * ci(convergence): disable wandb in the CI launcher The convergence recipes log to wandb.entity=nvidia, but the CI token (WANDB_AUTOMODEL_API_KEY) has no models-write access to that org, so wandb.init() raises CommError on rank0 in recipe.setup(). wandb.init runs on rank0 only, so the other ranks proceed to the checkpoint-consolidation gloo barrier and wait for the now-dead rank0 until the 30-min c10d store timeout (DistStoreError). Set WANDB_MODE=disabled in the launcher; the gate is downstream IFEval, and the recipe's wandb config is untouched for manual runs. * Revert "ci(convergence): resolve template include ref at generation time" Revert the generation-time include resolution (cf80a1e / 06ac139) back to a bare local include. The convergence flow will rely on the nemo-ci template changes (convergence stage + .convergence_test) landing on nemo-ci main first, after which a bare local include resolves correctly for scheduled pipelines. * ci(convergence): log convergence runs to wandb (Nemo-automodel) Add a wandb block to the moonlight/qwen3-moe/gemma4 convergence recipes with entity Nemo-automodel (the org the CI wandb key can write to) and project automodel_convergence_runs. Recipes keep `enable: false` (required by the example-yaml linter); the launcher flips it on for CI via a `--wandb.enable true` CLI override instead of WANDB_MODE, so the runs log while direct/manual runs stay off unless opted in. * Add gemma4 curves to readme * ci(convergence): prefilter tulu3 for LLM recipes; wire gemma4 chat template The LLM convergence recipes (moonlight/qwen) train on raw allenai/tulu-3-sft-mixture with truncation: false; over-length samples spike memory on the large-vocab MoEs and OOM in CI. Add convergence_prefilter.py (reuse an existing seq2048 cache, else build one) and call it from the launcher for LLM recipes so training runs on filtered data, matching the measured baselines. gemma4 (VLM, packed with drop_long_samples) is skipped. Also supply gemma4's chat template via dataset.chat_template (the VLM loader applies it to the processor) so the base -it template no longer needs manual staging onto the model dir; refresh the now-stale yaml/README comments. gemma4 stays allow_failure until a green run verifies train + eval + gate. * fix(moe): AC-safe expert-load accumulation; disable wandb in convergence CI Two fixes for the Tulu-3 convergence CI: 1) moe/layers.py: make the aux-loss-free bias-update accumulation out-of-place (`self._cumulative_expert_load = self._cumulative_expert_load + ...`). Under activation checkpointing the gate forward is recomputed in backward, and the in-place `+=` on non-checkpointed module state trips torch's guard ("aten.add_.Tensor ... not found in storage"). The update uses sign(avg-load), so the recompute double-count is a no-op. Reproduced + verified fixed in the CI container (moonlight/qwen train past it). 2) convergence_tests_launcher.sh: stop enabling wandb. The CI token has no write access to the recipes' wandb entity, so wandb.init() raised CommError on rank0 and strands the other ranks at the consolidation gloo barrier -- killing all three recipes before training. The gate is downstream IFEval, not wandb. * ci(convergence): fetch IFEval online during eval; make gemma4 a blocking gate The convergence eval failed for all recipes on `ConnectionError: Couldn't reach 'google/IFEval' on the Hub (OfflineModeIsEnabled)` -- the CI HF cache does not pre-warm the IFEval dataset and the env defaults to offline. Export HF_HUB_OFFLINE=0 / HF_DATASETS_OFFLINE=0 in the eval step so lm-eval can fetch it (training already ran; only the eval step needs Hub access). Also drop gemma4's ci.allow_failure now that its chat template is recipe-wired, so it gates like moonlight/qwen; refresh the yaml/README references. * ci(convergence): gate on results file and tear down eval to avoid timeout lm-eval + vLLM (tp>1) hangs on shutdown *after* writing results_*.json, so an otherwise-passing eval ran the SLURM job to its wall-clock limit and failed on timeout (moonlight/qwen produced in-band IFEval scores but the jobs were killed DUE TO TIME LIMIT). convergence_eval.py now launches run_eval.sh in its own session, polls for the results file carrying the metric, and SIGTERM/SIGKILLs the whole eval process tree as soon as the score is written -- then gates: within k*stderr of baseline -> exit 0 (PASS, green CI); otherwise exit 1 with "eval score out of threshold". A CONVERGENCE_EVAL_TIMEOUT_S guard fails the run if no valid results appear. * ci(convergence): re-enable wandb logging for CI runs Flip wandb back on in the launcher via `--wandb.enable true` so convergence runs log to Nemo-automodel/automodel_convergence_runs. Requires WANDB_AUTOMODEL_API_KEY to have write access to that entity; if it does not, wandb.init() will error on rank0 (previously why it was disabled). * ci(convergence): floor eval vllm/cutlass-dsl for gemma4 FA4 kernel gemma4's head_dim makes vLLM auto-select the FA4 cute kernel (vllm_flash_attn.cute), which fails to compile on the cutlass-dsl 4.5.x that vllm 0.25.x resolves transitively (GPUModuleOp / cudaErrorIllegalAddress). Floor the eval install to vllm>=0.26.0 and nvidia-cutlass-dsl>=4.6.0, which compiles the kernel cleanly (verified: gemma4 IFEval prompt_level_strict_acc=0.5360), so the weekly convergence run stays on a known-good combination. * ci(convergence): disable wandb to unblock jobs (key lacks org write access) Enabling wandb (--wandb.enable true) failed every convergence job: the WANDB_AUTOMODEL_API_KEY account has no write access to the recipes' Nemo-automodel entity, so wandb.init() raises "user does not have models write access for this org" on rank0, which then strands the remaining ranks at the checkpoint-consolidation gloo barrier until the 1800s timeout. Drop the override so training runs; re-enable once the key is granted access. * ci(convergence): widen gemma4 eval gate to k=3 for SFT run variance gemma4 tulu3 SFT shows larger run-to-run spread (0.4713..0.5360) than the eval-sampling band (2*stderr=+/-0.043) can hold, so a converged run fails the gate on seed variance. Bump k to 3 (+/-0.0645). Eval validated end-to-end in the CI container (base 0.231 -> SFT ~0.50 -> gemma-4-31B-it 0.906), so this reflects training variance, not an eval artifact. moonlight/qwen keep k=2. * ci(convergence): re-enable wandb after key refresh WANDB_AUTOMODEL_API_KEY was refreshed; re-add --wandb.enable true to the training command to log convergence runs to Nemo-automodel/automodel_convergence_runs. Reverts the temporary disable (4b926a2). If the refreshed key still lacks org write access, wandb.init() will CommError on rank0 and strand ranks at the gloo barrier -- disable again in that case. --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: athitten <abhishreetm@gmail.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ecision (3255)` into `r0.6.0` (#3310) fix(distributed): preserve ERNIE router and dense Qwen3.5 SSM precision (#3255) * fix(moe): preserve ERNIE router precision * fix(moe): preserve complete ERNIE FP32 contracts * docs(distributed): clarify FP32 FSDP input contract * test(ernie): restore default HF KL threshold * test(ernie): verify router parity after FSDP * test(moe): exercise ERNIE router through FSDP root --------- Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Yuhe Zhang <yuhez@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
…3314) perf(moe): add scoped partial CUDA graphs (#2917) * perf(moe): add scoped partial CUDA graphs Capture only parameterless fixed-shape attention, router, and HybridEP preprocessing boundaries while leaving dispatch, expert compute, and combine eager. Based on the scoped partial graph implementation by Hemil Desai. * refactor(cuda-graphs): simplify partial graph state * feat(cuda-graphs): capture whole attention * perf(cuda-graphs): benchmark partial graph replay * test(moe): align cuda graph rope config expectation * fix(cuda-graphs): make partial capture transactional * refactor(cuda-graphs): move partial graphs into component * refactor(recipes): gate partial cuda graph lifecycle * refactor(config): group CUDA graph options * refactor(recipes): use CUDA graph API directly * refactor(recipes): close CUDA graphs after training loop * fix(cuda-graphs): restore CI compatibility --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Yu Yao <153118171+akoumpa@users.noreply.github.com> Co-authored-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
…3323) fix(config): repair GLM-5.2 LoRA recipe (#3313) * fix(config): remove unsupported GLM experts storage * fix(config): preserve environment strings when copied * fix(config): align GLM batch size with topology * fix(config): use GLM collater callable * fix(config): cap GLM LoRA sequences at 4K * fix(config): align GLM LoRA memory contract * ci(config): bound GLM release coverage * fix(config): match __reduce_ex__ signature * chore(config): update GLM-5.2 recipe owner * chore(ci): increase GLM-5.2 timeout --------- Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Yuhe Zhang <yuhez@nvidia.com>
…3334) fix(kd): use mesh-safe gradient clipping (#3302) * fix(kd): use mesh-safe gradient clipping * fix(kd): pass expert TP replication factor to grad clipping The KD optimizer steps called scale_grads_and_clip_grad_norm without expert_tp_replication_factor, so the factor defaulted to 1. For a custom-MoE model with tp_size > 1 and _nemo_moe_tp_requires_replica_sync set, the token path is replicated across TP ranks and each expert gradient is accumulated tp_size times, leaving expert grads over-counted relative to train_ft.py. Both the non-PP and PP KD steps now pass get_expert_tp_replication_factor(self.model_parts, self.device_mesh), matching the base finetune recipe. --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
…el stages (3329)` into `r0.6.0` (#3343) refactor(docker): build torchao & FlashAttention as isolated wheel stages (#3329) * chore(docker): trim Dockerfile comment blocks to ≤2 lines * refactor(docker): build torchao/FA wheels in isolated stages * fix: move system cve resolution * fix(docker): bound wheel-builder compile parallelism to avoid OOM * fix(docker): serialize torchao/FA compiles in one wheel_builder stage --------- Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…r0.6.0` (#3338) fix(training): prewarm Mamba SSD autotune kernels (#3296) * fix(training): prewarm Mamba SSD autotune kernels * test: update VLM prewarm setup mock --------- Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Yuhe Zhang <yuhez@nvidia.com>
…into `r0.6.0` (#3361) fix(pp): preserve VLM media cursor with static metadata (#3344) * fix(pp): preserve VLM media cursor with static metadata * fix(ci): read static PP metadata in checkpoint parity * fix(checkpoint): ignore base FP8 metadata on resume * ci(vlm): extend Mistral4 checkpoint timeout --------- Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Yuhe Zhang <yuhez@nvidia.com>
#3363) fix(deps): resolve 26.08 rc2 container CVEs (#3346) * fix(deps): resolve 26.08 rc2 container CVEs * fix(ci): isolate uv installation test * fix(ci): seed pip in uv installation test * test(mlflow): use SQLite tracking backend * docs(mlflow): document SQLite tracking backend --------- Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: OpenAI Codex <codex@openai.com>
…6.0` (#3364) fix(pp): route Qwen3.5 MoE pre-embedded inputs (#3294) * fix(pp): route Qwen3.5 MoE pre-embedded inputs * fix(qwen3.5): declare causal LM tie support Register the text-only Qwen3.5 MoE model under the existing family docs coverage and enforce its untied embedding policy with a CPU regression test. * test(qwen3.5): cover text-only adapter routing Document the causal LM pipeline tensor contract and verify that text-only expert keys round-trip without a language-model prefix. * fix(qwen3.5): avoid duplicate text root forwards Keep multimodal pre-embedding on the separately sharded embedding module. This avoids invoking the text FSDP root twice per pipeline microbatch and removes the private _pre_embed_only forward control path. * test(qwen3.5): execute text-only causal LM on CPU Cover construction, weight initialization, and an unfused forward/backward for a tiny text-only Qwen3.5-MoE model so the production happy path remains exercised by CPU coverage jobs. --------- Signed-off-by: huiyingl <huiyingl@nvidia.com> Signed-off-by: HuiyingLi <willwin.lee@gmail.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Huiying <willwin.lee@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…(3295)` into `r0.6.0` (#3362) fix(kimi_k25_vl): don't int4 quantize LoRA adapter keys on save (#3295) _is_quantized_expert_key tested ".weight" as a substring, so expert LoRA tensors such as mlp.experts.0.gate_proj.lora_A.weight matched alongside the real expert weights. A PEFT save therefore wrote the adapter out as weight_packed/weight_scale/weight_shape triplets, leaving no lora_A/lora_B keys for PEFT to load, and quantize_to_int4 asserts outright when the LoRA rank is not divisible by 8 (examples/vlm_benchmark/kimi/kimi25vl_lora.yaml yields rank 4). Skip keys containing ".lora_" and require the key to end with ".weight". The endswith check also stops already-emitted ".weight_packed"/".weight_scale" keys from matching a second time. Expert weights are still quantized exactly as before, which is intended since the Kimi HF checkpoint is quantized. Fixes #3282 Signed-off-by: stanley1208 <stanley.mei08@gmail.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: stanley1208 <54892792+stanley1208@users.noreply.github.com>
…ls (3339)` into `r0.6.0` (#3366) fix(diffusion): use spawn start method for GPU preprocessing pools (#3339) * fix(diffusion): use spawn start method for GPU preprocessing pools The image and video preprocessing pools in tools/diffusion/preprocessing_multiprocess.py used the default fork start method. Since #3217 added the qwen_image_edit processor to the tools.diffusion.processors package import chain, the parent process ends up with an initialized CUDA context before the pools fork, and every CUDA-using worker dies with 'Cannot re-initialize CUDA in forked subprocess'. In the flux/qwen-image/hunyuan jobs the same failure surfaces masked by the transformers/diffusers lazy-import wrappers as "Could not import module '<class>'" errors. Create both pools from an explicit spawn context. The shard workers already initialize their own state from picklable arguments and main() is __main__-guarded, so spawn is a drop-in change that makes the tool immune to any parent-side CUDA initialization. Fixes the nightly diffusion recipe failures tracked in AMINT-212, AMINT-213, AMINT-214, and AMINT-215. Validated on 8xGPU with nvcr.io/nvidian/nemo-automodel:nightly-2026-07-31: stock code reproduces the CI failure; with this change the wan2_1_t2v_flow preprocessing stage completes and writes the latent cache. * fix(diffusion): select worker GPUs explicitly under spawn With spawn start method, the worker's module re-imports (including diffusers) can touch the CUDA driver before _init_worker/_init_video_worker run, after which setting CUDA_VISIBLE_DEVICES inside the worker is ignored and every worker binds cuda:0 to physical GPU 0, OOMing on multi-GPU nodes. Select the assigned GPU explicitly with torch.cuda.set_device and a per-worker cuda:{gpu_id} device string instead. * fix(diffusion): defer diffusers import in qwen_image_edit preprocessing Importing diffusers initializes the CUDA driver as a side effect. This module is pulled into the tools.diffusion.processors package import chain (the preprocessing CLI parent process), which must stay CUDA-free so its multiprocessing workers can initialize CUDA — the eager module-level safe_import here is what regressed the fork-based preprocessing pools in the 2026-07-28 nightly. Move the safe_import into _load_pipeline, the only consumer, restoring a side-effect-free import. * test(diffusion): cover lazy diffusers import in qwen_image_edit _load_pipeline Covers both branches of the deferred safe_import: the ImportError raised when diffusers is unavailable, and the happy path asserting the pipeline loads with transformer=None and a frozen eval-mode VAE/text-encoder pair on the requested device and dtype. --------- Signed-off-by: Pranav Prashant Thombre <pthombre@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Pranav Thombre <pthombre@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
… `r0.6.0` (#3375) fix(tokenizer): make special token insertion opt-in (#3337) * fix(tokenizer): make special token insertion opt-in * fix(examples): preserve Baichuan special tokens * test(tokenizer): opt in to EOS in formatting coverage --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
…into `r0.6.0` (#3376) fix(distributed): stabilize SAC replay with TE and FSDP (#3330) * fix(distributed): restore TE cache during AC recompute * fix(distributed): exclude FSDP runtime ops from SAC * fix(distributed): exclude FSDP all-gather from SAC * fix(distributed): exclude FSDP views from SAC * fix(distributed): exclude FSDP allocations from SAC * refactor(distributed): centralize SAC ignored ops * refactor(distributed): resolve SAC ops by dotted path * refactor(distributed): group selective AC ops * fix(distributed): snapshot TE cache in shared selective AC --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
#3381) fix(ci): use a single container cache donor (#3377) Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com>
…r (3328)` into `r0.6.0` (#3384) fix(fsdp): resolve fp32 master-weight compute dtype per parameter (#3328) * fix(fsdp): resolve fp32 master-weight compute dtype per parameter Custom MoE models (e.g. Qwen3.5/3.6-MoE) never record `_hf_compute_dtype`, so every parameter falls to the compute-dtype fallback in `_make_compute_dtype_fn`. The fallback keyed on a module-global `storage_is_uniform` flag, which a single native-bf16 sibling (`shared_expert_gate`) flips false -- misclassifying the layer's fp32 master weights as fp32 compute. With #3255's `cast_forward_inputs=False`, the misclassified fp32 unit no longer casts its bf16 input and the forward crashes (`bf16 @ fp32`). Resolve the fallback per parameter: unhinted fp32 storage under a lower-precision policy is an fp32 master weight and computes in the policy dtype; any other storage keeps its own dtype. A single bf16 sibling no longer forces the layer's fp32 master weights into fp32 compute. The `(storage, compute)` grouping still keeps each FSDP unit storage-uniform, so `_init_mp_dtypes` is unaffected. Fixes #3327 * fix(qwen3_5_moe): pin _fp32_params compute, build shared_expert_gate in dtype Follow-up to the per-parameter compute-dtype fallback (#3328). Two gaps that the resolver change exposes on Qwen3.5-MoE: - The MoE parallelizer keys the fp32 compute pin on `_keep_in_fp32_modules_strict`, which the model never set, so the SSM-gating A_log/dt_bias fell to the policy dtype (bf16) instead of fp32. Add the strict entry, matching the inkling/kimi_k3 siblings. - `shared_expert_gate` was built at the `initialize_linear_module` default (bf16) even under fp32 master-weight storage, making it the lone bf16 sibling the #3327 fallback has to tolerate. Build it in `config.dtype`. Add a compute-contract regression test for the mixed-storage, no-HF-hint layout (fp32 masters + a genuinely-bf16 shared_expert_gate + the fp32 holder): ordinary weights and the gate compute bf16; only the pinned A_log/dt_bias stay fp32. Fixes #3327 * fix(qwen3_next): pin _fp32_params fp32 compute via strict contract Qwen3-Next uses the same intrinsic-fp32 `_fp32_params` holder but declared it only in `_keep_in_fp32_modules`. The MoE parallelizer keys the fp32 compute pin on `_keep_in_fp32_modules_strict`, so under the generalized fallback its A_log/dt_bias would compute in bf16. Add the strict entry, matching Qwen3.5-MoE. Add a model-owned assertion guarding the strict declaration on every model with the holder, so a future model that forgets the contract fails a test rather than silently dropping to bf16. Fixes #3327 --------- Signed-off-by: NancyFyong <88076188+NancyFyong@users.noreply.github.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: NancyFyong <88076188+NancyFyong@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
…`r0.6.0` (#3389) perf(checkpoint): reduce distributed save overhead (#3369) * perf(checkpoint): reduce distributed save overhead * fix(checkpoint): preserve node-local save behavior * fix(checkpoint): scope planner caches per checkpointer --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
…0` (#3391) fix(ci): enable LTX-2.3 diffusion finetuning (#3372) * fix(ci): enable LTX-2.3 diffusion finetuning * fix(docs): map LTX-2.3 publisher directory * Apply suggestions from code review --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com> Co-authored-by: jgerh <163925524+jgerh@users.noreply.github.com>
….6.0` (#3395) ci: tune Nemotron single-GPU model load threads (#3370) * ci: bound checkpoint CI worker threads * fix(ci): limit CPU threads to single-worker jobs * fix(ci): scope CPU threads to Nemotron single-GPU * chore: log CPU threading configuration * ci: finalize Nemotron thread tuning --------- Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Yuhe Zhang <yuhez@nvidia.com>
…ng (3348)` into `r0.6.0` (#3393) fix(model): correct Mistral4 attention and distributed MoE routing (#3348) * fix(model): align Mistral4 attention scaling with HF * fix(moe): synchronize expert load bias updates * test(vlm): calibrate Mistral4 parity thresholds * test(vlm): bound Mistral4 fresh-runtime parity * ci(vlm): install media dependencies directly * fix(moe): replicate routing bias as DTensor * fix(mistral4): preserve routing bias in fp32 * fix(moe): load routing bias into replicated DTensor --------- Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Yuhe Zhang <yuhez@nvidia.com>
…3474)` into `r0.6.0` (#3490) perf(benchmarks): recompute deterministic MoE routers under AC (#3474) * perf(benchmarks): recompute deterministic MoE routers under AC * rm --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
chore: Bump gitpython to >= 3.1.59 (#3482) * Bump gitpython to >= 3.1.59 * Update uv lock --------- Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: NeMo Bot <nemo-bot@nvidia.com>
…ting (#3492) Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
feat(models): make Inkling implementation standalone (#3358) * fix(pp): preserve VLM media cursor with static metadata * fix(ci): read static PP metadata in checkpoint parity * fix(checkpoint): ignore base FP8 metadata on resume * ci(vlm): extend Mistral4 checkpoint timeout * feat(models): make Inkling implementation standalone * fix(models): align Inkling attention dtypes * fix(models): preserve Inkling residual dtype * fix(models): make Inkling training pipeline-safe --------- (cherry picked from commit c2b929f) Signed-off-by: Yuhe Zhang <yuhez@nvidia.com> Signed-off-by: HuiyingLi <willwin.lee@gmail.com> Co-authored-by: Yuhe Zhang <yuhez@nvidia.com>
…6.0` (#3502) fix(models): preserve lm-head dtype boundaries (#3491) * fix(models): preserve lm-head dtype boundaries * fix(models): infer lm-head compute dtype * test(models): cover NeMo RL lm-head dtypes --------- Signed-off-by: larkzhang-nv <larkz@nvidia.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: jQizhang <larkz@nvidia.com> Co-authored-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
…r0.6.0` (#3503) fix(checkpoint): validate PEFT adapter-only state (#3501) * fix(ci): keep Nemotron Flash robustness on TP1 * test(ci): isolate Qwen MoE checkpoint integrity gates * test(ci): diagnose Qwen source parity instability * test(checkpoint): compare PEFT adapter-only state * test(ci): verify Qwen source expert weights * test(ci): tolerate FSDP expert parameter prefixes * test(ci): inspect Qwen expert state dict * test(ci): remove Qwen parity diagnostics --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
… (3493)` into `r0.6.0` (#3505) ci(convergence): fix gemma4 eval setup and re-baseline Qwen3-MoE (#3493) * ci(convergence): fix gemma4 eval setup and re-baseline Qwen3-MoE gemma4: the launcher body runs on every node, so the lm-eval clone, venv build and full vLLM eval ran once per node. In pipeline 61927701 the clone failed on 2 of 4 nodes with an auth challenge; those tasks exited non-zero and srun tore down the eval the healthy nodes were still running. Gate the eval phase to node 0 and retry the clone. qwen3: #3359 cut this recipe's MoE aux-loss backward scale 64x (8 -> cp/microbatches=1/8), leaving the step-0 forward identical but changing every optimizer step after it: final train loss 0.6625 -> 0.5832, val 0.7993 -> 0.7008, IFEval 0.6137 -> 0.6580. Re-baseline from CI 61927701. * ci(convergence): widen Qwen3-MoE baseline to 0.6380 pending confirmation Set below the observed post-#3359 score of 0.6580 so the gate spans both the pre- and post-fix observations while the new score is confirmed on a second run. * ci(convergence): make the downstream-eval gate one-sided The gate is a regression detector, so only a drop should fail it. Both #3028 (fused RoPE) and #3359 (MoE aux-loss scaling) were correctness fixes that raised these scores and failed the two-sided gate from above. PASS iff score > baseline - k*stderr. Keeps Qwen3-MoE at 0.6380 (floor 0.5972). * ci(convergence): trim the Qwen3-MoE baseline comment * ci(convergence): pin vllm to 0.26.0 * ci(convergence): pin lm-eval's transformers to 5.14.1 This is the transformers in lm-eval's own venv, which resolves separately from the container's; the training stack is unaffected. --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
… (3513)` into `r0.6.0` (#3515) fix(checkpoint): preserve FSDP2 mixed precision during recompute (#3513) * fix(checkpoint): preserve FSDP2 mixed precision during recompute * test(checkpoint): cover FSDP2 full-layer checkpointing * style(checkpoint): format full-layer checkpoint condition * fix(checkpoint): canonicalize wrapped parameter names --------- Signed-off-by: larkzhang-nv <larkz@nvidia.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: jQizhang <larkz@nvidia.com> Co-authored-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
….6.0` (#3524) fix(deps): resolve Starlette and GitPython CVEs (#3523) Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com>
MiniMaxM3CPSparseAttention derives per-document ids from packed position ids, builds its own FlexAttention block mask, and shards the packed sequence inside forward, but the model declared none of that. `ModelSupports` therefore reported `supports_sequence_packing=False` and the CP + packing gate from #3414 rejected `minimax_m3_vl_sft_tulu3_text_cp8_16k` at recipe.setup(). Carry over the two capability clauses that recognise a model-owned packed and CP attention path, and declare both on the model. On main these clauses already exist (added for Kimi Linear in #3243), so the equivalent change there is the model flags alone. Signed-off-by: Abhishree <abhishreetm@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
….0` (#3552) feat(gemma4): add E-series tensor parallelism (#3512) * feat(gemma4): add E-series tensor parallelism * chore(gemma4): remove mock TP parity recipe * perf(gemma4): default E4B recipe to TP2 CP2 * fix(ci): add Gemma4 recipe owner * refactor(gemma4): delegate E-series mask preparation * chore(gemma4): name E4B recipe for TP2 CP2 --------- Signed-off-by: HuiyingLi <willwin.lee@gmail.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Huiying <willwin.lee@gmail.com>
ci: route DGX Spark recipes to GB10 (#3538) * ci: route DGX Spark recipes to GB10 * fix(ci): use HF FlashAttention for Qwen Spark --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
feat(vlm): support packed THD vlm context parallelism (#3322) * feat(vlm): support packed THD context parallelism * refactor(distributed): simplify THD payload preservation * fix(vlm): account for CP padding during THD packing * refactor(distributed): keep THD indices in shard layout Remove the unused batch payload and rely on the existing ShardLayout contract for the data-dependent THD token map. --------- Signed-off-by: HuiyingLi <willwin.lee@gmail.com> Signed-off-by: qiaochuz <qiaochuz@nvidia.com> Co-authored-by: Huiying <willwin.lee@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…540)` into `r0.6.0` (#3561) fix(fsdp): uniform reduce dtype and EP-local expert gradients (#3540) * fix(fsdp): keep reduce-scatter gradient dtype uniform under accumulation FSDPParamGroup.post_backward feeds foreach_reduce from two slots that hold different dtypes: unsharded_accumulated_grad, upcast to reduce_dtype (e.g. fp32) for parameters that received a gradient during an earlier no-sync micro-batch, and the raw unsharded_param.grad in param_dtype (e.g. bf16) for parameters whose gradient joined only now. A group holding both trips AssertionError: FSDP reduce-scatter expects uniform gradient dtype but got {torch.bfloat16, torch.float32} Context parallelism makes this reachable: patch_fsdp_unused_param_reduction fills locally unused parameters with a param-dtype zero, which lands next to peers that already hold reduce-dtype accumulations. Any cp_size > 1 recipe with gradient accumulation can reach that state (NVBug 6599894: GLM-5.2 glm_moe_dsa cp8 plus pp4, which never completes step 0). Widen the minority gradients inside foreach_reduce itself, the last point before the reduce. At that point the gradients are already plain local tensors, FSDP2's own bookkeeping is untouched, and foreach_reduce copies them into a reduce_dtype buffer regardless, so no value changes. Uniform groups pass straight through, so the upstream assertion still catches genuinely inconsistent gradients such as fp8 weights that fail to produce higher-precision ones. * fix(fsdp): reduce unused expert gradients from local tensors * fix(fsdp): localize residual DTensor gradients * fix(fsdp): prefer public unused-param reduction --------- Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
…563)` into `r0.6.0` (#3566) fix(deps): resolve msgpack, wandb, and mistune container CVEs (#3563) * fix(deps): resolve msgpack, wandb, and mistune container CVEs Address the 26.08.rc8 nSpect container scan: - msgpack>=1.2.1 constraint for GHSA-6v7p-g79w-8964 (out-of-bounds read on Unpacker reuse); msgpack reaches the venv through ray. - wandb>=0.28.2 constraint for GO-2026-5970; the 0.28.2 wandb-core binary embeds golang.org/x/text v0.41.0, past the fixed 0.39.0. - mistune>=3.3.0 in the base-image pip block for GHSA-qcq2-496w-v96p (quadratic-time parse_link_text DoS). opentelemetry advances to 1.44.0 because wandb 0.28.2 requires >=1.43.0. * Update uv lock --------- Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: NeMo Bot <nemo-bot@nvidia.com>
#3611) fix(deps): resolve 26.08 rc9 container CVEs (#3607) * fix(deps): resolve 26.08 rc9 container CVEs Address the 26.08.rc9 nSpect container scan: - pyasn1>=0.6.4 for GHSA-hm4w-wwcw-mr6r (uncontrolled resource consumption decoding REAL values) and GHSA-8ppf-4f7h-5ppj (quadratic OBJECT IDENTIFIER parsing). - soupsieve>=2.8.4 for GHSA-836r-79rf-4m37 (selector-parser ReDoS) and GHSA-2wc2-fm75-p42x (memory exhaustion on large selector lists); soupsieve reaches the venv through beautifulsoup4 and resolves to 2.9.2. - Drop ray/jars from the venv for GHSA-r7wm-3cxj-wff9 in the jackson-core vendored by ray_dist.jar. Ray still ships jackson 2.18.6 as of 2.57.0, so no version bump clears it; Ray is a transitive Python-only dependency here and Automodel never launches Java workers, so the jars are unused. This also removes the earlier jackson-databind findings from the same jar. * docs(docker): condense ray jar removal comment --------- (cherry picked from commit 1e40293) Conflict resolution: r0.6.0 predates the non-root runtime chown in the uv sync layer (#3484), so the ray/jars removal ends that RUN chain here instead of preceding the chown. Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: gitlab-runner <gitlab-runner@gitlab-master.nvidia.com>
svcnvidia-nemo-ci
requested review from
a team,
HuiyingLi,
akoumpa,
athitten,
jgerh and
snowmanwwg
as code owners
August 22, 2026 09:06
jgerh
reviewed
Aug 25, 2026
jgerh
left a comment
jgerh
left a comment
Contributor
There was a problem hiding this comment.
Completed tech pubs review of docs/model-coverage/diffusion/lightricks/ltx-2-3.mdx and provided a few copyedits.
Contributor
There was a problem hiding this comment.
Suggested change
| description: "Fine-tune and generate synchronized video and audio with LTX-2.3 in NeMo AutoModel." |
Contributor
There was a problem hiding this comment.
Suggested change
| NeMo AutoModel supports [LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3), a text-to-video diffusion model from Lightricks. Its dual-stream transformer denoises video and audio latents together so generated clips can include synchronized sound. |
Contributor
There was a problem hiding this comment.
Suggested change
| The generation command writes an MP4 and muxes the generated waveform into it. See the [Diffusion Fine-Tuning Guide](/recipes-e2e-examples/diffusion-fine-tuning) and the [Diffusion Dataset Preparation](/datasets/diffusion-dataset) guide for shared training and dataset options. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
beep boop 🤖: Updating transformers to latest version on pypi