Adding Cosmos 3 to Diffusers#13818

Merged

yiyixuxu merged 112 commits into

huggingface:mainfrom

atharvajoshi10:cosmos3/video-gen-with-sound

May 28, 2026

Commits on May 27, 2026

Adding Cosmos 3
atharvajoshi10
committed
removed dead code
atharvajoshi10
committed
Change customer TimeEmbedding Layer to DIffusers Time Embedding
atharvajoshi10
committed
removed dependency on hugging face transformers
atharvajoshi10
committed
refactor 1
atharvajoshi10
committed
Fixed Attention Pattern
atharvajoshi10
committed
Removed from Pretrain overrides
atharvajoshi10
committed
Removing normalization from the audio Tokenizer
atharvajoshi10
committed
fixed diffusers checkpoint
atharvajoshi10
committed
fixed video save uint conversion
atharvajoshi10
committed
added forward hook for cpu offload case
atharvajoshi10
committed
removed dead params for sound tokenizer
atharvajoshi10
committed
renaming audio encoder for readability
atharvajoshi10
committed
ruff format
atharvajoshi10
committed
Fix checkpoint conversion script for sound tokenizer

yzhautouskay
authored and
atharvajoshi10
committed
Audio Decoder trim and removing some dead code
atharvajoshi10
committed
removing dead sequence packing code
atharvajoshi10
committed
refactor pipeline to diffusers style formatting
atharvajoshi10
committed
removing use of cosmos3 audio encoder
atharvajoshi10
committed
Revert "removing use of cosmos3 audio encoder"
atharvajoshi10
committed
refactor audio encoder
atharvajoshi10
committed
inline remaining sequence packing functions and lint
atharvajoshi10
committed
Removed GenerationDataClean class and Action logic
atharvajoshi10
committed
inlined default args
atharvajoshi10
committed
removed dead code and refactoring
atharvajoshi10
committed
drop pipeline-helper @no_grad, inline derive helper, move guidance check to check_inputs

atharvajoshi10
and
claude
committed
drop unused list bookkeeping from PackedSequence

atharvajoshi10
and
claude
committed
extract pack-time build state from PackedSequence dataclass

atharvajoshi10
and
claude
committed
drop private-API isinstance/shape asserts

atharvajoshi10
and
claude
committed
build PackedSequence tensors on target device, drop to_cuda

atharvajoshi10
and
claude
committed
drop SequencePlan, skip_text_tokens, and bos_token_id branch

atharvajoshi10
and
claude
committed
use retrieve_latents helper in _encode_video

atharvajoshi10
and
claude
committed
drop get_data_and_condition + data_batch dict scaffolding

atharvajoshi10
and
claude
committed
drop _load_image_as_tensor; use VideoProcessor for conditioning frame

atharvajoshi10
and
claude
committed
move encode/decode helpers + transformer forward into Cosmos3OmniTransformer

atharvajoshi10
and
claude
committed
remove Cosmos3VLTextModel; flatten transformer layout

atharvajoshi10
and
claude
committed
move save_img_or_video/save_wav to cosmos/export_utils.py

atharvajoshi10
and
claude
committed
drop @torch.no_grad on pipeline __call__

atharvajoshi10
and
claude
committed
follow standard transformer conventions in Cosmos3OmniTransformer

atharvajoshi10
and
claude
committed
restore @torch.no_grad() on Cosmos3OmniDiffusersPipeline.__call__

atharvajoshi10
and
claude
committed
drop training-only dummy projection paths in transformer decode helpers

atharvajoshi10
and
claude
committed
inline modality encode/decode helpers into transformer forward

atharvajoshi10
and
claude
committed
trim dead flags and idioms in Cosmos3 pipeline __call__

atharvajoshi10
and
claude
committed
introduce Cosmos3Condition for prepare_latents return shape

atharvajoshi10
and
claude
committed
restructure pack helpers as data-returning pipeline methods

atharvajoshi10
and
claude
committed
collapse builder-pattern PackedSequence/ModalityData into flat dataclasses

atharvajoshi10
and
claude
committed
exclude dtype from saved transformer config via ignore_for_config

atharvajoshi10
and
claude
committed
add Cosmos3 pipeline and transformer docs pages; retire JSON example inputs

atharvajoshi10
and
claude
committed
restore full prompts in cosmos3 docs from original example JSONs

atharvajoshi10
and
claude
committed
Renamed Cosmos3 module attributes

MaciejBalaNV
authored and
atharvajoshi10
committed
bugfix

MaciejBalaNV
authored and
atharvajoshi10
committed
Removed unnecessary helper function; added extra comments

MaciejBalaNV
authored and
atharvajoshi10
committed
Moved from encode_prompt to tokenize_prompt

MaciejBalaNV
authored and
atharvajoshi10
committed
Bring back video system prompt

MaciejBalaNV
authored and
atharvajoshi10
committed
Remove multi frame conditioning for now

MaciejBalaNV
authored and
atharvajoshi10
committed
Removed default negative prompts from code

MaciejBalaNV
authored and
atharvajoshi10
committed
Remove Cosmos3condition; simplify sequence pack

MaciejBalaNV
authored and
atharvajoshi10
committed
Clean up multiple parameters

MaciejBalaNV
authored and
atharvajoshi10
committed
Simplify decode video; remove remainings of batching

MaciejBalaNV
authored and
atharvajoshi10
committed
simple renames

MaciejBalaNV
authored and
atharvajoshi10
committed
Refactored schedulers for sound

MaciejBalaNV
authored and
atharvajoshi10
committed
Remove unnecessary autocast

MaciejBalaNV
authored and
atharvajoshi10
committed
Update sound example

MaciejBalaNV
authored and
atharvajoshi10
committed
Simplify loops in transformer_cosmos3.py

MaciejBalaNV
authored and
atharvajoshi10
committed
Remove unused config attributes

MaciejBalaNV
authored and
atharvajoshi10
committed
Cleanup audio decoder

MaciejBalaNV
authored and
atharvajoshi10
committed
Reuse encoder_video from LTX2 for Cosmos3

MaciejBalaNV
authored and
atharvajoshi10
committed
Fixed a few nits

MaciejBalaNV
authored and
atharvajoshi10
committed
Moved to RMSNorm for Cosmos3

MaciejBalaNV
authored and
atharvajoshi10
committed
Remove meta_tensor usage

MaciejBalaNV
authored and
atharvajoshi10
committed
Improved rope handling

MaciejBalaNV
authored and
atharvajoshi10
committed
Improved prompt templates

MaciejBalaNV
authored and
atharvajoshi10
committed
Added extra docs for templatete

MaciejBalaNV
authored and
atharvajoshi10
committed
remove dataclasses

Zhylkaaa
authored and
atharvajoshi10
committed
Cleanup after merging

MaciejBalaNV
authored and
atharvajoshi10
committed
Added guardrails

MaciejBalaNV
authored and
atharvajoshi10
committed
bugfixed guardrails

MaciejBalaNV
authored and
atharvajoshi10
committed
Bugfix guardrails v2

MaciejBalaNV
authored and
atharvajoshi10
committed
Simplified input_timestep

MaciejBalaNV
authored and
atharvajoshi10
committed
Add TODO

authored and
atharvajoshi10
committed
simplify conditional mask generation

Zhylkaaa
authored and
atharvajoshi10
committed
Inlined _postprocess_latents

MaciejBalaNV
authored and
atharvajoshi10
committed
removed pack_input_sequence helper
atharvajoshi10
committed
restore export utils with deprecation warning
atharvajoshi10
committed
moved sampling rate to pipeline attribute
atharvajoshi10
committed
inlined sound and image condition mask
atharvajoshi10
committed
seperating static and timestep based sound and vision token packing
atharvajoshi10
committed
unpack transformer args
atharvajoshi10
committed
enabled selection of cosmos3 super
atharvajoshi10
committed
Update src/diffusers/pipelines/cosmos/pipeline_cosmos3_omni.py

atharvajoshi10
and
yiyixuxu
committed
Update src/diffusers/pipelines/cosmos/pipeline_cosmos3_omni.py

atharvajoshi10
and
yiyixuxu
committed
Apply suggestions from code review

atharvajoshi10
and
yiyixuxu
committed
fixed sound and vision conditioning use from prepare latents
atharvajoshi10
committed
ruff format and doc builder
atharvajoshi10
committed
ran fix copies
atharvajoshi10
committed
move typing to python3.10

Zhylkaaa
authored and
atharvajoshi10
committed
move special token application from pack_text_tokens to tokenize_prompt

Zhylkaaa
authored and
atharvajoshi10
committed
rename packing methods to process methods

Zhylkaaa
authored and
atharvajoshi10
committed
remove guidance_scale check

Zhylkaaa
authored and
atharvajoshi10
committed
fix nits

Zhylkaaa
authored and
atharvajoshi10
committed
respect vae dtype in the pipeline

Zhylkaaa
authored and
atharvajoshi10
committed
use vae dtype for vae normalization stats

Zhylkaaa
authored and
atharvajoshi10
committed
skip CFG if guidance_scale is 1

Zhylkaaa
authored and
atharvajoshi10
committed
Remove unnecessary parameter

MaciejBalaNV
authored and
atharvajoshi10
committed
fix CFG for sound

Zhylkaaa
authored and
atharvajoshi10
committed
bugfix for sound CFG

Zhylkaaa
authored and
atharvajoshi10
committed
ruff format
atharvajoshi10
committed
Fix apply_chat_template return dict arg to return BatchEncoding

yzhautouskay
authored and
atharvajoshi10
committed
added option to select attention processor
atharvajoshi10
committed
docs: refresh Cosmos 3 pipeline intro
atharvajoshi10
committed
docs: document Cosmos3OmniPipeline.__call__ arguments
atharvajoshi10
committed
style: doc-builder reflow on Cosmos3OmniPipeline.__call__ docstring
atharvajoshi10
committed