Skip to content

Multi-LoRA SFT support FSDP2 #155

Draft
kevssim wants to merge 14 commits intomodelscope:mainfrom
kevssim:multilora_fsdp
Draft

Multi-LoRA SFT support FSDP2 #155
kevssim wants to merge 14 commits intomodelscope:mainfrom
kevssim:multilora_fsdp

Conversation

@kevssim
Copy link
Copy Markdown
Collaborator

@kevssim kevssim commented Apr 14, 2026

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

Write the detail information belongs to this PR.

Experiment results

Paste your experiment result here(if needed).

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements FSDP2 support for MultiLoraTransformersModel by integrating it into the shared strategy and lazy-wrap lifecycle and introducing sharding-aware parameter access helpers. Review feedback identifies critical bugs in the distributed tensor handling: _write_param_tensor may incorrectly double-shard local data, set_state_dict risks shape mismatches when applying global state to local shards, and get_state_dict returns sharded tensors that could lead to corrupt checkpoints. Furthermore, the model's initialization should be refactored to properly use the parent class, and internal imports should be moved to the module level.

Comment thread src/twinkle/model/multi_lora.py
Comment thread src/twinkle/model/multi_lora.py Outdated
Comment thread src/twinkle/model/multi_lora.py
Comment thread src/twinkle/model/transformers/multi_lora_transformers.py
Comment thread src/twinkle/model/multi_lora.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant