How to use DeepSpeed for training in this code repository?

[ysl0310]

How to use DeepSpeed for training in this code repository?

[generalfyl]

I want to know either. It seems that Zero1 is easy to integrate. But Zero2/3 seems hard to deal with. Any suggestions? @luyug

[luyug]

have you tried fsdp2 natively from pytorch? that should be more natural an integration