data-prallelism

Here is 1 public repository matching this topic...

Zare2001 / ProjectWorldWide

Pre-training one LLM across two supercomputers at once - AMD/ROCm and NVIDIA/CUDA, different queues, no shared filesystem. DiLoCo outer steps merged by a GPU-less VM via Flower/FedMom; DARL leases the corpus so no token is trained twice; membership is elastic, so a site stuck in the Slurm queue costs nothing.

hpc flower distributed-training federated-learning llm diloco torchtitan data-prallelism snellius

Updated Aug 11, 2026
Python

Improve this page

Add a description, image, and links to the data-prallelism topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the data-prallelism topic, visit your repo's landing page and select "manage topics."

Learn more