Copyright Notice:

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

C. Chen, T. Chen, J. Duan, Q. Zhu, Z. Wang, Q. Hu, P. Sun, X. Li, C. Yang, T. Hoefler:

 Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training

(arXiv:2509.21841v2. Sep. 2025)

Abstract

Training large language models (LLMs) with increasingly long and varying sequence lengths introduces severe load imbalance challenges in large-scale data-parallel training. Recent frameworks attempt to mitigate these issues through data reorganization or hybrid parallel strategies. However, they often overlook how computational and communication costs scale with sequence length, resulting in suboptimal performance. We identify three critical challenges: (1) varying computation-to-communication ratios across sequences of different lengths in distributed attention, (2) mismatch between static NIC-GPU affinity and dynamic parallel workloads, and (3) distinct optimal partitioning strategies required for quadratic attention versus linear components. To address these challenges, we present Zeppelin, a novel training system that integrates three key techniques: (1) a hierarchical sequence partitioning method for the attention module that reduces communication overhead and balances computation, supported by an efficient attention engine that applies divergent parallel strategies; (2) a routing layer that orchestrates inter-node transfers to fully utilize NIC bandwidth; and (3) a remapping layer that transforms sequence layouts between attention and linear modules, ensuring high computational efficiency across both. Comprehensive evaluations across diverse configurations show that Zeppelin delivers an average 2.80x speedup over state-of-the-art methods.

Documents

download article:
access preprint on arxiv:
 

BibTeX

@article{chang2025zeppelin,
  author={Chang Chen and Tiancheng Chen and Jiangfei Duan and Qianchao Zhu and Zerui Wang and Qinghao Hu and Peng Sun and Xiuhong Li and Chao Yang and Torsten Hoefler},
  title={{Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training}},
  journal={arXiv:2509.21841v2},
  year={2025},
  month={09},
  doi={10.48550/arXiv.2509.21841},
}