Copyright Notice:

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

J. Chen, S. Li, R. Gun, J. Yuan, T. Hoefler:

 AutoDDL: Automatic Distributed Deep Learning With Near-Optimal Bandwidth Cost

(IEEE Transactions on Parallel and Distributed Systems. Vol 35, Nr. 8, pages 1331-1344, IEEE Press, Aug. 2024)

Publisher Reference

Abstract

Recent advances in deep learning are driven by the growing scale of computation, data, and models. However, efficiently training large-scale models on distributed systems requires an intricate combination of data, operator, and pipeline parallelism, which exerts heavy burden on machine learning practitioners. To this end, we propose AutoDDL, a distributed training framework that automatically explores and exploits new parallelization schemes with near-optimal bandwidth cost. AutoDDL facilitates the description and implementation of different schemes by utilizing OneFlow's Split, Broadcast, and Partial Sum (SBP) abstraction. AutoDDL is equipped with an analytical performance model combined with a customized Coordinate Descent algorithm, which significantly reduces the scheme searching overhead. We conduct evaluations on Multi-Node-Single-GPU and Multi-Node-Multi-GPU machines using different models, including VGG and Transformer. Compared to the expert-optimized implementations, AutoDDL reduces the end-to-end training time by up to 31.1% and 10% for Transformer and up to 17.7% and 71.5% for VGG on the two parallel systems, respectively.

Documents

download article:
access preprint on arxiv:
 

BibTeX

@article{chen2024autoddl,
  author={Jinfan Chen and Shigang Li and Ran Gun and Jinhui Yuan and Torsten Hoefler},
  title={{AutoDDL: Automatic Distributed Deep Learning With Near-Optimal Bandwidth Cost}},
  journal={IEEE Transactions on Parallel and Distributed Systems},
  year={2024},
  month={8},
  pages={1331-1344},
  volume={35},
  number={8},
  publisher={IEEE Press},
  doi={10.1109/TPDS.2024.3397800},
}