• Home
• People
• Jobs
• Visit
• Thesis Topics
• Publications
• Conference Rankings
• Research
• Bcast
• Teaching
• Contact

SPONSORS

COLLABORATIONS

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

G. Vlassis, S. Ashkboos, A. Volkova, T. Hoefler, D. Alistarh:
	Beyond Outliers: A Study of Optimizers Under Quantization (In The Fourteenth International Conference on Learning Representations, presented in Rio de Janeiro, Brazil, Apr. 2026) Publisher Reference Abstract As new optimizers gain traction and model quantization becomes standard for efficient deployment, a key question arises: how does the choice of optimizer affect model performance in the presence of quantization? Despite progress in both areas, systematic evidence on optimizer-quantization interactions remains limited. To fill this gap, we study the impact of optimizer choice on model robustness under quantization, considering both post-training quantization (PTQ), and quantization-aware training (QAT). We first train full-precision models, ranging from 50M to 1.5B parameters, with six optimizers, to explore the hyperparameter landscape, and establish well-tuned baselines. We then apply PTQ to evaluate how model performance degrades when trained with different optimizers. We find that outlier-related metrics, such as the max-to-mean ratio (MMR) and Kurtosis, fail to predict the PTQ performance across different optimizers. We show analytically that this is due to the MMR capturing only isolated layer errors, while ignoring how quantization errors accumulate and propagate through the network. To study the QAT degradation, we train quantized models from scratch and compare them to our original-precision baselines. We find that optimizers performing well in the original pretraining setup may not remain optimal under QAT, and that models trained with Shampoo show the lowest accuracy degradation. Finally, we derive scaling laws for quantization-aware training under different optimizers, showing that Shampoo achieves the highest parameter efficiency of all tested optimizers. Documents download article: access preprint on arxiv:
	BibTeX
@inproceedings{vlassis2026beyond,   author={Georgios Vlassis and Saleh Ashkboos and Alexandra Volkova and Torsten Hoefler and Dan Alistarh},   title={{Beyond Outliers: A Study of Optimizers Under Quantization}},   year={2026},   month={04},   booktitle={The Fourteenth International Conference on Learning Representations},   location={Rio de Janeiro, Brazil},   doi={10.48550/arXiv.2509.23500}, }