Copyright Notice:

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

P. Okanovic, A. Kirsch, J. Kasper, T. Hoefler, A. Krause, N. Merve Gürel:

 All models are wrong, some are useful: Model Selection with Limited Labels

(In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS'25), presented in Mai Khao, Thailand, May 2025)

Publisher Reference

Abstract

We introduce MODEL SELECTOR, a framework for label-efficient selection of pretrained classifiers. Given a pool of unlabeled target data, MODEL SELECTOR samples a small subset of highly informative examples for labeling, in order to efficiently identify the best pretrained model for deployment on this target dataset. Through extensive experiments, we demonstrate that MODEL SELECTOR drastically reduces the need for labeled data while consistently picking the best or near-best performing model. Across 18 model collections on 16 different datasets, comprising over 1,500 pretrained models, MODEL SELECTOR reduces the labeling cost by up to 94.15% to identify the best model compared to the cost of the strongest baseline. Our results further highlight the robustness of MODEL SELECTOR in model selection, as it reduces the labeling cost by up to 72.41% when selecting a near-best model, whose accuracy is only within 1% of the best model.

Documents

download article:
access preprint on arxiv:
 

BibTeX

@inproceedings{okanovic2024modelswrongusefulmodel,
  author={Patrik Okanovic and Andreas Kirsch and Jannes Kasper and Torsten Hoefler and Andreas Krause and Nezihe Merve Gürel},
  title={{All models are wrong, some are useful: Model Selection with Limited Labels}},
  year={2025},
  month={05},
  booktitle={Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS'25)},
  location={Mai Khao, Thailand},
}