Title: A Simple and Provable Method to Adapt Pre-trained Model across Domains with Few Samples. Abstract: Adapting the pre-trained model across domains with few samples, known as cross-domain few-shot learning, is a challenging task in statistical machine learning. Most previous efforts focused on training robust and transferable feature representations but rarely explored how to train an accurate few-shot model from a given pre-trained model. In this paper, we are interested in the performance of training a cross-domain few-shot classifier with representations from different layers of a pre-trained model and the impact of reducing the dimensionality of these representations. Based on this, we propose a simple and provable method, Average Pooling Ensemble Few-shot Learning (APEF). We demonstrate the effectiveness of average pooling and ensemble in cross-domain few-shot image classification both theoretically and experimentally. In particular, we provide a theoretical analysis in the PAC-Bayesian framework to illustrate why our method works, and we also empirically evaluate our approach on the challenging CD-FSL benchmark, which shows that our proposed method consistently outperforms all baselines.