Federated learning has been popular for its ability to train centralized models while protecting clients’ data privacy. However, federated learning is highly susceptible to poisoning attacks, which can result in a decrease in model performance or even make it unusable. Most existing defense methods against poisoning attacks cannot achieve a good trade-off between robustness and training efficiency, especially on non-IID data. Therefore, this paper proposes an adaptive model filtering algorithm based on the Grubbs test in federated learning (FedGaf), which can achieve great trade-offs between robustness and efficiency against poisoning attacks. To achieve a trade-off between system robustness and efficiency, multiple child adaptive model filtering algorithms have been designed. Meanwhile, a dynamic decision mechanism based on global model accuracy is proposed to reduce additional computational costs. Finally, a global model weighted aggregation method is incorporated, which improves the convergence speed of the model. Experimental results on both IID and non-IID data show that FedGaf outperforms other Byzantine-robust aggregation rules in defending against various attack methods.