time sequences (we simply call it order split) [@riley2019three]. Actually, order split is closer to reality, because we always use historical data to predict the future condition in industry. Conversely, if we randomly split the data, it might be possible for the diagnosis algorithms to record the future patterns, and this might cause another pitfall with test leakage.

In this paper, we first collect most of the publicly available datasets and discuss whether it is suitable for intelligent fault diagnosis. Second, we release a code library of the data preparation for all datasets which are suitable for fault classification and the whole evaluation framework with different input formats, normalization methods, data split ways, augmentation methods, and DL-based models. Meanwhile, we also use some datasets to discuss the existing issues in intelligent fault diagnosis including class imbalance, generalization ability, interpretability, few-shot learning, and model selection. To the best of our knowledge, this is the first work to comprehensively perform the benchmark study and release the code library of DL-based intelligent algorithms. In summary, this work mainly focuses on evaluating various DL-based intelligent diagnosis algorithms for most of the publicly available datasets from several perspectives, providing the benchmark accuracy (it is worth 