Efficient fault diagnosis of rotating machinery is essential for the safe operation of equipment in the manufacturing industry. In this study, a robust and lightweight framework consisting of two lightweight temporal convolutional network (LTCN) backbones and a broad learning system with incremental learning (IBLS) classifier called LTCN-IBLS is proposed for the fault diagnosis of rotating machinery. The two LTCN backbones extract the fault’s time–frequency and temporal features with strict time constraints. The features are fused to obtain more comprehensive and advanced fault information and input into the IBLS classifier. The IBLS classifier is employed to identify the faults and exhibits a strong nonlinear mapping ability. The contributions of the framework’s components are analyzed by ablation experiments. The framework’s performance is verified by comparing it with other state-of-the-art models using four evaluation metrics (accuracy, macro-recall (MR), macro-precision (MP), and macro-F1 score (MF)) and the number of trainable parameters on three datasets. Gaussian white noise is introduced into the datasets to evaluate the robustness of the LTCN-IBLS. The results show that our framework provides the highest mean values of the evaluation metrics (accuracy ≥ 0.9158, MP ≥ 0.9235, MR ≥ 0.9158, and MF ≥ 0.9148) and the lowest number of trainable parameters (≤0.0165 Mage), indicating its high effectiveness and strong robustness for fault diagnosis.