ObjectivesThis study aimed to explore and develop artificial intelligence approaches for efficient classification of pulmonary nodules based on CT scans.Materials and methodsA number of 1007 nodules were obtained from 551 patients of LIDC-IDRI dataset. All nodules were cropped into 64 × 64 PNG images , and preprocessing was carried out to clean the image from surrounding non-nodular structure. In machine learning method, texture Haralick and local binary pattern features were extracted. Four features were selected using principal component analysis (PCA) algorithm before running classifiers. In deep learning, a simple CNN model was constructed and transfer learning was applied using VGG-16 and VGG-19, DenseNet-121 and DenseNet-169 and ResNet as pre-trained models with fine tuning.ResultsIn statistical machine learning method, the optimal AUROC was 0.885 ± 0.024 with random forest classifier and the best accuracy was 0.819 ± 0.016 with support vector machine. In deep learning, the best accuracy reached 90.39% with DenseNet-121 model and the best AUROC was 96.0%, 95.39% and 95.69% with simple CNN, VGG-16 and VGG-19, respectively. The best sensitivity reached 90.32% using DenseNet-169 and the best specificity attained was 93.65% when applying the DenseNet-121 and ResNet-152V2.ConclusionDeep learning methods with transfer learning showed several benefits over statistical learning in terms of nodule prediction performance and saving efforts and time in training large datasets. SVM and DenseNet-121 showed the best performance when compared with their counterparts. There is still more room for improvement, especially when more data can be trained and lesion volume is represented in 3D.Clinical relevance statementMachine learning methods offer unique opportunities and open new venues in clinical diagnosis of lung cancer. The deep learning approach has been more accurate than statistical learning methods. SVM and DenseNet-121 showed superior performance in pulmonary nodule classification.Graphical abstractSupplementary InformationThe online version contains supplementary material available at 10.1186/s13244-023-01441-6.