The positron emission tomography (PET) with18F-flortaucipir can distinguish individuals with mild cognitive impairment (MCI) and Alzheimer’s disease (AD) from cognitively unimpaired (CU) individuals. This study aimed to evaluate the utility of18F-flortaucipir-PET images and multimodal data integration in the differentiation of CU from MCI or AD through DL. We used cross-sectional data (18F-flortaucipir-PET images, demographic and neuropsychological score) from the ADNI. All data for subjects (138 CU, 75 MCI, 63 AD) were acquired at baseline. The 2D convolutional neural network (CNN)-long short-term memory (LSTM) and 3D CNN were conducted. Multimodal learning was conducted by adding the clinical data with imaging data. Transfer learning was performed for classification between CU and MCI. The AUC for AD classification from CU was 0.964 and 0.947 in 2D CNN-LSTM and multimodal learning. The AUC of 3D CNN showed 0.947, and 0.976 in multimodal learning. The AUC for MCI classification from CU had 0.840 and 0.923 in 2D CNN-LSTM and multimodal learning. The AUC of 3D CNN showed 0.845, and 0.850 in multimodal learning. The18F-flortaucipir PET is effective for the classification of AD stage. Furthermore, the effect of combination images with clinical data increased the performance of AD classification.