Early, express, and reliable detection of cancer can provide a favorable prognosis and decrease mortality. Tumor biomarkers have been proven to be closely related to tumor occurrence and development. Conventional tumor biomarker detection based on genomic, proteomic, and metabolomic methods is time and equipment-consuming and always needs a specific target marker. Surface-enhanced Raman scattering (SERS), as a non-invasive ultrasensitive and label-free vibrational spectroscopy technique, can detect cancer-related biomedical changes in biofluids. In this paper, 110 serum samples were collected from 30 healthy controls and 80 cancer patients (including 30 bladder cancer (BC), 30 adrenal cancer (AC), and 20 acute myeloid leukemia (AML)). One microliter of blood serum was mixed with 1 μl silver colloid and then was air-dried for SERS measurements. After spectral data augmentation, one-dimensional convolutional neural network (1D-CNN) was proposed for precise and rapid identification of healthy and three different cancers with high accuracy of 98.27%. After gradient-weighted class activation mapping (Grad-CAM) based spectral interpretation, the contributions of SERS peaks corresponding to biochemical substances indicated the most potential biomarkers, i.e., L-tyrosine in bladder cancer; acetoacetate and riboflavin in adrenal cancer and phospholipids, amide-I, and α-Helix in acute myeloid leukemia, which might provide an insight into the mechanism of intelligent diagnosis of different cancers based on label-free serum SERS. The integration of label-free SERS and deep learning has great potential for the rapid, reliable, and non-invasive detection of cancers, which may significantly improve the precise diagnosis in clinical practice.Graphical abstract