Flower scent is one of the main ornamental characteristics of herbaceous peony, and the improvement of flower fragrance is a vital objective of herbaceous peony breeding. In this study, 87 herbaceous peony cultivars were divided into three groups (no/light fragrance, medium fragrance, and strong fragrance) based on their sensory evaluation scores, and 16 strong fragrance cultivars and one no fragrance cultivar were selected for subsequent analysis. Sixty-eight volatile components were detected in these 17 cultivars based on solid-phase microextraction (SPME) and gas chromatography/mass spectrometry (GC/MS), and 26 types were identified as important scent components. They were composed of terpenoids, benzenoids/phenylpropanoids, and fatty acid derivatives. According to the content and odor threshold of these main aroma components, the characteristic aroma substances of herbaceous peony were identified, including linalool, geraniol, citronellol, and phenylethyl alcohol (2-PE). The cultivars of strong scented herbaceous peony were divided into three types: rose scent, lily scent, and mixed scent. We explored the possible key genes of characteristic aroma substances in herbaceous peony petals with different odors through the qRT-PCR. The key genes encoding monoterpene biosynthesis were found to bePlDXS2,PlDXR1,PlMDS1,PlHDR1,PlGPPS3,andPlGPPS4. In addition, the linalool synthase (LIS) gene and the geraniol synthase (GES) gene were also found.PlAADC1,PlPAR1,andPlMAO1,related to the biosynthesis of 2-PE were detected, and the synthetic pathway of 2-PE was speculated. In conclusion, these findings revealed that the difference in gene expression of monoterpene and 2-PE synthesis pathway was related to the difference in the fragrance of herbaceous peony. This study explored the releasing pathway of herbaceous peony characteristic aroma substances and provided key genetic resources for fragrance improvement.