The Camellia oil tree (Camellia oleiferaAbel.) is an important nonwood forest species in China, and the majority of its cultivars are late-acting self-incompatibility (LSI) types. Although several studies have examined the mechanism of LSI, the process is quite complicated and unclear. In this study, pollen tube growth and fruit setting of two Camellia oil tree cultivars Huashuo (HS) and Huajin (HJ) were investigated after non and self-pollination, and transcriptomic analysis of the ovaries was performed 48 h after self-pollination to identify the potential genes implicated in the LSI of Camellia oil trees. The results showed that the fruit set of HS was significantly higher than that of HJ after self-pollination. Transcriptomic analysis revealed that plant hormone signal transduction, the phosphatidylinositol signaling system, ATP-binding cassette (ABC) transporters, reactive oxygen species (ROS) metabolism, and Ca2+signaling were mainly contributed in the LSI of reaction of Camellia oil tree. Moreover, nine RNase T2 genes were identified from the transcriptome analysis, which also showed that CoRNase7 participated in the self-incompatibility reaction in HS. Based on phylogenetic analysis, CoRNase6 was closely related to S-RNase from coffee, and CoRNase7 and CoRNase8 were closely related to S-RNase from Camellia sinensis. The 9 RNase T2 genes successfully produced proteins in prokaryotes. Subcellular localization indicated that CoRNase1 and CoRNase5 were cytoplasmic proteins, while CoRNase7 was a plasma membrane protein. These results screened the main metabolic pathways closely related to LSI in Camellia oil tree, and SI signal transduction might be regulated by a large molecular regulatory network. The discovery of T2 RNases provided evidence that Camellia oil tree might be under RNase-based gametophytic self-incompatibility.