This research utilized the sooty tern optimization algorithm–variational mode decomposition (STOA-VMD) optimization algorithm to extract the acoustic emission (AE) signal associated with damage in fiber-reinforced composite materials. The effectiveness of this optimization algorithm was validated through a tensile experiment on glass fiber/epoxy NOL-ring specimens. To solve the problems of a high degree of aliasing, high randomness, and a poor robustness of AE data of NOL-ring tensile damage, the signal reconstruction method of optimized variational mode decomposition (VMD) was first used to reconstruct the damage signal and the parameters of VMD were optimized by the sooty tern optimization algorithm. The optimal decomposition mode number K and penalty coefficient α were introduced to improve the accuracy of adaptive decomposition. Second, a typical single damage signal feature was selected to construct the damage signal feature sample set and a recognition algorithm was used to extract the feature of the AE signal of the glass fiber/epoxy NOL-ring breaking experiment to evaluate the effectiveness of the damage mechanism recognition. The results showed that the recognition rates of the algorithm in matrix cracking, fiber fracture, and delamination damage were 94.59%, 94.26%, and 96.45%, respectively. The damage process of the NOL-ring was characterized and the findings indicated that it was highly efficient in the feature extraction and recognition of polymer composite damage signals.