The utilization of allelochemicals to inhibit algal overgrowth is a promising approach for controlling harmful algal blooms (HABs). Quercetin has been found to have an allelopathic effect on algae. However, its responsive mechanism needs to be better understood. In the present study, the inhibitory effects of different quercetin concentrations onM. aeruginosawere evaluated, and the inhibition mechanisms were explored. The results demonstrated that quercetin significantly inhibitedM. aeruginosagrowth, and the inhibitory effect was concentration-dependent. The inhibition rate of 40 mg L−1quercetin on algal density reached 90.79% after 96 h treatment. The concentration of chlorophyll-a(chl-a) in treatment groups with quercetin concentrations of 10, 20, and 40 mg L−1decreased by 59.74%, 74.77%, and 80.66% at 96 h, respectively. Furthermore, quercetin affects photosynthesis and damages the cell membrane, respiratory system, and enzyme system. All photosynthetic fluorescence parameters, including the maximum photochemical quantum yield (Fv/Fm), the actual photochemical quantum yield (YII), the maximum relative electron transfer rate (rETRmax), and light use efficiency (α), exhibited a downtrend after exposure. After treatment with 20 mg L−1quercetin, the nucleic acid and protein content in the algal solution increased, and the respiration rate of algae decreased significantly. Additionally, superoxide dismutase (SOD) activities significantly increased as a response to oxidative stress. In comparison, the activities of ribulose 1,5-biphosphate carboxylase (Rubisco) and phosphoenolpyruvate carboxylase (PEPC) decreased significantly. These results revealed that quercetin could inhibitM. aeruginosaby affecting its photosynthesis, respiration, cell membrane, and enzymic system. These results are promising for controllingM. aeruginosaeffectively.