Rare ginsenosides are the major components of red ginseng. However, there has been little research into the relationship between the structure of ginsenosides and their anti-inflammatory activity. In this work, BV-2 cells induced by lipopolysaccharide (LPS) or nigericin, the anti-inflammatory activity of eight rare ginsenosides, and the target proteins expression of AD were compared. In addition, the Morris water maze test, HE staining, thioflavins staining, and urine metabonomics were used to evaluate the effect of Rh4 on AD mice. Our results showed that their configuration influences the anti-inflammatory activity of ginsenosides. Ginsenosides Rk1, Rg5, Rk3, and Rh4 have significant anti-inflammatory activity compared to ginsenosidesS-Rh1,R-Rh1,S-Rg3, andR-Rg3. GinsenosidesS-Rh1 andS-Rg3 have more pronounced anti-inflammatory activity than ginsenosidesR-Rh1 andR-Rg3, respectively. Furthermore, the two pairs of stereoisomeric ginsenosides can significantly reduce the level of NLRP3, caspase-1, and ASC in BV-2 cells. Interestingly, Rh4 can improve the learning ability of AD mice, improve cognitive impairment, reduce hippocampal neuronal apoptosis and Aβ deposition, and regulate AD-related pathways such as the tricarboxylic acid cycle and the sphingolipid metabolism. Our findings conclude that rare ginsenosides with a double bond have more anti-inflammatory activity than those without, and 20(S)-ginsenosides have more excellent anti-inflammatory activity than 20(R)-ginsenosides.