The hippocampus is affected early in Alzheimer’s disease (AD) and altered hippocampal functioning influences normal cognitive aging. Here, we used task-based functional MRI to assess if theAPOEɛ4 allele or a polygenic risk score (PRS) for AD was linked to longitudinal changes in memory-related hippocampal activation in normal aging (baseline age 50–95, n = 292; n = 182 at 4 years follow-up, subsequently non-demented for at least 2 years). Mixed-models were used to predict level and change in hippocampal activation byAPOEɛ4 status and PRS based on gene variants previously linked to AD atp≤ 1,p< 0.05, orp< 5e−8 (excludingAPOE).APOEɛ4 and PRSp<5e−8significantly predicted AD risk in a larger sample from the same study population (n = 1542), while PRSp≤1predicted memory decline.APOEɛ4 was linked to decreased hippocampal activation over time, with the most prominent effect in the posterior hippocampi, while PRS was unrelated to hippocampal activation at all p-thresholds. These results suggests a link forAPOEɛ4, but not for AD genetics in general, on functional changes of the hippocampi in normal aging.