While the physical improvements from exercise have been well documented over the years, the impact of physical activity on mental health has recently become an object of interest. Physical exercise improves cognition, particularly attention, memory, and executive functions. However, the mechanisms underlying these effects have yet to be fully understood. Consequently, we conducted a narrative literature review concerning the association between acute and chronic physical activity and cognition to provide an overview of exercise-induced benefits during the lifetime of a person. Most previous papers mainly reported exercise-related greater expression of neurotransmitter and neurotrophic factors. Recently, structural and functional magnetic resonance imaging techniques allowed for the detection of increased grey matter volumes for specific brain regions and substantial modifications in the default mode, frontoparietal, and dorsal attention networks following exercise. Here, we highlighted that physical activity induced significant changes in functional brain activation and cognitive performance in every age group and could counteract psychological disorders and neural decline. No particular age group gained better benefits from exercise, and a specific exercise type could generate better cognitive improvements for a selected target subject. Further research should develop appropriate intervention programs concerning age and comorbidity to achieve the most significant cognitive outcomes.