Physical activity (PA) might influence the risk or progression of chronic pain through pain tolerance. Hence, we aimed to assess whether habitual leisure-time PA level and PA change affects pain tolerance longitudinally in the population. Our sample (n = 10,732; 51% women) was gathered from the sixth (Tromsø6, 2007–08) and seventh (Tromsø7, 2015–16) waves of the prospective population-based Tromsø Study, Norway. Level of leisure-time PA (sedentary, light, moderate, or vigorous) was derived from questionnaires; experimental pain tolerance was measured by the cold-pressor test (CPT). We used ordinary, and multiple-adjusted mixed, Tobit regression to assess 1) the effect of longitudinal PA change on CPT tolerance at follow-up, and 2) whether a change in pain tolerance over time varied with level of LTPA. We found that participants with high consistent PA levels over the two surveys (Tromsø6 and Tromsø7) had significantly higher tolerance than those staying sedentary (20.4 s. (95% CI: 13.7, 27.1)). Repeated measurements show that light (6.7 s. (CI 3.4, 10.0)), moderate (CI 14.1 s. (9.9, 18.3)), and vigorous (16.3 s. (CI 6.0, 26.5)) PA groups had higher pain tolerance than sedentary, with non-significant interaction showed slightly falling effects of PA over time. In conclusion, being physically active at either of two time points measured 7–8 years apart was associated with higher pain tolerance compared to being sedentary at both time-points. Pain tolerance increased with higher total activity levels, and more for those who increased their activity level during follow-up. This indicates that not only total PA amount matters but also the direction of change. PA did not significantly moderate pain tolerance change over time, though estimates suggested a slightly falling effect possibly due to ageing. These results support increased PA levels as a possible non-pharmacological pathway towards reducing or preventing chronic pain.