The aim was to investigate associations between leisure-time physical activity (LTPA) and mortality, and associations between desire to increase LTPA and mortality within the low LTPA group. A public health survey questionnaire was sent in 2008 to a stratified random sample of the population aged 18–80 in southernmost Sweden, yielding a 54.1% response rate. Baseline 2008 survey data with 25,464 respondents was linked to cause of death register data to create a prospective cohort with 8.3-year follow-up. Associations between LTPA, desire to increase LTPA and mortality were analyzed in logistic regression models. An 18.4% proportion performed regular exercise (at least 90 min/week, leading to sweating), 23.2% moderate regular exercise (once or twice a week at least 30 min/occasion, leading to sweating), 44.3% moderate exercise (more than two hours walking or equivalent activity/week) and 14.1% reported low LTPA (less than two hours walking or equivalent activity/week). These four LTPA groups were significantly associated with covariates included in the multiple analyses. The results showed significantly higher all-cause, cardiovascular (CVD), cancer and other cause mortality for the low LTPA group but not for the moderate regular exercise and moderate exercise groups compared to the regular exercise group. Both the “Yes, but I need support” and the “No” fractions within the low LTPA group had significantly increased ORs of all-cause mortality compared to the “Yes, and I can do it myself” reference, while no significant associations were observed for CVD mortality. Physical activity promotion is particularly warranted in the low LTPA group.