A US study found men and women in their 50s and 60s with a history of alcohol abuse were more likely to have memory problems up to two decades later.
The study, in The American Journal of Geriatric Psychiatry, adds to growing evidence that excessive drinking can impair mental processing later.
Researchers say it is a public health issue that needs to be addressed.
Scientists questioned 6,500 US middle-aged adults about their past alcohol consumption.
They were asked three specific questions:
Those who answered yes to one of these questions were considered to have a problem with alcohol.
They had more than double the risk of developing severe memory impairment, the study found.
"We know that alcohol is bad for the brain in general, but it's not just how much you drink but how it affects you," lead researcher, Dr Iain Lang, from the University of Exeter Medical School, told the BBC.
"The amount that you drink is important - what is also important is if you experience any problems in your drinking or if other people tell you you have a problem."
He advised drinking within recommended daily and weekly amounts and to cut down if affected by any of the items in the questionnaire, as this could increase dementia risk.
Dr Doug Brown, director of research and development at the Alzheimer's Society charity, said there was a hidden cost of alcohol abuse, given mounting evidence that alcohol misuse can impact on cognition later in life.
"This small study shows that people who admitted to alcohol abuse at some point in their lives were twice as likely to have severe memory problems, and as the research relied on self-reporting that number may be even higher.
"This isn't to say that people need to abstain from alcohol altogether. As well as eating a healthy diet, not smoking and maintaining a healthy weight, the odd glass of red wine could even help reduce your risk of developing dementia."
Dr Eric Karran, science director at Alzheimer's Research UK, said:  "Although studies such as this one can be very useful for observing health trends, it's important to note that they are not able to show cause and effect, and it's not clear whether other factors may also have influenced these results."
Problem drinking in middle age doubles the risk of memory loss in later life, research suggests.