Background: Studies exploring the possible protective effect of coffee and tea consumption on dementia have shown inconsistent results so far. We aimed to investigate whether consumption of tea and different types of coffee at midlife are associated with dementia later in life and whether sex or ApoE4 influence such association. Methods: We included 7381 participants from the Norwegian HUNT Study. Self-reported questionnaires assessed daily consumption of coffee and tea at baseline. After 22 years, individuals 70 years or older were screened for cognitive impairment. Results: General coffee consumption and tea consumption was not associated with dementia risk. Compared to daily consumption of 0–1 cups of coffee, daily consumption of ≥8 cups of boiled coffee was associated with increased dementia risk in women (OR: 1.83, 95% CI: 1.10–3.04,p-value for trend = 0.03) and daily consumption of 4–5 cups of other types of coffee was associated with a decrease in dementia risk in men (OR: 0.48, 95% CI: 0.32–0.72,p-value for trend = 0.05). Furthermore, the association between boiled coffee and increased dementia risk was only found in ApoE4 non-carriers. Differences by sex or ApoE4 carrier status were not supported by strong statistical evidence for interaction. Tea consumption was not associated with dementia risk. Conclusion: type of coffee may play a role in the direction of the association between coffee-drinking habits and dementia later in life.