The bioactive compounds of coffee are involved in lipid metabolism, and sex differences may play an important role. This study aimed to evaluate the influence of sex differences on serum lipid profiles among habitual coffee drinkers. We conducted a nationwide cross-sectional study of 23,628 adults using data obtained from the Taiwan Biobank database. Adults who drank more than one cup of coffee per day and those who drank less than one cup per day were compared with non-drinkers. After adjusting for baseline demographics and lifestyle, a generalized linear model was used to estimate the change in serum lipid profiles in men and women and in postmenopausal and premenopausal women among different coffee-drinking behaviors. We found that habitual coffee consumption changed the serum lipid profiles of men and women. Further, coffee drinkers had higher serum total cholesterol, low-density lipoprotein cholesterol, high-density lipoprotein cholesterol, and lower serum triglyceride levels than non-drinkers. Compared with premenopausal women, both men and postmenopausal women had increased serum total cholesterol and low-density lipoprotein cholesterol levels. Menopausal status may play an important role in modulating the effect of habitual coffee intake on dyslipidemia. Moreover, premenopausal women potentially benefit more from habitual coffee drinking than men and postmenopausal women.