To quantify the subjective and cognitive impairment caused by wearing face masks at work, 20 men and 20 women (median age 47 years, range 19–65) were tested under different ergometer workloads while wearing surgical mask, community mask, FFP2 respirator or no mask in a randomized and partially double-blinded design. Masks were worn also at the workplace for four hours. Subjective impairment was recorded by questionnaires. Cognitive performance was tested before and after the workplace examination. Subjective feeling of heat, humidity, and difficult breathing increased with rising physical exertion and wearing time for all three mask types, most notably for FFP2. Even when blinded, participants with FFP2 reported difficult breathing already at rest. During physical exertion, individuals with low tolerance to discomfort reported significantly stronger impairment (OR 1.14, 95% CI 1.02–1.27). Regarding light work, older subjects (OR 0.95, 95% CI 0.92–0.98) and women (OR 0.84, 95% CI 0.72–0.99) showed significantly lower and atopic subjects stronger impairment (OR 1.16, 95% CI 1.06–1.27). No significant influence of mask wearing was detected on cognitive performance. Wearing a mask had no effect on cognitive performance, but led to discomfort which increased with physical exertion and wearing time. Individuals who tolerate discomfort poorly felt more impaired by wearing a mask during physical exertion.