Many individuals are experiencing the potentially stressful combination of providing care while still employed. In this study, the association between unpaid caregiving to another adult and self-reported stress among men and women aged 45–74 is investigated, using nationally representative time use diary data for Sweden (2000-01 and 2010–11, N = 6689). Multivariate regression analyses established that women were overall more stressed than men with the largest gender stress gap observed among intensive caregivers, providing >60 min of daily care and employed caregivers. The association between unpaid caregiving, employment, and self-reported stress is gendered. Among men, there is no caregiver effect regarding stress, but for women there is a net effect of 6–9%. Combining employment and unpaid caregiving (especially if intensive) is stressful for women but not for men. There are two potential mechanisms for this: less time for leisure and sleep. Unpaid caregiving is positively associated with stress among women when seen in relation to the way caregivers trade off time, not least to aid their recovery. These findings provide a more nuanced understanding of the time trade-offs carers make and uncover gender differences in the association between caregiving and stress that add to an existing gender stress gap. Given that unpaid caregivers are an important source of long-term care services, policymakers should consider that caregiving may be stressful and that stress impacts are gendered when designing and evaluating policies for longer working lives.