Social buffering, a phenomenon where social presence can reduce anxiety and fear-related autonomic responses, has been studied in numerous laboratory settings. The results suggest that the familiarity of the interaction partner influences social buffering while also providing some evidence for gender effects. In the laboratory, however, it is difficult to mimic the complexity of real-life social interactions. Consequently, the social modulation of anxiety and related autonomic responses in everyday life remains poorly understood. We used smartphone-based Ecological Momentary Assessment (EMA) combined with wearable electrocardiogram sensors to investigate how everyday-life social interactions affect state anxiety and related cardiac changes in women and men. On five consecutive days, 96 healthy young participants (53% women) answered up to six EMA surveys per day, indicating characteristics of their most recent social interaction and the respective interaction partner(s). In women, our results showed lower heart rate in the presence of a male interaction partner. Men showed the same effect with female interaction partners. Moreover, only women showed decreased heart rate and increased heart rate variability with increasing interaction partner familiarity. These findings specify the conditions under which social interactions reduce anxiety-related responses in women and men.