A growing body of research suggests that social or physical activity can affect the risk of Major depressive disorder (MDD). However, the bidirectional relationship between them remains to be clarified further, especially between inactivity and MDD. Here, we performed a two-sample Mendelian Randomization analysis using genetic variants associated with social/physical activities and MDD, and assessed the mediating effect of obesity-related measures and brain imaging phenotypes. The dataset on MDD, social activities, and physical activities included 500,199; 461,369; 460,376 individuals, respectively. Information regarding body mass index (BMI), body fat percentage (BFP), IDPs for 454,633; 461,460; 8,428 participants, respectively. We identified bidirectional causal relationships between sport clubs or gyms, strenuous sports, heavy do-it-youself, other exercises and MDD. We also observed that leisure/social inactivity (odds ratio [OR] = 1.64;P= 5.14 × 10−5) or physical inactivity (OR = 3.67;P= 1.99 × 10−5) caused an increased risk of MDD, which were partially mediated by BMI or BFP and masked by the weighted-mean orientation dispersion index of left acoustic radiation or volume of right caudate. Furthermore, we discovered that MDD increased the risk of leisure/social inactivity (OR = 1.03;P= 9.89 × 10−4) or physical inactivity (OR = 1.01;P= 7.96 × 10−4). In conclusions, we found that social/physical activities reduced the risk of MDD, while MDD in turn hindered social/physical activities. Inactivity may increase the risk of MDD, which was mediated or masked by brain imaging phenotypes. These results help to understand the manifestations of MDD and provide evidence and direction for the advancement of intervention and prevention.