PurposeWe investigated longitudinal associations of sedentary behavior, light-intensity physical activity (LPA) and moderate-to-vigorous physical activity (MVPA) with body composition in colorectal cancer (CRC) survivors, between 6 weeks and 24 months post treatment. In addition, we explored whether body composition mediated associations of sedentary behavior and MVPA with fatigue.MethodsA prospective cohort study was conducted in 459 stage I–III CRC patients recruited at diagnosis. Measurements were performed of accelerometer-assessed sedentary time (hours/day), self-reported LPA and MVPA (hours/week), anthropometric assessment of body mass index (BMI), waist circumference and fat percentage (measures of adiposity), and muscle circumference and handgrip strength (measures of muscle mass/function) repeated at 6 weeks, and 6, 12 and 24 months post treatment. Longitudinal associations of sedentary time and physical activity with body composition were analyzed using confounder-adjusted linear mixed models. Mediation analyses were performed to explore the role of body mass index (BMI) and handgrip strength as mediators in associations of sedentary time and MVPA with fatigue.ResultsLess sedentary time and LPA were, independent of MVPA, longitudinally associated with increased handgrip strength, but not with measures of adiposity. More MVPA was associated with increased adiposity and increased handgrip strength. Higher BMI partly mediated associations between higher sedentary time and more fatigue.ConclusionWithin the first two years after CRC treatment, changes in sedentary behavior, physical activity and body composition are interrelated and associated with fatigue. Intervention studies are warranted to investigate causality.Trial registrationThe EnCoRe study is registered at trialregister.nl as NL6904 (former ID: NTR7099).Supplementary InformationThe online version contains supplementary material available at 10.1007/s00432-022-04267-9.