Background: Physical activity is important for children with congenital heart defects (CHD), not only for somatic health, but also for neurologic, emotional, and psychosocial development. Swimming is a popular endurance sport which is in general suitable for most children with CHD. Since we have previously shown that children with CHD are less frequently physically active than their healthy peers, we hypothesized that the prevalence of non-swimmers is higher in CHD patients than in healthy children. Methods: To obtain representative data, we performed a nationwide survey in collaboration with the German National Register of Congenital Heart Defects (NRCHD) and the Institute for Sport Sciences of the Karlsruhe Institute for Technology (KIT). The questionnaire included questions capturing the prevalence of swimming skills and the timing of swim learning and was part of the “Motorik-Modul” (MoMo) from the German Health Interview and Examination Survey for Children and Adolescents (KiGGS). A representative age-matched subset of 4569 participants of the MoMo wave two study served as a healthy control group. Results: From 894 CHD-patients (mean age of 12.5 ± 3.1 years), the proportion of non-swimmers in children with CHD was significantly higher (16% versus 4.3%;p< 0.001) compared to healthy children and was dependent on CHD severity: Children with complex CHD had an almost five-fold increased risk (20.4%) of being unable to swim, whereas in children with simple CHD, the ability to swim did not differ significantly from their healthy reference group (5.6% vs. 4.3% non-swimmers (p= not significant). Conclusions: According to our results, one in five patients with complex CHD are non-swimmers, a situation that is concerning in regard of motoric development, inclusion and integration, as well as prevention of drowning accidents. Implementation of swim learning interventions for children with CHD would be a reasonable approach.