Although the association between cognitive performances and the onset of psychiatric disorders has been widely investigated, limited research on the role of childhood trauma or early life stress (CT/ELS), and whether this role differs between clinical and non-clinical cohorts is available. This systematic review aims at filling this gap, testing whether the occurrence of CT/ELS and its subtypes are associated with cognitive domains (general cognitive ability, executive functions, working memory, attention, processing speed, verbal/visual memory) in patients with psychiatric disorders and in non-clinical populations. This study followed the PRISMA 2020 guidelines and the Newcastle-Ottawa scale for quality assessment. The search was performed until May 2022. Seventy-four studies were classified as eligible. The graphical representations of the results reported an association between exposure to CT/ELS and worse general cognitive ability, verbal/visual memory, processing speed and attention in patients affected by anxiety, mood and psychotic disorders, and that specific CT/ELS subtypes (physical neglect, physical/sexual abuse) can differentially influence specific cognitive abilities (executive functions, attention, working memory, verbal/visual memory). In non-clinical cohorts we found associations between CT/ELS exposure and impairments in executive functions, processing speed and working memory, while physical neglect was related to general cognitive ability and working memory. Concerning the emotional abuse/neglect subtypes in both populations, the results indicated their involvement in cognitive functioning; however, the few studies conducted are not enough to reach definitive conclusions. These findings suggest an association of CT/ELS with specific cognitive deficits and psychopathology.