In utero, the developing brain is highly susceptible to the environment. For example, adverse maternal experiences during the prenatal period are associated with outcomes such as altered neurodevelopment and emotion dysregulation. Yet, the underlying biological mechanisms remain unclear. Here, we investigate whether the function of a network of genes co-expressed with the serotonin transporter in the amygdala moderates the impact of prenatal maternal adversity on the structure of the orbitofrontal cortex (OFC) in middle childhood and/or the degree of temperamental inhibition exhibited in toddlerhood. T1-weighted structural MRI scans were acquired from children aged 6–12 years. A cumulative maternal adversity score was used to conceptualize prenatal adversity and a co-expression based polygenic risk score (ePRS) was generated. Behavioural inhibition at 18 months was assessed using the Early Childhood Behaviour Questionnaire (ECBQ). Our results indicate that in the presence of a low functioning serotonin transporter gene network in the amygdala, higher levels of prenatal adversity are associated with greater right OFC thickness at 6–12 years old. The interaction also predicts temperamental inhibition at 18 months. Ultimately, we identified important biological processes and structural modifications that may underlie the link between early adversity and future deviations in cognitive, behavioural, and emotional development.