The association between cognitive flexibility and related neural functioning has been inconsistent. This is particularly true in young children, where previous studies have found heterogenous results linking behavior and neural function, raising the possibility of unexplored moderators. The current study explored the moderating role of dimensional irritability in the association between cognitive flexibility task performance and prefrontal activation in young children. A total of 106 3- to 7-year-old children were recruited to complete a custom-designed, child-adapted, cognitive flexibility task, and 98 of them were included in the data analysis. The children’s dorsolateral prefrontal cortex activation was monitored using functional near-infrared spectroscopy, and their levels of irritability were reported by parents using the MAP-DB Temper Loss subscale. Results indicated that the mean reaction time of the cognitive flexibility task was negatively correlated with concurrent prefrontal activation. No evidence was found for the association between task accuracy and prefrontal activation. Moreover, irritability moderated the association between the mean reaction time and prefrontal activation. Children high in irritability exhibited a stronger negative association between the mean reaction time and related prefrontal activation than children low in irritability. The moderating model suggested a novel affective–cognitive interaction to investigate the associations between cognitive task performance and their neural underpinnings.