Lady Murasaki is deeply concerned with this, and while Genji himself can't feel affection for the third princess because she is a mere child, he also can't turn down his older brother and the Emperor's offer.
紫の上の憂慮はひとかたならず、源氏自身もほんの少女にすぎない彼女に対して愛情を感じられないが、兄帝の願いを無下には断れない。