Maybe due to the issue of his mother's position in society, he was raised at the Karo's (chief retainer) house during his early years and was ultimately taken to the castle.
母の身分に問題があったためか、幼年は家老の元で育てられ、やがて城中へ引き取られた。