His five sons, such as the heir Yorimoto, who had been Izumo no kami (chief of the Izumo Province), were deprived of their Samurai ranks: Izumo no kami Yorimoto and the third son Iori were forced to forfeit their ranks and properties while the fifth son Kumazo, the sixth son Takekuro as well as the seventh son Mitsuyoshi were left in the care of relatives until they became 15.
嫡子出雲守頼元をはじめ男子5人は士籍を剥奪され、出雲守頼元、三男伊織は改易、五男熊蔵、六男武九郎、七男満吉は15歳まで縁者に預けられた。