When he was at the advanced age of 62 years, after the premature death of his first son, Naotsune, his second son, Naonori, was born in 1653, and Naoyoshi passed away while Naonori was still a child.
承応3年（1653年）に長男・直常が早世した後、62歳という高齢になってようやく次男・直矩が生まれたが、直矩が幼いうちに死去。