His four children were as follows: Mitsuyoshi YAGYU (Jubei), the eldest son, well-known as a one-eyed swordsman; Tomonori YAGYU, who won Iemitsu's favor, and died young before his father; Munefuyu YAGYU, who succeeded to the post of sword instructor to the Shogun family in place of Mitsuyoshi who died soon after his father's death; Gisen RETSUDO, who became the first chief priest of Hotoku-ji Temple, a family temple of the Yagyu clan.
子には隻眼の剣士として有名な長男の柳生三厳（十兵衛）、家光の寵愛を受けたが父に先立って早世した柳生友矩、父の死後まもなく没した三厳に代わって将軍家師範役を継いだ柳生宗冬、菩提寺芳徳寺の第一世住持となった列堂義仙の4子が知られる。