Soon after his father Munenori died in 1646, his oldest brother Mitsutoshi YAGYU (Jubei) who had inherited most of their father's property died young in 1650, and the second older brother Munefuyu YAGYU succeeded to the Yagyu family.
1646年に父・宗矩が亡くなって間もなく、1650年には父の遺領の大半を相続していた長兄の柳生三厳（十兵衛）が若くして没し、次兄の柳生宗冬が代わって柳生家を継承する。