He was disinherited because he criticized his father Arishige KANAMORI and others who supported the Tokugawa clan at the Siege of Osaka in 1614, and began to live in seclusion with his mother (daughter of Yoshitaka ENDO) in Kyoto.
慶長19年（1614年）大坂の役で徳川方につく父可重らを批判したことで廃嫡され、母（遠藤慶隆娘）と供に京都に隠棲する。