In 1660, however, as a result of taking an opposing position towards roju Nobutsuna MATSUDAIRA, also known as "Chie-Izu" (wisdom of Izu), Masanobu was sanctioned with a kaieki (dismissal and deprivation of position, privileges and properties) and his territory was forfeit.
しかし正信は万治3年（1660年）に老中・松平信綱こと『知恵伊豆』と対立したため、所領を没収され、改易となった。