The position was succeeded by Toyomune GOMI again, however, after his death in 1680, it was succeeded by Masanori KOBORI, and since then by the Kobori clan hereditarily.
その後、再び五味豊旨が継いだが、延宝8年（1680年）に五味豊旨の死去によって就任した小堀正憲以後は小堀氏による世襲となった。