When Junkei TSUTSUI died of disease and Sadatsugu TSUTSUI, his legitimate son, succeeded to the estate in 1584, Fujimasa started to serve for Sadatsugu but his uncle, Tonaga, resigned and left the Tsutsui family.
1584年（天正12年）、主君筒井順慶が病死し、嫡子筒井定次が家督を継ぐとこれに仕えるが伯父遠長は役を辞して筒井家を去る。