The family originally served as a private guard of the Muromachi bakufu (Japanese feudal government headed by a shogun) for successive generations, but in the generation of Nagatsura INOO, they went down to Suruga and became a hereditary vassal of the Imagawa clan.
元々は室町幕府の相伴衆の家柄であったが、飯尾長連の代に駿河に下り、今川氏譜代の家臣となった。