Since then, the Toyama clan had been regarded as one of Goyuishoke no rokke (the six families of sworn friends who supported Soun HOJO), as well as, one of Izushu nijukke (the twenty groups of families under the power of the Hojo clan), and enjoyed an important position together with the Matsuda clan and the Daidoji clan (they are also called Sankaroke (the big three chief retainer families of the Hojo clan).
以降遠山氏は後北条氏家中に於いて、御由緒家の六家および伊豆衆二十家の一つと数えられ、中でも松田氏・大道寺氏と並ぶ重きを成した(三家老家とも言われる)。