It is said that the clan descended from Ameno Oshihi-no-Mikoto, who served as a guide in Tensonkorin (the descent to Earth of the grandson of the Sun Goddess) and had a kindred relationship with the Saeki clan (the Saeki clan was generally regarded as a branch family of the Otomo clan, though some say the opposite is true).
天孫降臨の時に先導を行った天忍日命（あめのおしひのみこと）の子孫とされ、佐伯氏とは同族関係であるとされている（一般には佐伯氏を大伴氏の分家とするが、その逆とする説もある）。