The patriarch of the Matsui clan was MINAMOTO no Koreyoshi (Matsui kanja) who was a son of MINAMOTO no Tameyoshi of Seiwa-Genji (Minamoto clan), a Rokujo Hankan (the Rokujo office of Judicial Chief) and the clan was living in Yamashiro Province (it is said that their origin was Saiin Matsui, Kadono no kori, Yamashiro Province) as Gokenin (a lower-ranking vassal in the Kamakura and Edo periods) worked for the bakufu (Japanese feudal government headed by a shogun) during the Kamakura period.
松井氏は清和源氏六条判官源為義の子松井冠者源維義を祖に持ち、鎌倉時代は幕府の御家人として山城国に住んだ一族（山城国葛野郡西院松井に発すと云う）。