The family was started when Okura-kyo (Minister of the Treasury) Nobutoyo KAJUJI, a biological child of Ukyo no suke (Assistant Master of the Western Capital Offices) Okiyasu ABE (who was the fifth son of Haretoyo KAJUJI, a Jun-daijin (Vice Minister)) was adopted by his uncle Gon Dainagon (provisional major counselor) Mitsutoyo KAJUJI and was named SHIBAYAMA.
右京亮・阿部致康（准大臣・勧修寺晴豊の五男）の実子である大蔵卿・勧修寺宣豊が、叔父である権大納言・勧修寺光豊の猶子となり、芝山を称したのにはじまる。