The origin of the library is believed to be the archives consisting chiefly of old and contemporary Chinese literature that Yakatsugu, in the late part of his life, built in part of his estate as part of its renovation as Ashuku-ji Temple and that was made open to viewing to interested persons.
晩年、自分の邸宅を阿閦寺（あしゅくじ）として改築した際に敷地の一郭に古今の漢籍を中心とした書籍を収蔵し希望者に閲覧を許可したのが始まりとされている。