In its compound of about 10,000 square meters, there stand two storerooms of books that were built when this library was founded, its access office, and Kozanso villa of Sukiya-zukuri style (built in the style of a tea-ceremony house) was built in 1944.
約10,000㎡の敷地内には、文庫設立以来の建物である書庫2棟、閲覧事務所のほか、昭和19年（1944年）に建てられた数寄屋造の虎山荘が建つ。