The entire area is the site of the former Toba Rikyu Villa that served as a center of imperial cloistered rule at the end of the Heian period (11th-12th centuries), and Anrakujuin Temple is the successor of a Buddha hall that was built within the villa.
付近一帯は平安時代末期（11～12世紀）、院政の舞台となった鳥羽離宮の跡地であり、安楽寿院は離宮内に営まれた仏堂の後身である。