The temple is believed to have been the place in which Imperial Prince Sawara was confined after becoming suspected as being the mastermind of FUJIWARA no Tanetsugu's assassination and Kukai is also said to have briefly resided here.
また藤原種継暗殺事件の首謀者の嫌疑をかけられた早良親王が幽閉された場所でもあるとされ、空海が一時住したともいわれる。