The priory dates back to around the year 1100, when Norman nobleman Walter de Lacy reputedly came upon a ruined chapel of St. David in this location, and was inspired to devote himself to solitary prayer and study. He was joined by Ersinius, a former Chaplain to Queen Matilda, the wife of King Henry I, and then a band of followers. A church was built on the site, dedicated to St John the Baptist, and consecrated in 1108. By 1118, a group of around 40 monks from England founded there a priory of Canons Regular, the first in Wales.