In the center of Bury St Edmunds lie the remains of an abbey , surrounded by the Abbey Gardens , a park .
Bury St Edmunds Cathedral was created when the Diocese of St Edmundsbury and Ipswich was formed in 1914 .