In the 12th century Rugby was mentioned as having a castle. However the 'castle' was probably little more than a defended manor house. In any event the 'castle' was short lived. It was probably constructed during the reign of King Stephen (1135–1154), and demolished in around 1157 on the orders of King Henry II. According to local legend, the stones from the castle were used to construct the west tower of St Andrew's church, which bears strong resemblance to a castle.