The main fortress within Hay-on-Wye was situated on the great site commanding the town and river under the current ruins of the castle and mansion. This was undoubtedly the 'castello de haia' handed to Miles of Gloucester, 1st Earl of Hereford, in 1121 with Sibyl de Neufmarché, the daughter of Bernard de Neufmarché. It is most likely that the keep stood by this time. It is therefore possible that this is the oldest Norman tower in Wales, dating to the onslaught of William Fitz Osbern in 1070. During the anarchy (1136–54) in the reign of King Stephen a series of charters were passed by the Gloucesters concerning the castle. In 1165 the last of Miles of Gloucester's male descendants was killed at nearby Bronllys Castle and Hay-on-Wye Castle passed into the hands of William de Braose, 3rd Lord of Bramber and of New Radnor and Buellt. The de Braose dynasty were energetic lords and probably built the core of the gatehouse which now stands besides the keep. In the summer of 1198 a major English army formed here before marching off to victory at the Battle of Painscastle some four miles to the north.