In the 14th century, the Fleming family were given lands in the area by Robert the Bruce, whose cause they had supported. The Flemings built Boghall Castle, visible as a ruin until the early 20th century, but now only represented by a few mounds. The town continued to grow as an important market town, and in 1451 the town became a burgh. The market place remains the central focus of the town. The kirk was rebuilt as a Collegiate church in 1546 for Malcolm, 3rd Lord Fleming, the last to be established before the Reformation of 1560. The Flemings found themselves on the wrong side in the 16th century, when they supported Mary, Queen of Scots. Their lands remained in the Fleming family until the 18th century when the male line of succession ended. The lands passed into the Elphinstone family in 1735 on the marriage of the heiress Lady Clementina Fleming to Charles, Lord Elphinstone.