In October 1513, Cholmeley was appointed Lieutenant of the Tower of London and Supervisor General of Richmond Castle and eleven other Yorkshire castles and manors. There was no suitable house for Cholmeley and his family within the Tower precincts, and so he purchased a house in nearby Barking, where he lived while serving as Lieutenant of the Tower. The Lieutenant served under the Constable of the Tower, at the time Thomas Lovell. The duties of the Lieutenant included defence, maintenance, and custody of the prisoners at the Tower, including escorting prisoners of note to trial at Westminster Hall which, at that time, housed the courts of law. Cholmeley's maintenance works included the complete rebuilding of the chapel of St Peter ad Vincula, the parish church of the Tower of London, which had been largely destroyed by fire in 1512. The construction was carried out in 1519–20.