The castle was designed by the architects Calvert Vaux and Jacob Wrey Mould as an additional feature of the Greensward Plan, created by Vaux and Frederick Law Olmsted. Olmsted and Vaux were re-hired to their positions in mid-1865 after quitting abruptly several years before. In 1867, Vaux decided to develop this area by building Belvedere Castle on the top of the rock, overlooking the Croton Reservoir. The site already held a fire tower under the control of the Croton Aqueduct board, and so the fire tower was demolished.