The spot for the palace, a large lot of land on the Vistula escarpment directly below the Nowy Świat, was bought by Prince Janusz Ostrogski in early 17th century. As the area had been still a suburb of Warsaw and exempted from the laws of the city which prevented the inhabitants from building private fortifications, Ostrogski decided to build a small castle there. For that he financed a bastion on which the manor was to be constructed. However, it was not until after his death that the manor itself was started. Designed by Tylman of Gameren, the palace built on top of the bastion was to become one of the wings of a huge future palace. However, it was never completed and was bought by deputy chancellor of the crown Jan Gniński, who turned it into his seat.