The site was originally part of the "Outeiro das Vinhas", a property that fronted the beach of the Tagus River. D. Manuel of Portugal, a diplomat and poet who was the son of the 1st Count of Vimioso, acquired the land in 1559, naming it "Quinta de Belém" and constructing a building with three salons and two atria. By the mid-17th century the property was linked to a scion of the Royal Court, then transferred to the possession of the Counts of Aveiras and occupied by a convent.