The estate was inherited by his eldest son Jan Baptist, who acquired the title of patroon. He died in 1658 and his younger brother Jeremias van Rensselaer became patroon. Acknowledging the surrender of New Amsterdam and Fort Orange to England in 1664 following a surprise incursion by the English during a time of peace (which led to the Second Anglo–Dutch War), Jeremias took the oath of allegiance to the King of England that October. In 1666, he also built the original Manor House, also known as Fort Crailo for its defensive reinforcement, located north of Fort Orange. The Manor House was the seat of the patroonship and the home of the patroon until 1765.