In 1652 Sylvester constructed a house on the island for his 17-year-old bride, Grissel (also spelled Grizzel) Brinley from London. Her mother was Anna (Wase) Brinley and her father Thomas Brinley had been an auditor in the court of King Charles I. With the Revolution he had lost his position; Grissel had gone to the colony with her older sister Anne, who had married William Coddington, the governor of the Rhode Island colony. Archeological research in the 21st century has revealed there may have been two early house complexes. The Sylvesters had eleven surviving children. The more elaborate manor house, which survives today, was built in 1733 by a Sylvester grandson.