Morris retired from the British army in 1764 and settled in New York City on the southeast corner of Whitehall and Stone streets with his American wife, Mary Philipse, nicknamed "Charming Polly", whom he had married in 1758. Middle daughter of Frederick Philipse, second Lord of the Philipsburg Manor, she had been a possible love interest of George Washington, and owned a one-third share of the Philipse Patent, a vast landed estate on the Hudson River which later became Putnam County, New York.