The history of the present-day property on the famous "Gold Coast" of Long Island began between 1904 and 1912, when Helen MacGregor Byrne – wife of New York City lawyer James Byrne – purchased six farming properties which she collectively referred to as "Upper Planting Fields Farm". The Byrnes hired landscape architect James Leal Greenleaf between 1904 and 1910 to create hedges, perennial borders, and espaliered fruit trees. Though notable features from this period are the Rose Arbor, the Circular Pool and the Green Garden Court, Coe recalled in later life, "Mrs Byrne had done very little in the shape of landscaping. She had a small lawn around the ridge of the residence" and next to it a cornfield. Most of the property, Coe recalled, was "just a jungle of scrub, locusts, and other trees".