When the Lillys purchased the property in 1933 the gardens and grounds were just beginning to mature. The family left the gardens largely unaltered other than the addition of several sculptures throughout. Lilly also constructed a number of new buildings on the grounds, expanding the greenhouse significantly. In 1939 Newfield was constructed as a residence for Joe Lilly and his family. In 1940 a recreation building was built in the south pasture that housed an indoor pool, game room, and exercise room. Outside of the recreation building was an outdoor pool, tennis court, and Four Seasons garden.