In 1898 Palmer purchased the hereditary farm holdings of the Lamson family, some . This would become the estate in which he resided, known as "Willow Dale". The mansion was built in 1902 by Charles K. Cummings, who described the original structure as having, a rather unusual arrangement, the master’s house, the quarters for a farmer or caretaker, and the stable, all being joined together under one roof. This was done with a view to economy of construction and management; and especially during the winter months this grouping together of quarters, which more often appear as separate buildings, has been found convenient and agreeable In 1915, Palmer renovated the mansion, moving the stables and coach house to separate buildings and replacing them with a dining room and ballroom respectively. He also added an addition for a screened patio, guest rooms, and servants quarters. This ultimately gave Willowdale the form it has today.