The site today covers , which includes the original tract of Jackson's land. It is overseen and managed by The Andrew Jackson Foundation, formerly called the Ladies' Hermitage Association. The mansion is approached by a cedar-lined, wide, guitar-shaped carriage drive designed by Ralph E. W. Earl. The design made it easier to maneuver carriages in the narrow space. To the east of the house was a formal garden designed by Philadelphia-based gardener William Frost in 1819. Laid out in the English four-square kitchen garden style, it consists of four quadrants and a circular center bed contained by unusually long, beveled bricks and pebbled pathways. Originally, the garden was primarily used to produce food for the mansion and secondarily as an ornamental pleasure garden. It is surrounded by a white picket fence. On the north perimeter stands a brick privy that served as a status symbol as well as a garden feature.