The first structures on the site of Morven Park date to about 1780. A fieldstone house built by Wilson Cary Seldon is now a part of the north wing of the main house, stuccoed over to match the rest of the mansion. Judge Thomas Swann acquired the property around 1800. Around 1830 Swann built the center two-story portion of the house, with flanking pavilions. It is not known whether the pavilions were initially linked to the house, but the renovations included the prominent tetrastyle Greek Revival portico that dominates the front portico. While the brick structure remains, now stuccoed, none of the Swann interiors exist. Judge Swann's son, Thomas Swann Jr., began a remodeling program around 1850, using the Baltimore firm of E.G. Lind and William T. Murdock as architects, converting the Palladian house to the Italianate style with four towers, including one that was five stories tall, stated by "The Buildings of Virginia" to resemble Queen Victoria's Osborne House on the Isle of Wight. The pavilions were by this time linked to the main house. The towers were later removed by Thomas Swann Jr.'s daughter, Mary Mercer Swann Carter, and her husband, Dr. Shirley Carter. They made more changes, raising the height of the hyphens to two stories and reworking the interior.