Brown began construction of a home on the property shortly after purchasing it, though he was often away on government business in Philadelphia, then the capital of the United States. The architect of Liberty Hall is unknown, although Brown may have done some of the design. One of the earliest brick homes in Frankfort, this structure was made from bricks fired locally from clay dug from the cellar. The construction continued until 1800 when the house was substantially complete. It lacked only the glass windows, which were added in 1804. In addition to the main house, several dependent structures were built on the property, including a kitchen and laundry, smokehouse, a privy, stables, carriage house, and slave quarters.