Clearly seen from the river and aligned on the meridian for observation purposes, Building 20, also known as the Ball House, is the former observatory and time ball tower. It is a compact, tall, two story stone structure of several architectural shapes. The rectangular section with a gabled roof was the former observatory and the square section formerly housed the time ball installation topped with an antenna. The building is entered from the ground floor of its small porch, which joins the two buildings.