The construction of Shan Ha Wai was begun in 1847 by the wealthy granite merchant and stonemason Tsang Koon-Man as a stronghold for the Tsang clan, and took around 20 years to complete. The original granite, bricks and solid timber are still preserved today. The complex is rectangular and consists of three rows of houses enclosed by grey brick walls. The four corners each feature a three-story guard tower with openings in the wall through which guns could be fired back in the early days of the complex's role as a defensive fortification. All the houses are interlinked by passages and small courtyards. In the innermost row of houses is an ancestral hall where meetings and ceremonies took place. There are three arched entrances in the northern wall. The main or ceremonial entrance is in the middle and leads to the ancestral hall. In front of the main building is a massive courtyard, where villagers traditionally winnowed and dried their harvest. It also contained a study hall where the Tsang children were tutored before schools existed.