For its first twenty years, the "Banking House" was a three-story brick building formerly occupied by a hat and bonnet maker (the vault was an addition in the back yard) and by 1833, it had become completely inadequate for its purpose. After a burglar took $3,995 in gold and silver one night, an inspection found the building "in a decayed state and in many parts quite insecure." Despite some hasty repairs, the bank outgrew the building within two years, temporarily expanded into the house next door and began to buy land on the opposite side of Third Street. Unfortunately, it was only able to obtain two adjacent lots, resulting in a narrow site flanked by higher buildings on both sides.