In 1784, he moved to Philadelphia to enter the dry goods business with his brother, Stephen. He then moved to Richmond, Virginia to open a second dry goods store. In 1785, he married into the affluent iron mining family of Mary Brown, who then became known as Mary Brown Austin. The Austins' second child was born in 1793 and named Stephen Fuller Austin in honor of his father's brother and his mother's great uncle. Their daughter Emily Austin followed in 1795. A second son, James Elijah Brown Austin, was born in 1803.