Brownsville was founded by Irish settlers who had immigrated to Canada from Pennsylvania in the United States. It was named for its founder, businessman Thomas Brown (born 13 May 1802), who was one of twelve siblings born in Pennsylvania, and one of four who emigrated to Upper Canada. About 1830, his farmer brother John R. Brown (born 3 June 1811) settled on lot 26, concession 8, establishing the rural community. Thomas built the community's only flour mill in 1836, stimulating development. The mill was eventually bought by their brother Garrett, who also established the first bank in the community. For postal service, residents used the post office in the nearby community of Lloydtown.