Mathews was raised on a dairy farm near the unincorporated community of Auroraville, Wisconsin in Waushara County, Wisconsin. As a youth, he had little interest in farm life or in the cheese factory operated by his older brother. His parents sent Joseph to public school in nearby Berlin, a city that straddles the boundary between Waushara and Green Lake counties. He graduated from Omro High School (Omro, Winnebago County). Mathews studied briefly at Ripon College in Ripon, Wisconsin, and then transferred to the University of Wisconsin. Mathews earned the B.S. in chemistry in 1903, writing a senior thesis on nitrosyl selenic acid, based on research under the supervision of Victor Lenher (1873–1927).