Neumann's father, Philip Neumann, a stocking knitter from Obernburg am Main, moved to Prachatitz in the Kingdom of Bohemia (then part of the Austrian Empire, now in the Czech Republic) in 1802 at age 28 with his wife, Antonia Strakotinskou. Antonia died in November 1804, together with the child she bore. He married the daughter of a Czech harness maker, Agnes Lepší, on July 17, 1805, and Neumann was the third of their six children: Catherine, Veronica, John, Joan, Louise, and Wenceslaus (or Wenzel).