James Montgomery was born on December 22, 1814 in Austinburg, Ashtabula County, Ohio; his parents were James and Mary Baldwin Montgomery. He moved to Kentucky in 1837 with his parents, and he taught school there. He married, but his first wife died shortly after the wedding, so he married again to Clarinda Evans. They moved to Pike County, Missouri in 1852, and then to Jackson County, and finally to Bates County while waiting for Kansas to be open to settlement.