Montgomery was born at Irvine in south-west Scotland, the son of a pastor and missionary of the Moravian Brethren. He was sent to be trained for the ministry at the Moravian School at Fulneck, near Leeds, while his parents left for the West Indies, where both died within a year of each other. At Fulneck, secular studies were banned, but James nevertheless found means of borrowing and reading a good deal of poetry and made ambitious plans to write epics of his own.