Armour was born in Stockbridge, New York to Danforth Armour and Juliana Ann Brooks. He was one of eight children and grew up on his family's farm. Armour was descended from colonial settlers of Scottish and English origin, with his surname originating in Scotland. He was educated at Cazenovia Academy in New York until the school expelled him for taking a ride in a buggy. Among his first jobs was that of Driver on upstate New York's Chenango Canal which ran through Madison County at that time and would have been a busy thoroughfare. At the age of 19, Armour left New York with about 30 other people for California, joining the great California gold rush. Before the journey, Armour "had received several hundred dollars from his parents," making him, for the most part, "the financier of the party," according to biographer Edward N. Wentworth. In California, Armour eventually started his own business, employing out-of-work miners to construct sluices, which controlled the waters that flowed through the mined rivers. In only a few years, Armour had turned his business into a profitable enterprise, earning himself about $8,000 by the time he had turned 24.