Gordon was born in Woolwich, London, a son of Major General Henry William Gordon (1786–1865) and Elizabeth (Enderby) Gordon (1792–1873). The men of the Gordon family had served as officers in the British Army for four generations, and as a son of a general, Gordon was brought up to be the fifth generation; the possibility that Gordon would pursue anything other than a military career seems never to have been considered by his parents. All of Gordon's brothers also became Army officers.