Gordon was born in London in 1833, where his father was a major in the Royal Artillery. While still a very small child the family moved to Dublin so his father could work on modernising the guns at the Pigeon House Fort. In 1837 they moved to Edinburgh when his father was posted to Leith Fort. The family were then sent to the island of Corfu where his father had to arrange for the guns to be placed in new positions. Gordon 's parents wanted him to have an English education so he was sent to Fullands School at Taunton. When he turned 14 he attended the Royal Military Academy at Woolwich. He was often bullied, and he would become quiet and withdrawn. He go for several days without speaking to anyone. He would stand still and stare at the bullies until they felt uncomfortable and stopped. Gordon was accused of having bullied other students, and was punished by having to stay at the college for another term. He passed his exams and joined the Royal Engineers on the 23 June 1852.