Spence was born in the Shankill Road, Belfast, Northern Ireland, the son of William Edward Spence, who was born in Whitehaven, England and raised in the Tiger's Bay area of north Belfast before moving to the Shankill. William Edward Spence was a member of the Ulster Volunteers and had fought in the First World War. He married Isabella "Bella" Hayes, Gusty Spence's mother, in 1919. Spence was the sixth of seven children, their birth order being Billy, Cassie, Jim, Bobby, Ned junior, Gusty and Lily. The family home was 66 Joseph Street in an area of the lower Shankill known colloquially as "the Hammer". He was educated at the Riddel School on Malvern Street and the Hemsworth Square school, both on the Shankill, finishing his education aged fourteen. He was also a member of the Church Lads' Brigade, a Church of Ireland group, and the Junior Orange Order. His family had a long tradition of Orange Order membership.