After leaving Oxford in 1922 Armstrong briefly took up an appointment at Manchester Cathedral (assistant organist), during which time he worked with Hamilton Harty. The following year he was appointed organist of St. Peter's Church, Eaton Square, London. During this period he studied at the Royal College of Music with Holst and Vaughan Williams. In 1926 Armstrong married Hester Draper, daughter of the Rev W H Draper, Master of the Temple. They were married for 52 years and had two children: Robert (later Baron Armstrong of Ilminster) (1927-2020) and Helen Hilary (b. 1930).