Fisher was born at the rectory, Higham on the Hill, Leicestershire, youngest of the ten children of the Rev Henry Fisher and his wife Katherine, "née" Richmond. A Fisher had served as rector of Higham since 1772: Henry Fisher's father and grandfather had preceded him; his eldest son, Legh, later held the post. After a short time in the Higham village school Fisher was sent to Lindley Lodge, a local preparatory school and in September 1901, having gained a scholarship, he entered Marlborough College, a public school in Wiltshire. He was greatly influenced by the headmaster, Frank Fletcher, an inspiring teacher, under whose guidance he did well both academically and in sport.