Woolf was born in Newcastle-upon-Tyne, England, on 2 May 1933, to Alexander Susman Woolf and his wife Leah (née Cussins). His grandfather Harry was a naturalised Briton of Polish and Russian Jewish origins. His father had been a fine art dealer, but was persuaded to run his own building business instead by his wife. They had four children, but their first child died, and his mother was protective of the three surviving children. Woolf lived in Newcastle-upon-Tyne until he was about five years old, when his family moved to Glasgow, Scotland, where he attended Glasgow Academy going on to Fettes College, an Edinburgh public school, where he mostly enjoyed his time and had supportive friends.