From 1942 to 1947 Phillips attended Bonneville Road Primary School in Clapham. Whilst he was there he says that he "learned the word artist and discovered that an artist is someone who does not have to put his paints away, so decided to become one". Although he enjoyed school he was noted for his fascination with drawing and his refusal to conform. His mother recalled him buying a platform ticket every Sunday and taking long railway journeys when he was just eleven. In that year he progressed to Henry Thornton Grammar School, Clapham, where he developed his love of music, playing violin and bassoon in the school orchestra and singing solo baritone in school concerts and stage events. In 1954 he exhibited paintings for the first time, in an open art show on the railings of the Thames Embankment. A year later, at seventeen, he won a traveling scholarship to France and lived there for three months. His mother remembers him returning to London with a sack of horse bones from the first World War, but more significantly he bought himself a piano and started to teach himself to play. In 1957 he became a founder member of the Philharmonia Chorus.