Stewart was raised in Drummoyne, in the western suburbs of Sydney. He came from a comfortable middle-class background, and his father, employed as a health inspector, had a keen interest in Asia. Stewart displayed early promise as a poet after enrolling at Fort Street High School at the age of fifteen in 1932. Before attending Fort Street he studied the trumpet at the Sydney Conservatorium High School. A subtitle honouring Claude Debussy in 'Prelude: On the Quay,' written in the last year of high school, demonstrates that music was a formative poetic influence and one that provided a sense of organisation for his later poetry, which is most apparent in the fugue-like thematic structure of his spiritual autobiography "By the Old Walls of Kyoto". The reference to Debussy also points to the significant influence the French Symbolists had on shaping the affective Gothic mood of his early poetry.