George Harrison began writing "Within You Without You" in early 1967 while at the house of musician and artist Klaus Voormann, in the north London suburb of Hampstead. Harrison's immediate inspiration for the song came from a conversation they had shared over dinner about the metaphysical space that prevents individuals from recognising the natural forces uniting the world. Following this discussion, Harrison worked out the song's melody on a harmonium and came up with the opening line: "We were talking about the space between us all".