Jay was born in Plymouth, England, the son of two folk musicians, his mother Hilary from Newcastle upon Tyne and his father Tony from Wolverhampton. The family moved to Abergavenny, South Wales when he was ten due to his civil engineering father's work. Jay played bass in his parents folk band with his brother Eddy, and began song writing his own material at the age of 15. At the time his primary influences included The Beatles, Queen, David Bowie and The Beach Boys. Commenting on the music scene in Abergavenny, Jay commented that "I was into whatever I could buy at Woolworth's. They never had anything, you always had to order it. You have to remember that Abergavenny is a place where the music scene amounts to one room with sawdust on the floor, where they play 'Wonderwall' at the end of every night".