He may have come from Kortrijk, since an Oliver Joye, possibly his father, has been identified there in 1420. Gilles seems to have had an excellent musical education, probably at either Kortrijk or Bruges, where he was hired as a singer in 1449. Documents from the cathedral archives show that he was often in trouble: engaging in street brawls, frequenting brothels, refusing to take part in regular singing events, and in particular visiting a notorious prostitute of the town named "Rosabelle". In spite of these activities, he was made a priest, and became a canon at Cleves in 1453 and at St. Donatian in 1459.