McGhee began his career in his native Ayrshire with Cronberry in 1878 and then Boswell Thistle two years later before moving to Edinburgh side Hibernian in 1883. Hibernian had been founded for the benefit of Edinburgh's immigrant Irish Catholic population but the club quickly became a beacon for the Irish-Catholic minority throughout Scotland. They were able to attract players from Irish communities across the country and with this abundant supply of talent they rapidly became one of the dominant sides in Scotland. McGhee was one such player and, as a free-scoring inside forward, he developed into a key part of the side. He is still Hibs record goalscorer against Hearts with at least 24 goals. He scored a remarkable 5 goals in the first 25 minutes in the Rosebery Charity Cup final of 1887, Hibs winning 7–1.