The fall of the clan and loss of the Isle of Lewis, began with Ruairi and his marriage to a daughter of John Mackenzie of Kintail. This marriage had produced a son named Torquil "Connanach" (named after his residence among the Mackenzies in Strathconnan). Ruairi later disowned Torquil Connanach on account of the alleged adultery between his wife and the Morrison brieve of Lewis. Ruairi's wife later abandoned him and eloped with a cousin of his, John MacGillechallum of Raasay, after which Ruairi divorced her. In 1541 Ruairi married Barbara Stewart, daughter of Andrew Lord Avondale, and by her had a son named Torquil "Oighre" ("Heir" to distinguish him from the disowned Torquil).