After coming back to Australia, Willoughby joined his cousin's new band Coloured Stone. He played with this group from 1984, including during its tour of Scotland in 1986. They played at that year's Edinburgh Festival. They returned to Australia for the Australian Music Awards at the end of the year, and won the Best Indigenous Album award for "Human Love".