Though his late 1970s albums were well received by critics and his core fans, Stewart never established a large rapport with the general music audience. He was often labeled as "too country" for rock listeners and "too rock" for country fans. In 1980, he released the Chips Moman-produced "Cactus and a Rose" which featured Southern rockers Gregg Allman, Dickey Betts, Mike Lawler, and Bonnie Bramlett. It did not garner much airplay and RCA teamed Stewart up with songwriter Dean Dillon for a pair of duet records. Soon after, Stewart returned to Florida, where alcoholism and drug use kept him from recording for much of the 1980s. His son, Gary Joseph Stewart, committed suicide late in the decade as well. Stewart signed with the HighTone label in 1988 and recorded three albums over the next five years. These albums included fan favorites like the minor hit "An Empty Glass (That's the Way the Day Ends)" (written by Stewart and Dillon), "Let's Go Jukin'" (written by Stewart and Betts), and "Brand New Whiskey" (written by Stewart and his wife).