"Burn the Bastards" is a celebratory house music song based upon Sly Stone's "Dance to the Music": a trumpet break and drum line are sampled, and the lyrical structure of that song is also mirrored. Whereas "Dance to the Music" vocally introduces the instruments used, so "Burn the Bastards" has Drummond sing of The JAMs' methods, such as "All we need is a beatbox, for people who only need a beat". The choral line "Dance to the music" is modified to "JAMs have a party".