In 1994, Kramer signed to Brett Gurewitz's punk rock label Epitaph Records and began a solo career. He released solo records, including 1995's self-produced "The Hard Stuff", which features the band Claw Hammer on most songs, along with appearances from members of The Melvins and The Vandals. In 1996 he released "Dangerous Madness." In 1997, he released "Citizen Wayne," co-produced by David Was. In 1998, he played with Pere Ubu. In 1999, he released the live record "LLMF." In 2002, he released the studio album "Adult World."