Sweet began distancing himself from other people in the Athens music scene and in 1984 quit Oh-OK; by 1985, the Buzz of Delight had broken up. That year, Sweet got a record deal with CBS Records and moved to New York City. He was accused of being opportunistic and using his Athens connections to get a record deal and leave. Sweet maintains that when he went to CBS, he never claimed to have anything to do with Athens, so that nobody could say he used the town. He says that after months of living in Athens, he realized things weren't as happy there as everyone pretended, and that there was backstabbing going on. In 1993, he said, "Things really turned dark there when R.E.M. got famous, because everyone wanted that fame so bad. Maybe I wanted it too, but I had this musical goal all of my own and wasn't going to go along with the way it was done there." Everybody was telling him that he should be touring and building up a following before doing his record, like R.E.M. had done. However, more than making the record itself and becoming a rock star, Sweet's main motivation was to get money to buy studio gear.