The club initially played at Thornfield and then briefly at Hillmorton Recreation ground when Thornfield was taken over for the sports centre before groundsharing with the then Southern Football League side Rugby Town at Oakfield. VS Rugby moved into their own ground, Butlin Road, in 1973 the first competitive game there being a 0–0 draw against Burton Park Wanderers in the United Counties League. In 1975 the club was invited to join the West Midlands League, and took part in the FA Cup for the first time in 1976, being knocked out in the first qualifying round.