In 2004, his fifth solo album "Frozen Orange" was released on Arch Hill Recordings (NZ) and Merge Records (USA). For "Frozen Orange", Kilgour left his native New Zealand and recorded in Nashville with members of Lambchop and Jason Loewenstein of Sebadoh, along with his longtime backing band The Heavy Eights, consisting of Taane Tokona, Alan Haig, Tony De Raad and Thomas Bell. The album also marked the first recording that Kilgour did not produce himself, a task left to Mark Nevers (producer / engineer of Lambchop). The making of "Frozen Orange" was documented by filmmaker Bridget Sutherland in the film "Far Off Town: Dunedin to Nashville". The film screened at Raindance Film Festival, London; Nashville Film Festival, San Francisco Film Festival, Arrhus Festival of Independent Arts, Denmark and was nominated for ‘Best Feature Documentary’ at DOCNZ, 2007.