"Giving You Up" was released digitally as the second single from "Ultimate Kylie" on 25 March 2005. Three days later, it was released physically in the United Kingdom and Germany. It was released as a limited edition vinyl single in the United Kingdom. In Australia, the CD single was limited to 25,000 copies. All formats featured the B-side "Made of Glass", which was written by Minogue and Xenomania during the recording sessions for "Ultimate Kylie" in 2004. It was also recorded by British recording artist Rachel Stevens but was never released. "Made of Glass" appeared on the Australian compilation "Now Spring 2005".