Gibbs grew up in Ipswich, Queensland. His brother, Harry, was Chief Justice of Australia from 1981 to 1987. From 1947 until her death in 2015, he was married to Audrey Wald. The couple had eight children.