King was born in Parramatta, New South Wales. His parents were Harriet ("née" Lethbridge) and Phillip Parker King. His father was an explorer, Royal Navy officer and politician. He was named after his grandfather, also Philip Gidley King, the third governor of New South Wales.