Goh was given the Christian name Robert, which he disliked and refused to respond to. When he was two years old, his family moved from Malacca to Singapore where his maternal grandparents owned several properties. The Gohs later relocated to the Pasir Panjang rubber estate when his father found work there, and became manager in 1933. In common with many Peranakan families, the Gohs spoke both English and Malay at home; church services were held at home on Sundays in Malay. Goh's father Leng Inn and the latter's brothers-in-law Chew Cheng Yong and Goh Hood Keng taught in the Anglo-Chinese School for various periods, and were also involved in the Middle Road Baba Church while Hood Keng was pastor there. Goh himself attended this church as well.