Chan was born on 7 April 1954 in Hong Kong as Chan Kong-sang () to Charles and Lee-Lee Chan, refugees from the Chinese Civil War. His parents were so poor they had to borrow money from friends to pay the doctor. His parents nicknamed him "Pao-pao" () because the energetic child was always rolling around. His parents worked for the French ambassador in Hong Kong, and Chan spent his formative years within the grounds of the consul's residence in the Victoria Peak district.