Gibson was born in Omaha, the last of Pack and Victoria Gibson's seven children (five boys and two girls). Gibson's father died of tuberculosis three months prior to Gibson's birth, and Gibson was named Pack Robert Gibson in his father's honor. While he revered his father's legacy, Gibson disliked the name Pack, and later changed his first name to Robert. Despite a childhood that included health problems like rickets, and a serious case of either asthma or pneumonia when he was three, Gibson was active in sports in both informal and organized settings, particularly baseball and basketball. Gibson's brother Josh (no relation to the Negro leagues star player), who was 15 years his senior, had a profound effect on his early life, serving as a mentor to him. Gibson played on a number of youth basketball and baseball teams his brother coached, many of which were organized through the local YMCA.