Cromartie was the only child of Leroy and Marjorie Cromartie. Leroy played quarterback at Florida A&M College, and led his team to Southern Intercollegiate Athletic Conference championships in 1944 and 1945. Having also played basketball and baseball in high school, he left FAMC to play semi-pro baseball with the Miami Giants, which led to a brief stint as a second baseman with the Cincinnati/Indianapolis Clowns of the Negro leagues. After that, Leroy returned to FAMC and led the Rattlers to another SIAC championship in 1947 and a national championship in 1950.