Born in Hattiesburg, Mississippi, to an impoverished family, Brown was avidly interested in aircraft from a young age. He graduated as salutatorian of his high school, notwithstanding its racial segregation, and was later awarded a degree from Ohio State University. Brown enlisted in the U.S. Navy in 1946, becoming a midshipman. Brown earned his pilot wings on 21 October 1948 amid a flurry of press coverage; in January 1949 he was assigned to Fighter Squadron 32 aboard the aircraft carrier .