The brothers were both born on Governors Island in New York Harbor, where their father, Thomas B. Smothers, Jr., a West Point graduate and U.S. Army officer, was stationed. Tom was born on February 2, 1937, and Dick was born on November 20, 1939. Major Smothers served in the 45th Infantry Regiment and died during World War II, while being transported from a Japanese prisoner of war camp in Fukuoka, Japan, to a POW camp in Mukden, Manchukuo. They were raised by their mother in the Los Angeles area.