After World War II, Whitmore studied acting at the American Theatre Wing and the Actors Studio in New York. At this time, Whitmore met his first wife, Nancy Mygatt. They married in 1947, and the couple had three sons before their divorce in 1971. The eldest son, James III, found success as a television actor and director under the name James Whitmore, Jr. The second son, Stephen, became the public spokesman for the Los Angeles County Sheriff's Department. The youngest son, Daniel, was a Forest Service Snow Ranger and firefighter before he launched his own construction company. In 1979, Whitmore and Mygatt remarried, but they divorced again after two years.