Howard was widely known as a "ladies' man", and he once said that he "didn't chase women but … couldn't always be bothered to run away". He reportedly had affairs with Tallulah Bankhead when they appeared on stage in the UK in "Her Cardboard Lover" (1927), with Merle Oberon while filming "The Scarlet Pimpernel" (1934), and with Conchita Montenegro, with whom he had appeared in the film "Never the Twain Shall Meet" (1931). There were also rumours of affairs with Norma Shearer and Myrna Loy during filming of "The Animal Kingdom".