Gilmour was born in 1912 in Yokohama, Japan. Her father was unknown. Her mother, Léonie Gilmour, attended Bryn Mawr College and studied at the Sorbonne in Paris, then moved to New York City in the early 1900s to try to establish herself as a writer. In 1907, Léonie traveled to Japan at the behest of Yone Noguchi, the father of Ailes' older half-brother, Isamu, who had been born in 1904. However, by the time Léonie arrived in Tokyo, Yone was involved with a Japanese woman who had already borne the first of their nine children. Léonie's circumstances in Japan were always precarious. Nevertheless, she chose to stay there, teaching to support herself and Isamu, while continuing to edit Yone's writing. When Ailes was born, Léonie chose the name Ailes for her daughter from a poem "Beauty's a Flower" by Moira O'Neill, the pseudonym of Agnes Shakespeare Higginson. It is a striking coincidence that the words in that poem seemed to predict Ailes' career as a dancer. O'Neill wrote, "Ailes was a girl that stepped on two bare feet..." Léonie, Isamu and Ailes lived together in Japan until 1918, when Léonie sent Isamu back to the United States to attend a progressive school in Indiana.