Elinor Wylie was born Elinor Morton Hoyt on 7 September, 1885 in Somerville, New Jersey. She came from a prominent New Jersey family. Her parents were Henry Martyn Hoyt, Jr. and Anne Morton McMichael. Elinor was the oldest of five children.