Ackerley's memoir "My Father and Myself" begins: "I was born in 1896 and my parents were married in 1919." Registered at birth as Joe Ackerley, he later took the middle name Randolph after his uncle, Randolph Payne, first husband of his mother's sister Bunny. As an adult, he published under his first two initials and surname. His father, Roger Ackerley, was a successful fruit merchant known as the "Banana King" of London. Roger Ackerley was first married to a young Swiss woman of wealthy parentage named (Charlotte) Louise Burckhardt (1862-1892) who died probably of tuberculosis, before they had children. Louise was the subject of John Singer Sargent's painting, "Lady with the Rose".