Ellerman lived a secretive life in Mayfair and Eastbourne. Although he did purchase New Slains Castle in Scotland in 1916, he subsequently dismantled it and removed its roof to avoid roof tax, leaving it a ruin by 1925. He made no attempt to join high society or enter politics. One possible reason is that from the early 1890s he lived with a woman called Hannah Glover, and had a daughter by her in 1894, but did not marry her until 1908, the year before the birth of his only son, who was also called John Ellerman.