Ashley's early family life was loveless, a circumstance common among the British upper classes, and resembled in that respect the fictional childhood of Esther Summerson vividly narrated in the early chapters of Charles Dickens's novel "Bleak House". G.F.A Best in his biography "Shaftesbury" writes that: "Ashley grew up without any experience of parental love. He saw little of his parents, and when duty or necessity compelled them to take notice of him they were formal and frightening." Even as an adult, he disliked his father and was known to refer to his mother as "a devil".