When the marriage of his sister Lady Caroline to William Lamb, 2nd Viscount Melbourne, began to break up, he strongly supported Caroline. Unfortunately he was not noted for tact or intelligence—Melbourne's sister Emily Lamb, Countess Cowper, described him as being universally regarded as "an ass and a jackanapes". Ponsonby reminded Melbourne that the Lamb family were socially "parvenu", and that his sister had married beneath her; although arguably true, these remarks were so tactless that Melbourne broke off any further dealings with him.