Bankes was an educated and affluent man, though at times somewhat mischievous and boisterous. He was born in 1786 to Frances Woodley (b. 1760 d. 1823) and Henry Bankes, MP, of Kingston Lacy and Corfe Castle in Dorset. Frances was the eldest daughter of William Woodley (MP for Great Bedwin and Marlborough), Governor and Captain-General of the Leeward Islands (1766–1771 and 1791–1793), and his wife Frances Payne of St Kitts. William's aunt was Maria Banks (Woodley) Riddell, a well-known poet, who was known to be the chief benefactor of the Scottish poet Robert Burns. William's sister, Anne Frances Bankes married Edward Boscawen, 1st Earl of Falmouth in 1810. In 1841, on the death of her husband, Lady Falmouth returned to reside at Kingston Lacey. It was Lady Falmouth, in the absence of her brother, who was responsible for the ongoing re-decoration of Kingston Lacy which had been commenced by her mother Frances (Woodley) Bankes.