He was the son of Lieutenant-General Sir Henry Pottinger of the British East India Company, and his wife Susanna Maria, née Cooke, of Dublin. Henry Pottinger was the first Governor of Hong Kong. Frederick went to Eton from 1844 to 1847. Between 1850 and 1854, Pottinger was an officer in the Grenadier Guards in England. He lost a lot of his mother's money gambling on horse races. In 1856 he took the place of his father as second baronet. He soon gambled away all his money. He owed people a lot of money.