Lubbock was the son of Sir John Lubbock, 3rd Baronet, and was brought up in the family home of High Elms, near Downe in Kent. In 1842 his father brought home a "great piece of news". Young John Lubbock thought it might be a new pony, and was disappointed that the news was that Charles Darwin was moving to Down House in the village. He was soon a frequent visitor to Down House, and became the closest of Darwin's younger friends.