William Fisher sold his commission the year John was born, and became a coffee planter and later chief superintendent of police. He incurred such debt on his two coffee plantations that he could barely support his growing family. At the age of six John (who was always known within the family as "Jack") was sent to England to live with his maternal grandfather, Charles Lambe, in New Bond Street, London. His grandfather had also lost money and the family survived by renting out rooms in their home. John's younger brother, Frederic William Fisher, joined the Royal Navy and reached the rank of admiral, and his youngest surviving sibling Philip became a navy lieutenant on before drowning in an 1880 storm.