Fisher was born and raised in Peterborough, Ontario, to parents Jim and Karen and is a devout Christian. He has two brothers (Rob and Gregory, aka "Bud") and one sister (Meredith). Fisher's uncle is David Fisher, a former chaplain to the Toronto Blue Jays baseball club. Fisher's brother Gregory is also a hockey player; he is a goaltender who played for Quinnipiac University in the ECAC. On October 23, 2009, Gregory replaced the Senators' then-starting goaltender Pascal Leclaire in net during a Senators' practice after Leclaire fell ill.