Fisher currently lives in Maryland with his wife Sorrel. They have four children. Two of them are also theoretical physicists: Daniel S. Fisher is professor of Applied Physics at Stanford, while Matthew P. A. Fisher is professor of Physics at the University of California, Santa Barbara.