I'm haunted by the faces of my victims.
In therapy, people will sometimes emotionally work through their guilt if they feel they have wronged people before, i.e., if they have victimized them; in extreme cases, they might be haunted by the memory of their victim's (human) faces. Here, the fisherman's victims are the many, many fish he killed during the course of his job that he can't stop thinking about. It's funny because people normally don't feel guilty for killing fish, let alone feeling guilty for each individual one (e.g., fish meat is usually measured in pounds, not lives).