David and Samantha McHattie, who run The Bridge Inn in Derbyshire, said since March they have been "bullied" by animal activists online and four protests were held outside the pub against the legal ingredient.
At first they refused to remove it and hit back at abuse.
Foie gras is produced from livers of force-fed ducks or geese.
Mrs McHattie, 35, said it started as "online bulling" and protests followed.
"The emotional toll was just too much. My sister has been abused, my mother, my 80-year-old father, and one of our customers has been called horrendous names and had photographs of her children posted on an animal activist website," she said.
Mr McHattie, 51, said: "The levels of intimidation were getting closer and closer to the line. It is a tough decision because they have not been harming the business."
About 40 protesters gathered outside the pub, in Calver, on Easter Sunday, while on the May bank holidays and last Sunday, 20 to 25 people protested.
Mr McHattie said an elderly customer walked in "visibly shaking" on Sunday, and that night they removed foie gras from the menu.
He said during the protests, abuse was shouted and messages were written on the floor, and while his brother was trying to remove chalk from the floor, water was poured over his head.
"We are not having my family and guests abused," Mr McHattie added.
The McHatties also received abusive phone calls and removed their Facebook page after it was "bombarded with fake reviews and hateful messages".
Nottingham Animal Rights, one of protest organisers, posted: "After a long campaign with the pub... activists have managed to get another establishment to see sense and remove the item.
"We do not see this as a victory for us, but a victory for the poor defenceless animal involved."
Pub owners have removed foie gras from the menu after their family, friends and guests received abuse.