Flames engulfed Stuart Russell's pigeon lofts and grain store at Hulk Moor Drove near Glastonbury on Sunday, causing Â£9,000 worth of damage.
Devon and Somerset Fire and Rescue said the fire was started deliberately and an investigation is under way.
The fire happened a year to the day after most of Mr Russell's prize-winning birds were stolen.
Mr Russell, 71, said he believes "parasites" from within the racing pigeon community were to blame.
"This is not a coincidence - I've been dreading something like this since last year," he said.
"It's happening throughout the UK - it's happening all the time - these parasites just want to stop you from doing what you enjoy doing and from winning."
Mr Russell, who has won 57 first prizes since 2005, was given a number of young birds by the racing community to replace those which were stolen a year ago.
He said he was "dreading" telling his grandchildren, who call him "Grandpa Pigeon".
He added: "I really haven't got the heart for it anymore - I've been left with absolutely nothing."
Lee Fribbins, editor of The Racing Pigeon, said competition among the UK's 50,000 fanciers was fierce and the price of birds had "gone through the roof".
He said while there had been "stories of jealous fanciers who have spoiled the enjoyment of others", the community "loves to applaud others and their achievements".
A prize-winning pigeon fancier is giving up the sport after arsonists set fire to his lofts, killing 20 birds.