Pigeon racer Simon Schofield said he believed 25 of his birds had been killed so far this year by raptors.
The RSPB said birds of prey were increasingly nesting in cities and would naturally kill racing pigeons.
But the charity said this accounted for only a small proportion of losses and many failed to return to lofts due to straying and collisions with buildings.
A pair of peregrine falcons are often seen nesting on buildings around Leeds University.
Peregrines have also been present on Wakefield Cathedral for a number of years.
The RSPB said such birds, which are a protected species, were colonising in cities because high rise buildings replicated natural habitats such as cliff edges
Mr Schofield, of Wetherby Road, Leeds, said: "I've got 60 birds now and I reckon I'll end up with 30.
"It's not just the emotional side, but the cost too. Racing pigeons aren't cheap and for each one that gets killed or injured you've to replace them or get them treated.
"This is my hobby but now I'm facing the possibility of having to give it up."
But birder Paul Wheatley, who takes pictures of the Leeds peregrines, said it was wrong to assume that pigeon losses were a result of birds of prey.
"Peregrines eat a variety of species and pigeons make up only a small amount of their meals.
"Also, as hard as it must be for pigeon fanciers to lose birds this way, it is also the natural order of things and there is not a lot we can do about that."
Birds of prey nesting in Leeds are threatening the future of pigeon racing in the city, fanciers fear.