Thousands of people have signed an online petition calling for the 25-year-old to be given a permanent remembrance site in Woolwich.
But local MP Nick Raynsford said a memorial would attract "undesirable interest from extremists".
The soldier was killed by two British Muslim converts on 22 May 2013.
Michael Adebolajo, 29, together with Michael Adebowale, 22, drove into Fusilier Rigby with a car before hacking him to death. Adebolajo was given a whole-life term and Adebowale was jailed for 45 years.
This week a number of religious groups including the Islamic Society of Britain and the Conservative Muslim Forum wrote to the Daily Telegraph to express support for a memorial in Woolwich.
They said: "If the family's desire is to have a memorial, neither they, nor the British public as a whole, should be denied the chance to commemorate Lee Rigby's service and sacrifice in a proper way."
Mr Raynsford, Labour MP for Greenwich and Woolwich, said: "It would not in my view be helpful for the site where Lee Rigby was murdered to be marked in perpetuity by a memorial, as this would be a continuing reminder of the brutal murder and might attract undesirable interest from extremists.
"It is worth remembering that the memorial to Stephen Lawrence, the teenager murdered during a racist attack in south-east London in 1993, has sadly been the victim of unwanted focus and vandalism on a number of occasions."
The MP added: "Lee Rigby, although murdered in Woolwich, originally came from Lancashire and this is where he is buried. His family, who have been amazingly brave in facing their loss, will be responsible for his memorial there.
"His regiment, the Royal Regiment of Fusiliers, has also made clear that Lee Rigby's memory will be honoured in perpetuity and I understand that his name is to be added to the list of fallen soldiers at the national arboretum."
A spokesman for the local council, the Royal Borough of Greenwich, said it would only consider requests made via the Army.
"The Army have told us that Fusilier Rigby's immediate family have been closely involved in all the memorial plans to date and they are satisfied that his memory will be properly preserved," the spokesman said. "They are not seeking an additional memorial in Woolwich."
An MP and council have rejected calls for a memorial to Fusilier Lee Rigby in the south-east London area where he was murdered.