The banner which read "supporting the London Olympics" breached the rules set out by Games organisers Locog.
The rules state that only official sponsors are allowed to use logos and text defined in the organisation's  
          2012 UK statutory marketing rights.
The sign was being used to promote Olympic events in the city.
The torch relay is due to visit Derby on 29 June and Peter Allen, director of marketing for the university, said he was keen to support the event.
He said: "As you can imagine it's difficult to do this without making reference in some way to the Olympics."
But he added that the university was working closely with Locog and the city council to ensure it followed brand guidelines.
Several independent businesses in the East Midlands have also used Olympic logos in their shop windows.
Hobsons shoe shop on the Strand in Derby has put a display with five coloured rings in its window.
The shop's owner said he would keep the rings until he was told to take them down.
The owner of a neighbouring wool shop, John Sallis, had planned to create his own woollen Olympic rings to put on display.
He said: "It's absolutely ridiculous - all I want to do is celebrate the Olympics by putting the five rings in the window to celebrate the Olympic runners coming past."
One shop owner in Melton Mowbray, Leicestershire, was told to take down her plastic rings by Trading Standards.
Julie Swain, who runs a lingerie shop, said: "We were hoping to have a competition for window designs for when the torch relay comes past. Obviously, I don't think we will be able to do that now."
Keith Regan, from Leicestershire Trading Standards, said: "The Olympic rings are very much protected."
He added that people could get involved by using different types of displays  - as long as Olympic logos were not used.
The University of Derby has removed a sign supporting the London 2012 Olympics because it had not been officially approved.