Humberside Police and Hull City Council have written to the owners of the Lord Line building asking them to improve security at the site.
The former trawler offices at St Andrew's Quay has stood empty for a number of years.
Owners Manor Property Group said it had spent more than Â£200,000 on security.
Insp Kirsty Tock  said police had been called to the building 17 times in 21 days.
She said young people had been walking and cycling on the roof, as well throwing stones at cars.
"Young people who are going inside the building, which is derelict and in quite a dangerous condition, they are placing themselves in danger, " she said.
"My biggest fear is that someone either very seriously injures themselves, or worse case scenario is that somebody dies, and that's a daily fear basically."
Ms Tock said the site had featured on urban explorer websites which had encouraged people to visit from across the region.
The building is located in the former fishing docks in the west of the city.
Manor Property Group said it planned to redevelop the site into a "maritime educational campus".
The company said it was "disappointed" to have received the letter from the council and police.
It added that it had "always co-operated with the council when requested and would like to have a working group with the local authority to discuss any issues."
Police have said they are concerned somebody could be killed while trespassing in a derelict building in Hull.