As it was an urban castle inserted into a pre-existing settlement, buildings were probably cleared to make way for the castle. Chichester Castle was of timber construction; although some timber castles were rebuilt in stone, there is no evidence that this was the case at Chichester.
Since Chichester Castle was situated within a pre-existing town and was made of wood, it's likely that buildings had to be cleared to make room for the fortress. Unlike other wooden fortresses that were later rebuilt with stone, there is no indication that this occurred at Chichester.