the enemy were now in the town in some numbers and remnants of the guards brigade , who only hours before had been landed at boulogne to protect it for use as an evacuation port , were engaged in fierce hand to hand fighting in and around the old warehouses .