If they learn of the caves we're going to be fighting with our backs to the wall.
If they find the caves, they'll pin us in them and we won't stand a chance.
Those caves can never be found, and even if they know about them, we'll be fine.