I admire your imagination.  Let me try to reply.

1) Hawaiians weren't Buddhists or Catholics, so the behavior of modern Buddhists and Catholics probably isn't a good model for the behavior of the adze makers and other Hawaiians who traveled to the mountain.

2) The top of the mountain including the adze quarry is above the tree line, so no huts.  Transport costs for timber were prohibitive.  People took shelter in caves to get out of the weather.

3) Shrines are typically found outside the caves, often on high spots so their uprights are silhouetted against the sky.

4) There are no midden piles in the quarry.  Archaeologists have recovered scant food remains.

5) The piles are waste from adze manufacture at the largest traditional quarry in the world.  You can see them clearly on Google Earth.

6) In my experience, the mountain has an unusually high proportion of shrines to other feature types.

Thus, the claim of sacredness seems completely plausible to me.