This worship appears to allude to their superstitious observances. While in vain seems to means that they do not obtain any fruit from their worship.
Their religious practices seem to be motivated by fear or ignorance, whereas the phrase "in vain" suggests that they do not achieve any positive outcomes from their worship.