This worship appears to allude to their superstitious observances. While in vain seems to means that they do not obtain any fruit from their worship.
The passage suggests that this form of worship is driven by irrational beliefs and superstitions, whereas the phrase "in vain" implies that their efforts are ultimately unsuccessful or unfruitful.