Wafer map defect pattern classification is essential in semiconductor manufacturing processes for increasing production yield and quality by providing key root-cause information. However, manual diagnosis by field experts is difficult in large-scale production situations, and existing deep-learning frameworks require a large quantity of data for learning. To address this, we propose a novel rotation- and flip-invariant method based on the labeling rule that the wafer map defect pattern has no effect on the rotation and flip of labels, achieving class discriminant performance in scarce data situations. The method utilizes a convolutional neural network (CNN) backbone with a Radon transformation and kernel flip to achieve geometrical invariance. The Radon feature serves as a rotation-equivariant bridge for translation-invariant CNNs, while the kernel flip module enables the model to be flip-invariant. We validated our method through extensive qualitative and quantitative experiments. For qualitative analysis, we suggest a multi-branch layer-wise relevance propagation to properly explain the model decision. For quantitative analysis, the superiority of the proposed method was validated with an ablation study. In addition, we verified the generalization performance of the proposed method to rotation and flip invariants for out-of-distribution data using rotation and flip augmented test sets.