Title: Group-Equivariant Transformers Without Positional Encoding. Abstract: Self-attention is a permutation-equivariant operator in its basic form and can further extend to achieve equivariance for a specific symmetry group by incorporating group-invariant positional encoding. In this work, we propose an effective group-equivariant transformer without positional encoding. Instead of injecting group-invariant position encoding to the transformer, we replace point-wise MLPs with group-equivariant convolutions that act as both a group mixer and an implicit positional encoding. This allows to reduce the group of self-attention to translation only while preserving group equivariance, resulting in less computation and memory. Our strategy not only retains dynamic long-range interactions of transformers but also incorporates the static effective kernel learning of convolution, resulting in a significant accuracy gain. We also find that adopting a group-equivariant convolution stem and a translation-equivariant pooling further improves the performance. The proposed method sets a new state of the art in standard benchmarks, outperforming the existing group-equivariant transformers by a large margin.