<p>There are two main properties of transformers that makes them so appealing compared to convolutions:</p>
<ol>
<li>A transformer is <em>permutation</em> equivariant. This makes transformers very useful for set predictions. For sequences and images where order does matter, positional encoding/embedding are used.</li>
<li>The receptive field of a transformer is the <em>entire</em> input (!) as opposed to the very limited receptive field of a convolution layer.</li>
</ol>
<p>See sec. 3 and fig. 3 in:<br />
<em>Shir Amir, Yossi Gandelsman, Shai Bagon and Tali Dekel</em> <a href="https://arxiv.org/pdf/2112.05814.pdf" rel="nofollow noreferrer"><strong>Deep ViT Features as Dense Visual Descriptors</strong></a> (arXiv 2021).</p>
For an image or sequence, what is the properties transformers use?
<p>Today my teacher ask me a question: he said the CNN is use the translation invariance of the images or matrixs. So what is the properties Transformer uses ???</p>