<p>The reason why convolutions are more efficient than fully connected layers is <em>beacause</em> they are translation invariant. If you wish to have convolutions which are dependent on location you would need to add two extra parameters to the convolution i.e. having N+2 input channels where x, y coord are the values of the two additonal channels (as in e.g. <a href="https://arxiv.org/abs/1807.03247" rel="nofollow noreferrer">CoordConv</a>, or <a href="https://arxiv.org/pdf/1510.02927.pdf" rel="nofollow noreferrer">Location Biased Convolutions</a>).</p>
How to implement a location specific convolutional filters in Tensorflow or Pytorch?
<p>I want to implement a convolutional layer, with a different convolutional filter for each output location. Specifically, think of the case when the output is of 16*16*128 (W * H * C). Instead of having a 3*3*128 filter we have 16*16 filters; each with size 3*3*128. This would lead to huge amount of parameters, but it can the case be that each of the 3*3*128 filter may be the same except scaled by a different constant, and the constants can be learned through a side network. In this way the number of parameters won't be too much.</p>

<p>The similar idea is briefly in <a href="https://arxiv.org/abs/1605.09673" rel="nofollow noreferrer">Dynamic Filter Networks</a>, but I cannot find an implementation of location specific filters. My question is, if we want a location specific convolutional filter how do I implement it in Tensorflow or Pytorch? Do I need to write my own operation or there is some smart way to use the functions provided? If I have to write an OP is there any trick that can easily achieve this idea? Any help is appreciated!</p>