<p>The reason why convolutions are more efficient than fully connected layers is <strong>because</strong> they are translation invariant. If you wish to have convolutions which are dependent on location you would need to add two extra parameters to the convolution i.e. having N+2 input channels where x, y coord are the values of the two additonal channels (as in e.g. <a href="https://arxiv.org/abs/1807.03247" rel="nofollow noreferrer">CoordConv</a>).</p>
<p>As for alternative solutions, is the gradient meaningful? If not, and it is uniform across all images, it might be better to just manually remove it in the pre-processing stage (similar to orientation correction, cropping etc). If it is not (e.g. differences in lighting, shadows) then including other layers under the assumption they would learn the invariance of different lightings is a common hands-off approach.</p>
Is location-dependent convolution filter possible in PyTorch or TensorFlow?
<p>Let's pretend that in plus of having an image, I also have a gradient from left to right on the X axis of an image, and another gradient from top to bottom on the Y axis. Those two gradients are of the same size of the image, and could both range from -0.5 to 0.5. </p>

<p>Now, I'd like to make the convolution kernel (a.k.a. convolution filter, or convolution weights) depend on the <code>(x, y)</code> location in the gradient. So the kernel is a function of the gradient as if the kernel was the output of a nested mini-neural net. This would make the weights of the filter to be different in every position, but slightly similar to their neighbors. How do I do that within PyTorch or TensorFlow? </p>

<p>Sure, I could compute a <a href="https://en.wikipedia.org/wiki/Toeplitz_matrix#Discrete_convolution" rel="nofollow noreferrer">Toeplitz matrix (a.k.a. diagonal-constant matrix) </a> by myself, but the matrix multiplication would take <code>O(n^3)</code> operations if pretending <code>x==y==n</code>, whereas convolutions can be implemented in <code>O(n^2) normally</code>. Or I could maybe iterate on every element myself and do the multiplications in an unvectorized fashion. </p>

<p>Any better ideas? I'd like to see creativity here, thinking about how could this be implemented neatly. I believe coding that would be an interesting way to build a network layer capable of doing things similar to a simplified version of a <a href="https://arxiv.org/abs/1506.02025" rel="nofollow noreferrer">Spatial Transformer Networks</a>, but which's spatial transformation would be independent of the image. </p>