<p>If I've understood correctly, there are two questions here:</p>

<ol>
<li><p>Is it possible to determine what features in the input have activated neurons?</p></li>
<li><p>If so, is it possible to use this information to generate samples from <code>p(x|y)</code>?</p></li>
</ol>

<p>Regarding <code>1</code>, a basic way to determine if a neuron is sensitive to an input feature <code>x_i</code> is to compute the gradient of this neuron's output w.r.t <code>x_i</code>. A high gradient will indicate sensitivity to a particular input element. There is a rich literature on the subject, for example, you can have a look at guided backpropagation or at <a href="https://arxiv.org/abs/1610.02391" rel="nofollow noreferrer">GradCam</a> (the latter is about classification with convnets, but it does contain useful ideas).</p>

<p>As for <code>2</code>, I don't think that your approach to "reversing the problem" is correct. The problem is that your network is discriminative and what it outputs can be seen as <code>argmax_y p(y|x)</code>. Note that this is a point-wise estimation, not a full modeling of the distribution. However, the inverse problem that you're interested in seems to be sampling from </p>

<pre><code>p(x|y)=constant*p(y|x)p(x).
</code></pre>

<p>You don't know how to sample from <code>p(y|x)</code> and you don't know anything about <code>p(x)</code>. Even if you use a method to discover correlations between the neurons and specific input features, you have only discovered which features where more important to the networks prediction, but depending on the nature of <code>y</code> this might be insufficiant. Consider a toy example where your inputs <code>x</code> are 2d points distributed according to some distribution in <code>R^2</code> and where the output <code>y</code> is binary, such that any <code>(a,b) in R^2</code> is classified as <code>1</code> if <code>a&lt;1</code> and it is classified as <code>0</code> if <code>a&gt;1</code>. Then a discriminative network could learn the vertical line <code>x=1</code> as its decision boundary. Inspecting correlations between neurons and input features will reveal that only the first coordinate was useful in this prediction, but this information is not sufficient for sampling from the full 2d distribution of inputs. </p>

<p>I think that <a href="https://en.wikipedia.org/wiki/Autoencoder#Variational_autoencoder_(VAE)" rel="nofollow noreferrer">Variational autoencoders</a> could be what you're looking for. </p>
Can you reverse a PyTorch neural network and activate the inputs from the outputs?
<p>Can we activate the outputs of a NN to gain insight into how the neurons are connected to input features?</p>

<p>If I take a basic NN example from the PyTorch tutorials. Here is an example of a <code>f(x,y)</code> training example.</p>

<pre class="lang-py prettyprint-override"><code>import torch

N, D_in, H, D_out = 64, 1000, 100, 10

x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)

loss_fn = torch.nn.MSELoss(reduction='sum')

learning_rate = 1e-4
for t in range(500):
    y_pred = model(x)
    loss = loss_fn(y_pred, y)
    model.zero_grad()
    loss.backward()
    with torch.no_grad():
        for param in model.parameters():
            param -= learning_rate * param.grad
</code></pre>

<p>After I've finished training the network to predict <code>y</code> from <code>x</code> inputs. Is it possible to reverse the trained NN so that it can now predict <code>x</code> from <code>y</code> inputs?</p>

<p>I don't expect <code>y</code> to match the original <em>inputs</em> that trained the <code>y</code> outputs. So I expect to see what <em>features</em> the model activates on to match <code>x</code> and <code>y</code>.</p>

<p>If it is possible, then how do I rearrange the <code>Sequential</code> model without breaking all the weights and connections?</p>