<p>This behavior is expected. You are using the Adam optimizer, which means the parameter updates are not based solely on the gradient at the current step but on an average of past squared gradients. You can read more about it in the original paper: <a href="https://arxiv.org/abs/1412.6980" rel="nofollow noreferrer"><em>Adam: A Method for Stochastic Optimization</em></a>.</p>
<p>In contrast, if you use a simpler algorithm such as SGD (without momentum) you will get the same parameter values between the third and last logs.</p>
<p>Alternatively, you could unset the gradients of <code>net.fc2</code> all together instead of having them as zero tensors (because of the <code>zero_grad</code>). You can do this by adding in:</p>
<pre><code>net.fc2.weight.grad = None
net.fc2.bias.grad = None
</code></pre>
<p>This will prevent the optimizer from updating the parameter which is desired in your case.</p>
In pytorch model training, how to freeze, unfreeze and freeze again some params?
<p>Following is a toy example explaining what actually I want to do.</p>
<pre class="lang-py prettyprint-override"><code>import torch
from torch import nn
from torch.autograd import Variable
import torch.nn.functional as F
import torch.optim as optim


# toy feed-forward net
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.fc1 = nn.Linear(10, 5)
        self.fc2 = nn.Linear(5, 5)
        self.fc3 = nn.Linear(5, 1)

    def forward(self, x):
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        return x


# define random data
random_input = Variable(torch.randn(10,))
random_target = Variable(torch.randn(1,))

# define net
net = Net()

# print the initial fc2 weight
print('fc2 weight before train :')
print(net.fc2.weight)

# we want to freeze the fc2 layer: only train fc1 and fc3
net.fc2.weight.requires_grad = False
net.fc2.bias.requires_grad = False

criterion = nn.MSELoss()
optimizer = optim.Adam(net.parameters(), lr=0.1)

for i in range(10):
    net.zero_grad()
    output = net(random_input)
    loss = criterion(output, random_target)
    loss.backward()
    optimizer.step()

# print the trained fc2 weight
# note that the weight is same as the one before training: only fc1 &amp; fc3 changed
print('fc2 weight (frozen) after retrain:')
print(net.fc2.weight)

# let's unfreeze the fc2 layer this time for extra tuning
net.fc2.weight.requires_grad = True
net.fc2.bias.requires_grad = True

# re-retrain
for i in range(10):
    net.zero_grad()
    output = net(random_input)
    loss = criterion(output, random_target)
    loss.backward()
    optimizer.step()

# print the re-retrained fc2 weight
# note that this time the fc2 weight also changed
print('fc2 weight (unfrozen) after re-retrain:')
print(net.fc2.weight)

# let's freeze the fc2 layer again
net.fc2.weight.requires_grad = False
net.fc2.bias.requires_grad = False

# re-retrain
for i in range(10):
    net.zero_grad()
    output = net(random_input)
    loss = criterion(output, random_target)
    loss.backward()
    optimizer.step()

# print the re-retrained fc2 weight
# note that this time the fc2 weight also changed, BUT why?
print('fc2 weight (freeze again) after re-retrain:')
print(net.fc2.weight)
</code></pre>
<p>The output of the above code is as follows:</p>
<pre><code>fc2 weight before train :
Parameter containing:
tensor([[-0.0335, -0.1526,  0.1972,  0.3360,  0.2845],
        [ 0.2449,  0.3305, -0.0060, -0.0302, -0.0060],
        [-0.3496,  0.2047,  0.2549,  0.1363,  0.3202],
        [ 0.0900,  0.1425, -0.1090,  0.2983,  0.3481],
        [ 0.2390, -0.1817,  0.0885,  0.0562,  0.1787]], requires_grad=True)
fc2 weight (frozen) after retrain:
Parameter containing:
tensor([[-0.0335, -0.1526,  0.1972,  0.3360,  0.2845],
        [ 0.2449,  0.3305, -0.0060, -0.0302, -0.0060],
        [-0.3496,  0.2047,  0.2549,  0.1363,  0.3202],
        [ 0.0900,  0.1425, -0.1090,  0.2983,  0.3481],
        [ 0.2390, -0.1817,  0.0885,  0.0562,  0.1787]])
fc2 weight (unfrozen) after re-retrain:
Parameter containing:
tensor([[-0.1092, -0.1510,  0.2565,  0.3626,  0.0869],
        [ 0.3208,  0.4498, -0.0719, -0.0643,  0.5945],
        [-0.1369,  0.1388,  0.0623, -0.0110,  0.5612],
        [-0.0655,  0.1785,  0.0269,  0.3923,  0.1261],
        [ 0.1571, -0.1942,  0.1538,  0.0924, -0.0485]], requires_grad=True)
fc2 weight (freeze again) after re-retrain:
Parameter containing:
tensor([[-0.1465, -0.2145,  0.2829,  0.3678, -0.0570],
        [ 0.4450,  0.1603, -0.1964, -0.1753,  0.5486],
        [-0.0317,  0.1698, -0.0310, -0.0769,  0.7227],
        [-0.1470,  0.1200,  0.0956,  0.4346, -0.0379],
        [ 0.1705, -0.1075,  0.1387,  0.0685, -0.0168]])
</code></pre>
<p>fc2's weight is freezed in the first 10 iters as we have freezed it, and fc2's weight has changed in the second 10 iters as we have unfreezed it. But we freeze it again in the third 10 iters, why does it change again? What's the proper way to freeze, unfreeze and freeze again some params?</p>