This would be unexpected from the users point of view. The bias is not sharded, yet the grad of the bias received by the user is sharded.    I think we should narrow and apply the bias using an autograd function. The forward part of that autograd function would be exactly what we do in this PR where we narrow the bias and add it. In the backward pass we would gather the grad of the biases using this logic above. That way the user always gets a non-sharded grad for the bias.