<p>There is an end-to-end training method combying a neural network with a CRF that does exactly what you want it to do. The paper is this one: <a href="https://arxiv.org/pdf/1611.10229.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1611.10229.pdf</a>. </p>
Using gradient ascent for image segmentation
<p>I am working with image segmentation using saliency maps via gradient ascent.
Here is an image of the process: <a href="https://imgur.com/a/h8vBZ" rel="nofollow noreferrer">http://imgur.com/a/h8vBZ</a></p>

<p>I have a trained model that can accurately predict my classes. This model is then used to compute a gradient of an input picture with gradient ascent wrt loss. To me, the gradient produced here is a representation of what the model is focusing on in prediction. </p>

<p>I run a quantile filter to pick out the the gradient values (pixels) that are most related to the class, then produce a binary mask from this. This works well, but find that the map could be more accurate and tighter around the class within the image. I read about Conditional Random Fields as a mechanism to generate more accurate and smooth segmentation results and am attempting to implement this, but feel as if I don't have a full understanding of the gradient produced here. </p>

<p>My question is: what exactly does the gradient represent in this case? My guess is that these values are essentially pixel level predictions/pixel labels. Is this equivalent to unary potentials?</p>