This is understandable; while cross entropy loss gradient is applied to all pixels, topological gradient is only applied to a sparse set of critical pixels.
the weight needs to be much smaller to avoid overfitting with these critical pixels.