This issue was specifically related to a problem in the LLVM compilation which resulted in layers such as softmax giving nonsense results. It was fully resolved for me by upgrading from LLVM 4.0 to LLVM 6.0 for all the networks I tested. I wouldn't expect this issue to affect depthwise_conv2d, so it sounds like you have a different problem.