<p>As it turned out, the problem is in low-level implementation of SpatialDepthWiseConvolution in torch/nn. I've created an issue: <a href="https://github.com/torch/nn/issues/1307" rel="nofollow noreferrer">https://github.com/torch/nn/issues/1307</a></p>

<p>For the moment (03 of March 2018) this issue is not solved yet.
Of course, I hope that somebody will fix mistake in low-level implementation.
But for now I know two ways to overcome this problem:</p>

<ul>
<li>emulate via torch containers</li>
<li>use my pure-lua implementation</li>
</ul>

<p>Here is a way to emulate this module using containers <strong>Concat</strong> and <strong>Parallel</strong> along with standard <strong>SpatialConvolution</strong> module:</p>

<pre><code>  local depth_wise_conv = nn.Concat(2)
  for o = 1, nOutputPlane do
    local out = nn.Parallel(2, 2)
    for i = 1, nInputPlane do
      local seq = nn.Sequential()
      local conv = nn.SpatialConvolution(1, 1, kW, kH, dW, dH, pW, pH):noBias()
      seq:add( nn.Reshape(1, inputHeight, inputWidth) )
      seq:add( conv )
      out:add( seq )
    end
    depth_wise_conv:add( out )
  end
</code></pre>

<p>Note that depth_wise_conv module in the code above is supposed to get batch-input with 4 dimensions: <code>batchSize x nInputPlane x inputHeight x inputWidth</code>.</p>

<p>But I also spend some time and created a pure-lua implementation of <strong>SpatialDepthWiseConvolution</strong> as a module. You may find it here:
<a href="https://gist.github.com/diovisgood/36ce5a6c5e9dd4cb20b13dd2a28c1f71" rel="nofollow noreferrer">https://gist.github.com/diovisgood/36ce5a6c5e9dd4cb20b13dd2a28c1f71</a></p>

<p>There are also: <strong>SpatialConvolution</strong> implementation and unit tests.
I published these modules under MIT license, so anyone can use them.</p>

<p>Please note that these modules have not been heavily tested!
So any help or advice is appreciated.</p>

<p>One more thing. There is a great paper that explains convolution and transposed convolution by Vincent Dumoulin and Francesco Visin:
<a href="https://arxiv.org/pdf/1603.07285.pdf" rel="nofollow noreferrer">'A guide to convolution arithmetic for deep learning'</a></p>
SpatialDepthWiseConvolution Xception backpropagation error
<p>I'm trying to implement NN inspired by Xception ideas.
Can't understand what is wrong with my model...</p>

<pre><code>local torch = require 'torch'
local nn = require 'nn'

dofile('GlobalAveragePooling.lua')

local model = nn.Sequential()
-- Entry convolution 
model:add( nn.SpatialConvolution(3, 64, 3, 3, 2, 2, 1, 1) )
model:add( nn.SpatialBatchNormalization(64) )
model:add( nn.ReLU() )

-- Xception Unit with "skip-path"
local seq = nn.Sequential()
seq:add( nn.SpatialDepthWiseConvolution(64, 1, 3, 3, 1, 1, 1, 1) )
seq:add( nn.SpatialConvolution(64, 512, 1, 1, 1, 1, 0, 0) )
seq:add( nn.SpatialBatchNormalization(512) )
seq:add( nn.SpatialMaxPooling(3, 3, 2, 2, 1, 1) )

local con = nn.ConcatTable()
con:add( nn.SpatialConvolution(64, 512, 1, 1, 2, 2, 0, 0) )
con:add( seq )

model:add( con )
model:add( nn.CAddTable() )
model:add( nn.ReLU() )

-- Exit fully-connected layers for softmax(3) output
model:add( nn.GlobalAveragePooling() )
model:add( nn.Reshape(512) )

model:add( nn.Linear(512, 3) )
model:add( nn.LogSoftMax() )

print(tostring(model))

local X = torch.randn(10, 3, 16, 8)
local Y = torch.LongTensor(10):random(1,3)

local criterion = nn.ClassNLLCriterion()

local Yhat = model:forward(X)

local loss = criterion:forward(Yhat, Y)
local gradLoss = criterion:backward(Yhat, Y)
model:backward(X, gradLoss)
</code></pre>

<p>The model works good at forward() step.
But fails when it comes to model:backward(X, gradLoss) with error:</p>

<pre><code>    /nn/THNN.lua:110: Need gradOutput of dimension 5 and gradOutput.size[3] == 8 but got gradOutput to be of shape: [10 x 64 x 1 x 4 x 8] at ../THNN/generic/SpatialDepthWiseConvolution.c:53
stack traceback:
[C]: in function 'v'    ../nn/THNN.lua:110: in function 'SpatialDepthWiseConvolution_updateGradInput'   ../nn/SpatialDepthWiseConvolution.lua:80:
in function 'updateGradInput' ../Module.lua:31:
in function &lt;../nn/Module.lua:29&gt;
[C]: in function 'xpcall'   ../nn/Container.lua:63:
in function 'rethrowErrors'     ../nn/Sequential.lua:88:
in function &lt;../nn/Sequential.lua:78&gt;
[C]: in function 'xpcall'   ../Container.lua:63:
in function 'rethrowErrors'     ../nn/ConcatTable.lua:66:
in function &lt;../ConcatTable.lua:30&gt;
[C]: in function 'xpcall'   ../nn/Container.lua:63:
in function 'rethrowErrors'     ../nn/Sequential.lua:84:
in function 'backward'  test.lua:45:
in main chunk   [C]: at 0x00405d50
</code></pre>