title: Shape mismatch in two equivalent networks?
body: <p>So, the answer is probably "the two networks are not equivalent" and there's something obvious I'm missing, but in my understanding they should do the same thing but the dimensions of intermediate outputs are not the same. My main issue is with pytorch <code>ConvTranspose3d</code> and <code>Conv3d</code>. In particular, I have an input that has shape <code>(1,44,68,120)</code>, where 44 is the depth dimension while 68 and 120 are width and height. I use strided conv/convtranspose to go down/up in dimensions. Shouldn't this:</p>
  
  <pre><code>self.conv3d1_1 = nn.Conv3d(in_channels=1, out_channels=32, groups=1, stride=(2,2,2), kernel_size=3, padding=1)
  </code></pre>
  
  <p>be equivalent to these two layers in terms of output dimensions?</p>
  
  <pre><code>self.conv3d1_1 = nn.Conv3d(in_channels=1, out_channels=32, groups=1, stride=(2,1,1), kernel_size=3, padding=1)
  self.conv3d1_2 = nn.Conv3d(in_channels=32, out_channels=32, groups=1, stride=(1,2,2), kernel_size=3, padding=1)
  </code></pre>
  
  <p>where the first layer halves all the dimensions simultaneously, while the second block decreases the temporal dimension first, and the spatial dimensions after?</p>
  
  <p>So, first network:</p>
  
  <pre><code>    self.conv3d1_1 = nn.Conv3d(in_channels=1, out_channels=32, groups=1, stride=(2,2,2), kernel_size=3, padding=1)  # depthwise convolution (1 ch, 44 depth dimension, h, w)
      self.conv3d1_2 = nn.Conv3d(in_channels=32, out_channels=64, groups=1, kernel_size=3, padding=1)
  
      self.conv3d2_1 = nn.Conv3d(in_channels=64, out_channels=128, groups=1, stride=(2, 2, 2), kernel_size=3, padding=1)
      self.conv3d2_2 = nn.Conv3d(in_channels=128, out_channels=128, groups=1, kernel_size=3, padding=1)
  
      self.conv3d3_1 = nn.ConvTranspose3d(in_channels=128, out_channels=128, stride=(2, 2, 2), groups=1, kernel_size=4, padding=(1,1,1))
      self.conv3d3_2 = nn.Conv3d(in_channels=128, out_channels=128, groups=1, kernel_size=3, padding=1)
  
      self.conv3d4_1 = nn.ConvTranspose3d(in_channels=128, out_channels=128, stride=(2, 2, 2), groups=1, kernel_size=4, padding=1)
      self.conv3d4_2 = nn.Conv3d(in_channels=128, out_channels=64, groups=1, kernel_size=3, padding=1)
  </code></pre>
  
  <p>Produces these intermediate dimensions:</p>
  
  <pre><code>torch.Size([2, 1, 44, 68, 120])
  torch.Size([2, 32, 22, 34, 60])
  torch.Size([2, 64, 22, 34, 60])
  torch.Size([2, 128, 11, 17, 30])
  torch.Size([2, 128, 11, 17, 30])
  torch.Size([2, 128, 22, 34, 60])
  torch.Size([2, 128, 22, 34, 60])
  torch.Size([2, 128, 44, 68, 120])
  torch.Size([2, 64, 44, 68, 120])
  </code></pre>
  
  <p>Everything looks fine.</p>
  
  <p>Second network (only 1 decrease in dimensions to keep it short, but same and worst happens when going down 2 times (so 4x))</p>
  
  <pre><code>    elf.conv3d1_1 = nn.Conv3d(in_channels=1, out_channels=32, groups=1, stride=(2,1,1), kernel_size=3, padding=1)  # depthwise convolution (1 ch, 44 depth dimension, h, w)
      self.conv3d1_2 = nn.Conv3d(in_channels=32, out_channels=64, groups=1, kernel_size=3, padding=1)
  
      self.conv3d2_1 = nn.Conv3d(in_channels=64, out_channels=128, groups=1, stride=(1, 2, 2), kernel_size=3, padding=1)
      self.conv3d2_2 = nn.Conv3d(in_channels=128, out_channels=128, groups=1, kernel_size=3, padding=1)
  
      self.conv3d3_1 = nn.ConvTranspose3d(in_channels=128, out_channels=128, stride=(1, 2, 2), groups=1, kernel_size=4, padding=(1,1,1))
      self.conv3d3_2 = nn.Conv3d(in_channels=128, out_channels=128, groups=1, kernel_size=3, padding=1)
  
      self.conv3d4_1 = nn.ConvTranspose3d(in_channels=128, out_channels=128, stride=(2, 1, 1), groups=1, kernel_size=4, padding=1)
      self.conv3d4_2 = nn.Conv3d(in_channels=128, out_channels=64, groups=1, kernel_size=3, padding=1)
  </code></pre>
  
  <p>And these are the intermediate dimensions:</p>
  
  <pre><code>torch.Size([2, 1, 44, 68, 120])
  torch.Size([2, 32, 22, 68, 120])
  torch.Size([2, 64, 22, 68, 120])
  torch.Size([2, 128, 22, 34, 60])
  torch.Size([2, 128, 22, 34, 60])
  torch.Size([2, 128, 23, 68, 120])
  torch.Size([2, 128, 23, 68, 120])
  torch.Size([2, 128, 46, 69, 121])
  torch.Size([2, 64, 46, 69, 121])
  </code></pre>
  
  <p>For some reason, the first <code>ConvTranspose3d</code> increases the temporal dimension, while it should be only working on the spatial dimensions? I initially thought it was a padding issue but changing the padding doesn't fix the issue. Same happens with the temporal <code>ConvTranspose3d</code>, which increases the spatial dimensions by 1.</p>
  
  <p>Any clues? Thanks in advance.</p>