<p>It is really a broad question, asking for answers relying mostly on opinions. Here are my two cents though, which you might find interesting as it does not go along the previous answers here and on datascience.</p>

<p>First, I wouldn't go with separate columns for each input. AFAIK, when different inputs are processed by different columns, it is almost always the case that the network is some sort of Siemese network and the columns share the same weights; or at least the columns all need to produce a similar code. It is not your case here, so I would simply not bother.</p>

<p>Second, you are blessed with a problem with a dense output <em>and</em> no need to learn a code. This should direct you straight to <a href="https://arxiv.org/abs/1505.04597" rel="nofollow noreferrer">U-nets</a>, which outperforms any bottleneck-designed network without much effort. U-nets were introduced for dense segmentation but they shine at any dense-output problem really.</p>

<p>In short, just stack your inputs together and use a U-net.</p>
Multiple-input multiple-output CNN with custom loss function
<p>I have a set of 2D input arrays <code>m x n</code> namely <code>A,B,C</code> and I have to predict two 2D output arrays namely <code>d,e</code> for which I do have the expected values. You can think of the inputs/outputs as grey images if you like.</p>

<p>Because of the spatial information is relevant (these are actually 2D physical domains) I want to use a Convolutional Neural Network to predict <code>d</code> and <code>e</code>. My design (not tested yet) looks as follows:</p>

<p><a href="https://i.stack.imgur.com/BnDZv.png" rel="noreferrer"><img src="https://i.stack.imgur.com/BnDZv.png" alt="enter image description here"></a></p>

<p>Because I have multiple inputs, I guess I should use multiple columns (or branches) to find different features for each of the inputs (they look fairly different). Each of these columns follows a encoding-decoding architecture used in segmentation (see SegNet): Conv2D block involves a convolution+batch normalisation+ReLU layer. Deconv2D involves a deconvolution+batch normalisation+ReLU.</p>

<p>Then, I can merge the output of each column by either concatenating, averaging or taking the maximum for example. To obtain the original <code>m x n</code> shape for each of the outputs I have seen I could do this with a <code>1 x 1</code> kernel convolution.</p>

<p>I want to predict the two outputs from that single layer. Is that okay from the network structure point of view? Finally my loss function depends on the outputs themselves compared to the target plus another relation I want to impose.</p>

<p>A would like to have some expert opinion on this since this is my first design of a CNN and I am not sure if I it makes sense as it is now and/or if there are better approaches (or network architectures) to this problem.</p>

<p>I posted this originally in <a href="https://datascience.stackexchange.com/questions/42798/multiple-input-multiple-output-cnn-with-custom-loss-function">datascience</a> but I did not get much feedback. I am now posting it here since there is a bigger community on these topics plus I would be very grateful to receive implementation tips beside network architectural ones. Thanks.</p>