<blockquote>
<p>How do experts design CNN's? How could I choose between Inception Modules, Dropout Regularization, Batch Normalization, convolutional filter size, size and depth of convolutional channels, number of fully-connected layers, activations neurons, etc? How do people navigate this large optimization problem in a scientific manner? The combinations are endless.</p>
</blockquote>
<p>You said truly that the combinations are huge in number. And without approaching rightly you may end up with nowhere. A great one said machine Learning is an art, not science. Results are data-dependent. Here are a few tips regarding your above concern.</p>
<ul>
<li><p><strong>Log Everything</strong>: In the training time, save necessary logs of every experiment such as training loss, validation loss, weight files, execution times, visualization, etc. Some of them can be saved with <code>CSVLogger</code>, <code>ModelCheckpoint</code> etc. <code>TensorBoard</code> is a great tool for inspecting both training log and visualization and many more.</p>
</li>
<li><p><strong>Strong Validation Strategies</strong>: This is very important. To build a stable <strong>Cross-Validation (CV)</strong>, we must have a good understanding of the data and the challenges faced. We’ll check and make sure the <strong>validation set</strong> has a <strong>similar distribution</strong> to the <strong>training set</strong> and <strong>test set</strong>. And We’ll try to make sure our models improve <strong>both</strong> on our <strong>CV</strong> and on the <strong>test set</strong> (if <code>gt</code> is available for the test set). Basically, partitioning the data randomly is usually not enough to satisfy this. Understanding the data and how we can partition it without introducing a <strong>data leakage</strong> in our CV is key to avoid <strong>overfitting</strong>.</p>
</li>
<li><p><strong>Change Only One</strong>: During the experiment, change one thing at a time and save the observations (<code>logs</code>) for those changes. For example: change the image size gradually from <code>224</code> (for example) to higher and observe the results. We should start with a small combination. While experimenting with image size, fix others like <code>model</code> architecture, <code>learning rate</code>, etc. The same goes for the <code>learning rate</code> part or <code>model</code> architectures. However, later we also may need to change more than one when we get some promising combinations. In kaggle competition, these are very common approaches one would follow. Below is a very simple example regarding this. But it's not limited any way.</p>
</li>
</ul>
<p><a href="https://i.stack.imgur.com/jqgf6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jqgf6.png" alt="enter image description here" /></a></p>
<hr />
<p>However, as you said, your Ph.D. project is to <strong>reduce CO2 emissions on Earth.</strong> In my understanding, these are more <strong>application-specific</strong> problems and less than the <strong>algorithm-specific</strong> problems. So, we think it's better to take benefit from well-recognized pre-trained models.</p>
<p>In case if we wish to write our <code>CNN</code> on our own, we should give a decent time on it. Start with a very simple one, for example:</p>
<pre><code>Conv2D (16,  3, 'relu') - &gt; MaxPool (2)
Conv2D (32,  3, 'relu') - &gt; MaxPool (2) 
Conv2D (64,  3, 'relu') - &gt; MaxPool (2)
Conv2D (128, 3, 'relu') - &gt; MaxPool (2)
</code></pre>
<p>Here we gradually increase the depth but reducing the feature dimension. By the end layer, more semantic information would emerge. While stacking <code>Conv2D</code> layers, it's common practice to increase the channel depth in such order <code>16, 32, 64, 128</code> etc. If we want to impute <code>Inception</code> or <code>Residual Block</code> inside our network, I think, we should do some basic math first about what feature properties will come out of this, etc. Following a concept like this, we may also wish to look at approaches like <code>SENet</code>, <code>ResNeSt</code> etc. About <code>Dropout</code>, if we observe that our model is getting overfitted during training, then we should add some. In the final layer, we may want to choose <code>GlobalAveragePooling</code> over the <code>Flatten</code> layer (<strong>FCC</strong>). We can probably now understand that there are lots of ablation studies that need to be done to get a satisfactory <code>CNN</code> model.</p>
<p>In this regard, We suggest you explore the two most important things: (<strong>1</strong>). Read one of the pre-trained model papers/blogs/videos about their strategies to build the algorithm. For example: check out this <a href="https://www.youtube.com/watch?v=3svIm5UC94I" rel="nofollow noreferrer">EfficientNet Explained</a>. (<strong>2</strong>). Next, explore the source code of it. That would give your more sense and encourage you to build your own giant.</p>
<hr />
<p>We like to end this with one last working example. See the model diagram below, it's a <strong>small inception network</strong>, <a href="https://arxiv.org/pdf/1611.03530.pdf" rel="nofollow noreferrer">source</a>. If we look closely, we will see, it consists of the following three modules.</p>
<ul>
<li>Conv Module</li>
<li>Inception Module</li>
<li>Downsample Modul</li>
</ul>
<p>Take a close look at each module's configuration such as <strong>filter size</strong>, <strong>strides</strong>, etc. Let's try to understand and implement this module. Before that, here are two good references (<a href="https://www.youtube.com/watch?v=C86ZXvgpejM" rel="nofollow noreferrer">1</a>, <a href="https://www.youtube.com/watch?v=KfV8CJh7hE0" rel="nofollow noreferrer">2</a>) for the Inception concept to refresh the concept.</p>
<p><a href="https://i.stack.imgur.com/Vtm4f.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Vtm4f.png" alt="enter image description here" /></a></p>
<h2>Conv Module</h2>
<p>From the diagram we can see, it consists of <strong>one convolutional</strong> network, <strong>one batch normalization</strong>, and <strong>one relu</strong> activation. Also, it produces <code>C</code> times feature maps with <code>K x K</code> filters and <code>S x S</code> strides. To do that, we will create a class object that will inherit the <code>tf.keras.layers.Layer</code> classes</p>
<pre><code>class ConvModule(tf.keras.layers.Layer):
    def __init__(self, kernel_num, kernel_size, strides, padding='same'):
        super(ConvModule, self).__init__()
        # conv layer
        self.conv = tf.keras.layers.Conv2D(kernel_num, 
                        kernel_size=kernel_size, 
                        strides=strides, padding=padding)

        # batch norm layer
        self.bn   = tf.keras.layers.BatchNormalization()


    def call(self, input_tensor, training=False):
        x = self.conv(input_tensor)
        x = self.bn(x, training=training)
        x = tf.nn.relu(x)
        
        return x
</code></pre>
<h2>Inception Module</h2>
<p>Next comes the <strong>Inception</strong> module. According to the above graph, it consists of <em>two convolutional modules</em> and then <strong>merges</strong> together. Now as we know to merge, here we need to ensure that the output feature maps dimension ( <code>height</code> and <code>width</code> ) needs to be the <strong>same</strong>.</p>
<pre><code>class InceptionModule(tf.keras.layers.Layer):
    def __init__(self, kernel_size1x1, kernel_size3x3):
        super(InceptionModule, self).__init__()
        
        # two conv modules: they will take same input tensor 
        self.conv1 = ConvModule(kernel_size1x1, kernel_size=(1,1), strides=(1,1))
        self.conv2 = ConvModule(kernel_size3x3, kernel_size=(3,3), strides=(1,1))
        self.cat   = tf.keras.layers.Concatenate()


    def call(self, input_tensor, training=False):
        x_1x1 = self.conv1(input_tensor)
        x_3x3 = self.conv2(input_tensor)
        x = self.cat([x_1x1, x_3x3])
        return x 
</code></pre>
<p>Here you may notice that we are now hard-coded the exact <strong>kernel size</strong> and <strong>strides</strong> number for both convolutional layers according to the network (diagram). And also in <code>ConvModule</code>, we have already set padding to the <code>same</code>, so that the dimension of the feature maps will be the same for both (<code>self.conv1</code> and <code>self.conv2</code>); which is required in order to concatenate them to the end.</p>
<p>Again, in this module, two variable performs as the placeholder, <code>kernel_size1x1</code>, and <code>kernel_size3x3</code>. This is for the purpose of course. Because we will need different numbers of feature maps to the different stages of the entire model. If we look into the diagram of the model, we will see that <code>InceptionModule</code> takes a different number of filters at different stages in the model.</p>
<h2>Downsample Module</h2>
<p>Lastly the <strong>downsampling module</strong>. The main intuition for downsampling is that we hope to get more relevant feature information that highly represents the inputs to the model. As it tends to remove the unwanted feature so that model can focus on the most relevant. There are many ways we can reduce the dimension of the feature maps (or inputs). For example: using <code>strides 2</code> or using the conventional <code>pooling</code> operation. There are many types of pooling operation, namely: <code>MaxPooling</code>, <code>AveragePooling</code>, <code>GlobalAveragePooling</code>.</p>
<p>From the diagram, we can see that the downsampling module contains <strong>one convolutional</strong> layer and <strong>one max-pooling</strong> layer which later merges together. Now, if we look closely at the diagram (<strong>top-right</strong>), we will see that the convolutional layer takes a <code>3 x 3</code> size filter with <code>strides 2 x 2</code>. And the pooling layer (here <code>MaxPooling</code>) takes pooling size <code>3 x 3</code> with <code>strides 2 x 2</code>. Fair enough, however, we also ensure that the dimension coming from each of them should be the same in order to merge at the end. Now, if we remember when we design the <code>ConvModule</code> we purposely set the value of the <code>padding</code> argument to <code>same</code>. But in this case, we need to set it to <code>valid</code>.</p>
<pre><code>class DownsampleModule(tf.keras.layers.Layer):
    def __init__(self, kernel_size):
        super(DownsampleModule, self).__init__()

        # conv layer
        self.conv3 = ConvModule(kernel_size, kernel_size=(3,3), 
                         strides=(2,2), padding=&quot;valid&quot;) 

        # pooling layer 
        self.pool  = tf.keras.layers.MaxPooling2D(pool_size=(3, 3), 
                         strides=(2,2))
        self.cat   = tf.keras.layers.Concatenate()

    def call(self, input_tensor, training=False):
        # forward pass 
        conv_x = self.conv3(input_tensor, training=training)
        pool_x = self.pool(input_tensor)
    
        # merged
        return self.cat([conv_x, pool_x])
</code></pre>
<p>Okay, now we have built all three modules, namely: <strong>ConvModule InceptionModule DownsampleModule</strong>. Let's initialize their parameter according to the diagram.</p>
<pre><code>class MiniInception(tf.keras.Model):
    def __init__(self, num_classes=10):
        super(MiniInception, self).__init__()

        # the first conv module
        self.conv_block = ConvModule(96, (3,3), (1,1))

        # 2 inception module and 1 downsample module
        self.inception_block1  = InceptionModule(32, 32)
        self.inception_block2  = InceptionModule(32, 48)
        self.downsample_block1 = DownsampleModule(80)
  
        # 4 inception module and 1 downsample module
        self.inception_block3  = InceptionModule(112, 48)
        self.inception_block4  = InceptionModule(96, 64)
        self.inception_block5  = InceptionModule(80, 80)
        self.inception_block6  = InceptionModule(48, 96)
        self.downsample_block2 = DownsampleModule(96)

        # 2 inception module 
        self.inception_block7 = InceptionModule(176, 160)
        self.inception_block8 = InceptionModule(176, 160)

        # average pooling
        self.avg_pool = tf.keras.layers.AveragePooling2D((7,7))

        # model tail
        self.flat      = tf.keras.layers.Flatten()
        self.classfier = tf.keras.layers.Dense(num_classes, activation='softmax')


    def call(self, input_tensor, training=True, **kwargs):
        # forward pass 
        x = self.conv_block(input_tensor)
        x = self.inception_block1(x)
        x = self.inception_block2(x)
        x = self.downsample_block1(x)

        x = self.inception_block3(x)
        x = self.inception_block4(x)
        x = self.inception_block5(x)
        x = self.inception_block6(x)
        x = self.downsample_block2(x)

        x = self.inception_block7(x)
        x = self.inception_block8(x)
        x = self.avg_pool(x)

        x = self.flat(x)
        return self.classfier(x)
</code></pre>
<p>The amount of <strong>filter</strong> number for each computational block is set according to the design of the model (see the diagram). After initialing all the blocks (in the <code>__init__</code> function), we connect them according to the design (in the <code>call</code> function).</p>
How to design an optimal CNN?
<p>I am working on a Ph.D. project, which objective is to reduce <code>CO2</code> emissions on Earth.</p>
<p>I have a dataset, and I was able to successfully implement a <code>CNN</code>, which gives <code>80%</code> accuracy (worst-case scenario). However, the field where I work is very demanding, and I have the impression that I could get better accuracy with a well-optimized CNN.</p>
<p>How do experts design <code>CNN's</code>? How could I choose between <code>Inception</code> Modules, <code>Dropout</code> Regularization, <code>Batch Normalization</code>, convolutional filter size, size and depth of convolutional channels, number of fully-connected layers, activations neurons, etc? How do people navigate this large optimization problem in a scientific manner? The combinations are endless. Are there any real-life examples where this problem is navigated, addressing its full complexity (not just optimizing a few hyper-parameters)?</p>
<p>Hopefully, my dataset is not too large, so the <code>CNN</code> models that I am considering should have very few parameters.</p>