<p>Larger model is often significantly harder to train [0], thus if you simply increase the size/stacked the model, don't expect much improvement compared to simpler model.</p>
<p>Furthermore, how large is your dataset? It seems that at both models, there is a sign of over fitting (or at least stagnating loss value for test).</p>
<p>[0] <a href="https://arxiv.org/abs/1512.03385" rel="nofollow noreferrer">https://arxiv.org/abs/1512.03385</a></p>
Stacked model gets worse results than base models
<p>I have made a stacking model using 5 efficientNet models for a Kaggle competition. Given below is the architecture of the stacking model:</p>
<pre><code>Model: &quot;model&quot;
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1_0 (InputLayer)          [(None, 600, 600, 3) 0                                            
__________________________________________________________________________________________________
input_3_1 (InputLayer)          [(None, 600, 600, 3) 0                                            
__________________________________________________________________________________________________
input_5_2 (InputLayer)          [(None, 600, 600, 3) 0                                            
__________________________________________________________________________________________________
input_7_3 (InputLayer)          [(None, 600, 600, 3) 0                                            
__________________________________________________________________________________________________
input_9_4 (InputLayer)          [(None, 600, 600, 3) 0                                            
__________________________________________________________________________________________________
effnet_layer0_0 (Functional)    (None, None, None, 2 64097680    input_1_0[0][0]                  
__________________________________________________________________________________________________
effnet_layer1_1 (Functional)    (None, None, None, 2 64097680    input_3_1[0][0]                  
__________________________________________________________________________________________________
effnet_layer2_2 (Functional)    (None, None, None, 2 64097680    input_5_2[0][0]                  
__________________________________________________________________________________________________
effnet_layer3_3 (Functional)    (None, None, None, 2 64097680    input_7_3[0][0]                  
__________________________________________________________________________________________________
effnet_layer4_4 (Functional)    (None, None, None, 2 64097680    input_9_4[0][0]                  
__________________________________________________________________________________________________
global_average_pooling2d_0 (Glo (None, 2560)         0           effnet_layer0_0[0][0]            
__________________________________________________________________________________________________
global_average_pooling2d_1_1 (G (None, 2560)         0           effnet_layer1_1[0][0]            
__________________________________________________________________________________________________
global_average_pooling2d_2_2 (G (None, 2560)         0           effnet_layer2_2[0][0]            
__________________________________________________________________________________________________
global_average_pooling2d_3_3 (G (None, 2560)         0           effnet_layer3_3[0][0]            
__________________________________________________________________________________________________
global_average_pooling2d_4_4 (G (None, 2560)         0           effnet_layer4_4[0][0]            
__________________________________________________________________________________________________
dropout_0 (Dropout)             (None, 2560)         0           global_average_pooling2d_0[0][0] 
__________________________________________________________________________________________________
dropout_1_1 (Dropout)           (None, 2560)         0           global_average_pooling2d_1_1[0][0
__________________________________________________________________________________________________
dropout_2_2 (Dropout)           (None, 2560)         0           global_average_pooling2d_2_2[0][0
__________________________________________________________________________________________________
dropout_3_3 (Dropout)           (None, 2560)         0           global_average_pooling2d_3_3[0][0
__________________________________________________________________________________________________
dropout_4_4 (Dropout)           (None, 2560)         0           global_average_pooling2d_4_4[0][0
__________________________________________________________________________________________________
dense_0 (Dense)                 (None, 4)            10244       dropout_0[0][0]                  
__________________________________________________________________________________________________
dense_1_1 (Dense)               (None, 4)            10244       dropout_1_1[0][0]                
__________________________________________________________________________________________________
dense_2_2 (Dense)               (None, 4)            10244       dropout_2_2[0][0]                
__________________________________________________________________________________________________
dense_3_3 (Dense)               (None, 4)            10244       dropout_3_3[0][0]                
__________________________________________________________________________________________________
dense_4_4 (Dense)               (None, 4)            10244       dropout_4_4[0][0]                
__________________________________________________________________________________________________
concatenate (Concatenate)       (None, 20)           0           dense_0[0][0]                    
                                                                 dense_1_1[0][0]                  
                                                                 dense_2_2[0][0]                  
                                                                 dense_3_3[0][0]                  
                                                                 dense_4_4[0][0]                  
__________________________________________________________________________________________________
dense (Dense)                   (None, 10)           210         concatenate[0][0]                
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 4)            44          dense[0][0]                      
==================================================================================================
Total params: 320,539,874
Trainable params: 254
Non-trainable params: 320,539,620
</code></pre>
<p><em>Performance metrics of the stacking model:</em></p>
<p><a href="https://i.stack.imgur.com/3JwV2.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3JwV2.png" alt="stacking model accuracy" /></a></p>
<p><a href="https://i.stack.imgur.com/VeF8a.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VeF8a.png" alt="stacking model loss" /></a></p>
<p><em>Performance metrics of a base model:</em></p>
<p><a href="https://i.stack.imgur.com/JzoFN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JzoFN.png" alt="accuracy" /></a></p>
<p><a href="https://i.stack.imgur.com/5FCM8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5FCM8.png" alt="loss" /></a></p>
<p>But, when I use the stacking model for the Kaggle predictions, I am getting a score of 0.551, whereas when I use one of the base models, I get a score of 0.581.</p>
<p><strong>Why does this happen? Isn't the stacking model supposed to give better results than the base model?</strong></p>