title: No gradients are provided / 'NoneType' object is not callable when trying to fit a multi-output model
body: <p>I'm fairly new to machine learning and I'm having quite a bit of difficulty with this issue. I'm using Kaggle notebook with tensorflow version 2.3.1. I need to train a model with face images and predict multiple attributes, wrinkles, freckles, hair colour etc, hair thickness and glasses, hence multi-output model. When I try to model.fit, on the first instance I get the error of &quot;No gradients are provided&quot;. Upon running the same code without any change gives me error of &quot;NoneType object is not callable&quot;. I'm stuck here for over a week now and so far no solution on the internet has been able to resolve this issue so I'm including as much detail as possible here. Some side info about the problem, Wrinkles and Freckles have values 0 or 1 while other outputs have values ranging from 0 to 3, 0 to 5 or 0 to 9. Here is the code.</p>
  <p>Setting up CNN:</p>
  <pre><code>IMAGE_SIZE = 100
  
  def base_hidden_layers(input_shape):
      model = Conv2D(64, kernel_size = (3, 3), padding = 'same', activation= 'relu')(input_shape)
      model = BatchNormalization(axis=-1)(model)
      model = MaxPooling2D(pool_size=(2, 2), strides=(2, 2))(model)
      model = Dropout(0.25)(model)
      
      model = Conv2D(128, kernel_size = (3, 3), padding = 'same', activation= 'relu')(model)
      model = BatchNormalization(axis=-1)(model)
      model = MaxPooling2D(pool_size=(2, 2), strides=(2, 2))(model)
      model = Dropout(0.25)(model)
      
      model = Conv2D(128, kernel_size = (3, 3), padding = 'same', activation= 'relu')(model)
      model = BatchNormalization(axis=-1)(model)
      model = MaxPooling2D(pool_size=(2, 2), strides=(2, 2))(model)
      model = Dropout(0.25)(model)
      
      return model
  
  def wrinkle_layers(input_shape):
      model = base_hidden_layers(input_shape)
      
      model = Flatten()(model)
      model = Dense(1024, activation = 'relu')(model)
      model = BatchNormalization()(model)
      model = Dropout(0.5)(model)
      model = Dense(2, activation = 'sigmoid', name = 'wrinkles')(model)
      
      return model
      
  def freckles_layers(input_shape):
      model = base_hidden_layers(input_shape)
      
      model = Flatten()(model)
      model = Dense(1024, activation = 'relu')(model)
      model = BatchNormalization()(model)
      model = Dropout(0.5)(model)
      model = Dense(2, activation = 'sigmoid', name = 'freckles')(model)
      
      return model
  
  def glasses_layers(input_shape):
      model = base_hidden_layers(input_shape)
      
      model = Flatten()(model)
      model = Dense(1024, activation = 'relu')(model)
      model = BatchNormalization()(model)
      model = Dropout(0.5)(model)
      model = Dense(3, activation = 'softmax', name = 'glasses')(model)
      
      return model
  
  def hair_color_layers(input_shape):
      model = base_hidden_layers(input_shape)
      
      model = Flatten()(model)
      model = Dense(1024, activation = 'relu')(model)
      model = BatchNormalization()(model)
      model = Dropout(0.5)(model)
      model = Dense(9, activation = 'softmax', name = 'hair_color')(model)
      
      return model
  
  def hair_top_layers(input_shape):
      model = base_hidden_layers(input_shape)
      
      model = Flatten()(model)
      model = Dense(1024, activation = 'relu')(model)
      model = BatchNormalization()(model)
      model = Dropout(0.5)(model)
      model = Dense(4, activation = 'softmax', name = 'hair_top')(model)
      
      return model
  
  def neural_net_model(image_size):
      shape = (image_size, image_size, 3)
      shape = Input(shape=shape)
      
      wrinkle_branch = wrinkle_layers(shape)
      freckles_branch = freckles_layers(shape)
      glasses_branch = glasses_layers(shape)
      hair_col_branch = hair_color_layers(shape)
      hair_top_branch = hair_top_layers(shape)
      
      model = Model(inputs=shape, outputs= [wrinkle_branch, freckles_branch, glasses_branch, hair_col_branch, hair_top_branch], name='face_net')
      
      return model
  
  model = neural_net_model(IMAGE_SIZE)
  </code></pre>
  <p>Compiling the model, setting up loss function:</p>
  <pre><code>model.compile(optimizer=Adam(lr=1e-4, decay= 1e-4 / 100),
               loss = {'wrinkles': 'binary_crossentropy',
                      'freckles': 'binary_crossentropy',
                      'glasses': 'categorical_crossentropy',
                      'hair_color': 'categorical_crossentropy',
                      'hair_top': 'categorical_crossentropy'}, metrics=['accuracy'])
  
  model.summary()
  </code></pre>
  <p>Output for model.summary():</p>
  <pre><code>Model: &quot;face_net&quot;
  __________________________________________________________________________________________________
  Layer (type)                    Output Shape         Param #     Connected to                     
  ==================================================================================================
  input_1 (InputLayer)            [(None, 100, 100, 3) 0                                            
  __________________________________________________________________________________________________
  conv2d (Conv2D)                 (None, 100, 100, 64) 1792        input_1[0][0]                    
  __________________________________________________________________________________________________
  conv2d_3 (Conv2D)               (None, 100, 100, 64) 1792        input_1[0][0]                    
  __________________________________________________________________________________________________
  conv2d_6 (Conv2D)               (None, 100, 100, 64) 1792        input_1[0][0]                    
  __________________________________________________________________________________________________
  conv2d_9 (Conv2D)               (None, 100, 100, 64) 1792        input_1[0][0]                    
  __________________________________________________________________________________________________
  conv2d_12 (Conv2D)              (None, 100, 100, 64) 1792        input_1[0][0]                    
  __________________________________________________________________________________________________
  batch_normalization (BatchNorma (None, 100, 100, 64) 256         conv2d[0][0]                     
  __________________________________________________________________________________________________
  batch_normalization_4 (BatchNor (None, 100, 100, 64) 256         conv2d_3[0][0]                   
  __________________________________________________________________________________________________
  batch_normalization_8 (BatchNor (None, 100, 100, 64) 256         conv2d_6[0][0]                   
  __________________________________________________________________________________________________
  batch_normalization_12 (BatchNo (None, 100, 100, 64) 256         conv2d_9[0][0]                   
  __________________________________________________________________________________________________
  batch_normalization_16 (BatchNo (None, 100, 100, 64) 256         conv2d_12[0][0]                  
  __________________________________________________________________________________________________
  max_pooling2d (MaxPooling2D)    (None, 50, 50, 64)   0           batch_normalization[0][0]        
  __________________________________________________________________________________________________
  max_pooling2d_3 (MaxPooling2D)  (None, 50, 50, 64)   0           batch_normalization_4[0][0]      
  __________________________________________________________________________________________________
  max_pooling2d_6 (MaxPooling2D)  (None, 50, 50, 64)   0           batch_normalization_8[0][0]      
  __________________________________________________________________________________________________
  max_pooling2d_9 (MaxPooling2D)  (None, 50, 50, 64)   0           batch_normalization_12[0][0]     
  __________________________________________________________________________________________________
  max_pooling2d_12 (MaxPooling2D) (None, 50, 50, 64)   0           batch_normalization_16[0][0]     
  __________________________________________________________________________________________________
  dropout (Dropout)               (None, 50, 50, 64)   0           max_pooling2d[0][0]              
  __________________________________________________________________________________________________
  dropout_4 (Dropout)             (None, 50, 50, 64)   0           max_pooling2d_3[0][0]            
  __________________________________________________________________________________________________
  dropout_8 (Dropout)             (None, 50, 50, 64)   0           max_pooling2d_6[0][0]            
  __________________________________________________________________________________________________
  dropout_12 (Dropout)            (None, 50, 50, 64)   0           max_pooling2d_9[0][0]            
  __________________________________________________________________________________________________
  dropout_16 (Dropout)            (None, 50, 50, 64)   0           max_pooling2d_12[0][0]           
  __________________________________________________________________________________________________
  conv2d_1 (Conv2D)               (None, 50, 50, 128)  73856       dropout[0][0]                    
  __________________________________________________________________________________________________
  conv2d_4 (Conv2D)               (None, 50, 50, 128)  73856       dropout_4[0][0]                  
  __________________________________________________________________________________________________
  conv2d_7 (Conv2D)               (None, 50, 50, 128)  73856       dropout_8[0][0]                  
  __________________________________________________________________________________________________
  conv2d_10 (Conv2D)              (None, 50, 50, 128)  73856       dropout_12[0][0]                 
  __________________________________________________________________________________________________
  conv2d_13 (Conv2D)              (None, 50, 50, 128)  73856       dropout_16[0][0]                 
  __________________________________________________________________________________________________
  batch_normalization_1 (BatchNor (None, 50, 50, 128)  512         conv2d_1[0][0]                   
  __________________________________________________________________________________________________
  batch_normalization_5 (BatchNor (None, 50, 50, 128)  512         conv2d_4[0][0]                   
  __________________________________________________________________________________________________
  batch_normalization_9 (BatchNor (None, 50, 50, 128)  512         conv2d_7[0][0]                   
  __________________________________________________________________________________________________
  batch_normalization_13 (BatchNo (None, 50, 50, 128)  512         conv2d_10[0][0]                  
  __________________________________________________________________________________________________
  batch_normalization_17 (BatchNo (None, 50, 50, 128)  512         conv2d_13[0][0]                  
  __________________________________________________________________________________________________
  max_pooling2d_1 (MaxPooling2D)  (None, 25, 25, 128)  0           batch_normalization_1[0][0]      
  __________________________________________________________________________________________________
  max_pooling2d_4 (MaxPooling2D)  (None, 25, 25, 128)  0           batch_normalization_5[0][0]      
  __________________________________________________________________________________________________
  max_pooling2d_7 (MaxPooling2D)  (None, 25, 25, 128)  0           batch_normalization_9[0][0]      
  __________________________________________________________________________________________________
  max_pooling2d_10 (MaxPooling2D) (None, 25, 25, 128)  0           batch_normalization_13[0][0]     
  __________________________________________________________________________________________________
  max_pooling2d_13 (MaxPooling2D) (None, 25, 25, 128)  0           batch_normalization_17[0][0]     
  __________________________________________________________________________________________________
  dropout_1 (Dropout)             (None, 25, 25, 128)  0           max_pooling2d_1[0][0]            
  __________________________________________________________________________________________________
  dropout_5 (Dropout)             (None, 25, 25, 128)  0           max_pooling2d_4[0][0]            
  __________________________________________________________________________________________________
  dropout_9 (Dropout)             (None, 25, 25, 128)  0           max_pooling2d_7[0][0]            
  __________________________________________________________________________________________________
  dropout_13 (Dropout)            (None, 25, 25, 128)  0           max_pooling2d_10[0][0]           
  __________________________________________________________________________________________________
  dropout_17 (Dropout)            (None, 25, 25, 128)  0           max_pooling2d_13[0][0]           
  __________________________________________________________________________________________________
  conv2d_2 (Conv2D)               (None, 25, 25, 128)  147584      dropout_1[0][0]                  
  __________________________________________________________________________________________________
  conv2d_5 (Conv2D)               (None, 25, 25, 128)  147584      dropout_5[0][0]                  
  __________________________________________________________________________________________________
  conv2d_8 (Conv2D)               (None, 25, 25, 128)  147584      dropout_9[0][0]                  
  __________________________________________________________________________________________________
  conv2d_11 (Conv2D)              (None, 25, 25, 128)  147584      dropout_13[0][0]                 
  __________________________________________________________________________________________________
  conv2d_14 (Conv2D)              (None, 25, 25, 128)  147584      dropout_17[0][0]                 
  __________________________________________________________________________________________________
  batch_normalization_2 (BatchNor (None, 25, 25, 128)  512         conv2d_2[0][0]                   
  __________________________________________________________________________________________________
  batch_normalization_6 (BatchNor (None, 25, 25, 128)  512         conv2d_5[0][0]                   
  __________________________________________________________________________________________________
  batch_normalization_10 (BatchNo (None, 25, 25, 128)  512         conv2d_8[0][0]                   
  __________________________________________________________________________________________________
  batch_normalization_14 (BatchNo (None, 25, 25, 128)  512         conv2d_11[0][0]                  
  __________________________________________________________________________________________________
  batch_normalization_18 (BatchNo (None, 25, 25, 128)  512         conv2d_14[0][0]                  
  __________________________________________________________________________________________________
  max_pooling2d_2 (MaxPooling2D)  (None, 12, 12, 128)  0           batch_normalization_2[0][0]      
  __________________________________________________________________________________________________
  max_pooling2d_5 (MaxPooling2D)  (None, 12, 12, 128)  0           batch_normalization_6[0][0]      
  __________________________________________________________________________________________________
  max_pooling2d_8 (MaxPooling2D)  (None, 12, 12, 128)  0           batch_normalization_10[0][0]     
  __________________________________________________________________________________________________
  max_pooling2d_11 (MaxPooling2D) (None, 12, 12, 128)  0           batch_normalization_14[0][0]     
  __________________________________________________________________________________________________
  max_pooling2d_14 (MaxPooling2D) (None, 12, 12, 128)  0           batch_normalization_18[0][0]     
  __________________________________________________________________________________________________
  dropout_2 (Dropout)             (None, 12, 12, 128)  0           max_pooling2d_2[0][0]            
  __________________________________________________________________________________________________
  dropout_6 (Dropout)             (None, 12, 12, 128)  0           max_pooling2d_5[0][0]            
  __________________________________________________________________________________________________
  dropout_10 (Dropout)            (None, 12, 12, 128)  0           max_pooling2d_8[0][0]            
  __________________________________________________________________________________________________
  dropout_14 (Dropout)            (None, 12, 12, 128)  0           max_pooling2d_11[0][0]           
  __________________________________________________________________________________________________
  dropout_18 (Dropout)            (None, 12, 12, 128)  0           max_pooling2d_14[0][0]           
  __________________________________________________________________________________________________
  flatten (Flatten)               (None, 18432)        0           dropout_2[0][0]                  
  __________________________________________________________________________________________________
  flatten_1 (Flatten)             (None, 18432)        0           dropout_6[0][0]                  
  __________________________________________________________________________________________________
  flatten_2 (Flatten)             (None, 18432)        0           dropout_10[0][0]                 
  __________________________________________________________________________________________________
  flatten_3 (Flatten)             (None, 18432)        0           dropout_14[0][0]                 
  __________________________________________________________________________________________________
  flatten_4 (Flatten)             (None, 18432)        0           dropout_18[0][0]                 
  __________________________________________________________________________________________________
  dense (Dense)                   (None, 1024)         18875392    flatten[0][0]                    
  __________________________________________________________________________________________________
  dense_1 (Dense)                 (None, 1024)         18875392    flatten_1[0][0]                  
  __________________________________________________________________________________________________
  dense_2 (Dense)                 (None, 1024)         18875392    flatten_2[0][0]                  
  __________________________________________________________________________________________________
  dense_3 (Dense)                 (None, 1024)         18875392    flatten_3[0][0]                  
  __________________________________________________________________________________________________
  dense_4 (Dense)                 (None, 1024)         18875392    flatten_4[0][0]                  
  __________________________________________________________________________________________________
  batch_normalization_3 (BatchNor (None, 1024)         4096        dense[0][0]                      
  __________________________________________________________________________________________________
  batch_normalization_7 (BatchNor (None, 1024)         4096        dense_1[0][0]                    
  __________________________________________________________________________________________________
  batch_normalization_11 (BatchNo (None, 1024)         4096        dense_2[0][0]                    
  __________________________________________________________________________________________________
  batch_normalization_15 (BatchNo (None, 1024)         4096        dense_3[0][0]                    
  __________________________________________________________________________________________________
  batch_normalization_19 (BatchNo (None, 1024)         4096        dense_4[0][0]                    
  __________________________________________________________________________________________________
  dropout_3 (Dropout)             (None, 1024)         0           batch_normalization_3[0][0]      
  __________________________________________________________________________________________________
  dropout_7 (Dropout)             (None, 1024)         0           batch_normalization_7[0][0]      
  __________________________________________________________________________________________________
  dropout_11 (Dropout)            (None, 1024)         0           batch_normalization_11[0][0]     
  __________________________________________________________________________________________________
  dropout_15 (Dropout)            (None, 1024)         0           batch_normalization_15[0][0]     
  __________________________________________________________________________________________________
  dropout_19 (Dropout)            (None, 1024)         0           batch_normalization_19[0][0]     
  __________________________________________________________________________________________________
  wrinkles (Dense)                (None, 2)            2050        dropout_3[0][0]                  
  __________________________________________________________________________________________________
  freckles (Dense)                (None, 2)            2050        dropout_7[0][0]                  
  __________________________________________________________________________________________________
  glasses (Dense)                 (None, 3)            3075        dropout_11[0][0]                 
  __________________________________________________________________________________________________
  hair_color (Dense)              (None, 9)            9225        dropout_15[0][0]                 
  __________________________________________________________________________________________________
  hair_top (Dense)                (None, 4)            4100        dropout_19[0][0]                 
  ==================================================================================================
  Total params: 95,540,500
  Trainable params: 95,527,060
  Non-trainable params: 13,440
  </code></pre>
  <p>Shape of training and validation sets:</p>
  <pre><code>print(valid_images.shape)
  print(valid_labels.shape)
  print(train_images.shape)
  print(train_labels.shape)
  print(type(valid_labels))
  </code></pre>
  <p>Output:</p>
  <pre><code>(319, 100, 100, 3)
  (319, 5)
  (1272, 100, 100, 3)
  (1272, 5)
  &lt;class 'numpy.ndarray'&gt;
  </code></pre>
  <p>The above shape for input means: (Number of images, image length, image width, image depth (RGB))</p>
  <p>Output means: (Number of Images, Number of Output columns)</p>
  <p>Printing the training input and output:</p>
  <pre><code>train_images
  </code></pre>
  <p>Output:</p>
  <pre><code>array([[[[0.41176471, 0.49019608, 0.44705882],
           [0.40784314, 0.49411765, 0.44705882],
           [0.41176471, 0.50588235, 0.45490196],
           ...,
           [0.77647059, 0.81568627, 0.81960784],
           [0.7372549 , 0.76862745, 0.78823529],
           [0.57254902, 0.6       , 0.63921569]]]])
  
  train_labels
  </code></pre>
  <p>Output:</p>
  <pre><code>array([[0, 0, 0, 2, 2],
         [0, 1, 0, 0, 2],
         [0, 0, 0, 0, 2],
         ...,
         [0, 0, 0, 1, 2],
         [0, 0, 0, 2, 2],
         [1, 0, 0, 2, 2]])
  </code></pre>
  <p>Fitting the Model:</p>
  <pre><code>history = model.fit((train_images, train_labels),
                      epochs = 100, 
                      validation_data=(valid_images, 
                                       valid_labels))
  
  plt.plot(history.history['accuracy'], label='accuracy')
  plt.plot(history.history['val_accuracy'], label = 'val_accuracy')
  plt.xlabel('Epoch')
  plt.ylabel('Accuracy')
  plt.show()
  
  test_loss, test_acc = model.evaluate(test_images,  test_labels)
  </code></pre>
  <p>Error Stack Trace on running model.fit for the first time since start of notebook:</p>
  <pre><code>Epoch 1/100
  ---------------------------------------------------------------------------
  ValueError                                Traceback (most recent call last)
  &lt;ipython-input-34-0dd91e39ee8e&gt; in &lt;module&gt;
        5                     epochs = 100,
        6                     validation_data=(valid_images, 
  ----&gt; 7                                      valid_labels))
        8 
        9 plt.plot(history.history['accuracy'], label='accuracy')
  
  /opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
      106   def _method_wrapper(self, *args, **kwargs):
      107     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
  --&gt; 108       return method(self, *args, **kwargs)
      109 
      110     # Running inside `run_distribute_coordinator` already.
  
  /opt/conda/
  ...
  
  ValueError: in user code:
  
      /opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
          return ...
      /opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:1271 _filter_grads
          ([v.name for _, v in grads_and_vars],))
  
      ValueError: No gradients provided for any variable: ['conv2d/kernel:0', 'conv2d/bias:0', 'conv2d_3/kernel:0', ... 'hair_top/kernel:0', 'hair_top/bias:0'].
  </code></pre>
  <p>Error on running code again:</p>
  <pre><code>Epoch 1/100
  ---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  &lt;ipython-input-38-356b6cd04e18&gt; in &lt;module&gt;
        2                     epochs = 100,
        3                     validation_data=(valid_images, 
  ----&gt; 4                                      valid_labels))
        5 
        6 plt.plot(history.history['accuracy'], label='accuracy')
  
  /opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
      106   def _method_wrapper(self, *args, **kwargs):
      107     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
  --&gt; 108       return method(self, *args, **kwargs)
      109 
      110     # Running inside `run_distribute_coordinator` already.
  
      ...
  TypeError: 'NoneType' object is not callable
  </code></pre>
  <p>Only important bits of stacktrace have been added because adding the whole stacktrace was making this post over 3000 characters.</p>