title: Multi-input Convolutional Neural Network for Images Classification
body: <p>I am beginner in deep learning and I want to create a multi-input Convolutional Neural Network (CNN) model in Keras for <strong>Images Classification</strong>. </p>
  
  <p>I am working to create a CNN model that takes <strong>two images</strong> and gives <strong>one</strong> output which is the <strong>class of the two images</strong>. </p>
  
  <p>I have two datasets: type1 and type2, and each dataset contains <strong>the same  classes</strong>, but <strong>the number of images in each class in the dataset type1 is higher than the number of images in each class in the dataset type2.</strong> 
  The model should take one image from Type1 dataset and one image from Type2 dataset and then classify these images to <strong>one class</strong> (ClassA or ClassB or------).</p>
  
  <p>The following is the structure of the datasets.</p>
  
  <pre><code>Type1 dataset
  |Train
                |ClassA
                               |image1
                               |image2
                               |image3
                               |image4
                              -----
                |ClassB
                               |image1
                               |image2
                               |image3
                               |image4
                              -----
                |ClassC
                               |image1
                               |image2
                               |image3
                               |image4
                              -----
                |ClassD
                               |image1
                               |image2
                               |image3
                               |image4
                              -----
         ----------------
  |Validate
              -----------
  |Test
             --------------
  
  Type2 dataset
  |Train
                |ClassA
                               |image1
                               |image2
                              -----
                |ClassB
                               |image1
                               |image2
                              -----
                |ClassC
                               |image1
                               |image2
                              -----
                |ClassD
                               |image1
                               |image2
                              -----
         ----------------
  |Validate
              -----------
  |Test
             --------------
  </code></pre>
  
  <p>The model is very similar to the model in this images, but it has more layers before flatten layer.</p>
  
  <p><a href="https://i.stack.imgur.com/Dg2QX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Dg2QX.png" alt="Multi-input"></a></p>
  
  <p>I created a custom generator that inputs two images (from type 1 and 2), and each image from type1 be paired with every images from type2 as long as these images belong to the <strong>same class (label)</strong>. </p>
  
  <p>The problem is when executing <code>fit_generator</code> I get infinite loop as the following:</p>
  
  <pre><code>  Found *** images belonging to 100 classes.
  
      Found *** images belonging to 100 classes.
      Found *** images belonging to 100 classes.
      Found *** images belonging to 100 classes.
      Found *** images belonging to 100 classes.
      Found *** images belonging to 100 classes.
      Found *** images belonging to 100 classes.
      Found *** images belonging to 100 classes.
      Found *** images belonging to 100 classes.
      Found *** images belonging to 100 classes.
      Found *** images belonging to 100 classes.
      Found *** images belonging to 100 classes. ......
  .................................................................
  </code></pre>
  
  <p>Here is my custom generator code:</p>
  
  <pre><code>input_imgen = ImageDataGenerator( 
                                    rotation_range=10,
                                    shear_range=0.2,
                                    zoom_range=0.1,
                                    width_shift_range=0.1,
                                    height_shift_range=0.1
                                    )
  
  
  
  test_imgen = ImageDataGenerator()
  
  
  
  def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
  
  
      genX1 = generator.flow_from_directory(dir1,
                                            target_size = (img_height,img_width),
                                            class_mode = 'categorical',
                                            batch_size = batch_size,
                                            shuffle=False, 
                                            seed=7)
  
      genX2 = generator.flow_from_directory(dir2,
                                            target_size = (img_height,img_width),
                                            class_mode = 'categorical',
                                            batch_size = batch_size,
                                            shuffle=False, 
                                            seed=7)
      while True:
        X2i = genX2.next() 
        Type1 = []
        Type2 = []
        image1 = []
        image2 = []
  
        while True:
          X1i = genX1.next() 
          for i in range(len(X2i[1])): #Type2
            for j in range(len(X1i[1])): #Type1
              if all(X2i[1][i]) == all(X1i[1][j]): # have same label
                image1.append(X1i[0][j]) # add image
                image1.append(X1i[1][j]) # add label
                image2.append(X2i[0][i]) # add image
                image2.append(X2i[1][i]) # add label
        Type1.append(image1)
        Type2.append(image2)
        yield [Type1 [0], Type2 [0]], Type2 [1]  #Yield both images and their mutual label
  
  
  inputgenerator=generate_generator_multiple(generator=input_imgen,
                                             dir1=train_iris_data,
                                             dir2=train_face_data,
                                             batch_size=32,
                                             img_height=224,
                                             img_width=224)       
  
  validgenerator=generate_generator_multiple(generator=test_imgen,
                                            dir1=valid_iris_data,
                                            dir2=valid_face_data,
                                            batch_size=32,
                                            img_height=224,
                                            img_width=224) 
  
  testgenerator=generate_generator_multiple(generator=test_imgen,
                                            dir1=test_face_data,
                                            dir2=test_face_data,
                                            batch_size=32,
                                            img_height=224,
                                            img_width=224)
  
  
      # compile the model
      multi_model.compile(
              loss='categorical_crossentropy',
              optimizer=Adam(lr=0.0001),
              metrics=['accuracy']
          )
  
  
  # train the model and save the history
  history = multi_model.fit_generator(
  inputgenerator,
  steps_per_epoch=len(train_data) // batch_size,
  epochs=10,
  verbose=1,
  validation_data=validgenerator,
  validation_steps=len(valid_data) // batch_size,
  use_multiprocessing=True,
  shuffle=False
  )
  </code></pre>
  
  <p>Could you please help me to solve this problem and create the custom generator??</p>