title: Multi-input and multi-output models based on keras2.0
body: <p>I'm searching for a long time on the net. But I don't found nothing. Please help me or try to give me some ideas about how to achieve this.</p>
  
  <p>I have built the 3-inputs and 2-outputs model which is shown below.</p>
  
  <pre><code>the code block is from def build_srgan_model in class SRGANNetwork:
  
      ip = Input(shape=(self.img_width, self.img_height,3), name='x_generator')
      ip_gan = Input(shape=(large_width, large_height,3), name='x_discriminator') 
      ip_vgg = Input(shape=(large_width, large_height,3), name='x_vgg')
  
      sr_output = self.generative_network.create_sr_model(ip)
      self.generative_model_ = Model(ip, sr_output)
  
      gan_output = self.discriminative_network.append_gan_network(ip_gan)
      self.discriminative_model_ = Model(ip_gan, gan_output)
  
      gan_output = self.discriminative_model_(self.generative_model_.output)
      vgg_output = self.vgg_network.append_vgg_network(self.generative_model_.output, ip_vgg)
  
      self.srgan_model_ = Model(inputs=[ip, ip_gan, ip_vgg], outputs=[gan_output, vgg_output])
  </code></pre>
  
  <p>this is complete code，it's too long.</p>
  
  <p>class VGGNetwork:</p>
  
  <pre><code>'''
  Helper class to load VGG and its weights to the FastNet model
  '''
  
  def __init__(self, img_width=384, img_height=384, vgg_weight=1.0):
      self.img_height = img_height
      self.img_width = img_width
      self.vgg_weight = vgg_weight
  
      self.vgg_layers = None
  
  def append_vgg_network(self, x_in, true_X_input, pre_train=False):
  
      # Append the initial inputs to the outputs of the SRResNet
      x = merge([x_in, true_X_input], mode='concat', concat_axis=0)
  
      # Normalize the inputs via custom VGG Normalization layer
      x = Normalize(name="normalize_vgg")(x)
  
      # Begin adding the VGG layers
      x = Convolution2D(64, 3, 3, activation='relu', name='vgg_conv1_1', border_mode='same')(x)
  
      x = Convolution2D(64, 3, 3, activation='relu', name='vgg_conv1_2', border_mode='same')(x)
      x = MaxPooling2D(name='vgg_maxpool1')(x)
  
      x = Convolution2D(128, 3, 3, activation='relu', name='vgg_conv2_1', border_mode='same')(x)
  
      if pre_train:
          vgg_regularizer2 = ContentVGGRegularizer(weight=self.vgg_weight)
          x = Convolution2D(128, 3, 3, activation='relu', name='vgg_conv2_2', border_mode='same',
                            activity_regularizer=vgg_regularizer2)(x)
      else:
          x = Convolution2D(128, 3, 3, activation='relu', name='vgg_conv2_2', border_mode='same')(x)
      x = MaxPooling2D(name='vgg_maxpool2')(x)
  
      x = Convolution2D(256, 3, 3, activation='relu', name='vgg_conv3_1', border_mode='same')(x)
      x = Convolution2D(256, 3, 3, activation='relu', name='vgg_conv3_2', border_mode='same')(x)
  
      x = Convolution2D(256, 3, 3, activation='relu', name='vgg_conv3_3', border_mode='same')(x)
      x = MaxPooling2D(name='vgg_maxpool3')(x)
  
      x = Convolution2D(512, 3, 3, activation='relu', name='vgg_conv4_1', border_mode='same')(x)
      x = Convolution2D(512, 3, 3, activation='relu', name='vgg_conv4_2', border_mode='same')(x)
  
      x = Convolution2D(512, 3, 3, activation='relu', name='vgg_conv4_3', border_mode='same')(x)
      x = MaxPooling2D(name='vgg_maxpool4')(x)
  
      x = Convolution2D(512, 3, 3, activation='relu', name='vgg_conv5_1', border_mode='same')(x)
      x = Convolution2D(512, 3, 3, activation='relu', name='vgg_conv5_2', border_mode='same')(x)
  
      if not pre_train:
          vgg_regularizer5 = ContentVGGRegularizer(weight=self.vgg_weight)
          x = Convolution2D(512, 3, 3, activation='relu', name='vgg_conv5_3', border_mode='same',
                        activity_regularizer=vgg_regularizer5)(x)
      else:
          x = Convolution2D(512, 3, 3, activation='relu', name='vgg_conv5_3', border_mode='same')(x)
      x = MaxPooling2D(name='vgg_maxpool5')(x)
  
      return x
  
  def load_vgg_weight(self, model):
      # Loading VGG 16 weights
      if K.image_dim_ordering() == "th":
          weights = get_file('vgg16_weights_th_dim_ordering_th_kernels_notop.h5', THEANO_WEIGHTS_PATH_NO_TOP,
                                 cache_subdir='models')
      else:
          weights = get_file('vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5', TF_WEIGHTS_PATH_NO_TOP,
                                 cache_subdir='models')
      f = h5py.File(weights)
  
      layer_names = [name for name in f.attrs['layer_names']]
  
      if self.vgg_layers is None:
          self.vgg_layers = [layer for layer in model.layers
                             if 'vgg_' in layer.name]
  
      for i, layer in enumerate(self.vgg_layers):
          g = f[layer_names[i]]  #g is i layer weight of .h5  //dict{'weight_names':wight]}
          weights = [g[name] for name in g.attrs['weight_names']]
          layer.set_weights(weights)
  
      # Freeze all VGG layers
      for layer in self.vgg_layers:
          layer.trainable = False
  
      return model
  </code></pre>
  
  <p>class DiscriminatorNetwork:</p>
  
  <pre><code>def __init__(self, img_width=384, img_height=384, adversarial_loss_weight=1, small_model=False):
      self.img_width = img_width
      self.img_height = img_height
      self.adversarial_loss_weight = adversarial_loss_weight
      self.small_model = small_model
  
      self.k = 3
      self.mode = 2
      self.weights_path = "weights/Discriminator weights.h5"
  
      self.gan_layers = None
  
  def append_gan_network(self, true_X_input):
  
      # Normalize the inputs via custom VGG Normalization layer
      x = Normalize(type="gan", value=127.5, name="gan_normalize")(true_X_input)
  
      x = Convolution2D(64, self.k, self.k, border_mode='same', name='gan_conv1_1')(x)
      x = LeakyReLU(0.3, name="gan_lrelu1_1")(x)
  
      x = Convolution2D(64, self.k, self.k, border_mode='same', name='gan_conv1_2', subsample=(2, 2))(x)
      x = LeakyReLU(0.3, name='gan_lrelu1_2')(x)
      x = BatchNormalization(mode=self.mode, axis=channel_axis, name='gan_batchnorm1_1')(x)
  
      filters = [128, 256] if self.small_model else [128, 256, 512]
  
      for i, nb_filters in enumerate(filters):
          for j in range(2):
              subsample = (2, 2) if j == 1 else (1, 1)
  
              x = Convolution2D(nb_filters, self.k, self.k, border_mode='same', subsample=subsample,
                                name='gan_conv%d_%d' % (i + 2, j + 1))(x)
              x = LeakyReLU(0.3, name='gan_lrelu_%d_%d' % (i + 2, j + 1))(x)
              x = BatchNormalization(mode=self.mode, axis=channel_axis, name='gan_batchnorm%d_%d' % (i + 2, j + 1))(x)
  
      x = Flatten(name='gan_flatten')(x)
  
      output_dim = 128 if self.small_model else 1024
  
      x = Dense(output_dim, name='gan_dense1')(x)
      x = LeakyReLU(0.3, name='gan_lrelu5')(x)
  
      gan_regulrizer = AdversarialLossRegularizer(weight=self.adversarial_loss_weight)
      x = Dense(2, activation="softmax", activity_regularizer=gan_regulrizer, name='gan_output')(x)
  
      return x
  
  def set_trainable(self, model, value=True):
      if self.gan_layers is None:
          disc_model = [layer for layer in model.layers
                        if 'model' in layer.name][0] # Only disc model is an inner model
  
          self.gan_layers = [layer for layer in disc_model.layers
                             if 'gan_' in layer.name]
  
      for layer in self.gan_layers:
          layer.trainable = value
  
  def load_gan_weights(self, model):
      f = h5py.File(self.weights_path)
  
      layer_names = [name for name in f.attrs['layer_names']]
      layer_names = layer_names[1:] # First is an input layer. Not needed.
  
      if self.gan_layers is None:
          self.gan_layers = [layer for layer in model.layers
                              if 'gan_' in layer.name]
  
      for i, layer in enumerate(self.gan_layers):
          g = f[layer_names[i]]
          weights = [g[name] for name in g.attrs['weight_names']]
          layer.set_weights(weights)
  
      print("GAN Model weights loaded.")
      return model
  
  def save_gan_weights(self, model):
      print('GAN Weights are being saved.')
      model.save_weights(self.weights_path, overwrite=True)
      print('GAN Weights saved.')
  </code></pre>
  
  <p>class GenerativeNetwork:</p>
  
  <pre><code>def __init__(self, img_width=96, img_height=96, batch_size=16, nb_upscales=2, small_model=False,
               content_weight=1, tv_weight=2e5, gen_channels=64):
      self.img_width = img_width
      self.img_height = img_height
      self.batch_size = batch_size
      self.small_model = small_model
      self.nb_scales = nb_upscales
  
      self.content_weight = content_weight
      self.tv_weight = tv_weight
  
      self.filters = gen_channels
      self.mode = 2
      self.init = 'glorot_uniform'
  
      self.sr_res_layers = None
      self.sr_weights_path = "weights/SRGAN.h5"
  
      self.output_func = None
  
  def create_sr_model(self, ip):
  
      x = Convolution2D(self.filters, 5, 5, activation='linear', border_mode='same', name='sr_res_conv1',
                        init=self.init)(ip)
      x = BatchNormalization(axis=channel_axis, mode=self.mode, name='sr_res_bn_1')(x)
      x = LeakyReLU(alpha=0.25, name='sr_res_lr1')(x)
  
      # x = Convolution2D(self.filters, 5, 5, activation='linear', border_mode='same', name='sr_res_conv2')(x)
      # x = BatchNormalization(axis=channel_axis, mode=self.mode, name='sr_res_bn_2')(x)
      # x = LeakyReLU(alpha=0.25, name='sr_res_lr2')(x)
  
      nb_residual = 5 if self.small_model else 15
  
      for i in range(nb_residual):
          x = self._residual_block(x, i + 1)
  
      for scale in range(self.nb_scales):
          x = self._upscale_block(x, scale + 1)
  
      scale = 2 ** self.nb_scales
      tv_regularizer = TVRegularizer(img_width=self.img_width * scale, img_height=self.img_height * scale,
                                     weight=self.tv_weight) #self.tv_weight)
  
      x = Convolution2D(3, 5, 5, activation='tanh', border_mode='same', activity_regularizer=tv_regularizer,
                        init=self.init, name='sr_res_conv_final')(x)
  
      x = Denormalize(name='sr_res_conv_denorm')(x)
  
      return x
  
  def _residual_block(self, ip, id):
      init = ip
  
      x = Convolution2D(self.filters, 3, 3, activation='linear', border_mode='same', name='sr_res_conv_' + str(id) + '_1',
                        init=self.init)(ip)
      x = BatchNormalization(axis=channel_axis, mode=self.mode, name='sr_res_bn_' + str(id) + '_1')(x)
      x = LeakyReLU(alpha=0.25, name="sr_res_activation_" + str(id) + "_1")(x)
  
      x = Convolution2D(self.filters, 3, 3, activation='linear', border_mode='same', name='sr_res_conv_' + str(id) + '_2',
                        init=self.init)(x)
      x = BatchNormalization(axis=channel_axis, mode=self.mode, name='sr_res_bn_' + str(id) + '_2')(x)
  
      m = merge([x, init], mode='sum', name="sr_res_merge_" + str(id))
  
      return m
  
  def _upscale_block(self, ip, id):
      '''
      As per suggestion from http://distill.pub/2016/deconv-checkerboard/, I am swapping out
      SubPixelConvolution to simple Nearest Neighbour Upsampling
      '''
      init = ip
  
      x = Convolution2D(128, 3, 3, activation="linear", border_mode='same', name='sr_res_upconv1_%d' % id,
                        init=self.init)(init)
      x = LeakyReLU(alpha=0.25, name='sr_res_up_lr_%d_1_1' % id)(x)
      x = UpSampling2D(name='sr_res_upscale_%d' % id)(x)
      #x = SubPixelUpscaling(r=2, channels=32)(x)
      x = Convolution2D(128, 3, 3, activation="linear", border_mode='same', name='sr_res_filter1_%d' % id,
                        init=self.init)(x)
      x = LeakyReLU(alpha=0.3, name='sr_res_up_lr_%d_1_2' % id)(x)
  
      return x
  
  def set_trainable(self, model, value=True):
      if self.sr_res_layers is None:
          self.sr_res_layers = [layer for layer in model.layers
                                  if 'sr_res_' in layer.name]
  
      for layer in self.sr_res_layers:
          layer.trainable = value
  
  def get_generator_output(self, input_img, srgan_model):
      if self.output_func is None:
          gen_output_layer = [layer for layer in srgan_model.layers
                              if layer.name == "sr_res_conv_denorm"][0]
          self.output_func = K.function([srgan_model.layers[0].input],
                                        [gen_output_layer.output])
  
      return self.output_func([input_img])
  </code></pre>
  
  <p>class SRGANNetwork:</p>
  
  <pre><code>def __init__(self, img_width=96, img_height=96, batch_size=16, nb_scales=2):
      self.img_width = img_width
      self.img_height = img_height
      self.batch_size = batch_size
      self.nb_scales = nb_scales  #factor 4X using nb_scales Upsampling2D
  
      self.discriminative_network = None # type: DiscriminatorNetwork
      self.generative_network = None # type: GenerativeNetwork
      self.vgg_network = None # type: VGGNetwork
  
      self.srgan_model_ = None # type: Model
      self.generative_model_ = None # type: Model
      self.discriminative_model_ = None #type: Model
  
  def build_srgan_pretrain_model(self, use_small_srgan=False):
      large_width = self.img_width * 4
      large_height = self.img_height * 4
  
      self.generative_network = GenerativeNetwork(self.img_width, self.img_height, self.batch_size, self.nb_scales,
                                                  use_small_srgan)
      self.vgg_network = VGGNetwork(large_width, large_height)
  
      ip = Input(shape=(3, self.img_width, self.img_height), name='x_generator')
      ip_vgg = Input(shape=(3, large_width, large_height), name='x_vgg')  # Actual X images
  
      sr_output = self.generative_network.create_sr_model(ip)
      self.generative_model_ = Model(ip, sr_output)
  
      vgg_output = self.vgg_network.append_vgg_network(sr_output, ip_vgg, pre_train=True)
  
      self.srgan_model_ = Model(input=[ip, ip_vgg],
                                output=vgg_output)
  
      self.vgg_network.load_vgg_weight(self.srgan_model_)
  
      srgan_optimizer = Adam(lr=1e-4)
      generator_optimizer = Adam(lr=1e-4)
  
      self.generative_model_.compile(generator_optimizer, dummy_loss)
      self.srgan_model_.compile(srgan_optimizer, dummy_loss)
  
      return self.srgan_model_
  
  
  def build_discriminator_pretrain_model(self, use_smal_srgan=False, use_small_discriminator=False):
      large_width = self.img_width * 4
      large_height = self.img_height * 4
  
      self.generative_network = GenerativeNetwork(self.img_width, self.img_height, self.batch_size, self.nb_scales,
                                                  use_small_srgan)
      self.discriminative_network = DiscriminatorNetwork(large_width, large_height,
                                                         small_model=use_small_discriminator)
  
      ip = Input(shape=(3, self.img_width, self.img_height), name='x_generator')
      ip_gan = Input(shape=(3, large_width, large_height), name='x_discriminator')  # Actual X images
  
      sr_output = self.generative_network.create_sr_model(ip)
      self.generative_model_ = Model(ip, sr_output)
      #self.generative_network.set_trainable(self.generative_model_, value=False)
  
      gan_output = self.discriminative_network.append_gan_network(ip_gan)
      self.discriminative_model_ = Model(ip_gan, gan_output)
  
      generator_out = self.generative_model_(ip)
      gan_output = self.discriminative_model_(generator_out)
  
      self.srgan_model_ = Model(input=ip, output=gan_output)
  
      srgan_optimizer = Adam(lr=1e-4)
      generator_optimizer = Adam(lr=1e-4)
      discriminator_optimizer = Adam(lr=1e-4)
  
      self.generative_model_.compile(generator_optimizer, loss='mse')
      self.discriminative_model_.compile(discriminator_optimizer, loss='categorical_crossentropy', metrics=['acc'])
      self.srgan_model_.compile(srgan_optimizer, loss='categorical_crossentropy', metrics=['acc'])
  
  
  
      return self.discriminative_model_
  
  
  def build_srgan_model(self, use_small_srgan=False, use_small_discriminator=False):
      large_width = self.img_width * 4
      large_height = self.img_height * 4
  
      self.generative_network = GenerativeNetwork(self.img_width, self.img_height, self.batch_size, nb_upscales=self.nb_scales,
                                                  small_model=use_small_srgan)
      self.discriminative_network = DiscriminatorNetwork(large_width, large_height,
                                                         small_model=use_small_discriminator)
      self.vgg_network = VGGNetwork(large_width, large_height)
  
      ip = Input(shape=(3, self.img_width, self.img_height), name='x_generator')
      ip_gan = Input(shape=(3, large_width, large_height), name='x_discriminator') # Actual X images
      ip_vgg = Input(shape=(3, large_width, large_height), name='x_vgg') # Actual X images
  
      sr_output = self.generative_network.create_sr_model(ip)
      self.generative_model_ = Model(ip, sr_output)
  
      gan_output = self.discriminative_network.append_gan_network(ip_gan)
      self.discriminative_model_ = Model(ip_gan, gan_output)
  
      gan_output = self.discriminative_model_(self.generative_model_.output)
      vgg_output = self.vgg_network.append_vgg_network(self.generative_model_.output, ip_vgg)
  
      self.srgan_model_ = Model(input=[ip, ip_gan, ip_vgg], output=[gan_output, vgg_output])
  
      self.vgg_network.load_vgg_weight(self.srgan_model_)
  
      srgan_optimizer = Adam(lr=1e-4)
      generator_optimizer = Adam(lr=1e-4)
      discriminator_optimizer = Adam(lr=1e-4)
  
      self.generative_model_.compile(generator_optimizer, dummy_loss)
      self.discriminative_model_.compile(discriminator_optimizer, loss='categorical_crossentropy', metrics=['acc'])
      self.srgan_model_.compile(srgan_optimizer, dummy_loss)
  
      return self.srgan_model_
  </code></pre>
  
  <p>but I got this error when running code with keras2.0.</p>
  
  <pre><code> Traceback (most recent call last):
  
      File "&lt;ipython-input-43-f26b38c03792&gt;", line 1, in &lt;module&gt;
  runfile('F:/keras_projects/Super-Resolution-using-Generative-Adversarial-Networks-master/models.py', wdir='F:/keras_projects/Super-Resolution-using-Generative-Adversarial-Networks-master')
  
      File "C:\Program Files\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 866, in runfile
  execfile(filename, namespace)
  
      File "C:\Program Files\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
  exec(compile(f.read(), filename, 'exec'), namespace)
  
      File "F:/keras_projects/Super-Resolution-using-Generative-Adversarial-Networks-master/models.py", line 778, in &lt;module&gt;
  srgan_network.build_srgan_model()
  
      File "F:/keras_projects/Super-Resolution-using-Generative-Adversarial-Networks-master/models.py", line 421, in build_srgan_model
  self.srgan_model_ = Model(inputs=[ip, ip_gan, ip_vgg], outputs=[gan_output, vgg_output])
  
      File "C:\Program Files\Anaconda3\lib\site-packages\keras\legacy\interfaces.py", line 88, in wrapper
  return func(*args, **kwargs)
  
      File "C:\Program Files\Anaconda3\lib\site-packages\keras\engine\topology.py", line 1676, in __init__
  build_map_of_graph(x, finished_nodes, nodes_in_progress)
  
  
      File "C:\Program Files\Anaconda3\lib\site-packages\keras\engine\topology.py", line 1666, in build_map_of_graph
  layer, node_index, tensor_index)
  
      File "C:\Program Files\Anaconda3\lib\site-packages\keras\engine\topology.py", line 1664, in build_map_of_graph
  next_node = layer.inbound_nodes[node_index]
  
      AttributeError: 'NoneType' object has no attribute 'inbound_nodes'
  </code></pre>
  
  <p>In my limited experience,I think it's caused by keras version. I’m so sorry that not much expression is included.</p>