title: tf.data for keras model with multiple inputs, one ctc output, ValueError
body: <p>I'm using tf.version '2.1.0-rc2'</p>
  
  <p>My model has 2 inputs and 1 output and I am struggling to feed it data.</p>
  
  <p>Example <a href="https://keras.io/examples/image_ocr/" rel="nofollow noreferrer">https://keras.io/examples/image_ocr/</a> uses a generator that output 
  a dictionary of numpy arrays.</p>
  
  <pre><code>inputs = {'the_input': X_data,
            'the_labels': labels,
            'input_length': input_length,
            'label_length': label_length,
            'source_str': source_str  # used for visualization only
            }
  outputs = {'ctc': np.zeros([size])} 
  return (inputs, outputs)
  </code></pre>
  
  <p>and fit_generator</p>
  
  <pre><code>   model.fit_generator(
          generator=img_gen.next_train(),
          steps_per_epoch=(words_per_epoch - val_words) // minibatch_size,
          epochs=stop_epoch,
          validation_data=img_gen.next_val(),
          validation_steps=val_words // minibatch_size,
          callbacks=[viz_cb, img_gen],
          initial_epoch=start_epoch)
  </code></pre>
  
  <p>but <a href="https://www.tensorflow.org/api_docs/python/tf/keras/Model" rel="nofollow noreferrer">https://www.tensorflow.org/api_docs/python/tf/keras/Model</a> indicates</p>
  
  <blockquote>
    <p>Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future
    version. Instructions for updating: Please use Model.fit, which
    supports generators.</p>
  </blockquote>
  
  <p>I believe td.data is the recommended way to feed data into tf.keras.
  So I would like to figure this out.</p>
  
  <p>First, a simplified dummy model</p>
  
  <pre><code>def why_fail(input_size, one_hot_output_size, model_output_shape):
      ''' A dummy model, to focus on issue of multiple inputs '''
  
      a = Input(name="the_image_input", shape=(320, 128, 1))
      b = Input(name="the_book_id_input", shape=(20,))
      print(f"the_image_input.shape = {a.shape}, the_book_id_input.shape = {b.shape}")
      # the_image_input.shape = (None, 320, 128, 1), the_book_id_input.shape = (None, 20)
  
      c = Flatten()(a)
      d = Concatenate()([b, c]) # ([b, c])
  
      # reduce dimension to something we can make 40*xxx
      e = Dense(units=model_output_shape[1]*2)(d )
      f = Reshape( (model_output_shape[1],2) )(e)
  
      output_data = Dense(units=one_hot_output_size, activation="softmax")(f )
  
      print(f"into ctc output.shape = {output_data.shape}")
      # into ctc output.shape = (None, 40, 72)
  
      return Model(inputs=[a,b],  outputs=output_data)
  </code></pre>
  
  <p>My generator based on tf.data</p>
  
  <pre><code>def get_tf_dataset(height, width, batch_size, alphabet, record_limit = 10000000, partition_name="train", max_string_length=128, angle_in_radians=30/180*math.pi, min_percent_of_target=0.95, dataset=None, full_word_only=True, expensive_safety_checks=False, debug_dir_path=None, sample_output_path=None):
  
      # my data fits in memory (for now at least)
      my_data = read_dataset(alphabet, record_limit=record_limit, partition_names=[partition_name], max_string_length=max_string_length, expensive_safety_checks=expensive_safety_checks, debug_dir_path=debug_dir_path)
  
      def generator():
          i = 0
          while True:
              i = (i + 1) % len(my_data[partition_name]["image"])
  
              img = my_data[partition_name]["image"][i]
              img = kiss_image.phase_2_img_rnd_rot_rnd_size_rnd_position_rnd_noise(image=img, id=id, height=height, width=width, angle_in_radians=angle_in_radians, min_percent_of_target=min_percent_of_target, expensive_safety_checks=expensive_safety_checks, debug_dir_path=debug_dir_path )
  
              book_id = my_data[partition_name]["book_id"][i]
              text_as_alphabet_index = my_data[partition_name]["text_as_alphabet_index"][i]
  
              if len(text_as_alphabet_index) &lt;= max_string_length:
  
                  text_as_alphabet_index_array = pad_sequences([text_as_alphabet_index], maxlen=max_string_length, padding="post")
                  text_as_alphabet_index_array = text_as_alphabet_index_array[0]
  
                  print(f"the_image_input.shape = {img.shape}")
                  print(f"the_book_id_input.shape = {book_id.shape}")
                  print(f"text_as_alphabet_index_array.shape = {text_as_alphabet_index_array.shape}")
  
                  # the_image_input.shape = (320, 128)
                  # the_book_id_input.shape = (20,)
                  # text_as_alphabet_index_array.shape = (18,)
  
                  yield {"the_image_input": img, "the_book_id_input": book_id}, text_as_alphabet_index_array
  
  
      dataset = tf.data.Dataset.from_generator(generator, output_types=({"the_image_input": tf.float64, "the_book_id_input": tf.float64}, tf.int64))
      dataset = dataset.batch(batch_size)
      return dataset
  </code></pre>
  
  <p>Instantiate a train and validation generator from the above</p>
  
  <pre><code>g_train = kiss_grabner.get_tf_dataset(partition_name="train", record_limit=1000*2*1, height=height, width=width, batch_size=batch_size, max_string_length=max_string_length, alphabet=alphabet, angle_in_radians=15/180*math.pi, full_word_only=full_word_only, min_percent_of_target=0.95, sample_output_path=sample_output_path)
  g_valid = kiss_grabner.get_tf_dataset(partition_name="valid", record_limit=1000*2*1, height=height, width=width, batch_size=batch_size, max_string_length=max_string_length, alphabet=alphabet, angle_in_radians=15/180*math.pi, full_word_only=full_word_only, min_percent_of_target=0.95, sample_output_path=sample_output_path)
  </code></pre>
  
  <p>Call fit with the td.data generators</p>
  
  <pre><code>model.fit(  x= g_train, #generator.next_train_batch_with_book_id(),
              steps_per_epoch=steps_per_epoch,
              epochs=1000,
              validation_data=g_valid, #generator.next_valid_batch(), #validation_data=generator.next_valid_batch_with_book_id(),
              validation_steps=1000,
              callbacks=callbacks
          )
  </code></pre>
  
  <p>The error</p>
  
  <pre><code>Traceback (most recent call last):
    File "kiss_run_model_on_grabner.py", line 371, in &lt;module&gt;
      callbacks=callbacks
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit
      use_multiprocessing=use_multiprocessing)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 235, in fit
      use_multiprocessing=use_multiprocessing)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 593, in _process_training_inputs
      use_multiprocessing=use_multiprocessing)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 706, in _process_inputs
      use_multiprocessing=use_multiprocessing)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/data_adapter.py", line 702, in __init__
      x = standardize_function(x)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 684, in standardize_function
      return dataset.map(map_fn, num_parallel_calls=dataset_ops.AUTOTUNE)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 1591, in map
      self, map_func, num_parallel_calls, preserve_cardinality=True)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 3926, in __init__
      use_legacy_function=use_legacy_function)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 3147, in __init__
      self._function = wrapper_fn._get_concrete_function_internal()
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2395, in _get_concrete_function_internal
      *args, **kwargs)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2389, in _get_concrete_function_internal_garbage_collected
      graph_function, _, _ = self._maybe_define_function(args, kwargs)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2703, in _maybe_define_function
      graph_function = self._create_graph_function(args, kwargs)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2593, in _create_graph_function
      capture_by_value=self._capture_by_value),
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/func_graph.py", line 978, in func_graph_from_py_func
      func_outputs = python_func(*func_args, **func_kwargs)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 3140, in wrapper_fn
      ret = _wrapper_helper(*args)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 3082, in _wrapper_helper
      ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
      raise e.ag_error_metadata.to_exception(e)
  ValueError: in converted code:
  
      /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py:677 map_fn
          batch_size=None)
      /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py:2410 _standardize_tensors
          exception_prefix='input')
      /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_utils.py:526 standardize_input_data
          standardize_single_array(x, shape) for (x, shape) in zip(data, shapes)
      /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_utils.py:526 &lt;listcomp&gt;
          standardize_single_array(x, shape) for (x, shape) in zip(data, shapes)
      /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_utils.py:451 standardize_single_array
          if (x.shape is not None and len(x.shape) == 1 and
      /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/tensor_shape.py:822 __len__
          raise ValueError("Cannot take the length of shape with unknown rank.")
  
      ValueError: Cannot take the length of shape with unknown rank.
  </code></pre>
  
  <p>I also tried the dictionary approach</p>
  
  <pre><code> def next_train_batch_with_book_id(self):
  
  train = self.dataset["train"]
  while True:
      xy = self.get_batch(train, return_ids_and_text_too=True, angle_in_radians=self.angle_in_radians, min_percent_of_target=self.min_percent_of_target, full_word_only=self.full_word_only)
  
      an_array_of_images = xy["image"]
      an_array_of_book_ids = xy["book_id"]
      an_array_of_1_hot  = xy["text_as_alphabet_index"]
  
      print(f"an_array_of_images.shape = {an_array_of_images.shape}")
      print(f"an_array_of_book_ids.shape = {an_array_of_book_ids.shape}")
      print(f"an_array_of_1_hot.shape = {an_array_of_1_hot.shape}")
      # an_array_of_images.shape = (32, 320, 128, 1)
      # an_array_of_book_ids.shape = (32, 20)
      #an_array_of_1_hot.shape = (32, 18)
  
      # dictionary based input/output to model
      inputs  = { 'the_image_input': an_array_of_images, "the_book_id_input": an_array_of_book_ids} 
      outputs = { 'the_text_as_alphabet_index_input' : an_array_of_1_hot }
  
      # just the data as input
      #inputs = an_array_of_images 
      outputs =  an_array_of_1_hot 
  
      xy = (inputs, outputs, [])
  
      yield xy
  </code></pre>
  
  <p>...</p>
  
  <p>with model.fit calling as recommended</p>
  
  <pre><code>model.fit(  x=generator.next_train_batch_with_book_id(),
              steps_per_epoch=steps_per_epoch,
              epochs=1000,
              validation_data=generator.next_valid_batch(), #validation_data=generator.next_valid_batch_with_book_id(),
              validation_steps=1000,
              callbacks=callbacks
          )
  </code></pre>
  
  <p>But this gives me the following error:</p>
  
  <pre><code>Traceback (most recent call last):
    File "kiss_run_model_on_grabner.py", line 371, in &lt;module&gt;
      callbacks=callbacks
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit
      use_multiprocessing=use_multiprocessing)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 235, in fit
      use_multiprocessing=use_multiprocessing)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 614, in _process_training_inputs
      distribution_strategy=distribution_strategy)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 706, in _process_inputs
      use_multiprocessing=use_multiprocessing)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/data_adapter.py", line 767, in __init__
      dataset = standardize_function(dataset)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 684, in standardize_function
      return dataset.map(map_fn, num_parallel_calls=dataset_ops.AUTOTUNE)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 1591, in map
      self, map_func, num_parallel_calls, preserve_cardinality=True)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 3926, in __init__
      use_legacy_function=use_legacy_function)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 3147, in __init__
      self._function = wrapper_fn._get_concrete_function_internal()
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2395, in _get_concrete_function_internal
      *args, **kwargs)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2389, in _get_concrete_function_internal_garbage_collected
      graph_function, _, _ = self._maybe_define_function(args, kwargs)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2703, in _maybe_define_function
      graph_function = self._create_graph_function(args, kwargs)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2593, in _create_graph_function
      capture_by_value=self._capture_by_value),
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/func_graph.py", line 978, in func_graph_from_py_func
      func_outputs = python_func(*func_args, **func_kwargs)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 3140, in wrapper_fn
      ret = _wrapper_helper(*args)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 3082, in _wrapper_helper
      ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
    File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
      raise e.ag_error_metadata.to_exception(e)
  ValueError: in converted code:
  
      /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py:677 map_fn
          batch_size=None)
      /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py:2410 _standardize_tensors
          exception_prefix='input')
      /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_utils.py:539 standardize_input_data
          str(data)[:200] + '...')
  
      ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), for inputs ['the_image_input', 'the_book_id_input'] but instead got the following list of 1 arrays: [&lt;tf.Tensor 'args_0:0' shape=(None, None, None, None) dtype=float32&gt;]...
  </code></pre>