title: Why does my model learn with Ragged Tensors but not Dense Tensors?
body: <p>I have a string of letters that follow a &quot;grammar.&quot; I also have boolean labels on my training set of whether the string follows &quot;the grammar&quot; or not. Basically, my model is trying to learn determine if a string of letters follows the rules. It's a fairly simple problem (I got it out of a textbook).</p>
  <p>I am generating my dataset like this:</p>
  <pre><code>def generate_dataset(size):
      good_strings = [string_to_ids(generate_string(embedded_reber_grammar))
                      for _ in range(size // 2)]
      bad_strings = [string_to_ids(generate_corrupted_string(embedded_reber_grammar))
                     for _ in range(size - size // 2)]
      all_strings = good_strings + bad_strings
      X = tf.ragged.constant(all_strings, ragged_rank=1)
  
      # X = X.to_tensor(default_value=0)
  
      y = np.array([[1.] for _ in range(len(good_strings))] +
                   [[0.] for _ in range(len(bad_strings))])
      return X, y
  </code></pre>
  <p>Notice the line <code>X = X.to_tensor(default_value=0)</code>. If this line is commented out, my model learns just fine. However, if it is not commented out, it fails to learn and the validation set performs the same as chance (50-50).</p>
  <p>Here is my actual model:</p>
  <pre><code>np.random.seed(42)
  tf.random.set_seed(42)
  
  embedding_size = 5
  
  model = keras.models.Sequential([
      keras.layers.InputLayer(input_shape=[None], dtype=tf.int32, ragged=True),
      keras.layers.Embedding(input_dim=len(POSSIBLE_CHARS) + 1, output_dim=embedding_size),
      keras.layers.GRU(30),
      keras.layers.Dense(1, activation=&quot;sigmoid&quot;)
  ])
  optimizer = keras.optimizers.SGD(lr=0.02, momentum = 0.95, nesterov=True)
  model.compile(loss=&quot;binary_crossentropy&quot;, optimizer=optimizer, metrics=[&quot;accuracy&quot;])
  history = model.fit(X_train, y_train, epochs=5, validation_data=(X_valid, y_valid))
  </code></pre>
  <p>I am using <code>0</code> as the default value for the dense tensors. The <code>strings_to_ids</code> doesn't use 0 for any of the values but instead starts at 1. Also, when I switch to using a Dense tensor I change <code>ragged=True</code> to <code>False.</code> I have no idea why using a dense tensor causes the model to fail, as I've used dense tensors before in similar exercises.</p>
  <p>For additional details, see the solution from the book (<a href="https://github.com/ageron/handson-ml2/blob/master/16_nlp_with_rnns_and_attention.ipynb" rel="nofollow noreferrer">exercise 8</a>) or my own colab <a href="https://colab.research.google.com/drive/13Za1YeCO8uEiyqcaUyWKm6zErOaWwpQC?usp=sharing" rel="nofollow noreferrer">notebook</a>.</p>