title: Tensorflow `x` and `y` must have the same dtype, even after casting to the correct type
body: <p>Hi all getting an error I'm not expecting when fitting a model even after casting to the correct type. Code snippet below showing the error and workflow.</p>
  <pre><code>        # print(asset_weight_dict[asset_name])
          x_train, x_val = X.iloc[train_idx], X.iloc[val_idx]
          y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]
          x_train = tf.cast(x_train, tf.float32)
          x_val = tf.cast(x_val, tf.float32)
          y_train = tf.cast(y_train, tf.float32)
          y_val = tf.cast(y_val, tf.float32)
          print(x_train.info())
          print()
          print(y_train.info())
          with strategy.scope(): model = build_model(fold, dim = x_train.shape[1], weight = asset_weight_dict[asset_name])
  
          # SAVE BEST MODEL EACH FOLD
          sv = tf.keras.callbacks.ModelCheckpoint('fold-%i.h5' % fold, monitor = 'val_loss', verbose = 0, save_best_only = True, save_weights_only = True, mode = 'min', save_freq = 'epoch')
  
          # TRAIN
  
  
          history = model.fit(x_train, y_train)
  </code></pre>
  <p>The output of x_train and y_train info shows both are in the float32 dtype:</p>
  <pre><code>tf.Tensor(
  [[-1.1736900e-18  8.4033599e-19  6.4855290e-18 ... -1.7973387e-03
    -9.7247097e-04 -8.3497784e-04]
   [-4.9040946e-03 -9.3674008e-03 -9.2475545e-03 ...  2.5425672e-03
     3.5907906e-03  3.6137134e-03]
   [-5.7132118e-03 -7.1952469e-03 -6.1273980e-03 ...  1.0056033e-02
     9.7314240e-03  9.3650799e-03]
   ...
   [ 2.7320897e-05  2.7972811e-03 -3.9492366e-03 ...  9.2556234e-03
     8.6822165e-03  9.2183622e-03]
   [-2.7372641e-03 -5.3076809e-03 -6.9548660e-03 ... -4.5336168e-03
    -3.8521911e-03 -3.2705730e-03]
   [ 1.2495884e-03  1.8994033e-03 -5.5942247e-03 ...  8.6857486e-05
    -6.3753058e-04 -5.0171453e-04]], shape=(32157, 121), dtype=float32)
  
  tf.Tensor(
  [[ 0.01257048]
   [ 0.00535373]
   [-0.00205016]
   ...
   [ 0.01012999]
   [-0.0218502 ]
   [-0.00389742]], shape=(32157, 1), dtype=float32)
  </code></pre>
  <p>Full error message after running showing that y (I think) is somehow being recast to int32 or I'm fundamentally missing something.</p>
  <pre><code>Input In [26], in get_Xy_and_model_for_asset(asset_name)
       49 sv = tf.keras.callbacks.ModelCheckpoint('fold-%i.h5' % fold, monitor = 'val_loss', verbose = 0, save_best_only = True, save_weights_only = True, mode = 'min', save_freq = 'epoch')
       51 # TRAIN
  ---&gt; 54 history = model.fit(x_train, y_train)
       55 model.load_weights('fold-%i.h5' % fold)
       57 # PREDICT OOF
  
  File ~\Anaconda3\lib\site-packages\keras\utils\traceback_utils.py:67, in filter_traceback.&lt;locals&gt;.error_handler(*args, **kwargs)
       65 except Exception as e:  # pylint: disable=broad-except
       66   filtered_tb = _process_traceback_frames(e.__traceback__)
  ---&gt; 67   raise e.with_traceback(filtered_tb) from None
       68 finally:
       69   del filtered_tb
  
  File ~\Anaconda3\lib\site-packages\tensorflow\python\framework\func_graph.py:1147, in func_graph_from_py_func.&lt;locals&gt;.autograph_handler(*args, **kwargs)
     1145 except Exception as e:  # pylint:disable=broad-except
     1146   if hasattr(e, &quot;ag_error_metadata&quot;):
  -&gt; 1147     raise e.ag_error_metadata.to_exception(e)
     1148   else:
     1149     raise
  
  TypeError: in user code:
  
      File &quot;C:\Users\dmartens\Anaconda3\lib\site-packages\keras\engine\training.py&quot;, line 1021, in train_function  *
          return step_function(self, iterator)
      File &quot;C:\Users\dmartens\AppData\Local\Temp\ipykernel_9956\295126875.py&quot;, line 10, in wcorr  *
          def wcorr(x, y): return tf_comp_metric(x, y ,w)
      File &quot;C:\Users\dmartens\AppData\Local\Temp\ipykernel_9956\295126875.py&quot;, line 8, in tf_comp_metric  *
          def tf_comp_metric(a, b, w): return tf_cov(a, b, w) / tf.sqrt(tf_cov(a, a, w) * tf_cov(b, b, w))
      File &quot;C:\Users\dmartens\AppData\Local\Temp\ipykernel_9956\295126875.py&quot;, line 7, in tf_cov  *
          def tf_cov(x, y, w): return (tf.reduce_sum(w * (x - tf.reduce_mean(x * w)) * (y - tf.reduce_mean(y * w))) / tf.reduce_sum(w))
  
      TypeError: `x` and `y` must have the same dtype, got tf.float32 != tf.int32.
  </code></pre>