title: Use a custom matrix in keras model which is not part of layer weight but like an confusion matrix but trainable and will be used in loss computation
body: <p>I am trying to implement <a href="https://rt416.github.io/pdf/trace_codes.pdf" rel="nofollow noreferrer">this</a> paper in keras. Base code is as follows:</p>
  <pre><code>import tensorflow as tf
  
  # inputs and annotators’ labels
  images = tf.placeholder(tf.float32, (None, image_shape))
  labels = tf.placeholder(tf.int32,(None, num_annotators, num_classes))
  
  # classifier for estimating true label distribution
  logits = classifier(images)
  
  # confusion matrices of annotators
  confusion_matrices = confusion_matrix_estimators(num_annotators, num_classes)
  
  # loss function
  # 1. weighted cross-entropy
  weighted_cross_entropy = cross_entropy_over_annotators(labels, logits, confusion_matrices)
  
  # 2. trace of confusion matrices:
  trace_norm = tf.reduce_mean(tf.trace(confusion_matrices))
  
  # final loss (eq.(4))
  total_loss = weighted_cross_entropy + scale * trace_norm
  </code></pre>
  <p>I am using keras Model API. Image and labels can be handled normally. My concern in <code>confusion_matrix</code>
  which is (1) trainable (2) used to compute the final output for loss computation. I am not able to introduce this in my model so that it can be trainable and same time get it as output for loss computation.</p>