{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20529","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20529/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20529/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20529/events","html_url":"https://github.com/tensorflow/tensorflow/issues/20529","id":338014892,"node_id":"MDU6SXNzdWUzMzgwMTQ4OTI=","number":20529,"title":"[tf.keras] Stateful Metrics assorted errors.","user":{"login":"brge17","id":33430930,"node_id":"MDQ6VXNlcjMzNDMwOTMw","avatar_url":"https://avatars3.githubusercontent.com/u/33430930?v=4","gravatar_id":"","url":"https://api.github.com/users/brge17","html_url":"https://github.com/brge17","followers_url":"https://api.github.com/users/brge17/followers","following_url":"https://api.github.com/users/brge17/following{/other_user}","gists_url":"https://api.github.com/users/brge17/gists{/gist_id}","starred_url":"https://api.github.com/users/brge17/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brge17/subscriptions","organizations_url":"https://api.github.com/users/brge17/orgs","repos_url":"https://api.github.com/users/brge17/repos","events_url":"https://api.github.com/users/brge17/events{/privacy}","received_events_url":"https://api.github.com/users/brge17/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"raymond-yuan","id":17325195,"node_id":"MDQ6VXNlcjE3MzI1MTk1","avatar_url":"https://avatars1.githubusercontent.com/u/17325195?v=4","gravatar_id":"","url":"https://api.github.com/users/raymond-yuan","html_url":"https://github.com/raymond-yuan","followers_url":"https://api.github.com/users/raymond-yuan/followers","following_url":"https://api.github.com/users/raymond-yuan/following{/other_user}","gists_url":"https://api.github.com/users/raymond-yuan/gists{/gist_id}","starred_url":"https://api.github.com/users/raymond-yuan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raymond-yuan/subscriptions","organizations_url":"https://api.github.com/users/raymond-yuan/orgs","repos_url":"https://api.github.com/users/raymond-yuan/repos","events_url":"https://api.github.com/users/raymond-yuan/events{/privacy}","received_events_url":"https://api.github.com/users/raymond-yuan/received_events","type":"User","site_admin":false},"assignees":[{"login":"raymond-yuan","id":17325195,"node_id":"MDQ6VXNlcjE3MzI1MTk1","avatar_url":"https://avatars1.githubusercontent.com/u/17325195?v=4","gravatar_id":"","url":"https://api.github.com/users/raymond-yuan","html_url":"https://github.com/raymond-yuan","followers_url":"https://api.github.com/users/raymond-yuan/followers","following_url":"https://api.github.com/users/raymond-yuan/following{/other_user}","gists_url":"https://api.github.com/users/raymond-yuan/gists{/gist_id}","starred_url":"https://api.github.com/users/raymond-yuan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raymond-yuan/subscriptions","organizations_url":"https://api.github.com/users/raymond-yuan/orgs","repos_url":"https://api.github.com/users/raymond-yuan/repos","events_url":"https://api.github.com/users/raymond-yuan/events{/privacy}","received_events_url":"https://api.github.com/users/raymond-yuan/received_events","type":"User","site_admin":false}],"milestone":null,"comments":9,"created_at":"2018-07-03T18:46:52Z","updated_at":"2018-08-01T02:00:50Z","closed_at":null,"author_association":"NONE","body":"I will break this issue down into several code snippets each displaying a different error. @fchollet. In total 3 issues. All of these issues are only relevant to ``tf.keras`` implementation. The ``keras`` implementation works as intended.\r\n\r\n### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: Yes\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 16.04\r\n- **TensorFlow installed from (source or binary)**: binary.\r\n- **TensorFlow version (use command below)**: 1.9.0\r\n- **Python version**: 3.6.5\r\n- **Bazel version (if compiling from source)**: n/a\r\n- **GCC/Compiler version (if compiling from source)**: n/a\r\n- **CUDA/cuDNN version**: n/a\r\n- **GPU model and memory**: n/a\r\n- **Exact command to reproduce**: n/a\r\n\r\n### Problem 1\r\nIssues with multi-input/multi-output and batch averaging. This happens for both train and validation metrics.\r\n\r\n\r\n### Source code/logs\r\n\r\n```python\r\nimport tensorflow as tf\r\n\r\nfrom tensorflow.python.keras.datasets import mnist\r\nfrom tensorflow.python.keras.models import Model\r\nfrom tensorflow.python.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense, UpSampling2D\r\n\r\n\r\nclass BatchCounter(tf.keras.layers.Layer):\r\n\r\n        def __init__(self, name='batch_counter', **kwargs):\r\n            super(BatchCounter, self).__init__(name=name, **kwargs)\r\n            self.stateful = True\r\n            self.batches = tf.keras.backend.variable(value=0, dtype='int32')\r\n\r\n        def reset_states(self):\r\n            tf.keras.backend.set_value(self.batches, 0)\r\n\r\n        def __call__(self, y_true, y_pred):\r\n            updates = [tf.keras.backend.update_add(self.batches, tf.keras.backend.variable(value=1, dtype='int32'))]\r\n            self.add_update(updates)\r\n            return self.batches\r\n\r\n\r\nbatch_size = 100\r\nnum_classes = 10\r\nepochs = 1\r\n\r\n# input image dimensions\r\nimg_rows, img_cols = 28, 28\r\n\r\n# Data\r\n(x_train, y_train), (x_test, y_test) = mnist.load_data()\r\nx_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1).astype('float32') / 255\r\nx_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1).astype('float32') / 255\r\ny_train = tf.keras.utils.to_categorical(y_train, num_classes)\r\ny_test = tf.keras.utils.to_categorical(y_test, num_classes)\r\n\r\n# Convolutional Encoder\r\ninput_img = Input(shape=(img_rows, img_cols, 1))\r\nconv_1 = Conv2D(16, (3, 3), activation='relu', padding='same')(input_img)\r\npool_1 = MaxPooling2D((2, 2), padding='same')(conv_1)\r\nconv_2 = Conv2D(8, (3, 3), activation='relu', padding='same')(pool_1)\r\npool_2 = MaxPooling2D((2, 2), padding='same')(conv_2)\r\nconv_3 = Conv2D(8, (3, 3), activation='relu', padding='same')(pool_2)\r\nencoded= MaxPooling2D((2, 2), padding='same')(conv_3)\r\n\r\n# Classification\r\nflatten = Flatten()(encoded)\r\nfc = Dense(128, activation='relu')(flatten)\r\nsoftmax = Dense(num_classes, activation='softmax', name='classification')(fc)\r\n\r\n# Decoder\r\nconv_4 = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)\r\nup_1 = UpSampling2D((2, 2))(conv_4)\r\nconv_5 = Conv2D(8, (3, 3), activation='relu', padding='same')(up_1)\r\nup_2 = UpSampling2D((2, 2))(conv_5)\r\nconv_6 = Conv2D(16, (3, 3), activation='relu')(up_2)\r\nup_3 = UpSampling2D((2, 2))(conv_6)\r\ndecoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same', name='autoencoder')(up_3)\r\n\r\nmodel = Model(inputs=input_img, outputs=[softmax, decoded])\r\n\r\nmodel.compile(loss={'classification': 'categorical_crossentropy',\r\n                    'autoencoder': 'binary_crossentropy'},\r\n              loss_weights={'classification': 1.0,\r\n                            'autoencoder': 0.5},\r\n              optimizer='adam',\r\n              metrics={'classification': 'accuracy', 'autoencoder': BatchCounter()})\r\n\r\nhistory = model.fit(x_train,\r\n          {'classification': y_train, 'autoencoder': x_train},\r\n          batch_size=batch_size,\r\n          epochs=epochs,\r\n          validation_data= (x_test, {'classification': y_test, 'autoencoder': x_test}),\r\n          verbose=1)\r\n```\r\n\r\n```\r\nEpoch 1/1\r\n60000/60000 [==============================] - 41s 677us/step - loss: 0.5086 - classification_loss: 0.4051 - autoencoder_loss: 0.2069 - classification_acc: 0.8755 - autoencoder_batch_counter: 299.7983 - val_loss: 0.2001 - val_classification_loss: 0.1242 - val_autoencoder_loss: 0.1518 - val_classification_acc: 0.9596 - val_autoencoder_batch_counter: 50.1000\r\n```\r\n\r\n``autoencoder_batch_counter`` & ``val_autoencoder_batch_counter`` should always be (600, 100) respectively.  These metrics are batch averaged. This does not happen in the Keras implementation."},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/409424133","html_url":"https://github.com/tensorflow/tensorflow/issues/20529#issuecomment-409424133","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20529","id":409424133,"node_id":"MDEyOklzc3VlQ29tbWVudDQwOTQyNDEzMw==","user":{"login":"brge17","id":33430930,"node_id":"MDQ6VXNlcjMzNDMwOTMw","avatar_url":"https://avatars3.githubusercontent.com/u/33430930?v=4","gravatar_id":"","url":"https://api.github.com/users/brge17","html_url":"https://github.com/brge17","followers_url":"https://api.github.com/users/brge17/followers","following_url":"https://api.github.com/users/brge17/following{/other_user}","gists_url":"https://api.github.com/users/brge17/gists{/gist_id}","starred_url":"https://api.github.com/users/brge17/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brge17/subscriptions","organizations_url":"https://api.github.com/users/brge17/orgs","repos_url":"https://api.github.com/users/brge17/repos","events_url":"https://api.github.com/users/brge17/events{/privacy}","received_events_url":"https://api.github.com/users/brge17/received_events","type":"User","site_admin":false},"created_at":"2018-08-01T02:00:49Z","updated_at":"2018-08-01T02:00:49Z","author_association":"NONE","body":"Thanks for the response @raymond-yuan. Assuming it is still an internal google branch.\r\n\r\nShould I make a new issue for: https://github.com/tensorflow/tensorflow/pull/21071\r\n\r\nThis is the ``TensorBoard`` issue with Stateful Metrics. It was merged into Keras https://github.com/keras-team/keras/pull/10673"}}
{"actor":{"display_login":"brge17"}}