{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19241","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19241/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19241/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19241/events","html_url":"https://github.com/tensorflow/tensorflow/issues/19241","id":322485922,"number":19241,"title":"[Eager] Fix for determining input / output shape of the model prior to Model.fit()","user":{"login":"titu1994","id":3048602,"avatar_url":"https://avatars1.githubusercontent.com/u/3048602?v=4","gravatar_id":"","url":"https://api.github.com/users/titu1994","html_url":"https://github.com/titu1994","followers_url":"https://api.github.com/users/titu1994/followers","following_url":"https://api.github.com/users/titu1994/following{/other_user}","gists_url":"https://api.github.com/users/titu1994/gists{/gist_id}","starred_url":"https://api.github.com/users/titu1994/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/titu1994/subscriptions","organizations_url":"https://api.github.com/users/titu1994/orgs","repos_url":"https://api.github.com/users/titu1994/repos","events_url":"https://api.github.com/users/titu1994/events{/privacy}","received_events_url":"https://api.github.com/users/titu1994/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"cy89","id":29663194,"avatar_url":"https://avatars0.githubusercontent.com/u/29663194?v=4","gravatar_id":"","url":"https://api.github.com/users/cy89","html_url":"https://github.com/cy89","followers_url":"https://api.github.com/users/cy89/followers","following_url":"https://api.github.com/users/cy89/following{/other_user}","gists_url":"https://api.github.com/users/cy89/gists{/gist_id}","starred_url":"https://api.github.com/users/cy89/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cy89/subscriptions","organizations_url":"https://api.github.com/users/cy89/orgs","repos_url":"https://api.github.com/users/cy89/repos","events_url":"https://api.github.com/users/cy89/events{/privacy}","received_events_url":"https://api.github.com/users/cy89/received_events","type":"User","site_admin":false},"assignees":[{"login":"cy89","id":29663194,"avatar_url":"https://avatars0.githubusercontent.com/u/29663194?v=4","gravatar_id":"","url":"https://api.github.com/users/cy89","html_url":"https://github.com/cy89","followers_url":"https://api.github.com/users/cy89/followers","following_url":"https://api.github.com/users/cy89/following{/other_user}","gists_url":"https://api.github.com/users/cy89/gists{/gist_id}","starred_url":"https://api.github.com/users/cy89/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cy89/subscriptions","organizations_url":"https://api.github.com/users/cy89/orgs","repos_url":"https://api.github.com/users/cy89/repos","events_url":"https://api.github.com/users/cy89/events{/privacy}","received_events_url":"https://api.github.com/users/cy89/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2018-05-12T05:59:41Z","updated_at":"2018-05-27T12:35:50Z","closed_at":null,"author_association":"NONE","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: Yes\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Windows 10 Home Edition\r\n- **TensorFlow installed from (source or binary)**: Windows binary\r\n- **TensorFlow version (use command below)**: Github version 'v1.8.0-0-g93bc2e2072' 1.8.0TF (GPU) \r\n- **Python version**: 3.5\r\n- **Bazel version (if compiling from source)**: N/A\r\n- **GCC/Compiler version (if compiling from source)**: N/A\r\n- **CUDA/cuDNN version**: CUDA 9.0 / cuDNN 7.0.5\r\n- **GPU model and memory**: NVIDIA GTX 980M\r\n- **Exact command to reproduce**: Provided below as a standalone script\r\n\r\nYou can collect some of this information using our environment capture script:\r\n\r\nhttps://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh\r\n\r\nYou can obtain the TensorFlow version with\r\n\r\npython -c \"import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)\"\r\n\r\n### Describe the problem\r\nThe issue is that in Eager mode, the two attributes of the Model subclass, `inputs` and `outputs` is undetermined until the first call to `Model.fit(...)`.\r\n\r\nWhen attempting to determine this prior to fitting the model, in the script at location **tensorflow/python/keras/_impl/keras/engine/training.py**, the entire input dataset `X` (passed to .fit(...)) is provided as the input `x` in Line 684 `if not self.inputs: self._set_inputs(x)` inside `_standardize_user_data`.\r\n\r\nDue to this, in eager execution mode, this call is deferred to `_eager_set_inputs(inputs)`. Here, `inputs` is the entire dataset numpy matrix / tensor, and a `Model.call(inputs)` is performed at line 909. \r\n\r\nSince the entire dataset is unable to fit in GPU memory for smaller GPU devices, it causes an OOM error.\r\n\r\nHowever, to determine the input / output shape/s of a model, a single sample tensor is sufficient.\r\n\r\nThe below fix shows that the solution is adequate, and can be implemented by simply extracting a single sample of the entire dataset to determine the input / output shape/s of a model during eager execution.\r\n\r\n**Note : **\r\nIn order to provide indicators to identify where the error occurs, I modified the above mentioned script to print 2 logs to the console, to describe the shape of the \"inputs\" parameter inside `_eager_set_inputs(inputs)`. The lines `Inside training._eager_set_inputs ...` and `***** Providing entire dataset into call *****` are the above logs.\r\n\r\n### Source code / logs\r\n\r\n```python\r\nimport os\r\nimport numpy as np\r\n\r\nimport tensorflow as tf\r\nfrom tensorflow.python.keras.datasets import mnist\r\nfrom tensorflow.contrib.eager.python import tfe\r\n\r\n# enable eager mode\r\ntf.enable_eager_execution()\r\ntf.set_random_seed(0)\r\nnp.random.seed(0)\r\n\r\n# constants\r\nbatch_size = 128\r\nepochs = 10\r\nnum_classes = 10\r\n\r\n# dataset loading\r\n(x_train, y_train), (x_test, y_test) = mnist.load_data()\r\nx_train = x_train.astype('float32') / 255.\r\nx_test = x_test.astype('float32') / 255.\r\nx_train = x_train.reshape((-1, 28, 28, 1))\r\nx_test = x_test.reshape((-1, 28, 28, 1))\r\n\r\n# one hot encode the labels. convert back to numpy as we cannot use a combination of numpy\r\n# and tensors as input to keras\r\ny_train_ohe = tf.one_hot(y_train, depth=num_classes).numpy()\r\ny_test_ohe = tf.one_hot(y_test, depth=num_classes).numpy()\r\n\r\nprint('x train', x_train.shape)\r\nprint('y train', y_train_ohe.shape)\r\nprint('x test', x_test.shape)\r\nprint('y test', y_test_ohe.shape)\r\n\r\nclass CNN(tf.keras.Model):\r\n\r\n    def __init__(self, num_classes):\r\n        super(CNN, self).__init__()\r\n\r\n        self.cnn1 = tf.keras.layers.Conv2D(16, (5, 5), padding='same', strides=(2, 2))\r\n        self.bn1 = tf.keras.layers.BatchNormalization()\r\n        self.cnn2 = tf.keras.layers.Conv2D(32, (5, 5), padding='same', strides=(2, 2))\r\n        self.bn2 = tf.keras.layers.BatchNormalization()\r\n        self.pool = tf.keras.layers.GlobalAveragePooling2D()\r\n        self.classifier = tf.keras.layers.Dense(num_classes)\r\n\r\n    def call(self, inputs, training=None, mask=None):\r\n        # Used to print out the input shape of the entire dataset prior to training loop\r\n        print(inputs.shape)\r\n\r\n        x = self.cnn1(inputs)\r\n        x = self.bn1(x)\r\n        x = tf.nn.relu(x)  # layer 1\r\n        x = tf.nn.relu(self.bn2(self.cnn2(x))) # layer 2\r\n        x = self.pool(x)\r\n        output = self.classifier(x)\r\n\r\n        # softmax op does not exist on the gpu, so always use cpu\r\n        with tf.device('/cpu:0'):\r\n            output = tf.nn.softmax(output)\r\n\r\n        return output\r\n\r\n\r\ndevice = '/cpu:0' if tfe.num_gpus() == 0 else '/gpu:0'\r\n\r\nwith tf.device(device):\r\n    # build model and optimizer\r\n    model = CNN(num_classes)\r\n    model.compile(optimizer=tf.train.AdamOptimizer(0.001), loss='categorical_crossentropy',\r\n                  metrics=['accuracy'])\r\n    \r\n    # suggested fix ; can be incorporated inside `_eager_set_inputs` or `_set_input`\r\n    # Fix = Use exactly one sample from the provided input dataset to determine \r\n    # input/output shape/s for the model\r\n\r\n    # dummy_x = np.zeros((1, 28, 28, 1))\r\n    # model._set_inputs(dummy_x)\r\n\r\n    # train\r\n    model.fit(x_train, y_train_ohe, batch_size=batch_size, epochs=epochs,\r\n              validation_data=(x_test, y_test_ohe), verbose=2)\r\n\r\n    # evaluate on test set\r\n    scores = model.evaluate(x_test, y_test_ohe, batch_size, verbose=2)\r\n    print(\"Final test loss and accuracy :\", scores)\r\n\r\n```\r\n\r\nTruncated log without the fix : (Only tensor allocation summary dump after OOM is truncated)\r\n```python\r\n2018-05-12 00:22:44.208127: I T:\\src\\github\\tensorflow\\tensorflow\\core\\platform\\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2\r\n2018-05-12 00:22:45.385242: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1356] Found device 0 with properties: \r\nname: GeForce GTX 980M major: 5 minor: 2 memoryClockRate(GHz): 1.1265\r\npciBusID: 0000:01:00.0\r\ntotalMemory: 4.00GiB freeMemory: 3.32GiB\r\n2018-05-12 00:22:45.385822: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1435] Adding visible gpu devices: 0\r\n2018-05-12 00:22:45.838116: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n2018-05-12 00:22:45.838626: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:929]      0 \r\n2018-05-12 00:22:45.838912: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:942] 0:   N \r\n2018-05-12 00:22:45.839762: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3050 MB memory) -> physical GPU (device: 0, name: GeForce GTX 980M, pci bus id: 0000:01:00.0, compute capability: 5.2)\r\nx train (60000, 28, 28, 1)\r\ny train (60000, 10)\r\nx test (10000, 28, 28, 1)\r\ny test (10000, 10)\r\n\r\nInside training._eager_set_inputs ip shape/s [(60000, 28, 28, 1)]  <= Here the input is the entire dataset\r\n***** Providing entire dataset into call *****\r\n(60000, 28, 28, 1)\r\n\r\n2018-05-12 00:22:58.125845: W T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 717.77MiB.  Current allocation summary follows.\r\n\r\n<<< Truncated >>>\r\n\r\n2018-05-12 00:22:58.229153: W T:\\src\\github\\tensorflow\\tensorflow\\core\\framework\\op_kernel.cc:1318] OP_REQUIRES failed at fused_batch_norm_op.cc:263 : Resource exhausted: OOM when allocating tensor with shape[60000,16,14,14] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc\r\nTraceback (most recent call last):\r\n  File \"D:/Users/Yue/PycharmProjects/eager-tutorials/tutorials/04_cnn.py\", line 83, in <module>\r\n    validation_data=(x_test, y_test_ohe), verbose=2)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\keras\\_impl\\keras\\engine\\training.py\", line 1147, in fit\r\n    batch_size=batch_size)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\keras\\_impl\\keras\\engine\\training.py\", line 685, in _standardize_user_data\r\n    self._set_inputs(x)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\keras\\_impl\\keras\\engine\\training.py\", line 871, in _set_inputs\r\n    self._eager_set_inputs(inputs)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\keras\\_impl\\keras\\engine\\training.py\", line 914, in _eager_set_inputs\r\n    ops.convert_to_tensor(inputs, dtype=K.floatx()))\r\n  File \"D:/Users/Yue/PycharmProjects/eager-tutorials/tutorials/04_cnn.py\", line 55, in call\r\n    x = self.bn1(x)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\keras\\_impl\\keras\\engine\\base_layer.py\", line 314, in __call__\r\n    output = super(Layer, self).__call__(inputs, *args, **kwargs)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\layers\\base.py\", line 717, in __call__\r\n    outputs = self.call(inputs, *args, **kwargs)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\keras\\_impl\\keras\\layers\\normalization.py\", line 113, in call\r\n    output = super(BatchNormalization, self).call(inputs, training=training)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\layers\\normalization.py\", line 501, in call\r\n    outputs = self._fused_batch_norm(inputs, training=training)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\layers\\normalization.py\", line 396, in _fused_batch_norm\r\n    training, _fused_batch_norm_training, _fused_batch_norm_inference)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\layers\\utils.py\", line 206, in smart_cond\r\n    pred, true_fn=true_fn, false_fn=false_fn, name=name)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\smart_cond.py\", line 56, in smart_cond\r\n    return false_fn()\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\layers\\normalization.py\", line 393, in _fused_batch_norm_inference\r\n    data_format=self._data_format)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\nn_impl.py\", line 904, in fused_batch_norm\r\n    name=name)\r\n  File \"D:\\Users\\Yue\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\ops\\gen_nn_ops.py\", line 3804, in _fused_batch_norm\r\n    _six.raise_from(_core._status_to_exception(e.code, message), None)\r\n  File \"<string>\", line 3, in raise_from\r\ntensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[60000,16,14,14] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:FusedBatchNorm]\r\n\r\n```\r\n\r\nPartial log with the aforementioned fix\r\n```python\r\n2018-05-12 00:27:23.607020: I T:\\src\\github\\tensorflow\\tensorflow\\core\\platform\\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2\r\n2018-05-12 00:27:24.748056: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1356] Found device 0 with properties: \r\nname: GeForce GTX 980M major: 5 minor: 2 memoryClockRate(GHz): 1.1265\r\npciBusID: 0000:01:00.0\r\ntotalMemory: 4.00GiB freeMemory: 3.32GiB\r\n2018-05-12 00:27:24.748771: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1435] Adding visible gpu devices: 0\r\n2018-05-12 00:27:25.196921: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n2018-05-12 00:27:25.197231: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:929]      0 \r\n2018-05-12 00:27:25.197570: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:942] 0:   N \r\n2018-05-12 00:27:25.197993: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3050 MB memory) -> physical GPU (device: 0, name: GeForce GTX 980M, pci bus id: 0000:01:00.0, compute capability: 5.2)\r\nx train (60000, 28, 28, 1)\r\ny train (60000, 10)\r\nx test (10000, 28, 28, 1)\r\ny test (10000, 10)\r\n\r\nInside training._eager_set_inputs ip shape/s [(1, 28, 28, 1)]  <= Here the \"dataset\" is the dummy batch\r\n***** Providing entire dataset into call *****\r\n(1, 28, 28, 1)\r\n\r\nTrain on 60000 samples, validate on 10000 samples\r\nEpoch 1/10\r\n(128, 28, 28, 1)\r\n(128, 28, 28, 1)\r\n(128, 28, 28, 1)\r\n(128, 28, 28, 1)\r\n(128, 28, 28, 1)\r\n... repeated for the entire training loop over the batch.\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/392327940","html_url":"https://github.com/tensorflow/tensorflow/issues/19241#issuecomment-392327940","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19241","id":392327940,"user":{"login":"tensorflowbutler","id":28546240,"avatar_url":"https://avatars2.githubusercontent.com/u/28546240?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflowbutler","html_url":"https://github.com/tensorflowbutler","followers_url":"https://api.github.com/users/tensorflowbutler/followers","following_url":"https://api.github.com/users/tensorflowbutler/following{/other_user}","gists_url":"https://api.github.com/users/tensorflowbutler/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflowbutler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflowbutler/subscriptions","organizations_url":"https://api.github.com/users/tensorflowbutler/orgs","repos_url":"https://api.github.com/users/tensorflowbutler/repos","events_url":"https://api.github.com/users/tensorflowbutler/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflowbutler/received_events","type":"User","site_admin":false},"created_at":"2018-05-27T12:35:50Z","updated_at":"2018-05-27T12:35:50Z","author_association":"MEMBER","body":"Nagging Assignee @cy89: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly."}}
{"actor":{"display_login":"tensorflowbutler"}}