{"action":"created","issue":{"url":"https://api.github.com/repos/pierluigiferrari/ssd_keras/issues/84","repository_url":"https://api.github.com/repos/pierluigiferrari/ssd_keras","labels_url":"https://api.github.com/repos/pierluigiferrari/ssd_keras/issues/84/labels{/name}","comments_url":"https://api.github.com/repos/pierluigiferrari/ssd_keras/issues/84/comments","events_url":"https://api.github.com/repos/pierluigiferrari/ssd_keras/issues/84/events","html_url":"https://github.com/pierluigiferrari/ssd_keras/issues/84","id":305716063,"number":84,"title":"Error deep in fit_generator when using SSD7_training notebook with TF v1.6.0","user":{"login":"fancypantsifier","id":8006003,"avatar_url":"https://avatars1.githubusercontent.com/u/8006003?v=4","gravatar_id":"","url":"https://api.github.com/users/fancypantsifier","html_url":"https://github.com/fancypantsifier","followers_url":"https://api.github.com/users/fancypantsifier/followers","following_url":"https://api.github.com/users/fancypantsifier/following{/other_user}","gists_url":"https://api.github.com/users/fancypantsifier/gists{/gist_id}","starred_url":"https://api.github.com/users/fancypantsifier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fancypantsifier/subscriptions","organizations_url":"https://api.github.com/users/fancypantsifier/orgs","repos_url":"https://api.github.com/users/fancypantsifier/repos","events_url":"https://api.github.com/users/fancypantsifier/events{/privacy}","received_events_url":"https://api.github.com/users/fancypantsifier/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-15T20:55:12Z","updated_at":"2018-03-15T21:23:45Z","closed_at":null,"author_association":"NONE","body":"First, thanks for an incredibly detailed and well-documented code base and tutorials. This is an amazing piece of work.\r\n\r\nI'm trying to retrain the SSD7 model on COCO data, by very slightly modifying the ssd7_training notebook. I'm getting the following error stack when I get to the `history = model.fit_generator()` step.\r\n\r\nI'm using the notebook as is, except that I'm setting up the train/val datasets from json instead of csv:\r\n```python\r\ntrain_dataset.parse_json(images_dirs=['{}/train2014'.format(coco_data_path)], \r\n                         annotations_filenames=['{}/annotations/instances_train2014.json'.format(coco_data_path)], \r\n                         ground_truth_available=True,\r\n                         include_classes='all')\r\n\r\nval_dataset.parse_json(images_dirs=['{}/val2014'.format(coco_data_path)], \r\n                       annotations_filenames=['{}/annotations/instances_val2014.json'.format(coco_data_path)], \r\n                       ground_truth_available=True,\r\n                       include_classes='all')\r\n```\r\n\r\nI'm using TensorFlow 1.6.0. Could this be a versioning issue, as it seems deep within TensorFlow (i.e., down in a call to `tf.clip_by_value` and `tf.sqrt`?  \r\n\r\n```python\r\nWARNING:tensorflow:Variable *= will be deprecated. Use variable.assign_mul if you want assignment to the variable value or 'x = x * y' if you want a new python Tensor object.\r\n---------------------------------------------------------------------------\r\nValueError                                Traceback (most recent call last)\r\n<ipython-input-24-22d018627654> in <module>()\r\n     21                                                              cooldown=0)],\r\n     22                               validation_data = val_generator,\r\n---> 23                               validation_steps = ceil(num_val_samples/batch_size))\r\n     24 \r\n     25 # TODO: Set the filename (without the .h5 file extension!) under which to save the model and weights.\r\n\r\n~/Code/tensorflow/lib/python3.6/site-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)\r\n     89                 warnings.warn('Update your `' + object_name +\r\n     90                               '` call to the Keras 2 API: ' + signature, stacklevel=2)\r\n---> 91             return func(*args, **kwargs)\r\n     92         wrapper._original_function = func\r\n     93         return wrapper\r\n\r\n~/Code/tensorflow/lib/python3.6/site-packages/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\r\n   2078 \r\n   2079         do_validation = bool(validation_data)\r\n-> 2080         self._make_train_function()\r\n   2081         if do_validation:\r\n   2082             self._make_test_function()\r\n\r\n~/Code/tensorflow/lib/python3.6/site-packages/keras/engine/training.py in _make_train_function(self)\r\n    988                     training_updates = self.optimizer.get_updates(\r\n    989                         params=self._collected_trainable_weights,\r\n--> 990                         loss=self.total_loss)\r\n    991                 updates = self.updates + training_updates + self.metrics_updates\r\n    992                 # Gets loss and metrics. Updates weights at each call.\r\n\r\n~/Code/tensorflow/lib/python3.6/site-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)\r\n     89                 warnings.warn('Update your `' + object_name +\r\n     90                               '` call to the Keras 2 API: ' + signature, stacklevel=2)\r\n---> 91             return func(*args, **kwargs)\r\n     92         wrapper._original_function = func\r\n     93         return wrapper\r\n\r\n~/Code/tensorflow/lib/python3.6/site-packages/keras/optimizers.py in get_updates(self, loss, params)\r\n    452 \r\n    453         t = K.cast(self.iterations, K.floatx()) + 1\r\n--> 454         lr_t = lr * (K.sqrt(1. - K.pow(self.beta_2, t)) /\r\n    455                      (1. - K.pow(self.beta_1, t)))\r\n    456 \r\n\r\n~/Code/tensorflow/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py in sqrt(x)\r\n   1459     zero = _to_tensor(0., x.dtype.base_dtype)\r\n   1460     inf = _to_tensor(np.inf, x.dtype.base_dtype)\r\n-> 1461     x = tf.clip_by_value(x, zero, inf)\r\n   1462     return tf.sqrt(x)\r\n   1463 \r\n\r\n~/Code/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/clip_ops.py in clip_by_value(t, clip_value_min, clip_value_max, name)\r\n     58   \"\"\"\r\n     59   with ops.name_scope(name, \"clip_by_value\",\r\n---> 60                       [t, clip_value_min, clip_value_max]) as name:\r\n     61     t = ops.convert_to_tensor(t, name=\"t\")\r\n     62 \r\n\r\n~/Code/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in __enter__(self)\r\n   5614       if self._values is None:\r\n   5615         self._values = []\r\n-> 5616       g = _get_graph_from_inputs(self._values)\r\n   5617       self._g_manager = g.as_default()\r\n   5618       self._g_manager.__enter__()\r\n\r\n~/Code/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _get_graph_from_inputs(op_input_list, graph)\r\n   5282         graph = graph_element.graph\r\n   5283       elif original_graph_element is not None:\r\n-> 5284         _assert_same_graph(original_graph_element, graph_element)\r\n   5285       elif graph_element.graph is not graph:\r\n   5286         raise ValueError(\"%s is not from the passed-in graph.\" % graph_element)\r\n\r\n~/Code/tensorflow/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _assert_same_graph(original_item, item)\r\n   5218   if original_item.graph is not item.graph:\r\n   5219     raise ValueError(\"%s must be from the same graph as %s.\" % (item,\r\n-> 5220                                                                 original_item))\r\n   5221 \r\n   5222 \r\n\r\nValueError: Tensor(\"training/Adam/Const:0\", shape=(), dtype=float32) must be from the same graph as Tensor(\"sub:0\", shape=(), dtype=float32).\r\n```"},"comment":{"url":"https://api.github.com/repos/pierluigiferrari/ssd_keras/issues/comments/373527468","html_url":"https://github.com/pierluigiferrari/ssd_keras/issues/84#issuecomment-373527468","issue_url":"https://api.github.com/repos/pierluigiferrari/ssd_keras/issues/84","id":373527468,"user":{"login":"pierluigiferrari","id":17439178,"avatar_url":"https://avatars1.githubusercontent.com/u/17439178?v=4","gravatar_id":"","url":"https://api.github.com/users/pierluigiferrari","html_url":"https://github.com/pierluigiferrari","followers_url":"https://api.github.com/users/pierluigiferrari/followers","following_url":"https://api.github.com/users/pierluigiferrari/following{/other_user}","gists_url":"https://api.github.com/users/pierluigiferrari/gists{/gist_id}","starred_url":"https://api.github.com/users/pierluigiferrari/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pierluigiferrari/subscriptions","organizations_url":"https://api.github.com/users/pierluigiferrari/orgs","repos_url":"https://api.github.com/users/pierluigiferrari/repos","events_url":"https://api.github.com/users/pierluigiferrari/events{/privacy}","received_events_url":"https://api.github.com/users/pierluigiferrari/received_events","type":"User","site_admin":false},"created_at":"2018-03-15T21:23:44Z","updated_at":"2018-03-15T21:23:44Z","author_association":"OWNER","body":"Hm, very strange. I haven't been able to reproduce this error in TensorFlow 1.4.1. I have a hunch though. Could you try to compile the model with a different optimizer?  `keras.optimizers.SGD` or something. Anything but Adam. Let me know what happens."}}
{"actor":{"display_login":"pierluigiferrari"}}