{"action":"created","issue":{"url":"https://api.github.com/repos/fizyr/keras-retinanet/issues/646","repository_url":"https://api.github.com/repos/fizyr/keras-retinanet","labels_url":"https://api.github.com/repos/fizyr/keras-retinanet/issues/646/labels{/name}","comments_url":"https://api.github.com/repos/fizyr/keras-retinanet/issues/646/comments","events_url":"https://api.github.com/repos/fizyr/keras-retinanet/issues/646/events","html_url":"https://github.com/fizyr/keras-retinanet/issues/646","id":354052833,"node_id":"MDU6SXNzdWUzNTQwNTI4MzM=","number":646,"title":"FailedPreconditionError: Attempting to use uninitialized value training_...","user":{"login":"Miguel-P","id":25351417,"node_id":"MDQ6VXNlcjI1MzUxNDE3","avatar_url":"https://avatars1.githubusercontent.com/u/25351417?v=4","gravatar_id":"","url":"https://api.github.com/users/Miguel-P","html_url":"https://github.com/Miguel-P","followers_url":"https://api.github.com/users/Miguel-P/followers","following_url":"https://api.github.com/users/Miguel-P/following{/other_user}","gists_url":"https://api.github.com/users/Miguel-P/gists{/gist_id}","starred_url":"https://api.github.com/users/Miguel-P/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Miguel-P/subscriptions","organizations_url":"https://api.github.com/users/Miguel-P/orgs","repos_url":"https://api.github.com/users/Miguel-P/repos","events_url":"https://api.github.com/users/Miguel-P/events{/privacy}","received_events_url":"https://api.github.com/users/Miguel-P/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-26T00:07:48Z","updated_at":"2018-08-28T04:00:58Z","closed_at":null,"author_association":"NONE","body":"I am using custom code based on the train.py script. I am attempting to create the resnet50_retinanet model and then load up the pretrained coco weights, freeze certian layers and then compile and train using a custom CSV datatset. I'm completely lost by this error message?\r\n\r\nMy code is here:\r\n...\r\n```python\r\nfrom keras_retinanet import models\r\nfrom keras_retinanet.models import resnet\r\n\r\nnum_classes = 5\r\nbase_model = resnet.resnet50_retinanet(num_classes , inputs = None )\r\nbase_model.load_weights(r\"resnet50_coco_best_v2.1.0.h5\" , by_name = True , skip_mismatch=True)\r\n\r\nfor layer in base_model.layers[:]:\r\n    layer.trainable = False    \r\n#make the last 9 layers trainable\r\nfor layer in base_model.layers[len(base_model.layers)-9:]:\r\n    layer.trainable = True\r\n\r\nfrom keras.optimizers import SGD #using adam here also causes same error\r\nfrom keras_retinanet import losses\r\nfrom keras_retinanet.models.retinanet import retinanet_bbox\r\n#training is same ss base because single GPU\r\ntraining_model = base_model\r\ntraining_model.compile(\r\n    loss = {\r\n        'regression'  : losses.smooth_l1(),\r\n        'classification': losses.focal()\r\n    },\r\n    optimizer = SGD(lr=1e-5, clipnorm=0.001)\r\n)\r\npred_model = retinanet_bbox(model=base_model)\r\n#After creating the callbacks and training and evaluation generators...\r\nsteps_epochs =  1250\r\nnum_epochs = 10\r\ntraining_model.fit_generator(\r\n    generator = train_gen,\r\n    steps_per_epoch = steps_epochs,\r\n    epochs = num_epochs,\r\n    verbose = 1, \r\n    callbacks = callbacks_dat\r\n)\r\n```\r\n\r\nAnd the error message is:\r\n\r\n> Epoch 1/10\r\n> ---------------------------------------------------------------------------\r\n> FailedPreconditionError                   Traceback (most recent call last)\r\n> <ipython-input-31-1eb9bc0ef87c> in <module>()\r\n>       7     epochs = num_epochs,\r\n>       8     verbose = 1,\r\n> ----> 9     callbacks = callbacks_dat\r\n>      10 )\r\n> \r\n> c:\\users\\user\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\keras\\legacy\\interfaces.py in wrapper(*args, **kwargs)\r\n>      89                 warnings.warn('Update your `' + object_name +\r\n>      90                               '` call to the Keras 2 API: ' + signature, stacklevel=2)\r\n> ---> 91             return func(*args, **kwargs)\r\n>      92         wrapper._original_function = func\r\n>      93         return wrapper\r\n> \r\n> c:\\users\\user\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\keras\\engine\\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\r\n>    1424             use_multiprocessing=use_multiprocessing,\r\n>    1425             shuffle=shuffle,\r\n> -> 1426             initial_epoch=initial_epoch)\r\n>    1427 \r\n>    1428     @interfaces.legacy_generator_methods_support\r\n> \r\n> c:\\users\\user\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\keras\\engine\\training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\r\n>     189                 outs = model.train_on_batch(x, y,\r\n>     190                                             sample_weight=sample_weight,\r\n> --> 191                                             class_weight=class_weight)\r\n>     192 \r\n>     193                 if not isinstance(outs, list):\r\n> \r\n> c:\\users\\user\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\keras\\engine\\training.py in train_on_batch(self, x, y, sample_weight, class_weight)\r\n>    1218             ins = x + y + sample_weights\r\n>    1219         self._make_train_function()\r\n> -> 1220         outputs = self.train_function(ins)\r\n>    1221         if len(outputs) == 1:\r\n>    1222             return outputs[0]\r\n> \r\n> c:\\users\\user\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py in __call__(self, inputs)\r\n>    2659                 return self._legacy_call(inputs)\r\n>    2660 \r\n> -> 2661             return self._call(inputs)\r\n>    2662         else:\r\n>    2663             if py_any(is_tensor(x) for x in inputs):\r\n> \r\n> c:\\users\\user\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py in _call(self, inputs)\r\n>    2629                                 symbol_vals,\r\n>    2630                                 session)\r\n> -> 2631         fetched = self._callable_fn(*array_vals)\r\n>    2632         return fetched[:len(self.outputs)]\r\n>    2633 \r\n> \r\n> c:\\users\\user\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\tensorflow\\python\\client\\session.py in __call__(self, *args)\r\n>    1452         else:\r\n>    1453           return tf_session.TF_DeprecatedSessionRunCallable(\r\n> -> 1454               self._session._session, self._handle, args, status, None)\r\n>    1455 \r\n>    1456     def __del__(self):\r\n> \r\n> c:\\users\\user\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\tensorflow\\python\\framework\\errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)\r\n>     517             None, None,\r\n>     518             compat.as_text(c_api.TF_Message(self.status.status)),\r\n> --> 519             c_api.TF_GetCode(self.status.status))\r\n>     520     # Delete the underlying status object from memory otherwise it stays alive\r\n>     521     # as there is a reference to status from this from the traceback due to\r\n> \r\n> FailedPreconditionError: Attempting to use uninitialized value training_2/SGD/Variable_43\r\n> \t [[Node: training_2/SGD/Variable_43/read = Identity[T=DT_FLOAT, _class=[\"loc:@training_2/SGD/Assign_86\"], _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](training_2/SGD/Variable_43)]]\r\n> \t [[Node: loss_2/regression_loss/Mean_2/_5469 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device_incarnation=1, tensor_name=\"edge_7035_loss_2/regression_loss/Mean_2\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]\r\n\r\nI'm still new to tensorflow and keras and have no idea what exactly is going on?? The same error occurs regardless of the optimizer I choose (\r\n\r\n> FailedPreconditionError: Attempting to use uninitialized value training_...\r\n\r\n)\r\nI have a windows machine and have installed the windows fork of the cocoapi. I also have tensorflow 1.9.0 and am sure that both keras-retinanet and tensorflow have been installed correctly"},"comment":{"url":"https://api.github.com/repos/fizyr/keras-retinanet/issues/comments/416443757","html_url":"https://github.com/fizyr/keras-retinanet/issues/646#issuecomment-416443757","issue_url":"https://api.github.com/repos/fizyr/keras-retinanet/issues/646","id":416443757,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNjQ0Mzc1Nw==","user":{"login":"avaghefi","id":7268410,"node_id":"MDQ6VXNlcjcyNjg0MTA=","avatar_url":"https://avatars0.githubusercontent.com/u/7268410?v=4","gravatar_id":"","url":"https://api.github.com/users/avaghefi","html_url":"https://github.com/avaghefi","followers_url":"https://api.github.com/users/avaghefi/followers","following_url":"https://api.github.com/users/avaghefi/following{/other_user}","gists_url":"https://api.github.com/users/avaghefi/gists{/gist_id}","starred_url":"https://api.github.com/users/avaghefi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/avaghefi/subscriptions","organizations_url":"https://api.github.com/users/avaghefi/orgs","repos_url":"https://api.github.com/users/avaghefi/repos","events_url":"https://api.github.com/users/avaghefi/events{/privacy}","received_events_url":"https://api.github.com/users/avaghefi/received_events","type":"User","site_admin":false},"created_at":"2018-08-28T04:00:58Z","updated_at":"2018-08-28T04:00:58Z","author_association":"NONE","body":"@Miguel-P did you manage to figure out the problem but don't know how to solve it. The problem probably is AdamOptimizer and some other optimizers, create addition variables internally, but they don't get initialized before training the model. See [This](https://stackoverflow.com/questions/33788989/tensorflow-using-adam-optimizer). When creating a model manually, then `tf.global_variables_initializer\r\nDuplicate of ` should work. But here, I don't know how to initialize variables. \r\n"}}
{"actor":{"display_login":"avaghefi"}}