{"action":"created","issue":{"url":"https://api.github.com/repos/bigboNed3/bert_serving/issues/1","repository_url":"https://api.github.com/repos/bigboNed3/bert_serving","labels_url":"https://api.github.com/repos/bigboNed3/bert_serving/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/bigboNed3/bert_serving/issues/1/comments","events_url":"https://api.github.com/repos/bigboNed3/bert_serving/issues/1/events","html_url":"https://github.com/bigboNed3/bert_serving/issues/1","id":393759084,"node_id":"MDU6SXNzdWUzOTM3NTkwODQ=","number":1,"title":"exporting extract","user":{"login":"mvss80","id":5709876,"node_id":"MDQ6VXNlcjU3MDk4NzY=","avatar_url":"https://avatars0.githubusercontent.com/u/5709876?v=4","gravatar_id":"","url":"https://api.github.com/users/mvss80","html_url":"https://github.com/mvss80","followers_url":"https://api.github.com/users/mvss80/followers","following_url":"https://api.github.com/users/mvss80/following{/other_user}","gists_url":"https://api.github.com/users/mvss80/gists{/gist_id}","starred_url":"https://api.github.com/users/mvss80/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mvss80/subscriptions","organizations_url":"https://api.github.com/users/mvss80/orgs","repos_url":"https://api.github.com/users/mvss80/repos","events_url":"https://api.github.com/users/mvss80/events{/privacy}","received_events_url":"https://api.github.com/users/mvss80/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-12-23T15:34:43Z","updated_at":"2018-12-28T02:47:23Z","closed_at":null,"author_association":"NONE","body":"I need to export BERT model for tensorflow_serving where I can get the embeddings as outputs as shown in extract_features.py.\r\n\r\nI followed your example of run_classifier.py and modified extract_features by adding flags 'do_export' and 'export_dir', added serving_input_fn and call estimator.export_savedmodel.\r\n\r\nWhen I call my function\r\n```\r\npython extract_features_export.py --input_file=/tmp/input.txt --output_file=/tmp/output.jsonl --vocab_file=$BERT_BASE_DIR/vocab.txt --bert_config_file=$BERT_BASE_DIR/bert_config.json --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt --layers=-1,-2,-3,-4 --max_seq_length=128 --batch_size=8 --do_export=True --export_dir='/models'\r\n```\r\nthe model runs but gives me an error when trying to export. \r\n```\r\nINFO:tensorflow:*** Example ***\r\nINFO:tensorflow:unique_id: 0\r\nINFO:tensorflow:tokens: [CLS] who was jim henson ? [SEP] jim henson was a puppet ##eer [SEP]\r\nINFO:tensorflow:input_ids: 101 2040 2001 3958 27227 1029 102 3958 27227 2001 1037 13997 11510 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\r\nINFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\r\nINFO:tensorflow:input_type_ids: 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\r\nWARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7f0d1b3811e0>) includes params argument, but params are not passed to Estimator.\r\nWARNING:tensorflow:Using temporary folder as model directory: /tmp/tmp3eh_rldt\r\nINFO:tensorflow:Using config: {'_model_dir': '/tmp/tmp3eh_rldt', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\r\ngraph_options {\r\n  rewrite_options {\r\n    meta_optimizer_iterations: ONE\r\n  }\r\n}\r\n, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f0d12f0b6d8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=2, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': None}\r\nINFO:tensorflow:_TPUContext: eval_on_tpu True\r\nWARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.\r\nINFO:tensorflow:Could not find trained model in model_dir: /tmp/tmp3eh_rldt, running initialization to predict.\r\nINFO:tensorflow:Calling model_fn.\r\nINFO:tensorflow:Running infer on CPU\r\nINFO:tensorflow:Done calling model_fn.\r\nINFO:tensorflow:Graph was finalized.\r\n2018-12-23 09:33:37.252118: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2018-12-23 09:33:37.252483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: \r\nname: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.797\r\npciBusID: 0000:01:00.0\r\ntotalMemory: 7.93GiB freeMemory: 7.09GiB\r\n2018-12-23 09:33:37.252510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0\r\n2018-12-23 09:33:37.435350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n2018-12-23 09:33:37.435377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 \r\n2018-12-23 09:33:37.435382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N \r\n2018-12-23 09:33:37.435506: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6834 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)\r\nINFO:tensorflow:Running local_init_op.\r\nINFO:tensorflow:Done running local_init_op.\r\nINFO:tensorflow:prediction_loop marked as finished\r\nINFO:tensorflow:prediction_loop marked as finished\r\nTraceback (most recent call last):\r\n  File \"extract_features_export.py\", line 431, in <module>\r\n    tf.app.run()\r\n  File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/platform/app.py\", line 125, in run\r\n    _sys.exit(main(argv))\r\n  File \"extract_features_export.py\", line 423, in main\r\n    estimator.export_savedmodel(FLAGS.export_dir, serving_input_fn, strip_default_attrs=True)\r\n  File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 663, in export_savedmodel\r\n    mode=model_fn_lib.ModeKeys.PREDICT)\r\n  File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 789, in _export_saved_model_for_mode\r\n    strip_default_attrs=strip_default_attrs)\r\n  File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 878, in _export_all_saved_models\r\n    raise ValueError(\"Couldn't find trained model at %s.\" % self._model_dir)\r\nValueError: Couldn't find trained model at /tmp/tmp3eh_rldt.\r\n\r\n```\r\nAny ideas on how I can get this fixed?"},"comment":{"url":"https://api.github.com/repos/bigboNed3/bert_serving/issues/comments/450277093","html_url":"https://github.com/bigboNed3/bert_serving/issues/1#issuecomment-450277093","issue_url":"https://api.github.com/repos/bigboNed3/bert_serving/issues/1","id":450277093,"node_id":"MDEyOklzc3VlQ29tbWVudDQ1MDI3NzA5Mw==","user":{"login":"bigboNed3","id":25605618,"node_id":"MDQ6VXNlcjI1NjA1NjE4","avatar_url":"https://avatars0.githubusercontent.com/u/25605618?v=4","gravatar_id":"","url":"https://api.github.com/users/bigboNed3","html_url":"https://github.com/bigboNed3","followers_url":"https://api.github.com/users/bigboNed3/followers","following_url":"https://api.github.com/users/bigboNed3/following{/other_user}","gists_url":"https://api.github.com/users/bigboNed3/gists{/gist_id}","starred_url":"https://api.github.com/users/bigboNed3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bigboNed3/subscriptions","organizations_url":"https://api.github.com/users/bigboNed3/orgs","repos_url":"https://api.github.com/users/bigboNed3/repos","events_url":"https://api.github.com/users/bigboNed3/events{/privacy}","received_events_url":"https://api.github.com/users/bigboNed3/received_events","type":"User","site_admin":false},"created_at":"2018-12-28T02:47:23Z","updated_at":"2018-12-28T02:47:23Z","author_association":"OWNER","body":"> Yes but I don't understand the error.\r\n> \r\n> I updated `run_config` to include an `output_dir` as follows:\r\n> \r\n> ```\r\n>   run_config = tf.contrib.tpu.RunConfig(\r\n>       master=FLAGS.master,\r\n>       model_dir=FLAGS.output_dir,\r\n>       tpu_config=tf.contrib.tpu.TPUConfig(\r\n>           num_shards=FLAGS.num_tpu_cores,\r\n>           per_host_input_for_training=is_per_host))\r\n> ```\r\n> When I now call the function by giving `--output_dir=$BERT_BASE_DIR`, I still get the same \"`ValueError: Couldn't find trained model at ...`\" error.\r\n> \r\n> To try something different, I called `export_saved_model` instead of `export_savedmodel` where I can give the model checkpoint\r\n> \r\n> ```\r\n> estimator.export_saved_model(FLAGS.export_dir, serving_input_fn, checkpoint_path=FLAGS.init_checkpoint)\r\n> ```\r\n> but then I get the following errors -\r\n> \r\n> ```\r\n> 2018-12-27 10:45:59.152572: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key global_step not found in checkpoint\r\n> Traceback (most recent call last):\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 1334, in _do_call\r\n>     return fn(*args)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 1319, in _run_fn\r\n>     options, feed_dict, fetch_list, target_list, run_metadata)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 1407, in _call_tf_sessionrun\r\n>     run_metadata)\r\n> tensorflow.python.framework.errors_impl.NotFoundError: Key global_step not found in checkpoint\r\n> \t [[{{node save/RestoreV2}} = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]\r\n> \r\n> During handling of the above exception, another exception occurred:\r\n> \r\n> Traceback (most recent call last):\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1546, in restore\r\n>     {self.saver_def.filename_tensor_name: save_path})\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 929, in run\r\n>     run_metadata_ptr)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 1152, in _run\r\n>     feed_dict_tensor, options, run_metadata)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 1328, in _do_run\r\n>     run_metadata)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/client/session.py\", line 1348, in _do_call\r\n>     raise type(e)(node_def, op, message)\r\n> tensorflow.python.framework.errors_impl.NotFoundError: Key global_step not found in checkpoint\r\n> \t [[node save/RestoreV2 (defined at /home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py:2172)  = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]\r\n> \r\n> Caused by op 'save/RestoreV2', defined at:\r\n>   File \"extract_features_export.py\", line 443, in <module>\r\n>     tf.app.run()\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/platform/app.py\", line 125, in run\r\n>     _sys.exit(main(argv))\r\n>   File \"extract_features_export.py\", line 435, in main\r\n>     estimator.export_saved_model(FLAGS.export_dir, serving_input_fn, checkpoint_path=FLAGS.init_checkpoint)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 734, in export_saved_model\r\n>     strip_default_attrs=True)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 663, in export_savedmodel\r\n>     mode=model_fn_lib.ModeKeys.PREDICT)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 789, in _export_saved_model_for_mode\r\n>     strip_default_attrs=strip_default_attrs)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 907, in _export_all_saved_models\r\n>     mode=model_fn_lib.ModeKeys.PREDICT)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py\", line 2172, in _add_meta_graph_for_mode\r\n>     check_variables=check_variables))\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 1011, in _add_meta_graph_for_mode\r\n>     graph_saver = estimator_spec.scaffold.saver or saver.Saver(sharded=True)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1102, in __init__\r\n>     self.build()\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1114, in build\r\n>     self._build(self._filename, build_save=True, build_restore=True)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1151, in _build\r\n>     build_save=build_save, build_restore=build_restore)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 789, in _build_internal\r\n>     restore_sequentially, reshape)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 459, in _AddShardedRestoreOps\r\n>     name=\"restore_shard\"))\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 406, in _AddRestoreOps\r\n>     restore_sequentially)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 862, in bulk_restore\r\n>     return io_ops.restore_v2(filename_tensor, names, slices, dtypes)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py\", line 1466, in restore_v2\r\n>     shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py\", line 787, in _apply_op_helper\r\n>     op_def=op_def)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py\", line 488, in new_func\r\n>     return func(*args, **kwargs)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 3274, in create_op\r\n>     op_def=op_def)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 1770, in __init__\r\n>     self._traceback = tf_stack.extract_stack()\r\n> \r\n> NotFoundError (see above for traceback): Key global_step not found in checkpoint\r\n> \t [[node save/RestoreV2 (defined at /home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py:2172)  = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]\r\n> \r\n> \r\n> During handling of the above exception, another exception occurred:\r\n> \r\n> Traceback (most recent call last):\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1556, in restore\r\n>     names_to_keys = object_graph_key_mapping(save_path)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1830, in object_graph_key_mapping\r\n>     checkpointable.OBJECT_GRAPH_PROTO_KEY)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py\", line 371, in get_tensor\r\n>     status)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py\", line 528, in __exit__\r\n>     c_api.TF_GetCode(self.status.status))\r\n> tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint\r\n> \r\n> During handling of the above exception, another exception occurred:\r\n> \r\n> Traceback (most recent call last):\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 1018, in _add_meta_graph_for_mode\r\n>     graph_saver.restore(session, checkpoint_path)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1562, in restore\r\n>     err, \"a Variable name or other graph key that is missing\")\r\n> tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:\r\n> \r\n> Key global_step not found in checkpoint\r\n> \t [[node save/RestoreV2 (defined at /home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py:2172)  = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]\r\n> \r\n> Caused by op 'save/RestoreV2', defined at:\r\n>   File \"extract_features_export.py\", line 443, in <module>\r\n>     tf.app.run()\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/platform/app.py\", line 125, in run\r\n>     _sys.exit(main(argv))\r\n>   File \"extract_features_export.py\", line 435, in main\r\n>     estimator.export_saved_model(FLAGS.export_dir, serving_input_fn, checkpoint_path=FLAGS.init_checkpoint)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 734, in export_saved_model\r\n>     strip_default_attrs=True)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 663, in export_savedmodel\r\n>     mode=model_fn_lib.ModeKeys.PREDICT)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 789, in _export_saved_model_for_mode\r\n>     strip_default_attrs=strip_default_attrs)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 907, in _export_all_saved_models\r\n>     mode=model_fn_lib.ModeKeys.PREDICT)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py\", line 2172, in _add_meta_graph_for_mode\r\n>     check_variables=check_variables))\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 1011, in _add_meta_graph_for_mode\r\n>     graph_saver = estimator_spec.scaffold.saver or saver.Saver(sharded=True)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1102, in __init__\r\n>     self.build()\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1114, in build\r\n>     self._build(self._filename, build_save=True, build_restore=True)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1151, in _build\r\n>     build_save=build_save, build_restore=build_restore)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 789, in _build_internal\r\n>     restore_sequentially, reshape)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 459, in _AddShardedRestoreOps\r\n>     name=\"restore_shard\"))\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 406, in _AddRestoreOps\r\n>     restore_sequentially)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 862, in bulk_restore\r\n>     return io_ops.restore_v2(filename_tensor, names, slices, dtypes)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py\", line 1466, in restore_v2\r\n>     shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py\", line 787, in _apply_op_helper\r\n>     op_def=op_def)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py\", line 488, in new_func\r\n>     return func(*args, **kwargs)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 3274, in create_op\r\n>     op_def=op_def)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 1770, in __init__\r\n>     self._traceback = tf_stack.extract_stack()\r\n> \r\n> NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:\r\n> \r\n> Key global_step not found in checkpoint\r\n> \t [[node save/RestoreV2 (defined at /home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py:2172)  = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]\r\n> \r\n> \r\n> During handling of the above exception, another exception occurred:\r\n> \r\n> Traceback (most recent call last):\r\n>   File \"extract_features_export.py\", line 443, in <module>\r\n>     tf.app.run()\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/platform/app.py\", line 125, in run\r\n>     _sys.exit(main(argv))\r\n>   File \"extract_features_export.py\", line 435, in main\r\n>     estimator.export_saved_model(FLAGS.export_dir, serving_input_fn, checkpoint_path=FLAGS.init_checkpoint)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 734, in export_saved_model\r\n>     strip_default_attrs=True)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 663, in export_savedmodel\r\n>     mode=model_fn_lib.ModeKeys.PREDICT)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 789, in _export_saved_model_for_mode\r\n>     strip_default_attrs=strip_default_attrs)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 907, in _export_all_saved_models\r\n>     mode=model_fn_lib.ModeKeys.PREDICT)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py\", line 2172, in _add_meta_graph_for_mode\r\n>     check_variables=check_variables))\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 1026, in _add_meta_graph_for_mode\r\n>     raise ValueError(msg)\r\n> ValueError: Could not load all requested variables from checkpoint. Please make sure your model_fn does not expect variables that were not saved in the checkpoint.\r\n> \r\n> Encountered error with mode `infer` while restoring checkpoint from: `/home/m/Downloads/BERT/uncased_L-24_H-1024_A-16/bert_model.ckpt`. Full Traceback:\r\n> \r\n> Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:\r\n> \r\n> Key global_step not found in checkpoint\r\n> \t [[node save/RestoreV2 (defined at /home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py:2172)  = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]\r\n> \r\n> Caused by op 'save/RestoreV2', defined at:\r\n>   File \"extract_features_export.py\", line 443, in <module>\r\n>     tf.app.run()\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/platform/app.py\", line 125, in run\r\n>     _sys.exit(main(argv))\r\n>   File \"extract_features_export.py\", line 435, in main\r\n>     estimator.export_saved_model(FLAGS.export_dir, serving_input_fn, checkpoint_path=FLAGS.init_checkpoint)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 734, in export_saved_model\r\n>     strip_default_attrs=True)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 663, in export_savedmodel\r\n>     mode=model_fn_lib.ModeKeys.PREDICT)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 789, in _export_saved_model_for_mode\r\n>     strip_default_attrs=strip_default_attrs)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 907, in _export_all_saved_models\r\n>     mode=model_fn_lib.ModeKeys.PREDICT)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py\", line 2172, in _add_meta_graph_for_mode\r\n>     check_variables=check_variables))\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py\", line 1011, in _add_meta_graph_for_mode\r\n>     graph_saver = estimator_spec.scaffold.saver or saver.Saver(sharded=True)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1102, in __init__\r\n>     self.build()\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1114, in build\r\n>     self._build(self._filename, build_save=True, build_restore=True)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 1151, in _build\r\n>     build_save=build_save, build_restore=build_restore)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 789, in _build_internal\r\n>     restore_sequentially, reshape)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 459, in _AddShardedRestoreOps\r\n>     name=\"restore_shard\"))\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 406, in _AddRestoreOps\r\n>     restore_sequentially)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/training/saver.py\", line 862, in bulk_restore\r\n>     return io_ops.restore_v2(filename_tensor, names, slices, dtypes)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py\", line 1466, in restore_v2\r\n>     shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py\", line 787, in _apply_op_helper\r\n>     op_def=op_def)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py\", line 488, in new_func\r\n>     return func(*args, **kwargs)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 3274, in create_op\r\n>     op_def=op_def)\r\n>   File \"/home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 1770, in __init__\r\n>     self._traceback = tf_stack.extract_stack()\r\n> \r\n> NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:\r\n> \r\n> Key global_step not found in checkpoint\r\n> \t [[node save/RestoreV2 (defined at /home/m/anaconda3/envs/gpu-dev/lib/python3.6/site-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py:2172)  = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]\r\n> ```\r\n> Have you tried exporting the model for feature extraction? If you were successful, can you share the saved model?\r\n\r\nsorry, I haven't tried that. /(ㄒoㄒ)/~~"}}
{"actor":{"display_login":"bigboNed3"}}