{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/benchmarks/issues/243","repository_url":"https://api.github.com/repos/tensorflow/benchmarks","labels_url":"https://api.github.com/repos/tensorflow/benchmarks/issues/243/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/benchmarks/issues/243/comments","events_url":"https://api.github.com/repos/tensorflow/benchmarks/issues/243/events","html_url":"https://github.com/tensorflow/benchmarks/issues/243","id":353073044,"node_id":"MDU6SXNzdWUzNTMwNzMwNDQ=","number":243,"title":"[NotFoundError] tf_cnn_benchmarks.py cannot evaluate trained model, throwing affine0/biases NotFoundError","user":{"login":"kdatta","id":4193374,"node_id":"MDQ6VXNlcjQxOTMzNzQ=","avatar_url":"https://avatars2.githubusercontent.com/u/4193374?v=4","gravatar_id":"","url":"https://api.github.com/users/kdatta","html_url":"https://github.com/kdatta","followers_url":"https://api.github.com/users/kdatta/followers","following_url":"https://api.github.com/users/kdatta/following{/other_user}","gists_url":"https://api.github.com/users/kdatta/gists{/gist_id}","starred_url":"https://api.github.com/users/kdatta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kdatta/subscriptions","organizations_url":"https://api.github.com/users/kdatta/orgs","repos_url":"https://api.github.com/users/kdatta/repos","events_url":"https://api.github.com/users/kdatta/events{/privacy}","received_events_url":"https://api.github.com/users/kdatta/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-08-22T18:34:12Z","updated_at":"2018-08-23T01:15:27Z","closed_at":"2018-08-22T22:50:53Z","author_association":"NONE","body":"Hi,\r\n\r\nI trained ResNet50 model using the tf_cnn_benchmarks.py using the command line:\r\n$ python tf_cnn_benchmarks.py --batch_size=64 --model=resnet50 --num_intra_threads $OMP_NUM_THREADS --num_inter_threads 2 --display_every 5 --data_format NCHW --optimizer momentum --device cpu --num_batches=100 --mkl=true --variable_update horovod --horovod_device cpu --local_parameter_device cpu --kmp_blocktime=1 --data_dir=/scratch/TF_Records --data_name=imagenet --save_model_secs 60 --train_dir /tmp/\r\n\r\nTried to evaluate a model checkpoint:\r\n$ python tf_cnn_benchmarks.py --eval=True --model=resnet50 --data_name=imagenet --data_dir=/scratch/TF_Records --batch_size=32 --mkl=True --device=cpu --use_datasets=True --kmp_blocktime=0 --train_dir=/tmp --num_inter_threads=2 --num_intra_threads=40\r\n\r\nThis caused the NotFoundError:\r\nTensorFlow:  1.9\r\nModel:       resnet50\r\nDataset:     imagenet\r\nMode:        evaluation\r\nSingleSess:  False\r\nBatch size:  32 global\r\n             32 per device\r\nNum batches: 100\r\nNum epochs:  0.06\r\nDevices:     ['/cpu:0']\r\nData format: NCHW\r\nOptimizer:   sgd\r\nVariables:   parameter_server\r\n==========\r\nGenerating model\r\n2018-08-22 11:33:49.410834: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F\r\n\r\nUser settings:\r\n\r\n   KMP_AFFINITY=granularity=fine,verbose,compact,1,0\r\n   KMP_BLOCKTIME=0\r\n   KMP_SETTINGS=1\r\n   OMP_NUM_THREADS=40\r\n\r\nEffective settings:\r\n\r\n   KMP_ABORT_DELAY=0\r\n   KMP_ADAPTIVE_LOCK_PROPS='1,1024'\r\n   KMP_ALIGN_ALLOC=64\r\n   KMP_ALL_THREADPRIVATE=320\r\n   KMP_ATOMIC_MODE=2\r\n   KMP_BLOCKTIME=0\r\n   KMP_CPUINFO_FILE: value is not defined\r\n   KMP_DETERMINISTIC_REDUCTION=false\r\n   KMP_DEVICE_THREAD_LIMIT=2147483647\r\n   KMP_DISP_NUM_BUFFERS=7\r\n   KMP_DUPLICATE_LIB_OK=false\r\n   KMP_FORCE_REDUCTION: value is not defined\r\n   KMP_FOREIGN_THREADS_THREADPRIVATE=true\r\n   KMP_FORKJOIN_BARRIER='2,2'\r\n   KMP_FORKJOIN_BARRIER_PATTERN='hyper,hyper'\r\n   KMP_FORKJOIN_FRAMES=true\r\n   KMP_FORKJOIN_FRAMES_MODE=3\r\n   KMP_GTID_MODE=3\r\n   KMP_HANDLE_SIGNALS=false\r\n   KMP_HOT_TEAMS_MAX_LEVEL=1\r\n   KMP_HOT_TEAMS_MODE=0\r\n   KMP_INIT_AT_FORK=true\r\n   KMP_INIT_WAIT=2048\r\n   KMP_ITT_PREPARE_DELAY=0\r\n   KMP_LIBRARY=throughput\r\n   KMP_LOCK_KIND=queuing\r\n   KMP_MALLOC_POOL_INCR=1M\r\n   KMP_NEXT_WAIT=1024\r\n   KMP_NUM_LOCKS_IN_BLOCK=1\r\n   KMP_PLAIN_BARRIER='2,2'\r\n   KMP_PLAIN_BARRIER_PATTERN='hyper,hyper'\r\n   KMP_REDUCTION_BARRIER='1,1'\r\n   KMP_REDUCTION_BARRIER_PATTERN='hyper,hyper'\r\n   KMP_SCHEDULE='static,balanced;guided,iterative'\r\n   KMP_SETTINGS=true\r\n   KMP_SPIN_BACKOFF_PARAMS='4096,100'\r\n   KMP_STACKOFFSET=64\r\n   KMP_STACKPAD=0\r\n   KMP_STACKSIZE=4M\r\n   KMP_STORAGE_MAP=false\r\n   KMP_TASKING=2\r\n   KMP_TASKLOOP_MIN_TASKS=0\r\n   KMP_TASK_STEALING_CONSTRAINT=1\r\n   KMP_TEAMS_THREAD_LIMIT=80\r\n   KMP_TOPOLOGY_METHOD=all\r\n   KMP_USER_LEVEL_MWAIT=false\r\n   KMP_VERSION=false\r\n   KMP_WARNINGS=true\r\n   OMP_CANCELLATION=false\r\n   OMP_DEFAULT_DEVICE=0\r\n   OMP_DISPLAY_ENV=false\r\n   OMP_DYNAMIC=false\r\n   OMP_MAX_ACTIVE_LEVELS=2147483647\r\n   OMP_MAX_TASK_PRIORITY=0\r\n   OMP_NESTED=false\r\n   OMP_NUM_THREADS='40'\r\n   OMP_PLACES: value is not defined\r\n   OMP_PROC_BIND='intel'\r\n   OMP_SCHEDULE='static'\r\n   OMP_STACKSIZE=4M\r\n   OMP_THREAD_LIMIT=2147483647\r\n   OMP_WAIT_POLICY=PASSIVE\r\n   KMP_AFFINITY='verbose,warnings,respect,granularity=fine,compact,1,0'\r\n\r\nI0822 11:33:49.421849 140450627757888 tf_logging.py:115] Restoring parameters from /nfs/pdx/home/kdatta1/int8_inference/resnet_trained_model/cpu-trained-model/random_weights_tfcnnbm_100batches_20180822/model.ckpt-110\r\n2018-08-22 11:33:49.718046: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key v/cg/affine0/biases not found in checkpoint\r\nTraceback (most recent call last):\r\n  File \"./tf_cnn_benchmarks.py\", line 60, in <module>\r\n    app.run(main)  # Raises error on invalid flags, unlike tf.app.run()\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/absl/app.py\", line 274, in run\r\n    _run_main(main, args)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/absl/app.py\", line 238, in _run_main\r\n    sys.exit(main(argv))\r\n  File \"./tf_cnn_benchmarks.py\", line 56, in main\r\n    bench.run()\r\n  File \"/ec/site/disks/aipg_lab_home_pool_01/kdatta1/int8_inference/skx/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py\", line 1514, in run\r\n    return self._run_eval()\r\n  File \"/ec/site/disks/aipg_lab_home_pool_01/kdatta1/int8_inference/skx/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py\", line 1546, in _run_eval\r\n    input_producer_op, enqueue_ops, fetches, summary_op)\r\n  File \"/ec/site/disks/aipg_lab_home_pool_01/kdatta1/int8_inference/skx/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py\", line 1560, in _eval_once\r\n    global_step = load_checkpoint(saver, sess, self.params.train_dir)\r\n  File \"/ec/site/disks/aipg_lab_home_pool_01/kdatta1/int8_inference/skx/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py\", line 803, in load_checkpoint\r\n    saver.restore(sess, model_checkpoint_path)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py\", line 1768, in restore\r\n    six.reraise(exception_type, exception_value, exception_traceback)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py\", line 1752, in restore\r\n    {self.saver_def.filename_tensor_name: save_path})\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 900, in run\r\n    run_metadata_ptr)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 1135, in _run\r\n    feed_dict_tensor, options, run_metadata)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 1316, in _do_run\r\n    run_metadata)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py\", line 1335, in _do_call\r\n    raise type(e)(node_def, op, message)\r\ntensorflow.python.framework.errors_impl.NotFoundError: Key v/cg/affine0/biases not found in checkpoint\r\n\t [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_INT64, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]\r\n\r\nCaused by op u'save/RestoreV2', defined at:\r\n  File \"./tf_cnn_benchmarks.py\", line 60, in <module>\r\n    app.run(main)  # Raises error on invalid flags, unlike tf.app.run()\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/absl/app.py\", line 274, in run\r\n    _run_main(main, args)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/absl/app.py\", line 238, in _run_main\r\n    sys.exit(main(argv))\r\n  File \"./tf_cnn_benchmarks.py\", line 56, in main\r\n    bench.run()\r\n  File \"/ec/site/disks/aipg_lab_home_pool_01/kdatta1/int8_inference/skx/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py\", line 1514, in run\r\n    return self._run_eval()\r\n  File \"/ec/site/disks/aipg_lab_home_pool_01/kdatta1/int8_inference/skx/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py\", line 1530, in _run_eval\r\n    saver = tf.train.Saver(self.variable_mgr.savable_variables())\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py\", line 1284, in __init__\r\n    self.build()\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py\", line 1296, in build\r\n    self._build(self._filename, build_save=True, build_restore=True)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py\", line 1333, in _build\r\n    build_save=build_save, build_restore=build_restore)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py\", line 781, in _build_internal\r\n    restore_sequentially, reshape)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py\", line 400, in _AddRestoreOps\r\n    restore_sequentially)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py\", line 832, in bulk_restore\r\n    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py\", line 1463, in restore_v2\r\n    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py\", line 787, in _apply_op_helper\r\n    op_def=op_def)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py\", line 3414, in create_op\r\n    op_def=op_def)\r\n  File \"/nfs/pdx/home/kdatta1/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py\", line 1740, in __init__\r\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\r\n\r\nNotFoundError (see above for traceback): Key v/cg/affine0/biases not found in checkpoint\r\n\t [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_INT64, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]"},"comment":{"url":"https://api.github.com/repos/tensorflow/benchmarks/issues/comments/415240207","html_url":"https://github.com/tensorflow/benchmarks/issues/243#issuecomment-415240207","issue_url":"https://api.github.com/repos/tensorflow/benchmarks/issues/243","id":415240207,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNTI0MDIwNw==","user":{"login":"reedwm","id":6510203,"node_id":"MDQ6VXNlcjY1MTAyMDM=","avatar_url":"https://avatars2.githubusercontent.com/u/6510203?v=4","gravatar_id":"","url":"https://api.github.com/users/reedwm","html_url":"https://github.com/reedwm","followers_url":"https://api.github.com/users/reedwm/followers","following_url":"https://api.github.com/users/reedwm/following{/other_user}","gists_url":"https://api.github.com/users/reedwm/gists{/gist_id}","starred_url":"https://api.github.com/users/reedwm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reedwm/subscriptions","organizations_url":"https://api.github.com/users/reedwm/orgs","repos_url":"https://api.github.com/users/reedwm/repos","events_url":"https://api.github.com/users/reedwm/events{/privacy}","received_events_url":"https://api.github.com/users/reedwm/received_events","type":"User","site_admin":false},"created_at":"2018-08-23T01:15:27Z","updated_at":"2018-08-23T01:15:27Z","author_association":"MEMBER","body":"I also do not use horovod, but when evaluating, try running the exact same command, but adding `--eval` and potentially reducing `--num_batches`. In your case, try:\r\n\r\n```\r\n$ python tf_cnn_benchmarks.py --batch_size=64 --model=resnet50 --num_intra_threads $OMP_NUM_THREADS --num_inter_threads 2 --display_every 5 --data_format NCHW --optimizer momentum --device cpu --num_batches=100 --mkl=true --variable_update horovod --horovod_device cpu --local_parameter_device cpu --kmp_blocktime=1 --data_dir=/scratch/TF_Records --data_name=imagenet --save_model_secs 60 --train_dir /tmp/\r\n\r\n$ python tf_cnn_benchmarks.py --batch_size=64 --model=resnet50 --num_intra_threads $OMP_NUM_THREADS --num_inter_threads 2 --display_every 5 --data_format NCHW --optimizer momentum --device cpu --num_batches=100 --mkl=true --variable_update horovod --horovod_device cpu --local_parameter_device cpu --kmp_blocktime=1 --data_dir=/scratch/TF_Records --data_name=imagenet --save_model_secs 60 --train_dir /tmp/ --eval\r\n```"}}
{"actor":{"display_login":"reedwm"}}