title: Not able to deploy saved model into AWS Sagemaker
body: <p>I am following this article <a href="https://machinelearningmastery.com/how-to-perform-object-detection-with-yolov3-in-keras/" rel="nofollow noreferrer">https://machinelearningmastery.com/how-to-perform-object-detection-with-yolov3-in-keras/</a> to deploy YOLOv3 inside the AWS Sagemaker. I have the model.weights having weights and model.json having model structure and also model.h5 having model structure + weights. When I convert these files to protobuf format so that I can tar them and deploy on Sagemaker, this error comes in. </p>
  
  <pre><code>UnexpectedStatusException: Error hosting endpoint sagemaker-tensorflow-2020-04-12-10-57-05- 
  567: Failed. Reason:  The primary container for production variant AllTraffic did not pass the 
  ping health check. Please check CloudWatch logs for this endpoint..
  </code></pre>
  
  <p>Here is my code:</p>
  
  <hr>
  
  <pre><code>import tensorflow
  tensorflow.__version__
  
  Output:
  '1.7.0'
  </code></pre>
  
  <hr>
  
  <pre><code>import boto3, re
  from sagemaker import get_execution_role
  
  role = get_execution_role()
  
  from tensorflow.keras.models import model_from_json
  </code></pre>
  
  <hr>
  
  <pre><code>!ls keras_model/
  </code></pre>
  
  <hr>
  
  <pre><code>import tensorflow as tf
  
  json_file = open('/home/ec2-user/SageMaker/keras_model/' + 'model.json', 'r')
  loaded_model_json = json_file.read()
  json_file.close()
  loaded_model = model_from_json(loaded_model_json, custom_objects={"GlorotUniform": tf.keras.initializers.glorot_uniform})
  
  loaded_model.load_weights('/home/ec2-user/SageMaker/keras_model/model_weights.h5')
  print("Loaded model from disk ")
  </code></pre>
  
  <hr>
  
  <pre><code>from tensorflow.python.saved_model import builder
  from tensorflow.python.saved_model.signature_def_utils import predict_signature_def
  from tensorflow.python.saved_model import tag_constants
  
  # this directory sturcture will be followed as below. Do not change it.
  model_version = '1' 
  export_dir = 'export/Servo/' + model_version
  
  #Build the protocol buffer savedmodel at export_dir
  build = builder.SavedModelBuilder(export_dir)
  </code></pre>
  
  <hr>
  
  <pre><code>print(loaded_model.inputs)  
  print([t for t in loaded_model.outputs])
  
  Output:
  [&lt;tf.Tensor 'input_1:0' shape=(?, ?, ?, 3) dtype=float32&gt;]
  [&lt;tf.Tensor 'conv_81/BiasAdd:0' shape=(?, ?, ?, 255) dtype=float32&gt;, &lt;tf.Tensor 'conv_93/BiasAdd:0' shape=(?, ?, ?, 255) dtype=float32&gt;, &lt;tf.Tensor 'conv_105/BiasAdd:0' shape=(?, ?, ?, 255) dtype=float32&gt;]
  </code></pre>
  
  <hr>
  
  <pre><code>tf.convert_to_tensor(loaded_model.output)
  
  Output:
  &lt;tf.Tensor 'packed:0' shape=(3, ?, ?, ?, 255) dtype=float32&gt;
  </code></pre>
  
  <hr>
  
  <pre><code>signature = predict_signature_def(inputs={"input_image": loaded_model.input}, 
                                    outputs={t.name: t for t in loaded_model.outputs})
  </code></pre>
  
  <hr>
  
  <pre><code>from tensorflow.keras import backend as K
  with K.get_session() as sess:
      build.add_meta_graph_and_variables(sess=sess, tags=[tag_constants.SERVING], 
                                         signature_def_map={"serving_default":signature} )
      build.save()
  </code></pre>
  
  <hr>
  
  <pre><code>!ls export/Servo/1/variables/
  
  Output:
  variables.data-00000-of-00001  variables.index
  </code></pre>
  
  <hr>
  
  <pre><code>import tarfile 
  with tarfile.open('model.tar.gz', mode='w:gz') as archive:
      archive.add('export', recursive=True)
  </code></pre>
  
  <hr>
  
  <pre><code>import sagemaker
  
  sagemaker_session = sagemaker.Session()
  inputs = sagemaker_session.upload_data(path='model.tar.gz', key_prefix='model')
  </code></pre>
  
  <hr>
  
  <pre><code>!touch train.py
  
  from sagemaker.tensorflow.model import TensorFlowModel
  sagemaker_model = TensorFlowModel(model_data='s3://' + sagemaker_session.default_bucket() + '/model/model.tar.gz',
                                   role = role,
                                   entry_point= 'train.py')
  
  %%time
  predictor = sagemaker_model.deploy(initial_instance_count=1, 
                                     instance_type='ml.t2.large')
  </code></pre>
  
  <hr>
  
  <p>Error:</p>
  
  <pre><code>-----------------------------*
  ---------------------------------------------------------------------------
  UnexpectedStatusException                 Traceback (most recent call last)
  &lt;timed exec&gt; in &lt;module&gt;()
  
  ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sagemaker/model.py in deploy(self, initial_instance_count, instance_type, accelerator_type, endpoint_name, update_endpoint, tags, kms_key, wait, data_capture_config)
      478                 kms_key=kms_key,
      479                 wait=wait,
  --&gt; 480                 data_capture_config_dict=data_capture_config_dict,
      481             )
      482 
  
  ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sagemaker/session.py in endpoint_from_production_variants(self, name, production_variants, tags, kms_key, wait, data_capture_config_dict)
     2849 
     2850             self.sagemaker_client.create_endpoint_config(**config_options)
  -&gt; 2851         return self.create_endpoint(endpoint_name=name, config_name=name, tags=tags, wait=wait)
     2852 
     2853     def expand_role(self, role):
  
  ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sagemaker/session.py in create_endpoint(self, endpoint_name, config_name, tags, wait)
     2381         )
     2382         if wait:
  -&gt; 2383             self.wait_for_endpoint(endpoint_name)
     2384         return endpoint_name
     2385 
  
  ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/sagemaker/session.py in wait_for_endpoint(self, endpoint, poll)
     2638                 ),
     2639                 allowed_statuses=["InService"],
  -&gt; 2640                 actual_status=status,
     2641             )
     2642         return desc
  
  UnexpectedStatusException: Error hosting endpoint sagemaker-tensorflow-2020-04-12-10-57-05-567: Failed. Reason:  The primary container for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint..
  </code></pre>
  
  <hr>
  
  <p>I think that error is due to the difference between the Tensor shape of <code>loaded_model.inputs</code> and <code>loaded_model.outputs</code>. But I am still not sure about what do the 3 and 255 represent in the shape. Any help will be greatly appreciated. </p>
  
  <pre><code>print(loaded_model.inputs)
  [&lt;tf.Tensor 'input_1:0' shape=(?, ?, ?, 3) dtype=float32&gt;]
  
  print([t for t in loaded_model.outputs])
  [&lt;tf.Tensor 'conv_81/BiasAdd:0' shape=(?, ?, ?, 255) dtype=float32&gt;, 
   &lt;tf.Tensor 'conv_93/BiasAdd:0' shape=(?, ?, ?, 255) dtype=float32&gt;, 
   &lt;tf.Tensor 'conv_105/BiasAdd:0' shape=(?, ?, ?, 255) dtype=float32&gt;]
  </code></pre>
  
  <p>Cloudwatch log:</p>
  
  <pre><code>2020-04-12 11:01:33,439 INFO - root - running container entrypoint
  2020-04-12 11:01:33,440 INFO - root - starting serve task
  2020-04-12 11:01:33,440 INFO - container_support.serving - reading config
  Downloading s3://sagemaker-us-east-1-611475884433/sagemaker-tensorflow-2020-04-12-10-57-05-375/sourcedir.tar.gz to /tmp/script.tar.gz
  2020-04-12 11:01:33,828 INFO - container_support.serving - importing user module
  2020-04-12 11:01:33,828 INFO - container_support.serving - loading framework-specific dependencies
  2020-04-12 11:01:35,795 INFO - container_support.serving - starting nginx
  2020-04-12 11:01:35,797 INFO - container_support.serving - nginx config: 
  worker_processes auto;
  daemon off;
  pid /tmp/nginx.pid;
  error_log /var/log/nginx/error.log;
  worker_rlimit_nofile 4096;
  events {
    worker_connections 2048;
  }
  http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    access_log /var/log/nginx/access.log combined;
  
    upstream gunicorn {
      server unix:/tmp/gunicorn.sock;
    }
  
    server {
      listen 8080 deferred;
      client_max_body_size 0;
  
      keepalive_timeout 3;
  
      location ~ ^/(ping|invocations) {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://gunicorn;
      }
  
      location / {
        return 404 "{}";
      }
    }
  }
  
  2020-04-12 11:01:35,815 INFO - container_support.serving - starting gunicorn
  2020-04-12 11:01:35,820 INFO - container_support.serving - inference server started. waiting on processes: set([24, 23])
  2020-04-12 11:01:35.904746: I tensorflow_serving/model_servers/server.cc:82] Building single TensorFlow model file config:  model_name: generic_model model_base_path: /opt/ml/model/export/Servo
  2020-04-12 11:01:35.905995: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
  2020-04-12 11:01:35.906148: I tensorflow_serving/model_servers/server_core.cc:517]  (Re-)adding model: generic_model
  2020-04-12 11:01:35.907173: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: generic_model version: 1}
  2020-04-12 11:01:35.907349: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: generic_model version: 1}
  2020-04-12 11:01:35.907422: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: generic_model version: 1}
  2020-04-12 11:01:35.907578: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:360] Attempting to load native SavedModelBundle in bundle-shim from: /opt/ml/model/export/Servo/1
  2020-04-12 11:01:35.907687: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /opt/ml/model/export/Servo/1
  2020-04-12 11:01:35.939232: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
  2020-04-12 11:01:35.980215: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
  [2020-04-12 11:01:36 +0000] [24] [INFO] Starting gunicorn 19.9.0
  2020-04-12 11:01:36.048327: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:259] SavedModel load for tags { serve }; Status: fail. Took 140502 microseconds.
  2020-04-12 11:01:36.048617: E tensorflow_serving/util/retrier.cc:37] Loading servable: {name: generic_model version: 1} failed: Not found: Op type not registered 'FusedBatchNormV3' in binary running on model.aws.local. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
  </code></pre>