title: How to a make a model ready for TensorFlow Serving REST interface with a base64 encoded image?
body: <p>My understanding is that I should be able to grab a TensorFlow model from Google's AI Hub, deploy it to TensorFlow Serving and use it to make predictions by POSTing images via REST requests using curl.</p>
  
  <p>I could not find any bbox predictors on AI Hub at this time but I did find one on the TensorFlow model zoo:</p>
  
  <p><a href="http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_coco_2018_03_29.tar.gz" rel="noreferrer">http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_coco_2018_03_29.tar.gz</a></p>
  
  <p>I have the model deployed to TensorFlow serving, but the documentation is unclear with respect to exactly what should be included in the JSON of the REST request.</p>
  
  <p>My understanding is that   </p>
  
  <ol>
  <li>The SignatureDefinition of the model determines what the JSON should look like     </li>
  <li>I should base64 encode the images    </li>
  </ol>
  
  <p>I was able to get the signature definition of the model like so:</p>
  
  <pre><code>&gt;python tensorflow/tensorflow/python/tools/saved_model_cli.py show --dir /Users/alexryan/alpine/git/tfserving-tutorial3/model-volume/models/bbox/1/ --all
  
  MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
  
  signature_def['serving_default']:
    The given SavedModel SignatureDef contains the following input(s):
      inputs['in'] tensor_info:
          dtype: DT_UINT8
          shape: (-1, -1, -1, 3)
          name: image_tensor:0
    The given SavedModel SignatureDef contains the following output(s):
      outputs['out'] tensor_info:
          dtype: DT_FLOAT
          shape: unknown_rank
          name: detection_boxes:0
    Method name is: tensorflow/serving/predict
  </code></pre>
  
  <p>I <em>think</em> the shape info here is telling me that the model can handle images of any dimensions?</p>
  
  <p>The input layer looks like this in Tensorboard:
  <a href="https://i.stack.imgur.com/fM5Xr.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/fM5Xr.jpg" alt="enter image description here"></a></p>
  
  <p>But how do I convert this SignatureDefinition to a valid JSON request?<br>
  I'm assuming that I'm supposed to use the predict API ...</p>
  
  <p>and <a href="https://www.tensorflow.org/tfx/serving/api_rest" rel="noreferrer">Google's doc</a> says ...</p>
  
  <blockquote>
    <p>URL</p>
    
    <p>POST
        <a href="http://host:port/v1/models/" rel="noreferrer">http://host:port/v1/models/</a>${MODEL_NAME}[/versions/${MODEL_VERSION}]:predict</p>
    
    <p>/versions/${MODEL_VERSION} is optional. If omitted the latest version
    is used.</p>
    
    <p>Request format<br>
    The request body for predict API must be JSON object
    formatted as follows:</p>
  
  <pre><code>{
    // (Optional) Serving signature to use.
    // If unspecifed default serving signature is used.
    "signature_name": &lt;string&gt;,  
    // Input Tensors in row ("instances") or columnar ("inputs") format.
    // A request can have either of them but NOT both.
    "instances": &lt;value&gt;|&lt;(nested)list&gt;|&lt;list-of-objects&gt;
    "inputs": &lt;value&gt;|&lt;(nested)list&gt;|&lt;object&gt;
  }
  </code></pre>
    
    <p>Encoding binary values JSON uses UTF-8 encoding. If you have input
    feature or tensor values that need to be binary (like image bytes),
    you must Base64 encode the data and encapsulate it in a JSON object
    having b64 as the key as follows:</p>
  
  <pre><code>{ "b64": "base64 encoded string" }
  </code></pre>
    
    <p>You can specify this object as a value for an input feature or tensor.
    The same format is used to encode output response as well.</p>
    
    <p>A classification request with image (binary data) and caption features
    is shown below:</p>
  </blockquote>
  
  <pre><code>{   "signature_name": "classify_objects",   "examples": [
      {
        "image": { "b64": "aW1hZ2UgYnl0ZXM=" },
        "caption": "seaside"
      },
      {
        "image": { "b64": "YXdlc29tZSBpbWFnZSBieXRlcw==" },
        "caption": "mountains"
      }   ] }
  </code></pre>
  
  <p>Uncertainties include:</p>
  
  <ul>
  <li>should I use "instances" in my JSON </li>
  <li>should I base64 encode a JPG or PNG or something else? </li>
  <li>Should the image be of a particular
  width and height?</li>
  </ul>
  
  <p>In <a href="https://medium.com/@tmlabonte/serving-image-based-deep-learning-models-with-tensorflow-servings-restful-api-d365c16a7dc4" rel="noreferrer">Serving Image-Based Deep Learning Models with TensorFlow-Serving’s RESTful API</a> this format is suggested:</p>
  
  <pre><code>{
    "instances": [
                    {"b64": "iVBORw"},
                    {"b64": "pT4rmN"},
                    {"b64": "w0KGg2"}
                   ]
  }
  </code></pre>
  
  <p>I used this image:
  <a href="https://tensorflow.org/images/blogs/serving/cat.jpg" rel="noreferrer">https://tensorflow.org/images/blogs/serving/cat.jpg</a></p>
  
  <p>and base64 encoded it like so:</p>
  
  <pre><code>  # Download the image
    dl_request = requests.get(IMAGE_URL, stream=True)
    dl_request.raise_for_status()
  
    # Compose a JSON Predict request (send JPEG image in base64).
    jpeg_bytes = base64.b64encode(dl_request.content).decode('utf-8')
    predict_request = '{"instances" : [{"b64": "%s"}]}' % jpeg_bytes
  </code></pre>
  
  <p>But when I use curl to POST the base64 encoded image like so:</p>
  
  <pre><code>{"instances" : [{"b64": "/9j/4AAQSkZJRgABAQAASABIAAD/4QBYRXhpZgAATU0AKgAA
  ...
  KACiiigAooooAKKKKACiiigAooooA//Z"}]}
  </code></pre>
  
  <p>I get a response like this:</p>
  
  <pre><code>&gt;./test_local_tfs.sh 
  HEADER=|Content-Type:application/json;charset=UTF-8|
     URL=|http://127.0.0.1:8501/v1/models/saved_model/versions/1:predict|
  *   Trying 127.0.0.1...
  * TCP_NODELAY set
  * Connected to 127.0.0.1 (127.0.0.1) port 8501 (#0)
  &gt; POST /v1/models/saved_model/versions/1:predict HTTP/1.1
  &gt; Host: 127.0.0.1:8501
  &gt; User-Agent: curl/7.54.0
  &gt; Accept: */*
  &gt; Content-Type:application/json;charset=UTF-8
  &gt; Content-Length: 85033
  &gt; Expect: 100-continue
  &gt; 
  &lt; HTTP/1.1 100 Continue
  * We are completely uploaded and fine
  &lt; HTTP/1.1 400 Bad Request
  &lt; Content-Type: application/json
  &lt; Date: Tue, 17 Sep 2019 10:47:18 GMT
  &lt; Content-Length: 85175
  &lt; 
  { "error": "Failed to process element: 0 of \'instances\' list. Error: Invalid argument: JSON Value: {\n    \"b64\": \"/9j/4AAQSkZJRgABAQAAS
  ...
  ooooA//Z\"\n} Type: Object is not of expected type: uint8" }
  </code></pre>
  
  <p>I've tried converting a local version of the same file to base64 like so (confirming that the dtype is uint8) ...</p>
  
  <pre><code>  img = cv2.imread('cat.jpg')   
    print('dtype: ' +  str(img.dtype))                                                                                                                                                                                
    _, buf = cv2.imencode('.jpg', img)
    jpeg_bytes = base64.b64encode(buf).decode('utf-8')
    predict_request = '{"instances" : [{"b64": "%s"}]}' % jpeg_bytes
  </code></pre>
  
  <p>But posting this JSON generates the same error.</p>
  
  <p>However, when the json is formated like so ...</p>
  
  <pre><code>{'instances': [[[[112, 71, 48], [104, 63, 40], [107, 70, 20], [108, 72, 21], [109, 77, 0], [106, 75, 0], [92, 66, 0], [106, 80, 0], [101, 80, 0], [98, 77, 0], [100, 75, 0], [104, 80, 0], [114, 88, 17], [94, 68, 0], [85, 54, 0], [103, 72, 11], [93, 62, 0], [120, 89, 25], [131, 101, 37], [125, 95, 31], [119, 91, 27], [121, 93, 29], [133, 105, 40], [119, 91, 27], [119, 96, 56], [120, 97, 57], [119, 96, 53], [102, 78, 36], [132, 103, 44], [117, 88, 28], [125, 89, 4], [128, 93, 8], [133, 94, 0], [126, 87, 0], [110, 74, 0], [123, 87, 2], [120, 92, 30], [124, 95, 33], [114, 90, 32], 
  ...
  , [43, 24, 33], [30, 17, 36], [24, 11, 30], [29, 20, 38], [37, 28, 46]]]]}
  </code></pre>
  
  <p>... it works.
  The problem is this json file is >11 MB in size. </p>
  
  <p>How do I make the base64 encoded version of the json work? </p>
  
  <p><strong>UPDATE: It seems that we have to edit the pretrained model to accept base64 images at the input layer</strong></p>
  
  <p>This article describes how to edit the model ...
  <a href="https://medium.com/@tmlabonte/serving-image-based-deep-learning-models-with-tensorflow-servings-restful-api-d365c16a7dc4" rel="noreferrer">Medium: Serving Image-Based Deep Learning Models with TensorFlow-Serving’s RESTful API</a>
  ... unfortunately, it assumes that we have access to the code which generated the model.</p>
  
  <p>user260826's solution provides a work-around using an estimator but it assumes the model is a keras model. Not true in this case.</p>
  
  <p>Is there a generic method to make a model ready for TensorFlow Serving REST interface with a base64 encoded image that works with any of the TensorFlow model formats?</p>