title: AWS IoT Python Device SDK Shadow Update Timeouts
body: <p>I have been developing AWS IoT device shadow updates for our devices for some time.  We are using the AWS IoT Python device SDK located at <a href="https://github.com/aws/aws-iot-device-sdk-python" rel="nofollow noreferrer">https://github.com/aws/aws-iot-device-sdk-python</a> on an ARM distribution of Linux (made with Yocto/Bitbake) and Python 2.7.3 (which is the only version available provided by our hardware vendor for the Yocto we use for our ARM board).</p>
  
  <p>I downloaded the sample shadow update code from the AWS repo, modified it to suite our needs, and put it on our device build.  It works, most of the time, I would say, but has a failure rate that I am still uncomfortable with.  Note, in this code, I added the "OfflinePublishQueueing" setting to handle when it takes a while for the device to connect.  This was recommended by one site to prevent "PublishQueueDisabled" errors that we were getting.  Here's the initialization code.  I will add that I know that the certificates we are using are good, otherwise there would be no success ever.</p>
  
  <pre><code>self.AWSIoTMQTTShadowClient = AWSIoTMQTTShadowClient("InCoIoTShadowUpdate")
  self.AWSIoTMQTTShadowClient.configureEndpoint(endpoint, 8883)
  self.AWSIoTMQTTShadowClient.configureCredentials(rootCAPath, privateKeyFile, 
  jointCertificateFile)
  # AWSIoTMQTTShadowClient configuration
  self.AWSIoTMQTTShadowClient.configureAutoReconnectBackoffTime(1, 32, 20)
  self.AWSIoTMQTTShadowClient.configureConnectDisconnectTimeout(10)  # 10 sec
  self.AWSIoTMQTTShadowClient.configureMQTTOperationTimeout(15)  # 15 sec
  MQTTClient = self.AWSIoTMQTTShadowClient.getMQTTConnection()
  MQTTClient.configureOfflinePublishQueueing(5, DROP_OLDEST)
  </code></pre>
  
  <p>And later on, the code to update the shadow, noting that the ThingId is set elsewhere from local device config.</p>
  
  <pre><code>def ConnectAndUpdate(self):
  
      deviceState = InovaIoTDeviceState()
      log = logging.getLogger("InovaIoTDeviceClient:connectAndUpdate")
  
      # Connect and subscribe to AWS IoT
      try:
          self.AWSIoTMQTTShadowClient.connect()
      except (connectError, connectTimeoutException):
          log.error("Error connecting to AWS IoT service")
          return False
  
      # Create a deviceShadow with persistent subscription
      updateBot = self.AWSIoTMQTTShadowClient.createShadowHandlerWithName(self.ThingId, True)
      JSONPayload = deviceState.GetDeviceShadowDocument()
  
      try:
          updateBot.shadowUpdate(JSONPayload, self.ShadowUpdateCallback, 15)
      # This is bad if these errors are thrown, probably either an initial device registration failure
      # or global problem with Inova AWS IoT console configuration or lambda function
      except (publishError, subscribeError):
  
          log.error('Publish or subscribe error..')
          return False
  
      except (publishTimeoutException, subscribeTimeoutException):
  
          # It is possible we are here due to a temporary snafu in AWS
          log.error('Publish or subscribe timeout..')
          return False
  
      except publishQueueDisabledException:
  
          # From time to time, AWS will randomly disconnect.
          log.error('Publish Queue disabled..')
          return False
  
      while not self.ResponseReceived:
          time.sleep(1)
  
      # reset state
      self.ResponseReceived = False
  
      try:
          self.AWSIoTMQTTShadowClient.disconnect()
      except (disconnectError, disconnectTimeoutException):
          log.error('Error attempting to disconnect')
  
      return self.UpdateSuccess
  </code></pre>
  
  <p>and finally, the callback code</p>
  
  <pre><code># Custom MQTT message callback
  def ShadowUpdateCallback(self, payload, responseStatus, token):
      log = logging.getLogger("InovaIoTDeviceClient:shadowUpdateCallback")
      if responseStatus == "timeout":
          log.info("Shadow update timeout")
          self.UpdateSuccess = False
      elif responseStatus == "accepted":
          log.info("Shadow update successful")
          self.successive_errors = 0
          self.UpdateSuccess = True
      elif responseStatus == "rejected":
          log.info("Rejected shadow update")
          self.UpdateSuccess = False
  
      self.ResponseReceived = True
  </code></pre>
  
  <p>Here are the various errors we get, which are relatively common. I would say that the success rate of the shadow update is only around 60%.  Here's a timeout reported from AWS:</p>
  
  <pre><code>2017-09-06 11:15:13: (INFO:AWSIoTPythonSDK.core.shadow.deviceShadow) 
  Subscribed to update accepted/rejected topics for deviceShadow: 
  qqpba4fgsfazl2zfgqq8zkavj (Line:372)
  2017-09-06 11:15:13: (INFO:AWSIoTPythonSDK.core.protocol.mqttCore) Offline 
  publish request detected. (Line:343)
  2017-09-06 11:15:13: (INFO:AWSIoTPythonSDK.core.protocol.mqttCore) Try 
  queueing up this request... (Line:347)
  2017-09-06 11:15:28: (INFO:AWSIoTPythonSDK.core.shadow.deviceShadow) Shadow 
  request with token: InCoIoTShadowUpdate_qqpba4fgsfazl2zfgqq8zkavj_0_acbxa 
  has timed out. (Line:202)
  2017-09-06 11:15:28: (INFO:InCoIoTDeviceClient:shadowUpdateCallback) Shadow 
  update timeout (Line:188)
  </code></pre>
  
  <p>Here's a timeout error from the actual shadow update call</p>
  
  <pre><code>2017-09-06 12:40:11: (INFO:AWSIoTPythonSDK.core.protocol.mqttCore) Connected 
  to AWS IoT. (Line:302)
  2017-09-06 12:40:27: (ERROR:AWSIoTPythonSDK.core.protocol.mqttCore) No 
  feedback detected for subscribe request 1. Timeout and failed. (Line:413)
  2017-09-06 12:40:27: (ERROR:InCoIoTDeviceClient:connectAndUpdate) Publish 
  or subscribe timeout.. (Line:147)
  2017-09-06 12:40:27: (INFO:cycle) Unsuccessful shadow update... (Line:173)
  </code></pre>
  
  <p>So those are the timeouts, which are essentially all the errors.  As an additional wrinkle, we wiresharked our device's connection and publish attempts, not to see the content (since it is TLS 1.2 encrypted) but to see the connection behavior.  We noted that the behavior we see from the wireshark output is that the AWS endpoint is served by at least 8 different IP addresses.  In the timeout cases, we always see a connection happen <em>somewhere</em>.  In the case that we subscribe and publish to the shadow update topic, but receive a timeout, it only connects to one of the addresses.  In the subscribe or publish timeout, it tries three different addresses.</p>
  
  <p>It may be that all of this is on AWS and the performance at their end, but I wondered if anyone had seen this and been able to work around.  Upping the timeout values doesn't seem to help.</p>