title: Why would I get a Forbidden message from AWS API Gateway, even though things are working internally?
body: <p>I have AWS API gateway setup for a public endpoint with no auth. It connects to a websocket that triggers a Lambda.</p>
  
  <p>I was creating connections with Python's <code>websocket-client</code> lib at <a href="https://pypi.org/project/websocket_client/" rel="noreferrer">https://pypi.org/project/websocket_client/</a>.</p>
  
  <p>I noticed that connections would fail ~10% of the time, and get worse as I increased load. I can't find anywhere that would be throttling me seeing as my general API Gateway settings say <code>Your current account level throttling rate is 10000 requests per second with a burst of 5000 requests.</code>. That’s beside the point that just 2-3 requests per second would trigger issue fairly often. </p>
  
  <p>Meanwhile the failure response would be like <code>{u'message': u'Forbidden', u'connectionId': u'Z2Jp-dR5vHcCJkg=', u'requestId': u'Z2JqAEJRvHcFzvg='}</code></p>
  
  <p>I went into my CloudWatch log insights and searched for the connection ID and request ID. The log group for the API gateway would find no results with either ID. Yet a search on my Lambda that fires on websocket connect, would have a log with that connection ID. The log showed everything running as expected on our side. The lambda simply runs a MySQL query that fires. </p>
  
  <p>Why would I get a response of forbidden, despite the lambda working as expected?</p>
  
  <p>The existing question over at <a href="https://stackoverflow.com/questions/40988051/getting-message-forbidden-reply-from-aws-api-gateway">getting message: forbidden reply from AWS API gateway</a>, seems to address if it's ALWAYS returning forbidden for some private endpoints. Nothing lined up with my use case.</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>I think this may be related to <code>locust.io</code>, or python, which I'm using to connect every second. I installed <a href="https://www.npmjs.com/package/wscat" rel="noreferrer">https://www.npmjs.com/package/wscat</a> on my machine and am connecting and closing as fast as possible repeatedly. I am not getting a <code>Forbidden</code> message. It's just extra confusing since I'm not sure how the way I connect would randomly spit back a <code>Forbidden</code> message <em>some of the time</em>.</p>
  
  <pre class="lang-py prettyprint-override"><code>class SocketClient(object):
      def __init__(self, host):
          self.host = host
          self.session_id = uuid4().hex
  
      def connect(self):
          self.ws = websocket.WebSocket()
          self.ws.settimeout(10)
          self.ws.connect(self.host)
  
          events.quitting += self.on_close
  
          data = self.attach_session({})
          return data
  
      def attach_session(self, payload):
          message_id = uuid4().hex
          start_time = time.time()
          e = None
          try:
              print("Sending payload {}".format(payload))
              data = self.send_with_response(payload)
              assert data['mykey']
  
          except AssertionError as exp:
              e = exp
          except Exception as exp:
              e = exp
              self.ws.close()
              self.connect()
          elapsed = int((time.time() - start_time) * 1000)
          if e:
              events.request_failure.fire(request_type='sockjs', name='send',
                                          response_time=elapsed, exception=e)
          else:
              events.request_success.fire(request_type='sockjs', name='send',
                                          response_time=elapsed,
                                          response_length=0)
          return data
  
      def send_with_response(self, payload):
          json_data = json.dumps(payload)
  
          g = gevent.spawn(self.ws.send, json_data)
          g.get(block=True, timeout=2)
          g = gevent.spawn(self.ws.recv)
          result = g.get(block=True, timeout=10)
  
          json_data = json.loads(result)
          return json_data
      def on_close(self):
          self.ws.close()
  
  class ActionsTaskSet(TaskSet):
      @task
      def streams(self):
          response = self.client.connect()
          logger.info("Connect Response: {}".format(response))
  
  class WSUser(Locust):
      task_set = ActionsTaskSet
      min_wait = 1000
      max_wait = 3000
  
      def __init__(self, *args, **kwargs):
          super(WSUser, self).__init__(*args, **kwargs)
          self.client = SocketClient('wss://mydomain.amazonaws.com/endpoint')
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/bY5nh.png" rel="noreferrer"><img src="https://i.stack.imgur.com/bY5nh.png" alt="enter image description here"></a></p>
  
  <p><strong>Update 2</strong></p>
  
  <p>I have enabled access logs, the one type of log that wasn't there before. I can now see that my lambdas are always getting a 200 with no issue. The 403 is coming from some <code>MESSAGE</code> <code>eventType</code> that doesn't hit an actual <code>routeKey</code>. Not sure where it comes from, but pretty sure finding that answer will solve this.</p>
  
  <p>I was also able to confirm there are no ENI issues.</p>
  
  <p><a href="https://i.stack.imgur.com/FO4IK.png" rel="noreferrer"><img src="https://i.stack.imgur.com/FO4IK.png" alt="enter image description here"></a></p>