{"action":"created","issue":{"url":"https://api.github.com/repos/eclipse/paho.mqtt.python/issues/114","repository_url":"https://api.github.com/repos/eclipse/paho.mqtt.python","labels_url":"https://api.github.com/repos/eclipse/paho.mqtt.python/issues/114/labels{/name}","comments_url":"https://api.github.com/repos/eclipse/paho.mqtt.python/issues/114/comments","events_url":"https://api.github.com/repos/eclipse/paho.mqtt.python/issues/114/events","html_url":"https://github.com/eclipse/paho.mqtt.python/pull/114","id":179423172,"number":114,"title":"[114] Fix large payload over Websockets","user":{"login":"PierreF","id":359833,"avatar_url":"https://avatars.githubusercontent.com/u/359833?v=3","gravatar_id":"","url":"https://api.github.com/users/PierreF","html_url":"https://github.com/PierreF","followers_url":"https://api.github.com/users/PierreF/followers","following_url":"https://api.github.com/users/PierreF/following{/other_user}","gists_url":"https://api.github.com/users/PierreF/gists{/gist_id}","starred_url":"https://api.github.com/users/PierreF/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PierreF/subscriptions","organizations_url":"https://api.github.com/users/PierreF/orgs","repos_url":"https://api.github.com/users/PierreF/repos","events_url":"https://api.github.com/users/PierreF/events{/privacy}","received_events_url":"https://api.github.com/users/PierreF/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-09-27T08:05:06Z","updated_at":"2016-10-18T06:18:52Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/eclipse/paho.mqtt.python/pulls/114","html_url":"https://github.com/eclipse/paho.mqtt.python/pull/114","diff_url":"https://github.com/eclipse/paho.mqtt.python/pull/114.diff","patch_url":"https://github.com/eclipse/paho.mqtt.python/pull/114.patch"},"body":"Related to issue #107, when payload is too big, Websocket connection does not work. Issue is that  [_buffrered_read](https://github.com/eclipse/paho.mqtt.python/blob/33b7c95df3df330ab2ffb7b325e94164db1159fd/src/paho/mqtt/client.py#L2760) assume that socket.recv(wanted_size) will return data of size wanted_size, which is not always true, especially when wanted_size is larged than the TCP window size.\r\n\r\nThis PR will fix this issue, large payload (tested up to 100 Mb) works correctly.\r\n\r\nTo reproduce the issue:\r\n\r\nMosquitto server:\r\n```\r\n$ cat mosquitto.conf\r\nlistener 1884\r\nlistener 1885\r\nprotocol websockets\r\n$ mosquitto -c mosquitto.conf\r\n```\r\n\r\nCreate data to send/recv:\r\n```\r\n$ dd if=/dev/urandom of=data.bin bs=1024 count=10240\r\n```\r\n\r\nStart the client:\r\n```\r\n$ cat client.py\r\nimport paho.mqtt.client\r\n\r\ndef on_connect(client, ud, flags, rc):\r\n    client.subscribe('topic')\r\n\r\ndef on_message(client, ud, message):\r\n    print('Got the message!')\r\n    data = open('data.bin', 'rb').read()\r\n    if message.payload != data:\r\n        print('Message mismatch')\r\n    client.disconnect()\r\n\r\nclient = paho.mqtt.client.Client(transport='websockets')\r\nclient.on_connect = on_connect\r\nclient.on_message = on_message\r\nclient.connect('127.0.0.1', 1885)\r\nclient.loop_forever()\r\n$ python client.py\r\n```\r\n\r\nSend the message:\r\n```\r\n$ mosquitto_pub -p 1884 -f data.bin -t topic\r\n```\r\n\r\n\r\nWithout the PR, the following error occur:\r\n```\r\nTraceback (most recent call last):\r\n  File \"client.py\", line 20, in <module>\r\n    client.loop_forever()\r\n  File \"/home/pierref/dev/ext/github.com/eclipse/paho.mqtt.python/src/paho/mqtt/client.py\", line 1354, in loop_forever\r\n    rc = self.loop(timeout, max_packets)\r\n  File \"/home/pierref/dev/ext/github.com/eclipse/paho.mqtt.python/src/paho/mqtt/client.py\", line 876, in loop\r\n    rc = self.loop_read(max_packets)\r\n  File \"/home/pierref/dev/ext/github.com/eclipse/paho.mqtt.python/src/paho/mqtt/client.py\", line 1159, in loop_read\r\n    rc = self._packet_read()\r\n  File \"/home/pierref/dev/ext/github.com/eclipse/paho.mqtt.python/src/paho/mqtt/client.py\", line 1708, in _packet_read\r\n    data = self._sock.recv(self._in_packet['to_process'])\r\n  File \"/home/pierref/dev/ext/github.com/eclipse/paho.mqtt.python/src/paho/mqtt/client.py\", line 2881, in recv\r\n    return self._recv_impl(length)\r\n  File \"/home/pierref/dev/ext/github.com/eclipse/paho.mqtt.python/src/paho/mqtt/client.py\", line 2825, in _recv_impl\r\n    payload[index] ^= mask_key[index % 4]\r\nIndexError: bytearray index out of range\r\n```\r\n\r\n\r\nA side note, receiving is slow on websocket, but this seems to be due to Mosquitto. Receiving that 10Mb payload took about 20 seconds when using mosquitto as MQTT broker. With mosca it took less than 1 second.\r\nMosa run with:\r\n```\r\n$ mosca -v --port 1884 --http-port 1885\r\n```\r\nIt seems that Mosquitto is sending a packet of 64kb every 100ms. The 100ms is so fixed that is really looks like a sleep(100ms) somewhere"},"comment":{"url":"https://api.github.com/repos/eclipse/paho.mqtt.python/issues/comments/254417071","html_url":"https://github.com/eclipse/paho.mqtt.python/pull/114#issuecomment-254417071","issue_url":"https://api.github.com/repos/eclipse/paho.mqtt.python/issues/114","id":254417071,"user":{"login":"minghuadev","id":5402039,"avatar_url":"https://avatars.githubusercontent.com/u/5402039?v=3","gravatar_id":"","url":"https://api.github.com/users/minghuadev","html_url":"https://github.com/minghuadev","followers_url":"https://api.github.com/users/minghuadev/followers","following_url":"https://api.github.com/users/minghuadev/following{/other_user}","gists_url":"https://api.github.com/users/minghuadev/gists{/gist_id}","starred_url":"https://api.github.com/users/minghuadev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/minghuadev/subscriptions","organizations_url":"https://api.github.com/users/minghuadev/orgs","repos_url":"https://api.github.com/users/minghuadev/repos","events_url":"https://api.github.com/users/minghuadev/events{/privacy}","received_events_url":"https://api.github.com/users/minghuadev/received_events","type":"User","site_admin":false},"created_at":"2016-10-18T06:18:52Z","updated_at":"2016-10-18T06:18:52Z","body":"Hi, Would you try to apply the change suggested by #185 and see how long it takes to transfer the same 10M data? Thanks. "}}
{"actor":{"display_login":"minghuadev"}}