{"action":"created","issue":{"url":"https://api.github.com/repos/python/asyncio/issues/394","repository_url":"https://api.github.com/repos/python/asyncio","labels_url":"https://api.github.com/repos/python/asyncio/issues/394/labels{/name}","comments_url":"https://api.github.com/repos/python/asyncio/issues/394/comments","events_url":"https://api.github.com/repos/python/asyncio/issues/394/events","html_url":"https://github.com/python/asyncio/issues/394","id":169467311,"number":394,"title":"Memory footprint StreamReader.readexactly","user":{"login":"frensjan","id":3985389,"avatar_url":"https://avatars.githubusercontent.com/u/3985389?v=3","gravatar_id":"","url":"https://api.github.com/users/frensjan","html_url":"https://github.com/frensjan","followers_url":"https://api.github.com/users/frensjan/followers","following_url":"https://api.github.com/users/frensjan/following{/other_user}","gists_url":"https://api.github.com/users/frensjan/gists{/gist_id}","starred_url":"https://api.github.com/users/frensjan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/frensjan/subscriptions","organizations_url":"https://api.github.com/users/frensjan/orgs","repos_url":"https://api.github.com/users/frensjan/repos","events_url":"https://api.github.com/users/frensjan/events{/privacy}","received_events_url":"https://api.github.com/users/frensjan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-08-04T20:34:12Z","updated_at":"2016-08-05T05:11:46Z","closed_at":null,"body":"```StreamReader.readexactly``` is a very handy method to implement reading frames ('chunks' of data preceded with the size of the 'chunk') from a stream. E.g.:\r\n```python\r\nframe_len = yield from reader.readexactly(4) # when using 32 bits sizes\r\nframe = yield from reader.readexactly(frame_len)\r\n```\r\n\r\n---\r\n\r\nHowever, it's memory footprint isn't ideal. A list of 'blocks' is kept until all data is read which is then joined with ```b''.join(blocks)```. This causes each 'frame' to reside in memory twice until the ```blocks``` list and the ```bytes``` objects it points to are freed (when leaving the ```readexactly``` method). So it's peak memory usage is ```O(2n)```.\r\n\r\nFor small 'frames' the only issue probably is some copying which could be avoided, but for larger 'frames' this hurts. E.g. when sending large files for processing to a machine which are larger than ```memory / #receivers / 2```.\r\n\r\n---\r\n\r\nA possible solution might be to use a ```bytearray``` to gather the data incrementally. E.g. something along the lines of:\r\n\r\n```python\r\n@asyncio.coroutine\r\ndef readexactly(self, n):\r\n    if self._exception is not None:\r\n        raise self._exception\r\n     \r\n    if not n:\r\n        return b''\r\n     \r\n    if len(self._buffer) >= n and not self._eof:\r\n        data = self._buffer[:n]\r\n        del self._buffer[:n]\r\n        self._maybe_resume_transport()\r\n        return data\r\n     \r\n    data = bytearray(n)\r\n    pos = 0\r\n     \r\n    while n:\r\n        if not self._buffer:\r\n            yield from self._wait_for_data('read')\r\n \r\n        if self._eof or not self._buffer:\r\n            raise asyncio.IncompleteReadError(data[:pos], pos + n)\r\n         \r\n        available = len(self._buffer)\r\n        if available <= n:\r\n            data[pos:pos + available] = self._buffer\r\n            self._buffer.clear()\r\n            n -= available\r\n            pos += available\r\n        else:\r\n            data[pos:pos + n] = self._buffer[:n]\r\n            del self._buffer[:n]\r\n            n = 0\r\n        self._maybe_resume_transport()\r\n     \r\n    return data\r\n```\r\n\r\nThis implementation would have a peak memory footprint of ```O(n)```. Or ```O(n + max(map(len, blocks)))```, not sure what the other constant factors might be, probably lots more lower in the stack.\r\n\r\nNote that the implementation above also does away with some unnecessary copying by duplicating some code from ```StreamReader.read```.\r\nNote as well that the implementation has a short path for ```readexactly``` when the required data is available in the ```_buffer``` of the ```StreamReader```.\r\n\r\nAn obvious issue with the implementation above is that it returns (mutable) ```bytebuffer``` instead of ```bytes``` objects. For my use case I considered this acceptable; I'm not sure if it would be acceptable in asyncio / the standard library. I'm not a aware of a copy-free mechanism to turn a ```bytebuffer``` into an immutable ```bytes```-like object (like [asReadOnlyBuffer](https://docs.oracle.com/javase/8/docs/api/java/nio/ByteBuffer.html#asReadOnlyBuffer--)).\r\n\r\nApart from being more memory friendly in terms of 'volume', because there is less copying, it's also a tad bit faster (around 20% on my machine, see test script here: https://gist.github.com/frensjan/27061b47e1423dd73eeec9f5cad5f4ad\r\n\r\nAn alternative direction could be to add a ```socket.recv_into```-like method which accepts a mutable buffer.\r\n\r\n---\r\n\r\nI hope the issue is clear. Curious to get feedback on relevance for the 'broader developer community' and to the solution. If desirable I'll whip up a PR.\r\n\r\nBest regards,\r\nFrens Jan"},"comment":{"url":"https://api.github.com/repos/python/asyncio/issues/comments/237756762","html_url":"https://github.com/python/asyncio/issues/394#issuecomment-237756762","issue_url":"https://api.github.com/repos/python/asyncio/issues/394","id":237756762,"user":{"login":"1st1","id":239003,"avatar_url":"https://avatars.githubusercontent.com/u/239003?v=3","gravatar_id":"","url":"https://api.github.com/users/1st1","html_url":"https://github.com/1st1","followers_url":"https://api.github.com/users/1st1/followers","following_url":"https://api.github.com/users/1st1/following{/other_user}","gists_url":"https://api.github.com/users/1st1/gists{/gist_id}","starred_url":"https://api.github.com/users/1st1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/1st1/subscriptions","organizations_url":"https://api.github.com/users/1st1/orgs","repos_url":"https://api.github.com/users/1st1/repos","events_url":"https://api.github.com/users/1st1/events{/privacy}","received_events_url":"https://api.github.com/users/1st1/received_events","type":"User","site_admin":false},"created_at":"2016-08-05T05:11:46Z","updated_at":"2016-08-05T05:11:46Z","body":"I think I saw it when I was running this benchmark: https://github.com/MagicStack/uvloop/blob/master/examples/bench/rlserver.py -- stresses `StreamReader.readline` -- it became slower in 3.5.2."}}
{"actor":{"display_login":"1st1"}}