{"action":"created","issue":{"url":"https://api.github.com/repos/ConsenSys/mythril/issues/97","repository_url":"https://api.github.com/repos/ConsenSys/mythril","labels_url":"https://api.github.com/repos/ConsenSys/mythril/issues/97/labels{/name}","comments_url":"https://api.github.com/repos/ConsenSys/mythril/issues/97/comments","events_url":"https://api.github.com/repos/ConsenSys/mythril/issues/97/events","html_url":"https://github.com/ConsenSys/mythril/pull/97","id":309422816,"number":97,"title":"fix #90: try speeding up ipc reading","user":{"login":"freewind","id":333781,"avatar_url":"https://avatars1.githubusercontent.com/u/333781?v=4","gravatar_id":"","url":"https://api.github.com/users/freewind","html_url":"https://github.com/freewind","followers_url":"https://api.github.com/users/freewind/followers","following_url":"https://api.github.com/users/freewind/following{/other_user}","gists_url":"https://api.github.com/users/freewind/gists{/gist_id}","starred_url":"https://api.github.com/users/freewind/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/freewind/subscriptions","organizations_url":"https://api.github.com/users/freewind/orgs","repos_url":"https://api.github.com/users/freewind/repos","events_url":"https://api.github.com/users/freewind/events{/privacy}","received_events_url":"https://api.github.com/users/freewind/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-03-28T15:41:30Z","updated_at":"2018-03-29T15:59:07Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/ConsenSys/mythril/pulls/97","html_url":"https://github.com/ConsenSys/mythril/pull/97","diff_url":"https://github.com/ConsenSys/mythril/pull/97.diff","patch_url":"https://github.com/ConsenSys/mythril/pull/97.patch"},"body":"I see in this issue: <https://github.com/ConsenSys/mythril/issues/90>, current myth is quite slow to read data from ipc socket after removing web.py.\r\n\r\nCurrent implementation of reading data from socket, has 2 issues with performance I think:\r\n\r\n1. it connects the ipc socket every time, and the operation is expensive\r\n2. it doesn't know if the response is complete, so it will still call `socket.recv` until the time is out, and the timeout is at least 0.2 second\r\n\r\nThe implementation is inspired from web3.py.\r\n\r\nWhen I run the code (by command `myth --init-db`), it show me about 1 second for each IPC call, which is too slow (if you add `print(blockNumber)` in `contractstorage.py#initialize`)\r\n\r\nIn this PR, I tried 2 improvements:\r\n\r\n1. reuse the socket all the time, across ipc calls, to reduce the connection cost. And the socket will be re-created only if we can't read data from it (raised `socket.timeout`)\r\n2. when we got data from `socket.recv`, we check it immediately if we have received a valid JSON-format string. We return it immediately if it's valid JSON, otherwise make another `socket.recv`\r\n\r\nAfter these changes, the `--init-db` command is much faster, here is my comparison:\r\n\r\n```\r\nIPC: 10000 blocks cost 108.00932908058167 seconds\r\nRPC: 10000 blocks cost 124.78162384033203 seconds\r\n```\r\nIPC is a little bit faster than RPC.\r\n\r\nAlso note the timeout setting of socket is actually not affect the speed, since we almost always got correct response in time.\r\n\r\nYou can see the code changes from this PR, and which is base on an old commit(the one before the pool version), and to see if the new speed is good enough. Since I'm not a python expert, I'm not sure if there is any issues in the new code.\r\n\r\nAnother question: if the speed is good enough, do we still need to support level-db: <https://github.com/ConsenSys/mythril/issues/15>"},"comment":{"url":"https://api.github.com/repos/ConsenSys/mythril/issues/comments/377283398","html_url":"https://github.com/ConsenSys/mythril/pull/97#issuecomment-377283398","issue_url":"https://api.github.com/repos/ConsenSys/mythril/issues/97","id":377283398,"user":{"login":"b-mueller","id":5194767,"avatar_url":"https://avatars1.githubusercontent.com/u/5194767?v=4","gravatar_id":"","url":"https://api.github.com/users/b-mueller","html_url":"https://github.com/b-mueller","followers_url":"https://api.github.com/users/b-mueller/followers","following_url":"https://api.github.com/users/b-mueller/following{/other_user}","gists_url":"https://api.github.com/users/b-mueller/gists{/gist_id}","starred_url":"https://api.github.com/users/b-mueller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b-mueller/subscriptions","organizations_url":"https://api.github.com/users/b-mueller/orgs","repos_url":"https://api.github.com/users/b-mueller/repos","events_url":"https://api.github.com/users/b-mueller/events{/privacy}","received_events_url":"https://api.github.com/users/b-mueller/received_events","type":"User","site_admin":false},"created_at":"2018-03-29T15:59:06Z","updated_at":"2018-03-29T15:59:06Z","author_association":"MEMBER","body":"Cool, thx."}}
{"actor":{"display_login":"b-mueller"}}