{"action":"created","issue":{"url":"https://api.github.com/repos/CityOfZion/neo-python/issues/723","repository_url":"https://api.github.com/repos/CityOfZion/neo-python","labels_url":"https://api.github.com/repos/CityOfZion/neo-python/issues/723/labels{/name}","comments_url":"https://api.github.com/repos/CityOfZion/neo-python/issues/723/comments","events_url":"https://api.github.com/repos/CityOfZion/neo-python/issues/723/events","html_url":"https://github.com/CityOfZion/neo-python/pull/723","id":382615116,"node_id":"MDExOlB1bGxSZXF1ZXN0MjMyMjc4NjM3","number":723,"title":"Network stability improvements","user":{"login":"ixje","id":6625537,"node_id":"MDQ6VXNlcjY2MjU1Mzc=","avatar_url":"https://avatars0.githubusercontent.com/u/6625537?v=4","gravatar_id":"","url":"https://api.github.com/users/ixje","html_url":"https://github.com/ixje","followers_url":"https://api.github.com/users/ixje/followers","following_url":"https://api.github.com/users/ixje/following{/other_user}","gists_url":"https://api.github.com/users/ixje/gists{/gist_id}","starred_url":"https://api.github.com/users/ixje/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ixje/subscriptions","organizations_url":"https://api.github.com/users/ixje/orgs","repos_url":"https://api.github.com/users/ixje/repos","events_url":"https://api.github.com/users/ixje/events{/privacy}","received_events_url":"https://api.github.com/users/ixje/received_events","type":"User","site_admin":false},"labels":[{"id":648322778,"node_id":"MDU6TGFiZWw2NDgzMjI3Nzg=","url":"https://api.github.com/repos/CityOfZion/neo-python/labels/enhancement","name":"enhancement","color":"84b6eb","default":true},{"id":648322779,"node_id":"MDU6TGFiZWw2NDgzMjI3Nzk=","url":"https://api.github.com/repos/CityOfZion/neo-python/labels/help%20wanted","name":"help wanted","color":"128A0C","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2018-11-20T11:15:09Z","updated_at":"2018-11-23T01:09:44Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/CityOfZion/neo-python/pulls/723","html_url":"https://github.com/CityOfZion/neo-python/pull/723","diff_url":"https://github.com/CityOfZion/neo-python/pull/723.diff","patch_url":"https://github.com/CityOfZion/neo-python/pull/723.patch"},"body":"**What current issue(s) does this address, or what feature is it adding?**\r\nAddresses #325 and #492 \r\nThere's a couple of issues with the current networking code that I'll only describe at a high level:\r\n- nodes get stuck/stop syncing at some point\r\n- debugging output is insufficient to investigate the reason of poor stability\r\n\r\n**How did you solve this problem?**\r\n- Added health checking of peers based on block/header requests vs actual responses within a given time threshold. Bad peers are disconnected from and moved into the `DEAD_ADDRS` list.\r\n- Added monitoring of connected peers and if we reach 0 we reset and queue up peers. Address selection is sorted from \"never connected to\" to \"recently disconnected from\" so we always try to pick a fresh one.\r\n- Added data throttling when our request speed exceeds a certain response speed threshold\r\n- Move some threading calls into their proper threads (mainly into the reactor) to avoid unexpected behaviour.\r\n- Improved the nodes output to give a bit more information on the clients and addresses. i.e.\r\n```\r\nneo> nodes\r\nTotal Connected: 2\r\nPeer 0  /NEO:2.8.0/ -  18.222.168.189:20333 - IO   0.323453 MB in /   0.009913 MB out\r\nPeer 1  /Neo:2.9.0/ -    47.52.168.73:20333 - IO   0.131664 MB in /   0.006687 MB out\r\n\r\nKNOWN_ADDRS: 174\r\nDEAD_ADDRS: 6\r\nQUEUED_ADDRS: 3\r\n```\r\n\r\n- Added a lot of debug output and nicely format them to make analysis easier in the future.\r\n- Added 2 \"hidden\" commands `disconnect_node <id>` and `handles` for debugging purposes. They do not show up in the `help` output of the prompt, but can be used.\r\n- Fixed an issue where block persisting would pass, but the notificationDB fails emitting the event which caused the BlockPersist task to cancel and the node gets 'stuck'. \r\n\r\n**How did you make sure your solution works?**\r\nFor the past 2 weeks I've been monitoring and adjusting this daily. \r\n\r\n**Are there any special changes in the code that we should be aware of?**\r\n1) @localhuman I'd say this one is specifically for you because it's notificationDB related which I think you work a lot with? Block `1231519` has one 'malformed' transaction that throws an exception in the NotificationDB when serialising a `Neo.Contract.Create` event.   \r\n\r\n2) test-cases have not been finalised. I've skipped 2 because I wanted to get this PR out so people can start testing if they want.\r\n\r\n**Known issues**\r\n1) minor - `DEAD_ADDRS` can contain duplicates. This has little effect on usage or stability.\r\n2) minor - about 1 in 200 client disconnects can have a connection that doesn't fully close (meaning the TCP socket stays open in `LISTENING` state shown by `netstat`). I'm trying to resolve this with the `Twisted` package authors. This can be a problem if it was the last node connected while having no fresh addresses to pick from. It then prevents reset. I've seen this happen twice over the course of 2 weeks of testing.\r\n3) major - header and block persistence blocks the main thread/health monitoring threads leading to nodes timing out if processing takes too long. This is something I want to address in a separate PR because it will require extensive testing again given that I don't know what the effect will be if I simply move it into its own thread. The impact can be that we switch out healthy nodes because they time-out or exceed our response threshold. Note that this will not get us in a stuck state, there's always a check that will reset the node if all nodes gone bad.\r\n4) don't enable `AcceptIncomingPeers` in your config file. It suffers from the same as point 3. and then some. It's on the todo list after addressing 3.\r\n\r\n**Please check the following, if applicable:**\r\n\r\n- [ ] Did you add any tests?\r\n- [X] Did you run `make lint`?\r\n- [X] Did you run `make test`?\r\n- [X] Are you making a PR to a feature branch or development rather than master?\r\n- [X] Did you add an entry to `CHANGELOG.rst`? (if not, please do)\r\n"},"comment":{"url":"https://api.github.com/repos/CityOfZion/neo-python/issues/comments/441139842","html_url":"https://github.com/CityOfZion/neo-python/pull/723#issuecomment-441139842","issue_url":"https://api.github.com/repos/CityOfZion/neo-python/issues/723","id":441139842,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MTEzOTg0Mg==","user":{"login":"jseagrave21","id":40873301,"node_id":"MDQ6VXNlcjQwODczMzAx","avatar_url":"https://avatars0.githubusercontent.com/u/40873301?v=4","gravatar_id":"","url":"https://api.github.com/users/jseagrave21","html_url":"https://github.com/jseagrave21","followers_url":"https://api.github.com/users/jseagrave21/followers","following_url":"https://api.github.com/users/jseagrave21/following{/other_user}","gists_url":"https://api.github.com/users/jseagrave21/gists{/gist_id}","starred_url":"https://api.github.com/users/jseagrave21/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jseagrave21/subscriptions","organizations_url":"https://api.github.com/users/jseagrave21/orgs","repos_url":"https://api.github.com/users/jseagrave21/repos","events_url":"https://api.github.com/users/jseagrave21/events{/privacy}","received_events_url":"https://api.github.com/users/jseagrave21/received_events","type":"User","site_admin":false},"created_at":"2018-11-23T01:09:44Z","updated_at":"2018-11-23T01:09:44Z","author_association":"CONTRIBUTOR","body":"On the current run my node has remained at 0 peers for quite some time. I see peer_check_loop doing its job and restarting NodeLeader, but the node is not connecting any mew peers. However, blockheight is still advancing and I was wondering how that was possible?"}}
{"actor":{"display_login":"jseagrave21"}}