{"action":"created","issue":{"url":"https://api.github.com/repos/openblockchain/obc-peer/issues/699","repository_url":"https://api.github.com/repos/openblockchain/obc-peer","labels_url":"https://api.github.com/repos/openblockchain/obc-peer/issues/699/labels{/name}","comments_url":"https://api.github.com/repos/openblockchain/obc-peer/issues/699/comments","events_url":"https://api.github.com/repos/openblockchain/obc-peer/issues/699/events","html_url":"https://github.com/openblockchain/obc-peer/issues/699","id":134443267,"number":699,"title":"Classic PBFT hangs under stress","user":{"login":"bcbrock","id":11968347,"avatar_url":"https://avatars.githubusercontent.com/u/11968347?v=3","gravatar_id":"","url":"https://api.github.com/users/bcbrock","html_url":"https://github.com/bcbrock","followers_url":"https://api.github.com/users/bcbrock/followers","following_url":"https://api.github.com/users/bcbrock/following{/other_user}","gists_url":"https://api.github.com/users/bcbrock/gists{/gist_id}","starred_url":"https://api.github.com/users/bcbrock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcbrock/subscriptions","organizations_url":"https://api.github.com/users/bcbrock/orgs","repos_url":"https://api.github.com/users/bcbrock/repos","events_url":"https://api.github.com/users/bcbrock/events{/privacy}","received_events_url":"https://api.github.com/users/bcbrock/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/openblockchain/obc-peer/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/openblockchain/obc-peer/labels/consensus","name":"consensus","color":"fbca04"},{"url":"https://api.github.com/repos/openblockchain/obc-peer/labels/sev2","name":"sev2","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"corecode","id":177979,"avatar_url":"https://avatars.githubusercontent.com/u/177979?v=3","gravatar_id":"","url":"https://api.github.com/users/corecode","html_url":"https://github.com/corecode","followers_url":"https://api.github.com/users/corecode/followers","following_url":"https://api.github.com/users/corecode/following{/other_user}","gists_url":"https://api.github.com/users/corecode/gists{/gist_id}","starred_url":"https://api.github.com/users/corecode/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/corecode/subscriptions","organizations_url":"https://api.github.com/users/corecode/orgs","repos_url":"https://api.github.com/users/corecode/repos","events_url":"https://api.github.com/users/corecode/events{/privacy}","received_events_url":"https://api.github.com/users/corecode/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/openblockchain/obc-peer/milestones/7","html_url":"https://github.com/openblockchain/obc-peer/milestones/Sprint%202","labels_url":"https://api.github.com/repos/openblockchain/obc-peer/milestones/7/labels","id":1646514,"number":7,"title":"Sprint 2","description":"","creator":{"login":"binhn","id":3342057,"avatar_url":"https://avatars.githubusercontent.com/u/3342057?v=3","gravatar_id":"","url":"https://api.github.com/users/binhn","html_url":"https://github.com/binhn","followers_url":"https://api.github.com/users/binhn/followers","following_url":"https://api.github.com/users/binhn/following{/other_user}","gists_url":"https://api.github.com/users/binhn/gists{/gist_id}","starred_url":"https://api.github.com/users/binhn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/binhn/subscriptions","organizations_url":"https://api.github.com/users/binhn/orgs","repos_url":"https://api.github.com/users/binhn/repos","events_url":"https://api.github.com/users/binhn/events{/privacy}","received_events_url":"https://api.github.com/users/binhn/received_events","type":"User","site_admin":false},"open_issues":8,"closed_issues":0,"state":"open","created_at":"2016-03-15T19:50:11Z","updated_at":"2016-03-21T15:06:25Z","due_on":"2016-04-01T04:00:00Z","closed_at":null},"comments":20,"created_at":"2016-02-18T00:26:08Z","updated_at":"2016-03-25T16:02:05Z","closed_at":null,"body":"Observed in\r\n\r\n```\r\ncommit 7abd041f5b9e5ce346d1608f26b6880f6e235a77\r\nMerge: 5077256 4fd8ae5\r\nAuthor: Binh Q Nguyen <binhn@us.ibm.com>\r\nDate:   Wed Feb 17 13:29:30 2016 -0500\r\n\r\n    Merge pull request #682 from adecaro/crash_issue669\r\n    \r\n    Fix for issue #669\r\n```\r\n\r\nSame setup and irritation as in #687. I have a 4-peer docker-compose network running classic PBFT using all default parameters. I send 1000 transactions as fast as possible. The good news is that the network usually does not crash (#696 is rare) and eventually arrives at the correct answer. The issue is that it takes 8 or 9 minutes after I send in the last transaction for the database to correctly record the last transaction. \r\n\r\nYou can see in this trace that the network is only processing 20 or 40 transactions every 30 seconds after the last transaction has been sent over the REST API. Note that security is *not* enabled in this network (#684).\r\n\r\n```\r\n[bcbrock@arlab178]$ driveEx02 172.17.0.1 -rest -n 1000 -wait 30 -reCheck\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -638, expected -1000 B: Got 638, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -658, expected -1000 B: Got 658, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -678, expected -1000 B: Got 678, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -698, expected -1000 B: Got 698, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -738, expected -1000 B: Got 738, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -758, expected -1000 B: Got 758, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -778, expected -1000 B: Got 778, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -798, expected -1000 B: Got 798, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -838, expected -1000 B: Got 838, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -858, expected -1000 B: Got 858, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -878, expected -1000 B: Got 878, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -898, expected -1000 B: Got 898, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -938, expected -1000 B: Got 938, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -958, expected -1000 B: Got 958, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -978, expected -1000 B: Got 978, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\n A: Got -998, expected -1000 B: Got 998, expected 1000\r\nWaiting 30 seconds for chain state to stabilize\r\nRuntime: 4.519867 seconds\r\nRate   : 221.2 transactions/second\r\n```\r\nNote 221 TX/sec is only for the initial send of 1000 TX.\r\n\r\nLooking at `top`, the peers/chaincode are practically idle during this 8-9 minute period. Thinking this might be due to I/O I took a `dstat` visualization; I/O is nominal during this period.\r\n![screen shot 2016-02-17 at 5 59 11 pm](https://cloud.githubusercontent.com/assets/11968347/13129475/d71efa3c-d5a2-11e5-96ea-a3bc28c40c3d.png)\r\n\r\nHere are the full DEBUG -level logs from the 4 peers.\r\n[logs.tgz.zip](https://github.com/openblockchain/obc-peer/files/135403/logs.tgz.zip)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/openblockchain/obc-peer/issues/comments/201344299","html_url":"https://github.com/openblockchain/obc-peer/issues/699#issuecomment-201344299","issue_url":"https://api.github.com/repos/openblockchain/obc-peer/issues/699","id":201344299,"user":{"login":"jeffgarratt","id":4991920,"avatar_url":"https://avatars.githubusercontent.com/u/4991920?v=3","gravatar_id":"","url":"https://api.github.com/users/jeffgarratt","html_url":"https://github.com/jeffgarratt","followers_url":"https://api.github.com/users/jeffgarratt/followers","following_url":"https://api.github.com/users/jeffgarratt/following{/other_user}","gists_url":"https://api.github.com/users/jeffgarratt/gists{/gist_id}","starred_url":"https://api.github.com/users/jeffgarratt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeffgarratt/subscriptions","organizations_url":"https://api.github.com/users/jeffgarratt/orgs","repos_url":"https://api.github.com/users/jeffgarratt/repos","events_url":"https://api.github.com/users/jeffgarratt/events{/privacy}","received_events_url":"https://api.github.com/users/jeffgarratt/received_events","type":"User","site_admin":false},"created_at":"2016-03-25T16:02:05Z","updated_at":"2016-03-25T16:02:05Z","body":"@jyellick there is a modification that needs to be made wrt to chat termination.  I can take a look at this shortly."}}