{"action":"created","issue":{"url":"https://api.github.com/repos/hyperledger/fabric/issues/1701","repository_url":"https://api.github.com/repos/hyperledger/fabric","labels_url":"https://api.github.com/repos/hyperledger/fabric/issues/1701/labels{/name}","comments_url":"https://api.github.com/repos/hyperledger/fabric/issues/1701/comments","events_url":"https://api.github.com/repos/hyperledger/fabric/issues/1701/events","html_url":"https://github.com/hyperledger/fabric/issues/1701","id":158327941,"number":1701,"title":"Inconsistent state after writing transactions to multiple peers concurrently","user":{"login":"yingfeng","id":7248,"avatar_url":"https://avatars.githubusercontent.com/u/7248?v=3","gravatar_id":"","url":"https://api.github.com/users/yingfeng","html_url":"https://github.com/yingfeng","followers_url":"https://api.github.com/users/yingfeng/followers","following_url":"https://api.github.com/users/yingfeng/following{/other_user}","gists_url":"https://api.github.com/users/yingfeng/gists{/gist_id}","starred_url":"https://api.github.com/users/yingfeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yingfeng/subscriptions","organizations_url":"https://api.github.com/users/yingfeng/orgs","repos_url":"https://api.github.com/users/yingfeng/repos","events_url":"https://api.github.com/users/yingfeng/events{/privacy}","received_events_url":"https://api.github.com/users/yingfeng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-06-03T09:34:01Z","updated_at":"2016-06-06T14:50:43Z","closed_at":null,"body":"## Description\r\nAccording to the definition of consensus, each fabric peer should have a consistent state at all time, or else, the decentralized ledger could not work at all. However, after I setup a fabric network, and perform the stress test, I've found the state of all fabric peer nodes are inconsistent at all.\r\n\r\n## Describe How to Reproduce\r\n\r\n- Setup N fabric peer nodes, and I've tried N=2, N=3, N=4, each of the peer adopts the default consensus plugin---classic PBFT\r\n\r\n- Deploy a chaincode `chaincode_example02`:\r\n\r\n```\r\n{\r\n  \"jsonrpc\": \"2.0\",\r\n  \"method\": \"deploy\",\r\n  \"params\": {\r\n    \"type\": 1,\r\n    \"chaincodeID\":{\r\n        \"path\":\"github.com/hyperledger/fabric/examples/chaincode/go/chaincode_example02\"\r\n    },\r\n    \"ctorMsg\": {\r\n        \"function\":\"init\",\r\n        \"args\":[\"a\", \"1000\", \"b\", \"2000\"]\r\n    }\r\n  },\r\n  \"id\": 1\r\n}\r\n```\r\n\r\n- The initial state of each peer is consistent:\r\n\r\n```\r\n{\r\n  \"jsonrpc\": \"2.0\",\r\n  \"method\": \"query\",\r\n  \"params\": {\r\n    \"type\": 1,\r\n    \"chaincodeID\":\r\n    {\r\n        \"name\":\"7b07c59e9b9405c1aef33493b63b9a766d9bb836989ded1730052de650aa8ce5654274d148ceff96a4e5bd43bca26aba099f55c400e4befdc8b2ee4c0a94e30b\",\r\n        \"path\":\"github.com/hyperledger/fabric/examples/chaincode/go/chaincode_example02\"\r\n    },\r\n    \"ctorMsg\": {\r\n        \"function\":\"query\",\r\n        \"args\":[\"a\"]\r\n    }\r\n  },\r\n  \"id\": 5\r\n}\r\n```\r\n\r\nThe initial status for all peer nodes is:\r\n\r\n```\r\n{\"jsonrpc\":\"2.0\",\"result\":{\"status\":\"OK\",\"message\":\"1000\"}\r\n```\r\n\r\n- Perform stress invoking tests to two peer nodes concurrently, say peer2 and peer4:\r\n\r\n```\r\n{\r\n  \"jsonrpc\": \"2.0\",\r\n  \"method\": \"invoke\",\r\n  \"params\": {\r\n    \"type\": 1,\r\n    \"chaincodeID\":\r\n    {\r\n        \"name\":\"7b07c59e9b9405c1aef33493b63b9a766d9bb836989ded1730052de650aa8ce5654274d148ceff96a4e5bd43bca26aba099f55c400e4befdc8b2ee4c0a94e30b\",\r\n        \"path\":\"github.com/hyperledger/fabric/examples/chaincode/go/chaincode_example02\"\r\n    },\r\n    \"ctorMsg\": {\r\n        \"function\":\"approach\",\r\n        \"args\":[\"a\", \"1000\", \"b\", \"2000\"]\r\n    }\r\n  },\r\n  \"id\": 3\r\n}\r\n```\r\n\r\n- Stop the stress invoking test, wait for some time, and query each peer's status, then I've got 4 different result using the same query request as above:\r\n\r\n```\r\n{\"jsonrpc\":\"2.0\",\"result\":{\"status\":\"OK\",\"message\":\"-507530\"}\r\n{\"jsonrpc\":\"2.0\",\"result\":{\"status\":\"OK\",\"message\":\"-465020\"}\r\n{\"jsonrpc\":\"2.0\",\"result\":{\"status\":\"OK\",\"message\":\"-518570\"}\r\n{\"jsonrpc\":\"2.0\",\"result\":{\"status\":\"OK\",\"message\":\"-500030\"}\r\n```\r\n\r\nI could see some errors in the outputs of each peer:\r\n```\r\n09:13:04.063 [peer] handleChat -> ERRO 11f95 Error handling message: Message channel for name:\"vp1\"  full, rejecting\r\n09:13:04.063 [consensus/handler] HandleMessage -> ERRO 11f96 Failed to queue consensus message because: Message channel for name:\"vp1\"  full, rejecting\r\n09:13:04.063 [peer] handleChat -> ERRO 11f97 Error handling message: Message channel for name:\"vp1\"  full, rejecting\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/hyperledger/fabric/issues/comments/223982406","html_url":"https://github.com/hyperledger/fabric/issues/1701#issuecomment-223982406","issue_url":"https://api.github.com/repos/hyperledger/fabric/issues/1701","id":223982406,"user":{"login":"vukolic","id":11092575,"avatar_url":"https://avatars.githubusercontent.com/u/11092575?v=3","gravatar_id":"","url":"https://api.github.com/users/vukolic","html_url":"https://github.com/vukolic","followers_url":"https://api.github.com/users/vukolic/followers","following_url":"https://api.github.com/users/vukolic/following{/other_user}","gists_url":"https://api.github.com/users/vukolic/gists{/gist_id}","starred_url":"https://api.github.com/users/vukolic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vukolic/subscriptions","organizations_url":"https://api.github.com/users/vukolic/orgs","repos_url":"https://api.github.com/users/vukolic/repos","events_url":"https://api.github.com/users/vukolic/events{/privacy}","received_events_url":"https://api.github.com/users/vukolic/received_events","type":"User","site_admin":false},"created_at":"2016-06-06T14:50:43Z","updated_at":"2016-06-06T14:50:43Z","body":"What @2016Nishi got is expected behavior w/o proper flow control - requests may be dropped but not more than 1 response should be inconsistent. \r\n\r\nWhat @yingfeng reported is really worrysome (state inconsistency) - so since this issue was not reproduced yet - we still need logs from that run."}}
{"actor":{"display_login":"vukolic"}}