{"action":"created","issue":{"url":"https://api.github.com/repos/tikv/tikv/issues/3747","repository_url":"https://api.github.com/repos/tikv/tikv","labels_url":"https://api.github.com/repos/tikv/tikv/issues/3747/labels{/name}","comments_url":"https://api.github.com/repos/tikv/tikv/issues/3747/comments","events_url":"https://api.github.com/repos/tikv/tikv/issues/3747/events","html_url":"https://github.com/tikv/tikv/issues/3747","id":377901633,"node_id":"MDU6SXNzdWUzNzc5MDE2MzM=","number":3747,"title":" err send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\")) in upgrade ","user":{"login":"evancao77","id":30570337,"node_id":"MDQ6VXNlcjMwNTcwMzM3","avatar_url":"https://avatars2.githubusercontent.com/u/30570337?v=4","gravatar_id":"","url":"https://api.github.com/users/evancao77","html_url":"https://github.com/evancao77","followers_url":"https://api.github.com/users/evancao77/followers","following_url":"https://api.github.com/users/evancao77/following{/other_user}","gists_url":"https://api.github.com/users/evancao77/gists{/gist_id}","starred_url":"https://api.github.com/users/evancao77/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evancao77/subscriptions","organizations_url":"https://api.github.com/users/evancao77/orgs","repos_url":"https://api.github.com/users/evancao77/repos","events_url":"https://api.github.com/users/evancao77/events{/privacy}","received_events_url":"https://api.github.com/users/evancao77/received_events","type":"User","site_admin":false},"labels":[{"id":305526950,"node_id":"MDU6TGFiZWwzMDU1MjY5NTA=","url":"https://api.github.com/repos/tikv/tikv/labels/T:%20Question","name":"T: Question","color":"1d76db","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2018-11-06T15:35:01Z","updated_at":"2018-11-12T09:30:00Z","closed_at":null,"author_association":"NONE","body":"## Bug Report\r\n\r\n**What version of Rust are you using?**\r\n\r\n\r\n**What operating system and CPU are you using?**\r\n\r\n\r\n**What did you do?**\r\nthe problems in upgrading process,I upgrading  from v1.0.8-1-gaacba4a to 2.0.8.\r\n\r\n**What did you expect to see?**\r\nsuccessfull\r\n\r\n**What did you see instead?**\r\n\r\ntikv1：\r\n2018/11/06 23:27:10.285 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:27:10.285 transport.rs:262: [INFO] resolve store 4 address ok, addr 172.xxx.xxx.31:20160\r\n2018/11/06 23:27:10.367 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:27:10.439 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:27:10.470 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:27:11.293 raft_client.rs:92: [WARN] send raftmessage to 172.xxx.xxx.31:20160 failed: Grpc(RpcFinished(Some(RpcStatus { status: Cancelled, details: Some(\"Cancelled\") })))\r\n2018/11/06 23:27:11.293 raft_client.rs:92: [WARN] send raftmessage to 172.xxx.xxx.31:20160 failed: Grpc(RpcFinished(Some(RpcStatus { status: Cancelled, details: Some(\"Cancelled\") })))\r\n2018/11/06 23:27:11.294 raft_client.rs:92: [WARN] send raftmessage to 172.xxx.xxx.31:20160 failed: Grpc(RpcFinished(Some(RpcStatus { status: Cancelled, details: Some(\"Cancelled\") })))\r\n2018/11/06 23:27:11.323 transport.rs:262: [INFO] resolve store 4 address ok, addr 172.xxx.xxx.31:20160\r\n2018/11/06 23:27:11.323 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:27:11.323 transport.rs:262: [INFO] resolve store 4 address ok, addr 172.xxx.xxx.31:20160\r\n2018/11/06 23:27:11.325 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:27:11.328 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n\r\ntikv2:\r\n2018/11/06 23:28:15.096 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:15.096 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:15.096 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:15.096 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:15.096 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:15.096 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:15.096 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:15.292 apply.rs:902: [INFO] [region 416973] 416975 execute admin command cmd_type: CompactLog compact_log {compact_index: 3805384 compact_term: 74} at [term: 74, index: 3805386]\r\n2018/11/06 23:28:18.349 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.349 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.349 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.349 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.349 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.349 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.349 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.356 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.356 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.363 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.412 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.412 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.414 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.414 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n2018/11/06 23:28:18.415 kv.rs:1062: [ERROR] send raft msg to raft store fail: Transport(Discard(\"Failed to send Raft Message due to full\"))\r\n\r\n\r\ntikv3:\r\n2018/11/06 23:29:29.357 raft_client.rs:92: [WARN] send raftmessage to 172.xxx.xxx.31:20160 failed: Grpc(RpcFinished(Some(RpcStatus { status: Cancelled, details: Some(\"Cancelled\") })))\r\n2018/11/06 23:29:29.358 raft_client.rs:92: [WARN] send raftmessage to 172.xxx.xxx.31:20160 failed: Grpc(RpcFinished(Some(RpcStatus { status: Cancelled, details: Some(\"Cancelled\") })))\r\n2018/11/06 23:29:29.362 raft_client.rs:92: [WARN] send raftmessage to 172.xxx.xxx.31:20160 failed: Grpc(RpcFinished(Some(RpcStatus { status: Cancelled, details: Some(\"Cancelled\") })))\r\n2018/11/06 23:29:29.376 transport.rs:262: [INFO] resolve store 4 address ok, addr 172.xxx.xxx.31:20160\r\n2018/11/06 23:29:29.376 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:29:29.376 transport.rs:262: [INFO] resolve store 4 address ok, addr 172.xxx.xxx.31:20160\r\n2018/11/06 23:29:29.536 raft_client.rs:92: [WARN] send raftmessage to 172.xxx.xxx.31:20160 failed: Grpc(RpcFinished(Some(RpcStatus { status: Cancelled, details: Some(\"Cancelled\") })))\r\n2018/11/06 23:29:29.538 transport.rs:262: [INFO] resolve store 4 address ok, addr 172.xxx.xxx.31:20160\r\n2018/11/06 23:29:29.538 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:29:29.564 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:29:29.592 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:29:31.606 raft_client.rs:92: [WARN] send raftmessage to 172.xxx.xxx.31:20160 failed: Grpc(RpcFinished(Some(RpcStatus { status: Cancelled, details: Some(\"Cancelled\") })))\r\n2018/11/06 23:29:31.606 raft_client.rs:92: [WARN] send raftmessage to 172.xxx.xxx.31:20160 failed: Grpc(RpcFinished(Some(RpcStatus { status: Cancelled, details: Some(\"Cancelled\") })))\r\n2018/11/06 23:29:31.606 raft_client.rs:92: [WARN] send raftmessage to 172.xxx.xxx.31:20160 failed: Grpc(RpcFinished(Some(RpcStatus { status: Cancelled, details: Some(\"Cancelled\") })))\r\n2018/11/06 23:29:31.606 raft_client.rs:92: [WARN] send raftmessage to 172.xxx.xxx.31:20160 failed: Grpc(RpcFinished(Some(RpcStatus { status: Cancelled, details: Some(\"Cancelled\") })))\r\n2018/11/06 23:29:31.635 transport.rs:262: [INFO] resolve store 4 address ok, addr 172.xxx.xxx.31:20160\r\n2018/11/06 23:29:31.635 transport.rs:262: [INFO] resolve store 4 address ok, addr 172.xxx.xxx.31:20160\r\n2018/11/06 23:29:31.635 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:29:31.655 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:29:32.162 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n2018/11/06 23:29:32.163 raft_client.rs:54: [INFO] server: new connection with tikv endpoint: 172.xxx.xxx.31:20160\r\n"},"comment":{"url":"https://api.github.com/repos/tikv/tikv/issues/comments/437813015","html_url":"https://github.com/tikv/tikv/issues/3747#issuecomment-437813015","issue_url":"https://api.github.com/repos/tikv/tikv/issues/3747","id":437813015,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzgxMzAxNQ==","user":{"login":"DorianZheng","id":8065637,"node_id":"MDQ6VXNlcjgwNjU2Mzc=","avatar_url":"https://avatars0.githubusercontent.com/u/8065637?v=4","gravatar_id":"","url":"https://api.github.com/users/DorianZheng","html_url":"https://github.com/DorianZheng","followers_url":"https://api.github.com/users/DorianZheng/followers","following_url":"https://api.github.com/users/DorianZheng/following{/other_user}","gists_url":"https://api.github.com/users/DorianZheng/gists{/gist_id}","starred_url":"https://api.github.com/users/DorianZheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DorianZheng/subscriptions","organizations_url":"https://api.github.com/users/DorianZheng/orgs","repos_url":"https://api.github.com/users/DorianZheng/repos","events_url":"https://api.github.com/users/DorianZheng/events{/privacy}","received_events_url":"https://api.github.com/users/DorianZheng/received_events","type":"User","site_admin":false},"created_at":"2018-11-12T09:30:00Z","updated_at":"2018-11-12T09:30:00Z","author_association":"MEMBER","body":"@evancao77 RocksDB's log path is /path_to_your_db/data/db/LOG*"}}
{"actor":{"display_login":"DorianZheng"}}