{"action":"created","issue":{"url":"https://api.github.com/repos/github/orchestrator/issues/696","repository_url":"https://api.github.com/repos/github/orchestrator","labels_url":"https://api.github.com/repos/github/orchestrator/issues/696/labels{/name}","comments_url":"https://api.github.com/repos/github/orchestrator/issues/696/comments","events_url":"https://api.github.com/repos/github/orchestrator/issues/696/events","html_url":"https://github.com/github/orchestrator/issues/696","id":378196844,"node_id":"MDU6SXNzdWUzNzgxOTY4NDQ=","number":696,"title":"Strange martini logs","user":{"login":"mia0x75","id":5111356,"node_id":"MDQ6VXNlcjUxMTEzNTY=","avatar_url":"https://avatars0.githubusercontent.com/u/5111356?v=4","gravatar_id":"","url":"https://api.github.com/users/mia0x75","html_url":"https://github.com/mia0x75","followers_url":"https://api.github.com/users/mia0x75/followers","following_url":"https://api.github.com/users/mia0x75/following{/other_user}","gists_url":"https://api.github.com/users/mia0x75/gists{/gist_id}","starred_url":"https://api.github.com/users/mia0x75/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mia0x75/subscriptions","organizations_url":"https://api.github.com/users/mia0x75/orgs","repos_url":"https://api.github.com/users/mia0x75/repos","events_url":"https://api.github.com/users/mia0x75/events{/privacy}","received_events_url":"https://api.github.com/users/mia0x75/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-11-07T09:01:34Z","updated_at":"2018-11-11T14:35:01Z","closed_at":null,"author_association":"NONE","body":"Built up an orchestrator raft cluster with 134(leader)/135/136, then add mysql servers, open web page (135), it takes too long to finish the request, after check the log, I found these error logs\r\n\r\n```\r\n2018/11/07 16:37:39 [DEBUG] raft: Node 10.100.97.135:10008 updated peer set (2): [10.100.97.134:10008 10.100.97.135:10008 10.100.97.136:10008]\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124:56522\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n[martini] Started GET /web/cluster/alias/cs11 for 10.100.97.124, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135, 10.100.97.135\r\n.\r\n.\r\n.\r\n.\r\n.\r\n.\r\n.\r\n\r\n2018-11-07 16:38:29 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018/11/07 16:38:30 [DEBUG] raft-net: 10.100.97.135:10008 accepted connection from: 10.100.97.134:10662\r\n2018-11-07 16:38:30 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018-11-07 16:38:31 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018-11-07 16:38:32 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018/11/07 16:38:34 [ERR] raft-net: Failed to flush response: write tcp 10.100.97.135:10008->10.100.97.134:10642: write: broken pipe\r\n2018-11-07 16:38:34 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018-11-07 16:38:40 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018-11-07 16:38:40 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018-11-07 16:38:42 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018-11-07 16:38:42 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018-11-07 16:38:43 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018-11-07 16:38:44 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018-11-07 16:38:52 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018/11/07 16:38:56 [WARN] raft: Heartbeat timeout from \"\" reached, but leadership suspended. Will not enter Candidate mode\r\n2018/11/07 16:41:19 [ERR] raft: Failed to make RequestVote RPC to 10.100.97.134:10008: dial tcp 10.100.97.134:10008: i/o timeout\r\n2018/11/07 16:41:19 [ERR] raft: Failed to make RequestVote RPC to 10.100.97.136:10008: dial tcp 10.100.97.136:10008: i/o timeout\r\n2018-11-07 16:41:19 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018-11-07 16:41:19 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018/11/07 16:41:21 [DEBUG] raft-net: 10.100.97.135:10008 accepted connection from: 10.100.97.134:10716\r\n2018-11-07 16:41:22 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018/11/07 16:41:23 http: proxy error: context canceled\r\n2018/11/07 16:41:24 [DEBUG] raft: Votes needed: 2\r\n2018/11/07 16:41:24 [DEBUG] raft: Vote granted from 10.100.97.135:10008. Tally: 1\r\n2018-11-07 16:41:24 ERROR Heath test is failing for over 5 seconds. raft yielding\r\n2018-11-07 16:41:25 FATAL Node is unable to register health. Please check database connnectivity.\r\n```\r\n\r\nNotice: machine cs11 is not 124, the error log is very strange."},"comment":{"url":"https://api.github.com/repos/github/orchestrator/issues/comments/437675106","html_url":"https://github.com/github/orchestrator/issues/696#issuecomment-437675106","issue_url":"https://api.github.com/repos/github/orchestrator/issues/696","id":437675106,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzY3NTEwNg==","user":{"login":"mia0x75","id":5111356,"node_id":"MDQ6VXNlcjUxMTEzNTY=","avatar_url":"https://avatars0.githubusercontent.com/u/5111356?v=4","gravatar_id":"","url":"https://api.github.com/users/mia0x75","html_url":"https://github.com/mia0x75","followers_url":"https://api.github.com/users/mia0x75/followers","following_url":"https://api.github.com/users/mia0x75/following{/other_user}","gists_url":"https://api.github.com/users/mia0x75/gists{/gist_id}","starred_url":"https://api.github.com/users/mia0x75/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mia0x75/subscriptions","organizations_url":"https://api.github.com/users/mia0x75/orgs","repos_url":"https://api.github.com/users/mia0x75/repos","events_url":"https://api.github.com/users/mia0x75/events{/privacy}","received_events_url":"https://api.github.com/users/mia0x75/received_events","type":"User","site_admin":false},"created_at":"2018-11-11T14:35:01Z","updated_at":"2018-11-11T14:35:01Z","author_association":"NONE","body":"No idea, but it is martini log, somehow I guess."}}
{"actor":{"display_login":"mia0x75"}}