{"action":"opened","issue":{"url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/2404","repository_url":"https://api.github.com/repos/Graylog2/graylog2-server","labels_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/2404/labels{/name}","comments_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/2404/comments","events_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/2404/events","html_url":"https://github.com/Graylog2/graylog2-server/issues/2404","id":161819606,"number":2404,"title":"After long GCs, getting NodePingThread - Did not find meta info of this node. Re-registering.","user":{"login":"gitfrederic","id":5217211,"avatar_url":"https://avatars.githubusercontent.com/u/5217211?v=3","gravatar_id":"","url":"https://api.github.com/users/gitfrederic","html_url":"https://github.com/gitfrederic","followers_url":"https://api.github.com/users/gitfrederic/followers","following_url":"https://api.github.com/users/gitfrederic/following{/other_user}","gists_url":"https://api.github.com/users/gitfrederic/gists{/gist_id}","starred_url":"https://api.github.com/users/gitfrederic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gitfrederic/subscriptions","organizations_url":"https://api.github.com/users/gitfrederic/orgs","repos_url":"https://api.github.com/users/gitfrederic/repos","events_url":"https://api.github.com/users/gitfrederic/events{/privacy}","received_events_url":"https://api.github.com/users/gitfrederic/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-06-23T01:07:23Z","updated_at":"2016-06-23T01:07:23Z","closed_at":null,"body":"\r\nWe're noticing that frequently when Graylog Server has a GC event (as reported in the logs by 'org.elasticsearch.monitor.jvm'), it is followed by a \"org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.\" warning.  For example:\r\n\r\n```shell\r\n00:53:35.097 [elasticsearch[graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d][scheduler][T#1]] WARN  org.elasticsearch.monitor.jvm - [graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d] [gc][young][175582][14187] duration [2.1s], collections [1]/[2.9s], total [2.1s]/[51.6m], memory [10.4gb]->[8.1gb]/[11.7gb], all_pools {[young] [2.3gb]->[17.7mb]/[2.4gb]}{[survivor] [5.4mb]->[5.2mb]/[307.1mb]}{[old] [8gb]->[8gb]/[9gb]}\r\n\r\n00:53:35.108 [scheduled-daemon-25] WARN  org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.\r\n```\r\n\r\nAside from trying to minimize GC durations, is there a way to increase the timeouts when Graylog Server queries MongoDb (or does Graylog rely on the MongoDB URI parameters for such things)?\r\n\r\n(Contiguous) Graylog Server log showing this occurring:\r\n\r\n```shell\r\n00:26:56.345 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 2 ms.\r\n00:28:10.946 [elasticsearch[graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d][scheduler][T#1]] WARN  org.elasticsearch.monitor.jvm - [graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d] [gc][young][174069][14180] duration [1.8s], collections [1]/[2.1s], total [1.8s]/[51.4m], memory [10.4gb]->[8.1gb]/[11.7gb], all_pools {[young] [2.3gb]->[12.7mb]/[2.4gb]}{[survivor] [44.3mb]->[40mb]/[307.1mb]}{[old] [8gb]->[8gb]/[9gb]}\r\n00:29:16.280 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 2 ms.\r\n00:31:34.100 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 2 ms.\r\n00:31:57.789 [elasticsearch[graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d][scheduler][T#1]] WARN  org.elasticsearch.monitor.jvm - [graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d] [gc][young][174294][14181] duration [2.1s], collections [1]/[2.8s], total [2.1s]/[51.4m], memory [10.5gb]->[8.1gb]/[11.7gb], all_pools {[young] [2.3gb]->[2.6mb]/[2.4gb]}{[survivor] [40mb]->[35.3mb]/[307.1mb]}{[old] [8gb]->[8gb]/[9gb]}\r\n00:33:47.866 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 2 ms.\r\n00:35:22.462 [elasticsearch[graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d][scheduler][T#1]] WARN  org.elasticsearch.monitor.jvm - [graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d] [gc][young][174497][14182] duration [1.9s], collections [1]/[2.6s], total [1.9s]/[51.5m], memory [10.4gb]->[8.1gb]/[11.7gb], all_pools {[young] [2.3gb]->[10.2mb]/[2.4gb]}{[survivor] [35.3mb]->[28mb]/[307.1mb]}{[old] [8gb]->[8gb]/[9gb]}\r\n00:35:22.467 [scheduled-daemon-26] WARN  org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.\r\n00:35:22.474 [rest-worker-11] WARN  org.graylog2.shared.rest.resources.ProxiedResource - Node <a0a3fb6e-ac36-48ad-bc10-e901d185228d> not found while trying to call org.graylog2.shared.rest.resources.system.RemoteMetricsResource on it.\r\n00:35:59.633 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 2 ms.\r\n00:38:14.099 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 2 ms.\r\n00:39:08.664 [elasticsearch[graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d][scheduler][T#1]] WARN  org.elasticsearch.monitor.jvm - [graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d] [gc][young][174722][14183] duration [2s], collections [1]/[2.1s], total [2s]/[51.5m], memory [10.5gb]->[8.1gb]/[11.7gb], all_pools {[young] [2.3gb]->[10mb]/[2.4gb]}{[survivor] [28mb]->[14.3mb]/[307.1mb]}{[old] [8gb]->[8gb]/[9gb]}\r\n00:39:08.671 [scheduled-daemon-4] WARN  org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.\r\n00:40:28.676 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 2 ms.\r\n00:42:41.242 [elasticsearch[graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d][scheduler][T#1]] WARN  org.elasticsearch.monitor.jvm - [graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d] [gc][young][174933][14184] duration [1.9s], collections [1]/[2.5s], total [1.9s]/[51.5m], memory [10.4gb]->[8.1gb]/[11.7gb], all_pools {[young] [2.3gb]->[6.1mb]/[2.4gb]}{[survivor] [14.3mb]->[11mb]/[307.1mb]}{[old] [8gb]->[8gb]/[9gb]}\r\n00:42:41.250 [scheduled-daemon-20] WARN  org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.\r\n00:42:44.588 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 3 ms.\r\n00:45:02.086 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 2 ms.\r\n00:46:24.988 [elasticsearch[graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d][scheduler][T#1]] WARN  org.elasticsearch.monitor.jvm - [graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d] [gc][young][175155][14185] duration [2.1s], collections [1]/[2.6s], total [2.1s]/[51.6m], memory [10.4gb]->[8.1gb]/[11.7gb], all_pools {[young] [2.3gb]->[27.1mb]/[2.4gb]}{[survivor] [11mb]->[5.7mb]/[307.1mb]}{[old] [8gb]->[8gb]/[9gb]}\r\n00:47:13.684 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 3 ms.\r\n00:49:22.110 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 6 ms.\r\n00:49:52.068 [elasticsearch[graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d][scheduler][T#1]] WARN  org.elasticsearch.monitor.jvm - [graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d] [gc][young][175361][14186] duration [1.9s], collections [1]/[2s], total [1.9s]/[51.6m], memory [10.4gb]->[8.1gb]/[11.7gb], all_pools {[young] [2.3gb]->[21.5mb]/[2.4gb]}{[survivor] [5.7mb]->[5.4mb]/[307.1mb]}{[old] [8gb]->[8gb]/[9gb]}\r\n00:49:52.075 [scheduled-daemon-1] WARN  org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.\r\n00:51:33.840 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 2 ms.\r\n00:53:35.097 [elasticsearch[graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d][scheduler][T#1]] WARN  org.elasticsearch.monitor.jvm - [graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d] [gc][young][175582][14187] duration [2.1s], collections [1]/[2.9s], total [2.1s]/[51.6m], memory [10.4gb]->[8.1gb]/[11.7gb], all_pools {[young] [2.3gb]->[17.7mb]/[2.4gb]}{[survivor] [5.4mb]->[5.2mb]/[307.1mb]}{[old] [8gb]->[8gb]/[9gb]}\r\n00:53:35.108 [scheduled-daemon-25] WARN  org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.\r\n00:53:48.543 [inputbufferprocessor-2] INFO  kafka.log.Log - Rolled new log segment for 'messagejournal-0' in 1 ms.\r\n```\r\n\r\nVersion info, etc.:\r\n```\r\n23:55:42.304 [main] INFO  org.elasticsearch.node - [graylog-a0a3fb6e-ac36-48ad-bc10-e901d185228d] version[2.3.2], pid[28391], build[b9e4a6a/2016-04-21T16:03:47Z]\r\n...\r\n23:55:47.575 [main] INFO  org.graylog2.bootstrap.ServerBootstrap - Graylog server 2.0.2 (4da1379) starting up\r\n23:55:47.576 [main] INFO  org.graylog2.bootstrap.ServerBootstrap - JRE: Oracle Corporation 1.8.0_72 on Linux 3.10.0-123.el7.x86_64\r\n23:55:47.576 [main] INFO  org.graylog2.bootstrap.ServerBootstrap - Deployment: unknown\r\n23:55:47.576 [main] INFO  org.graylog2.bootstrap.ServerBootstrap - OS: CentOS Linux 7 (Core) (centos)\r\n23:55:47.576 [main] INFO  org.graylog2.bootstrap.ServerBootstrap - Arch: amd64\r\n```"}}
{"actor":{"display_login":"gitfrederic"}}