{"action":"created","issue":{"url":"https://api.github.com/repos/spujadas/elk-docker/issues/230","repository_url":"https://api.github.com/repos/spujadas/elk-docker","labels_url":"https://api.github.com/repos/spujadas/elk-docker/issues/230/labels{/name}","comments_url":"https://api.github.com/repos/spujadas/elk-docker/issues/230/comments","events_url":"https://api.github.com/repos/spujadas/elk-docker/issues/230/events","html_url":"https://github.com/spujadas/elk-docker/issues/230","id":367267083,"node_id":"MDU6SXNzdWUzNjcyNjcwODM=","number":230,"title":"version 641 start.sh's tail dies after 'bootstrap checks' with no errors","user":{"login":"tedder","id":97914,"node_id":"MDQ6VXNlcjk3OTE0","avatar_url":"https://avatars2.githubusercontent.com/u/97914?v=4","gravatar_id":"","url":"https://api.github.com/users/tedder","html_url":"https://github.com/tedder","followers_url":"https://api.github.com/users/tedder/followers","following_url":"https://api.github.com/users/tedder/following{/other_user}","gists_url":"https://api.github.com/users/tedder/gists{/gist_id}","starred_url":"https://api.github.com/users/tedder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tedder/subscriptions","organizations_url":"https://api.github.com/users/tedder/orgs","repos_url":"https://api.github.com/users/tedder/repos","events_url":"https://api.github.com/users/tedder/events{/privacy}","received_events_url":"https://api.github.com/users/tedder/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-10-05T15:44:22Z","updated_at":"2018-10-05T20:10:16Z","closed_at":null,"author_association":"NONE","body":"Hi sebp,\r\n\r\nI have this crazy situation where the elk-docker startup script exits, which kills the container, but upon closer inspection it would still be alive. Here are some logs.\r\n\r\n\r\n```\r\nwaiting for Elasticsearch to be up (30/30)\r\nCouln't start Elasticsearch. Exiting.\r\nElasticsearch log follows below.\r\n[2018-10-05T15:05:28,181][INFO ][o.e.n.Node               ] [] initializing ...\r\n...\r\n[2018-10-05T15:05:34,187][INFO ][o.e.p.PluginsService     ] [B-yQOpV] loaded module [x-pack-upgrade]\r\n[2018-10-05T15:05:34,188][INFO ][o.e.p.PluginsService     ] [B-yQOpV] loaded module [x-pack-watcher]\r\n[2018-10-05T15:05:34,190][INFO ][o.e.p.PluginsService     ] [B-yQOpV] no plugins loaded\r\n[2018-10-05T15:05:49,603][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/134] [Main.cc@109] controller (64 bit): Version 6.4.1 (Build 1df3104bc26648) Copyright (c) 2018 Elasticsearch BV\r\n[2018-10-05T15:05:50,936][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security\r\n[2018-10-05T15:05:51,628][INFO ][o.e.d.DiscoveryModule    ] [B-yQOpV] using discovery type [zen]\r\n[2018-10-05T15:05:54,406][INFO ][o.e.n.Node               ] [B-yQOpV] initialized\r\n[2018-10-05T15:05:54,408][INFO ][o.e.n.Node               ] [B-yQOpV] starting ...\r\n[2018-10-05T15:05:54,850][INFO ][o.e.t.TransportService   ] [B-yQOpV] publish_address {172.17.0.6:9300}, bound_addresses {0.0.0.0:9300}\r\n[2018-10-05T15:05:54,896][INFO ][o.e.b.BootstrapChecks    ] [B-yQOpV] bound or publishing to a non-loopback address, enforcing bootstrap checks\r\n```\r\n\r\nNow, if I start the container with a CMD of /bin/bash, then run start.sh, start.sh will exit on that same line. I can look at `ps` and see Elasticseach is still running, and I can tail the logs and see they continue:\r\n\r\n```\r\n[2018-10-05T15:21:26,963][INFO ][o.e.n.Node               ] [NfnF_Sk] starting ...\r\n[2018-10-05T15:21:27,399][INFO ][o.e.t.TransportService   ] [NfnF_Sk] publish_address {172.17.0.6:9300}, bound_addresses {0.0.0.0:9300}\r\n[2018-10-05T15:21:27,447][INFO ][o.e.b.BootstrapChecks    ] [NfnF_Sk] bound or publishing to a non-loopback address, enforcing bootstrap checks\r\n[2018-10-05T15:21:57,507][WARN ][o.e.n.Node               ] [NfnF_Sk] timed out while waiting for initial discovery state - timeout: 30s\r\n[2018-10-05T15:21:57,565][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [NfnF_Sk] publish_address {172.17.0.6:9200}, bound_addresses {0.0.0.0:9200}\r\n[2018-10-05T15:21:57,566][INFO ][o.e.n.Node               ] [NfnF_Sk] started\r\n[2018-10-05T15:22:00,862][INFO ][o.e.d.z.ZenDiscovery     ] [NfnF_Sk] failed to send join request to master ...\r\n```\r\n\r\n\r\nI thought that maybe there was something in the logs that was causing `tail` to exit, like some weird control character. `xxd` doesn't show anything:\r\n```\r\n00001470: 7373 2c20 656e 666f 7263 696e 6720 626f  ss, enforcing bo\r\n00001480: 6f74 7374 7261 7020 6368 6563 6b73 0a5b  otstrap checks.[\r\n00001490: 3230 3138 2d31 302d 3035 5431 353a 3330  2018-10-05T15:30\r\n000014a0: 3a32 332c 3039 345d 5b57 4152 4e20 5d5b  :23,094][WARN ][\r\n```\r\n\r\nI also thought that maybe the fd was changing, so I changed from `tail -f` to `tail -F`. No change, it still exits.\r\n\r\nI tried changing some settings to isolate the container (it was getting discovered by another node), and it leaves one more log entry before exiting:\r\n```\r\n[2018-10-05T15:37:08,209][INFO ][o.e.b.BootstrapChecks    ] [beebs_docker] bound or publishing to a non-loopback address, enforcing bootstrap checks\r\n[2018-10-05T15:37:11,275][WARN ][o.e.d.z.ZenDiscovery     ] [beebs_docker] not enough master nodes discovered during pinging (found [[Candidate{node={beebs_docker}{..}{..}{192.168.0.23}{192.168.0.23:9300}{ml.machine_memory=16474714112, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again\r\n```\r\n\r\nWhen I set minimum masters to 1, 'bootstrap' is the last message I see.\r\n\r\nChecking versions, `632` doesn't die, nor does `640`, so it seems specific to `641`. Perhaps you can add `642`? :)\r\n\r\nMy settings for max_map_count and others are correct, and the fact that it continues outside of `start.sh` is proof (to me) it isn't that *type* of problem. I have no idea what it actually is, though. Any ideas?"},"comment":{"url":"https://api.github.com/repos/spujadas/elk-docker/issues/comments/427484906","html_url":"https://github.com/spujadas/elk-docker/issues/230#issuecomment-427484906","issue_url":"https://api.github.com/repos/spujadas/elk-docker/issues/230","id":427484906,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNzQ4NDkwNg==","user":{"login":"tedder","id":97914,"node_id":"MDQ6VXNlcjk3OTE0","avatar_url":"https://avatars2.githubusercontent.com/u/97914?v=4","gravatar_id":"","url":"https://api.github.com/users/tedder","html_url":"https://github.com/tedder","followers_url":"https://api.github.com/users/tedder/followers","following_url":"https://api.github.com/users/tedder/following{/other_user}","gists_url":"https://api.github.com/users/tedder/gists{/gist_id}","starred_url":"https://api.github.com/users/tedder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tedder/subscriptions","organizations_url":"https://api.github.com/users/tedder/orgs","repos_url":"https://api.github.com/users/tedder/repos","events_url":"https://api.github.com/users/tedder/events{/privacy}","received_events_url":"https://api.github.com/users/tedder/received_events","type":"User","site_admin":false},"created_at":"2018-10-05T20:10:16Z","updated_at":"2018-10-05T20:10:16Z","author_association":"NONE","body":"a little extra time has helped:\r\n\r\n```\r\nwaiting for Elasticsearch to be up (31/180)\r\nwaiting for Elasticsearch to be up (32/180)\r\nWaiting for Elasticsearch cluster to respond (1/30)\r\nlogstash started.\r\n * Starting Kibana5                                                                                              [ OK ]\r\n==> /var/log/elasticsearch/elasticsearch.log <==\r\n[2018-10-05T20:04:09,007][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [beebs_docker] Failed to clear cache for realms [[]]\r\n[2018-10-05T20:04:09,224][INFO ][o.e.g.GatewayService     ] [beebs_docker] recovered [0] indices into cluster_state\r\n[2018-10-05T20:04:10,075][INFO ][o.e.c.m.MetaDataIndexTemplateService] [beebs_docker] adding template [.watch-history-9] for index patterns [.watcher-history-9*]\r\n```\r\n\r\nWith kibana and logstash enabled, but ports \"hiding\" the instance, 640 takes 30-31 seconds 'to be up', 641 takes 31-32 seconds. That might explain why I've had difficulty reproducing it at times- it isn't actually a problem with 641.\r\n\r\nTesting a few starts of 642 now."}}
{"actor":{"display_login":"tedder"}}