{"action":"created","issue":{"url":"https://api.github.com/repos/CiscoCloud/mantl/issues/1195","repository_url":"https://api.github.com/repos/CiscoCloud/mantl","labels_url":"https://api.github.com/repos/CiscoCloud/mantl/issues/1195/labels{/name}","comments_url":"https://api.github.com/repos/CiscoCloud/mantl/issues/1195/comments","events_url":"https://api.github.com/repos/CiscoCloud/mantl/issues/1195/events","html_url":"https://github.com/CiscoCloud/mantl/issues/1195","id":137721104,"number":1195,"title":"elasticsearch-executor fails to load on a worker node.","user":{"login":"shali-mor","id":6791254,"avatar_url":"https://avatars.githubusercontent.com/u/6791254?v=3","gravatar_id":"","url":"https://api.github.com/users/shali-mor","html_url":"https://github.com/shali-mor","followers_url":"https://api.github.com/users/shali-mor/followers","following_url":"https://api.github.com/users/shali-mor/following{/other_user}","gists_url":"https://api.github.com/users/shali-mor/gists{/gist_id}","starred_url":"https://api.github.com/users/shali-mor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shali-mor/subscriptions","organizations_url":"https://api.github.com/users/shali-mor/orgs","repos_url":"https://api.github.com/users/shali-mor/repos","events_url":"https://api.github.com/users/shali-mor/events{/privacy}","received_events_url":"https://api.github.com/users/shali-mor/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/CiscoCloud/mantl/labels/addons/elk","name":"addons/elk","color":"fad8c7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2016-03-01T23:30:49Z","updated_at":"2016-03-03T17:28:00Z","closed_at":null,"body":"Having 3 controls , two worker and 1 edge machine, \r\nrunning addons/elk.yml ansible script , the elasticsearch executor fails to load :  (from stderr) \r\n```\r\nI0301 23:22:49.115537  9782 logging.cpp:172] INFO level logging started!\r\nI0301 23:22:49.115803  9782 fetcher.cpp:414] Fetcher Info: {\"cache_directory\":\"\\/tmp\\/mesos\\/fetch\\/slaves\\/3cb967f5-d1ee-475d-891f-be0efe0d50d6-S0\\/root\",\"items\":[{\"action\":\"BYPASS_CACHE\",\"uri\":{\"extract\":true,\"value\":\"http:\\/\\/mi-worker-001.cisco.com:56062\\/get\\/elasticsearch-mesos-executor.jar\"}}],\"sandbox_directory\":\"\\/var\\/lib\\/mesos\\/slaves\\/3cb967f5-d1ee-475d-891f-be0efe0d50d6-S0\\/frameworks\\/e1ea903b-12ca-4e7c-85e2-4210d02e5e2b-0000\\/executors\\/9d087794-db61-4c89-8f52-297e11abf77e\\/runs\\/e16beaeb-fa11-4c6c-983d-c77fe093c61d\",\"user\":\"root\"}\r\nI0301 23:22:49.117177  9782 fetcher.cpp:369] Fetching URI 'http://mi-worker-001.cisco.com:56062/get/elasticsearch-mesos-executor.jar'\r\nI0301 23:22:49.117192  9782 fetcher.cpp:243] Fetching directly into the sandbox directory\r\nI0301 23:22:49.117208  9782 fetcher.cpp:180] Fetching URI 'http://mi-worker-001.cisco.com:56062/get/elasticsearch-mesos-executor.jar'\r\nI0301 23:22:49.117221  9782 fetcher.cpp:127] Downloading resource from 'http://mi-worker-001.cisco.com:56062/get/elasticsearch-mesos-executor.jar' to '/var/lib/mesos/slaves/3cb967f5-d1ee-475d-891f-be0efe0d50d6-S0/frameworks/e1ea903b-12ca-4e7c-85e2-4210d02e5e2b-0000/executors/9d087794-db61-4c89-8f52-297e11abf77e/runs/e16beaeb-fa11-4c6c-983d-c77fe093c61d/elasticsearch-mesos-executor.jar'\r\nFailed to fetch 'http://mi-worker-001.cisco.com:56062/get/elasticsearch-mesos-executor.jar': Error downloading resource: Couldn't resolve host name\r\nFailed to synchronize with slave (it's probably exited)\r\n```\r\n\r\nfrom mesos : \r\n```\r\nelasticsearch_mi-worker-002.node.consul_20160301T232839.294Z\telasticsearch-executor\tFAILED\tjust now\tjust now\tmi-worker-002.node.consul\tSandbox\r\n```\r\nfrom marathon : \r\n```\r\nelastic search is running on worker1 only . \r\n```\r\n**\r\nplease note that the behaviour is consistence (tryed more that 5 times with exact same errors)  \r\n\r\n**"},"comment":{"url":"https://api.github.com/repos/CiscoCloud/mantl/issues/comments/191873291","html_url":"https://github.com/CiscoCloud/mantl/issues/1195#issuecomment-191873291","issue_url":"https://api.github.com/repos/CiscoCloud/mantl/issues/1195","id":191873291,"user":{"login":"ryane","id":7346,"avatar_url":"https://avatars.githubusercontent.com/u/7346?v=3","gravatar_id":"","url":"https://api.github.com/users/ryane","html_url":"https://github.com/ryane","followers_url":"https://api.github.com/users/ryane/followers","following_url":"https://api.github.com/users/ryane/following{/other_user}","gists_url":"https://api.github.com/users/ryane/gists{/gist_id}","starred_url":"https://api.github.com/users/ryane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryane/subscriptions","organizations_url":"https://api.github.com/users/ryane/orgs","repos_url":"https://api.github.com/users/ryane/repos","events_url":"https://api.github.com/users/ryane/events{/privacy}","received_events_url":"https://api.github.com/users/ryane/received_events","type":"User","site_admin":false},"created_at":"2016-03-03T17:28:00Z","updated_at":"2016-03-03T17:28:00Z","body":"As far as I can tell, the root of the problem is that your hostnames are not resolvable - the framework seems to depend on that. Are you on openstack? I think you have to tweak your setup so you don't end up with hostnames like *.cisco.com and instead get hostnames that are internally resolvable. But, I don't really know a lot about that."}}