{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/nomad/issues/1641","repository_url":"https://api.github.com/repos/hashicorp/nomad","labels_url":"https://api.github.com/repos/hashicorp/nomad/issues/1641/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/nomad/issues/1641/comments","events_url":"https://api.github.com/repos/hashicorp/nomad/issues/1641/events","html_url":"https://github.com/hashicorp/nomad/issues/1641","id":172531306,"number":1641,"title":"clients flap after upgrading to 0.4.1 from 0.3.2","user":{"login":"ketzacoatl","id":10122937,"avatar_url":"https://avatars.githubusercontent.com/u/10122937?v=3","gravatar_id":"","url":"https://api.github.com/users/ketzacoatl","html_url":"https://github.com/ketzacoatl","followers_url":"https://api.github.com/users/ketzacoatl/followers","following_url":"https://api.github.com/users/ketzacoatl/following{/other_user}","gists_url":"https://api.github.com/users/ketzacoatl/gists{/gist_id}","starred_url":"https://api.github.com/users/ketzacoatl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ketzacoatl/subscriptions","organizations_url":"https://api.github.com/users/ketzacoatl/orgs","repos_url":"https://api.github.com/users/ketzacoatl/repos","events_url":"https://api.github.com/users/ketzacoatl/events{/privacy}","received_events_url":"https://api.github.com/users/ketzacoatl/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hashicorp/nomad/labels/client","name":"client","color":"bfdadc"},{"url":"https://api.github.com/repos/hashicorp/nomad/labels/needs_investigation","name":"needs_investigation","color":"d93f0b"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/hashicorp/nomad/milestones/9","html_url":"https://github.com/hashicorp/nomad/milestones/v0.5.0","labels_url":"https://api.github.com/repos/hashicorp/nomad/milestones/9/labels","id":1952077,"number":9,"title":"v0.5.0","description":"","creator":{"login":"dadgar","id":1047914,"avatar_url":"https://avatars.githubusercontent.com/u/1047914?v=3","gravatar_id":"","url":"https://api.github.com/users/dadgar","html_url":"https://github.com/dadgar","followers_url":"https://api.github.com/users/dadgar/followers","following_url":"https://api.github.com/users/dadgar/following{/other_user}","gists_url":"https://api.github.com/users/dadgar/gists{/gist_id}","starred_url":"https://api.github.com/users/dadgar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dadgar/subscriptions","organizations_url":"https://api.github.com/users/dadgar/orgs","repos_url":"https://api.github.com/users/dadgar/repos","events_url":"https://api.github.com/users/dadgar/events{/privacy}","received_events_url":"https://api.github.com/users/dadgar/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":9,"state":"open","created_at":"2016-08-19T18:10:20Z","updated_at":"2016-09-27T23:50:02Z","due_on":null,"closed_at":null},"comments":29,"created_at":"2016-08-22T19:14:52Z","updated_at":"2016-09-29T23:05:01Z","closed_at":"2016-09-27T23:50:02Z","body":"I have Terraform modules to deploy consul/nomad servers and clients as auto-scaling groups on AWS. When I run nomad 0.3.2 with consul 0.6.4, and my own form of auto-discovery for the nomad servers and clients, nomad is stable and has run well. After upgrading to 0.4.1, I updated the config templates I have, to ensure nomad would use its native auto-discovery thru consul (and to disable my previous form of auto-discovery with consul). With that fiddling, I found the nomad servers to be stable, and reliably boot and form their quorum. However, I have been unable to get the clients to stabilize, I see them flap up and down (status from `down` to `ready` and back).\r\n\r\n### Nomad version\r\n\r\n`0.4.1`\r\n\r\n\r\n### Operating system and Environment details\r\n\r\nLinux Ubuntu 14.04  AMD64, `3.13.0-74-generic`, consul `0.6.4`, upstart, nomad linux executable (not docker).\r\n\r\n\r\n### Issue\r\n\r\nI expect the nomad clients to auto-discover their servers through consul, and I see log messages from the consul agent to confirm that is happening:\r\n\r\n```\r\n    2016/08/21 17:14:10.892388 [WARN] client.rpcproxy: No servers available\r\n    2016/08/21 17:14:10.892500 [WARN] client.rpcproxy: No servers available\r\n    2016/08/21 17:14:10.892749 [ERR] client: failed to query for node allocations: no known servers\r\n    2016/08/21 17:14:10.924250 [DEBUG] client.consul: bootstrap contacting following Consul DCs: [\"us-east-1\"]\r\n    2016/08/21 17:14:10.930018 [DEBUG] client.consul: bootstrap adding following Servers: [\"10.10.10.13:4647\" \"10.10.10.12:4647\" \"10.10.11.6:4647\"]\r\n    2016/08/21 17:14:15.290166 [DEBUG] client.consul: bootstrap contacting following Consul DCs: [\"us-east-1\"]\r\n    2016/08/21 17:14:15.297537 [DEBUG] client.consul: bootstrap adding following Servers: [\"10.10.10.13:4647\" \"10.10.10.12:4647\" \"10.10.11.6:4647\"]\r\n    2016/08/21 17:14:16.501042 [DEBUG] http: Request /v1/agent/servers (557.928µs)\r\n    2016/08/21 17:14:20.228296 [DEBUG] client.consul: bootstrap contacting following Consul DCs: [\"us-east-1\"]\r\n    2016/08/21 17:14:20.233438 [DEBUG] client.consul: bootstrap adding following Servers: [\"10.10.10.13:4647\" \"10.10.10.12:4647\" \"10.10.11.6:4647\"]\r\n    2016/08/21 17:14:24.998500 [DEBUG] client.consul: bootstrap contacting following Consul DCs: [\"us-east-1\"]\r\n    2016/08/21 17:14:25.003561 [DEBUG] client.consul: bootstrap adding following Servers: [\"10.10.10.13:4647\" \"10.10.10.12:4647\" \"10.10.11.6:4647\"]\r\n    2016/08/21 17:14:26.503302 [DEBUG] http: Request /v1/agent/servers (1.293321ms)\r\n    2016/08/21 17:14:29.488031 [DEBUG] client.consul: bootstrap contacting following Consul DCs: [\"us-east-1\"]\r\n    2016/08/21 17:14:29.492944 [DEBUG] client.consul: bootstrap adding following Servers: [\"10.10.10.13:4647\" \"10.10.10.12:4647\" \"10.10.11.6:4647\"]\r\n    2016/08/21 17:14:31.446417 [DEBUG] client: node registration complete\r\n```\r\n\r\nAt that point, the client can report out about jobs:\r\n\r\n```\r\nclient# nomad status\r\nNo running jobs\r\n```\r\n\r\nand the node can be seen on a nomad server:\r\n\r\n```\r\nserver# nomad node-status\r\nID      DC             Name    Class   Drain  Status\r\n537...  foo.us-east-1  ...     <none>  false  ready\r\nc05...  foo.us-east-1  ...     <none>  false  down\r\neb1...  foo.us-east-1  ...     <none>  false  ready\r\n9b6...  foo.us-east-1  ...     <none>  false  ready\r\n9fc...  foo.us-east-1  ...     <none>  false  ready\r\nc3f...  foo.us-east-1  ...     <none>  false  down\r\n834...  foo.us-east-1  ...     <none>  false  ready\r\n11d...  foo.us-east-1  ...     <none>  false  down\r\n```\r\n\r\n(these nodes ^ are flapping, with some up and some down, eventually they'll all be down for a while, before coming back online for short bits of time)\r\n\r\nEventually, the client goes `down`, and `nomad status` looks like:\r\n\r\n```\r\nclient# nomad status\r\nError querying jobs: Unexpected response code: 500 (no known servers)\r\n```\r\n\r\nand the client logs says:\r\n\r\n```\r\n    2016/08/22 18:52:43.457042 [WARN] client.rpcproxy: No servers available\r\n    2016/08/22 18:52:43.457201 [ERR] client: heartbeating failed: failed to update status: no known servers\r\n    2016/08/22 18:52:50.894438 [DEBUG] http: Request /v1/agent/servers (78.711µs)\r\n    2016/08/22 18:53:00.895614 [DEBUG] http: Request /v1/agent/servers (234.545µs)\r\n    2016/08/22 18:53:07.242744 [WARN] client.rpcproxy: No servers available\r\n    2016/08/22 18:53:07.242820 [ERR] http: Request /v1/jobs, error: no known servers\r\n```\r\n\r\nIf I wait a little while, the client will re-register to the servers auto-discovered via consul (same logs as shown above), and `nomad status` will work. The clients seem to be up for 30 seconds or so, and down for about the same, or maybe a little longer. Eventually, you'll catch them all `down`:\r\n\r\n```\r\nID      DC             Name     Class   Drain  Status\r\n537...  foo.us-east-1  ...      <none>  false  down\r\nc05...  foo.us-east-1  ...      <none>  false  down\r\neb1...  foo.us-east-1  ...      <none>  false  down\r\n9b6...  foo.us-east-1  ...      <none>  false  down\r\n9fc...  foo.us-east-1  ...      <none>  false  down\r\nc3f...  foo.us-east-1  ...      <none>  false  down\r\n834...  foo.us-east-1  ...      <none>  false  down\r\n11d...  foo.us-east-1  ...      <none>  false  down\r\n```\r\n\r\nBut then they'll start re-registering.\r\n\r\n\r\n### Reproduction steps\r\n\r\n1) deploy nomad to a cluster of servers, 3 nodes, using the config included below\r\n2) deploy a cluster running the nomad client, using the config included below\r\n3) tail the logs and watch the client / servers each, they will auto-discover and then fail\r\n\r\nI am running the executable on the host directly, with ubuntu 14.04 and upstart:\r\n\r\n```\r\nrespawn\r\nchdir /var/lib/nomad\r\nscript\r\n  exec /usr/local/bin/nomad agent -client -config /etc/nomad/config.json -config /etc/nomad/conf.d/\r\nend script\r\n```\r\n\r\n\r\n### Nomad Server config\r\n\r\n```\r\n{\r\n  \"addresses\": {\"serf\": \"10.10.11.6\",\"rpc\": \"10.10.11.6\",\r\n    \"http\": \"10.10.11.6\"\r\n  },\r\n  \"datacenter\": \"foo.us-east-1\",\r\n  \"data_dir\": \"/var/lib/nomad/tmp/\",\r\n  \"enable_debug\": false,\r\n  \"enable_syslog\": true,\r\n  \"log_level\": \"DEBUG\",\r\n  \"name\": \"server...\",\r\n  \"region\": \"us-east-1\",\r\n  \"syslog_facility\": \"LOCAL0\",\r\n  \"consul\": {\r\n    \"address\": \"10.10.11.6:8500\",\r\n    \"auto_advertise\": true,\r\n    \"client_auto_join\": true,\r\n    \"server_auto_join\": true,\r\n    \"server_service_name\": \"nomad\",\r\n    \"client_service_name\": \"nomad-client\",\r\n    \"token\": \"FOOBAR\"\r\n  },\r\n  \"server\": {\r\n    \"bootstrap_expect\": 3,\r\n    \"retry_join\": [\r\n      \"nomad.service.consul\"\r\n      ],\r\n    \"retry_interval\": \"15s\",\r\n    \"enabled\": true \r\n  }\r\n}\r\n```\r\n\r\n### Nomad Client config\r\n\r\n```\r\n{\r\n  \"addresses\": {\r\n    \"http\": \"10.10.13.141\"\r\n  },\r\n  \"datacenter\": \"foo.us-east-1\",\r\n  \"data_dir\": \"/var/lib/nomad/tmp/\",\r\n  \"enable_debug\": false,\r\n  \"enable_syslog\": true,\r\n  \"log_level\": \"DEBUG\",\r\n  \"name\": \"client...\",\r\n  \"region\": \"us-east-1\",\r\n  \"syslog_facility\": \"LOCAL0\",\r\n  \"consul\": {\r\n    \"address\": \"127.0.0.1:8500\",\r\n    \"auto_advertise\": true,\r\n    \"client_auto_join\": true,\r\n    \"server_auto_join\": true,\r\n    \"server_service_name\": \"nomad\",\r\n    \"client_service_name\": \"nomad-client\",\r\n    \"token\": \"FOOBAR\"\r\n  },\r\n  \"client\": {\r\n    \"options\": {\r\n      \"driver.raw_exec.enable\": \"1\"\r\n    },\r\n    \"meta\": {\r\n      \"tier\": \"...\"\r\n    },\r\n    \"enabled\": true \r\n  }\r\n}\r\n```\r\n\r\n### Nomad Client logs\r\n\r\nHere is a client starting up, searching consul for nomad, finding nomad, registering with the servers, and then losing that connection and failing to find any servers:\r\n\r\n```\r\n    Loaded configuration from /etc/nomad/config.json\r\n==> Starting Nomad agent...\r\n==> Nomad agent configuration:\r\n\r\n                 Atlas: <disabled>\r\n                Client: true\r\n             Log Level: TRACE\r\n                Region: us-east-1 (DC: foo.us-east-1)\r\n                Server: false\r\n\r\n==> Nomad agent started! Log data will stream in below:\r\n\r\n    2016/08/21 17:14:10.876786 [INFO] client: using state directory /var/lib/nomad/tmp/client\r\n    2016/08/21 17:14:10.876879 [INFO] client: using alloc directory /var/lib/nomad/tmp/alloc\r\n    2016/08/21 17:14:10.876910 [DEBUG] client: built-in fingerprints: [arch cgroup cpu env_aws env_gce host memory network nomad storage]\r\n    2016/08/21 17:14:10.877085 [INFO] fingerprint.cgroups: cgroups are available\r\n    2016/08/21 17:14:10.877235 [DEBUG] fingerprint.cpu: frequency: 2400 MHz\r\n    2016/08/21 17:14:10.877258 [DEBUG] fingerprint.cpu: core count: 2\r\n    2016/08/21 17:14:10.877353 [DEBUG] client: fingerprinting cgroup every 15s\r\n    2016/08/21 17:14:10.887966 [DEBUG] fingerprint.env_gce: Could not read value for attribute \"machine-type\"\r\n    2016/08/21 17:14:10.889513 [DEBUG] fingerprint.network: Detected interface eth0 with IP 10.10.13.141 during fingerprinting\r\n    2016/08/21 17:14:10.890253 [WARN] fingerprint.network: Unable to parse Speed in output of '/sbin/ethtool eth0'\r\n    2016/08/21 17:14:10.890360 [DEBUG] fingerprint.network: Unable to read link speed from /sys/class/net/eth0/speed\r\n    2016/08/21 17:14:10.890418 [DEBUG] fingerprint.network: Unable to read link speed; setting to default 100\r\n    2016/08/21 17:14:10.891269 [DEBUG] client: applied fingerprints [arch cgroup cpu env_aws host memory network nomad storage]\r\n    2016/08/21 17:14:10.891387 [DEBUG] driver.docker: using client connection initialized from environment\r\n    2016/08/21 17:14:10.892040 [DEBUG] driver.exec: exec driver is enabled\r\n    2016/08/21 17:14:10.892143 [DEBUG] client: available drivers [docker exec]\r\n    2016/08/21 17:14:10.892282 [DEBUG] client: fingerprinting docker every 15s\r\n    2016/08/21 17:14:10.892339 [DEBUG] client: fingerprinting exec every 15s\r\n    2016/08/21 17:14:10.892388 [WARN] client.rpcproxy: No servers available\r\n    2016/08/21 17:14:10.892500 [WARN] client.rpcproxy: No servers available\r\n    2016/08/21 17:14:10.892749 [ERR] client: failed to query for node allocations: no known servers\r\n    2016/08/21 17:14:10.924250 [DEBUG] client.consul: bootstrap contacting following Consul DCs: [\"us-east-1\"]\r\n    2016/08/21 17:14:10.930018 [DEBUG] client.consul: bootstrap adding following Servers: [\"10.10.10.13:4647\" \"10.10.10.12:4647\" \"10.10.11.6:4647\"]\r\n    2016/08/21 17:14:15.290166 [DEBUG] client.consul: bootstrap contacting following Consul DCs: [\"us-east-1\"]\r\n    2016/08/21 17:14:15.297537 [DEBUG] client.consul: bootstrap adding following Servers: [\"10.10.10.13:4647\" \"10.10.10.12:4647\" \"10.10.11.6:4647\"]\r\n    2016/08/21 17:14:16.501042 [DEBUG] http: Request /v1/agent/servers (557.928µs)\r\n    2016/08/21 17:14:20.228296 [DEBUG] client.consul: bootstrap contacting following Consul DCs: [\"us-east-1\"]\r\n    2016/08/21 17:14:20.233438 [DEBUG] client.consul: bootstrap adding following Servers: [\"10.10.10.13:4647\" \"10.10.10.12:4647\" \"10.10.11.6:4647\"]\r\n    2016/08/21 17:14:24.998500 [DEBUG] client.consul: bootstrap contacting following Consul DCs: [\"us-east-1\"]\r\n    2016/08/21 17:14:25.003561 [DEBUG] client.consul: bootstrap adding following Servers: [\"10.10.10.13:4647\" \"10.10.10.12:4647\" \"10.10.11.6:4647\"]\r\n    2016/08/21 17:14:26.503302 [DEBUG] http: Request /v1/agent/servers (1.293321ms)\r\n    2016/08/21 17:14:29.488031 [DEBUG] client.consul: bootstrap contacting following Consul DCs: [\"us-east-1\"]\r\n    2016/08/21 17:14:29.492944 [DEBUG] client.consul: bootstrap adding following Servers: [\"10.10.10.13:4647\" \"10.10.10.12:4647\" \"10.10.11.6:4647\"]\r\n    2016/08/21 17:14:31.446417 [DEBUG] client: node registration complete\r\n    2016/08/21 17:14:31.446500 [DEBUG] client: periodically checking for node changes at duration 5s\r\n    2016/08/21 17:14:34.233183 [DEBUG] client.consul: bootstrap contacting following Consul DCs: [\"us-east-1\"]\r\n    2016/08/21 17:14:34.237880 [DEBUG] client.consul: bootstrap adding following Servers: [\"10.10.10.13:4647\" \"10.10.10.12:4647\" \"10.10.11.6:4647\"]\r\n    2016/08/21 17:14:36.504826 [DEBUG] http: Request /v1/agent/servers (285.277µs)\r\n    2016/08/21 17:14:37.782610 [DEBUG] client: state updated to ready\r\n    2016/08/21 17:14:46.506000 [DEBUG] http: Request /v1/agent/servers (94.024µs)\r\n    2016/08/21 17:14:47.878397 [WARN] client.rpcproxy: No servers available\r\n    2016/08/21 17:14:47.878540 [ERR] client: heartbeating failed: failed to update status: no known servers\r\n    2016/08/21 17:14:49.652303 [WARN] client.rpcproxy: No servers available\r\n    2016/08/21 17:14:49.652400 [ERR] http: Request /v1/jobs, error: no known servers\r\n    2016/08/21 17:14:49.652665 [DEBUG] http: Request /v1/jobs (366.805µs)\r\n    2016/08/21 17:14:56.507070 [DEBUG] http: Request /v1/agent/servers (276.932µs)\r\n    2016/08/21 17:14:56.785218 [WARN] client.rpcproxy: No servers available\r\n    2016/08/21 17:14:56.785287 [ERR] client: failed to query for node allocations: no known servers\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/hashicorp/nomad/issues/comments/250617032","html_url":"https://github.com/hashicorp/nomad/issues/1641#issuecomment-250617032","issue_url":"https://api.github.com/repos/hashicorp/nomad/issues/1641","id":250617032,"user":{"login":"schmichael","id":113362,"avatar_url":"https://avatars.githubusercontent.com/u/113362?v=3","gravatar_id":"","url":"https://api.github.com/users/schmichael","html_url":"https://github.com/schmichael","followers_url":"https://api.github.com/users/schmichael/followers","following_url":"https://api.github.com/users/schmichael/following{/other_user}","gists_url":"https://api.github.com/users/schmichael/gists{/gist_id}","starred_url":"https://api.github.com/users/schmichael/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/schmichael/subscriptions","organizations_url":"https://api.github.com/users/schmichael/orgs","repos_url":"https://api.github.com/users/schmichael/repos","events_url":"https://api.github.com/users/schmichael/events{/privacy}","received_events_url":"https://api.github.com/users/schmichael/received_events","type":"User","site_admin":false},"created_at":"2016-09-29T23:05:00Z","updated_at":"2016-09-29T23:05:00Z","body":"Fantastic! 0.5 should be out in October."}}
{"actor":{"display_login":"schmichael"}}