{"action":"created","issue":{"url":"https://api.github.com/repos/linkerd/linkerd/issues/1873","repository_url":"https://api.github.com/repos/linkerd/linkerd","labels_url":"https://api.github.com/repos/linkerd/linkerd/issues/1873/labels{/name}","comments_url":"https://api.github.com/repos/linkerd/linkerd/issues/1873/comments","events_url":"https://api.github.com/repos/linkerd/linkerd/issues/1873/events","html_url":"https://github.com/linkerd/linkerd/issues/1873","id":307096904,"number":1873,"title":"Log Spam and Namerd creating too many connections on consul (file descriptors)","user":{"login":"cchatfield","id":565433,"avatar_url":"https://avatars0.githubusercontent.com/u/565433?v=4","gravatar_id":"","url":"https://api.github.com/users/cchatfield","html_url":"https://github.com/cchatfield","followers_url":"https://api.github.com/users/cchatfield/followers","following_url":"https://api.github.com/users/cchatfield/following{/other_user}","gists_url":"https://api.github.com/users/cchatfield/gists{/gist_id}","starred_url":"https://api.github.com/users/cchatfield/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cchatfield/subscriptions","organizations_url":"https://api.github.com/users/cchatfield/orgs","repos_url":"https://api.github.com/users/cchatfield/repos","events_url":"https://api.github.com/users/cchatfield/events{/privacy}","received_events_url":"https://api.github.com/users/cchatfield/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-03-21T01:49:18Z","updated_at":"2018-03-22T19:04:06Z","closed_at":null,"author_association":"NONE","body":"Issue Type:\r\n\r\n- [X] Bug report\r\n- [ ] Feature request\r\n\r\n**What happened**:\r\nWith 1.3.6 (docker pull buoyantio/namerd) I am seeing the observation error spam in the log.\r\n\r\nI am also seeing namerd is creating thousands of established connections over time to consul ultimately leading consul to run out of file descriptors. It seemed with 1.3.5 to create about ~200 a day. With 1.3.6 it creates about 500 a day. This ultimately brings down consul and vault rendering an environment unroutable.\r\n\r\n**What you expected to happen**:\r\n\r\nLimited number of consistent open connections to consul\r\n\r\n\r\nWith 1.3.6 (docker pull buoyantio/namerd) I am seeing the observation error spam in the log.\r\n\r\n```\r\nW 0317 03:14:07.864 UTC THREAD28 TraceId:83b2534e4e7d2f12: consul datacenter 'dc1' service 'service-payments' observation error Failure(service observation released, flags=0x02) with RemoteInfo -> Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: consul.service.consul/10.10.1.4:8500, Downstream label: client, Trace Id: 27d1a3600be2b20b.27d1a3600be2b20b<:27d1a3600be2b20b. Last known state is Bound(Set(Inet(/10.12.1.5:31562,Map(endpoint_addr_weight -> 1.0))),Map(authority -> service-payments.service.dc1.consul))\r\nW 0317 03:14:12.869 UTC THREAD27 TraceId:83b2534e4e7d2f12: consul datacenter 'dc1' service 'service-payments' observation error Failure(service observation released, flags=0x02) with RemoteInfo -> Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: consul.service.consul/10.10.1.4:8500, Downstream label: client, Trace Id: b700ed57ab93668d.b700ed57ab93668d<:b700ed57ab93668d. Last known state is Bound(Set(Inet(/10.12.1.5:31562,Map(endpoint_addr_weight -> 1.0))),Map(authority -> service-payments.service.dc1.consul))\r\nW 0317 03:14:18.709 UTC THREAD27 TraceId:83b2534e4e7d2f12: consul datacenter 'dc1' service 'service-payments' observation error Failure(service observation released, flags=0x02) with RemoteInfo -> Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: consul.service.consul/10.10.1.4:8500, Downstream label: client, Trace Id: fa573f54f71166af.fa573f54f71166af<:fa573f54f71166af. Last known state is Bound(Set(Inet(/10.12.1.5:31562,Map(endpoint_addr_weight -> 1.0))),Map(authority -> service-payments.service.dc1.consul))\r\nW 0317 03:14:19.711 UTC THREAD27 TraceId:83b2534e4e7d2f12: consul datacenter 'dc1' service 'service-payments' observation error Failure(service observation released, flags=0x02) with RemoteInfo -> Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: consul.service.consul/10.10.1.4:8500, Downstream label: client, Trace Id: 5f11725cfcb18726.5f11725cfcb18726<:5f11725cfcb18726. Last known state is Bound(Set(Inet(/10.12.1.5:31562,Map(endpoint_addr_weight -> 1.0))),Map(authority -> service-payments.service.dc1.consul))\r\nW 0317 03:14:37.093 UTC THREAD27 TraceId:83b2534e4e7d2f12: consul datacenter 'dc1' service 'service-payments' observation error Failure(service observation released, flags=0x02) with RemoteInfo -> Upstream Address: Not Available, Upstream id: Not Available, Downstream Address: consul.service.consul/10.10.1.4:8500, Downstream label: client, Trace Id: 38ab7ee46a211cd5.38ab7ee46a211cd5<:38ab7ee46a211cd5. Last known state is Bound(Set(Inet(/10.12.1.5:31562,Map(endpoint_addr_weight -> 1.0))),Map(authority -> service-payments.service.dc1.consul))\r\n\r\n```\r\n\r\n\r\nSample lsof -p [consul pid]\r\n```\r\nconsul  51242 root  525u     IPv6          264999301      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:47658 (ESTABLISHED)\r\nconsul  51242 root  526u     IPv6          264999302      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:47652 (ESTABLISHED)\r\nconsul  51242 root  528u     IPv6          264998500      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:47666 (ESTABLISHED)\r\nconsul  51242 root  529u     IPv6          264998501      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:47670 (ESTABLISHED)\r\nconsul  51242 root  530u     IPv6          264998502      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:47674 (ESTABLISHED)\r\nconsul  51242 root  532u     IPv6          264971322      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:41378 (ESTABLISHED)\r\nconsul  51242 root  533u     IPv6          264998503      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:47676 (ESTABLISHED)\r\nconsul  51242 root  535u     IPv6          264988129      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:52014 (ESTABLISHED)\r\nconsul  51242 root  536u     IPv6          264998504      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:47672 (ESTABLISHED)\r\nconsul  51242 root  537u     IPv6          264998505      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:47668 (ESTABLISHED)\r\nconsul  51242 root  538u     IPv6          264998507      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:47678 (ESTABLISHED)\r\nconsul  51242 root  539u     IPv6          264998508      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:47680 (ESTABLISHED)\r\nconsul  51242 root  540u     IPv6          264998510      0t0       TCP non-prod-one-instance-consul-0.node.dc1.consul:fmtp->non-prod-two-instance-nomad-client-0.node.dc1.consul:47686 (ESTABLISHED)\r\n```\r\n\r\n\r\n\r\nI am temporarily resolving this by restarting the namerd container on a daily basis.\r\n\r\nConsul 1.0.2\r\nNamerd Config:\r\n\r\n```\r\nnamers:\r\n- kind: io.l5d.consul\r\n  host: consul.service.consul\r\n  port: 8500\r\n  includeTag: false\r\n  useHealthCheck: true\r\n  setHost: true\r\n  consistencyMode: stale\r\nadmin:\r\n  port: 9991\r\n  ip: 0.0.0.0\r\nstorage:\r\n  kind: io.l5d.consul\r\n  host: consul.service.consul\r\ninterfaces:\r\n- kind: io.l5d.thriftNameInterpreter\r\n- kind: io.l5d.httpController\r\n  ip: 0.0.0.0\r\ntelemetry:\r\n- kind: io.l5d.prometheus\r\n\r\n```\r\n\r\nDtabs\r\n```\r\n[\r\n    {\r\n        \"prefix\": \"/svc/eventstore\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/eventstore\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/eventstore.service.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/eventstore\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/eventstore.service.dc1.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/eventstore\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/couchdb\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/couchdb\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/couchdb.service.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/couchdb\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/couchdb.service.dc1.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/couchdb\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-account-sync\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-account-sync\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-account-sync.service.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-account-sync\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-account-sync.service.dc1.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-account-sync\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-notification\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-notification\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-notification.service.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-notification\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-notification.service.dc1.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-notification\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-payments\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-payments\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-payments.service.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-payments\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-payments.service.dc1.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-payments\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-account-key\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-account-key\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-account-key.service.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-account-key\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-account-key.service.dc1.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-account-key\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-account-storage\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-account-storage\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-account-storage.service.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-account-storage\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-account-storage.service.dc1.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-account-storage\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-aggregate-account\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-aggregate-account\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-aggregate-account.service.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-aggregate-account\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-aggregate-account.service.dc1.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-aggregate-account\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/tyk\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/tyk\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/tyk.service.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/tyk\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/tyk.service.dc1.consul\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/tyk\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-account-sync\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-account-sync\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-notification\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-notification\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-payments\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-payments\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-account-key\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-account-key\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-account-storage\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-account-storage\"\r\n    },\r\n    {\r\n        \"prefix\": \"/svc/redacted-service-aggregate-account\",\r\n        \"dst\": \"/#/io.l5d.consul/dc1/redacted-service-aggregate-account\"\r\n    }\r\n]\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/linkerd/linkerd/issues/comments/375422589","html_url":"https://github.com/linkerd/linkerd/issues/1873#issuecomment-375422589","issue_url":"https://api.github.com/repos/linkerd/linkerd/issues/1873","id":375422589,"user":{"login":"cchatfield","id":565433,"avatar_url":"https://avatars0.githubusercontent.com/u/565433?v=4","gravatar_id":"","url":"https://api.github.com/users/cchatfield","html_url":"https://github.com/cchatfield","followers_url":"https://api.github.com/users/cchatfield/followers","following_url":"https://api.github.com/users/cchatfield/following{/other_user}","gists_url":"https://api.github.com/users/cchatfield/gists{/gist_id}","starred_url":"https://api.github.com/users/cchatfield/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cchatfield/subscriptions","organizations_url":"https://api.github.com/users/cchatfield/orgs","repos_url":"https://api.github.com/users/cchatfield/repos","events_url":"https://api.github.com/users/cchatfield/events{/privacy}","received_events_url":"https://api.github.com/users/cchatfield/received_events","type":"User","site_admin":false},"created_at":"2018-03-22T19:04:06Z","updated_at":"2018-03-22T19:04:06Z","author_association":"NONE","body":"For most cases, the current environment is only generating sporadic traffic. Namerd can just sit there with connections from linkerd and create all of the extra connections.\r\n\r\nFor my environment, just starting up namerd consistently creates the connections. If there are certain captures or steps you would like me to do in the environment, I can. It's a non production environment.\r\n"}}
{"actor":{"display_login":"cchatfield"}}