{"action":"created","issue":{"url":"https://api.github.com/repos/linkerd/linkerd/issues/1958","repository_url":"https://api.github.com/repos/linkerd/linkerd","labels_url":"https://api.github.com/repos/linkerd/linkerd/issues/1958/labels{/name}","comments_url":"https://api.github.com/repos/linkerd/linkerd/issues/1958/comments","events_url":"https://api.github.com/repos/linkerd/linkerd/issues/1958/events","html_url":"https://github.com/linkerd/linkerd/issues/1958","id":324704511,"node_id":"MDU6SXNzdWUzMjQ3MDQ1MTE=","number":1958,"title":"Traffic to gRPC endpoints gradually stop being routed","user":{"login":"yoitsro","id":3006852,"node_id":"MDQ6VXNlcjMwMDY4NTI=","avatar_url":"https://avatars1.githubusercontent.com/u/3006852?v=4","gravatar_id":"","url":"https://api.github.com/users/yoitsro","html_url":"https://github.com/yoitsro","followers_url":"https://api.github.com/users/yoitsro/followers","following_url":"https://api.github.com/users/yoitsro/following{/other_user}","gists_url":"https://api.github.com/users/yoitsro/gists{/gist_id}","starred_url":"https://api.github.com/users/yoitsro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yoitsro/subscriptions","organizations_url":"https://api.github.com/users/yoitsro/orgs","repos_url":"https://api.github.com/users/yoitsro/repos","events_url":"https://api.github.com/users/yoitsro/events{/privacy}","received_events_url":"https://api.github.com/users/yoitsro/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":30,"created_at":"2018-05-20T12:09:22Z","updated_at":"2018-06-04T21:46:49Z","closed_at":null,"author_association":"NONE","body":"Issue Type:\r\n\r\n- [x ] Bug report\r\n- [ ] Feature request\r\n\r\n**What happened**:\r\nWith linkerd being used as an ingress controller for gRPC, the daemonset gradually stops routing gRPC requests.\r\n\r\n**What you expected to happen**:\r\nLinkerd should not stop routing gRPC requests.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\nOur setup:\r\n3 node Kubernetes cluster running linkerd in a daemonset.\r\n\r\n**Anything else we need to know?**:\r\nStats for my daemonset over the past 7 days\r\n![image](https://user-images.githubusercontent.com/3006852/40278591-be461f38-5c2b-11e8-88e2-fb8c23d5fa13.png)\r\n\r\nThere is currently nothing collecting the telemetry data.\r\n\r\n**Environment**:\r\n- linkerd/namerd version, config files:\r\n\r\nThis configuration is based on the concept of having an internal namespace and a public namespace. The public namespace contains publicly accessible microservices which talk to the internal microservices using linkerd as the mesh to link them all.\r\n\r\n<details>\r\n<summary>Configuration</summary>\r\n<p>\r\n\r\n```\r\nadmin:\r\n  ip: 0.0.0.0\r\n  port: \r\n\r\nnamers:\r\n- kind: io.l5d.k8s\r\n  experimental: true\r\n  host: localhost\r\n  port: 8001\r\n\r\ntelemetry:\r\n- kind: io.l5d.prometheus\r\n- kind: io.l5d.recentRequests\r\n  sampleRate: 0.25\r\n\r\nusage:\r\n  orgId: l5d-linkerd\r\n\r\nrouters:\r\n- protocol: h2\r\n  label: out-public\r\n  experimental: true\r\n  dtab: |\r\n    /srv        => /#/io.l5d.k8s/production-public/grpc;\r\n    /grpc       => /srv;\r\n    /svc        => /$/io.buoyant.http.domainToPathPfx/grpc;\r\n  identifier:\r\n    kind: io.l5d.header.path\r\n    segments: 1\r\n  interpreter:\r\n    kind: default\r\n    transformers:\r\n    - kind: io.l5d.k8s.daemonset\r\n      namespace: production-l5d\r\n      port: in-public\r\n      service: \"l5d-linkerd-internal\"\r\n  servers:\r\n  - port: 4140\r\n    ip: 0.0.0.0\r\n    clearContext: true\r\n    tls:\r\n      certPath: /io.buoyant/linkerd/certs/tls.crt\r\n      keyPath: /io.buoyant/linkerd/certs/tls.key\r\n\r\n- protocol: h2\r\n  label: in-public\r\n  experimental: true\r\n  dtab: |\r\n    /srv        => /#/io.l5d.k8s/production-public/grpc;\r\n    /grpc       => /srv;\r\n    /svc        => /$/io.buoyant.http.domainToPathPfx/grpc;\r\n  identifier:\r\n    kind: io.l5d.header.path\r\n    segments: 1\r\n  interpreter:\r\n    kind: default\r\n    transformers:\r\n    - kind: io.l5d.k8s.localnode\r\n  servers:\r\n  - port: 4141\r\n    ip: 0.0.0.0\r\n\r\n- protocol: h2\r\n  label: out-internal\r\n  experimental: true\r\n  dtab: |\r\n    /srv        => /#/io.l5d.k8s/production-internal/grpc;\r\n    /grpc       => /srv;\r\n    /svc        => /$/io.buoyant.http.domainToPathPfx/grpc;\r\n  identifier:\r\n    kind: io.l5d.header.path\r\n    segments: 1\r\n  interpreter:\r\n    kind: default\r\n    transformers:\r\n    - kind: io.l5d.k8s.daemonset\r\n      namespace: production-l5d\r\n      port: in-internal\r\n      service: \"l5d-linkerd-internal\"\r\n  servers:\r\n  - port: 4142\r\n    ip: 0.0.0.0\r\n    clearContext: true\r\n\r\n- protocol: h2\r\n  label: in-internal\r\n  experimental: true\r\n  dtab: |\r\n    /srv        => /#/io.l5d.k8s/production-internal/grpc;\r\n    /grpc       => /srv;\r\n    /svc        => /$/io.buoyant.http.domainToPathPfx/grpc;\r\n  identifier:\r\n    kind: io.l5d.header.path\r\n    segments: 1\r\n  interpreter:\r\n    kind: default\r\n    transformers:\r\n    - kind: io.l5d.k8s.localnode\r\n  servers:\r\n  - port: 4143\r\n    ip: 0.0.0.0\r\n\r\n- protocol: http\r\n  label: http-out-int\r\n  dtab: |\r\n    /srv        => /#/io.l5d.k8s/production-internal/http;\r\n    /host       => /srv;\r\n    /svc        => /host;\r\n  interpreter:\r\n    kind: default\r\n    transformers:\r\n    - kind: io.l5d.k8s.daemonset\r\n      namespace: production-l5d\r\n      port: http-in-int\r\n      service: \"l5d-linkerd-internal\"\r\n  servers:\r\n  - port: 4240\r\n    ip: 0.0.0.0\r\n    clearContext: true\r\n\r\n- protocol: http\r\n  label: http-in-int\r\n  interpreter:\r\n    kind: default\r\n    transformers:\r\n    - kind: io.l5d.k8s.localnode\r\n  dtab: |\r\n    /srv        => /#/io.l5d.k8s/production-internal/http;\r\n    /host       => /srv;\r\n    /svc        => /host;\r\n  servers:\r\n  - port: 4241\r\n    ip: 0.0.0.0\r\n```\r\n</p>\r\n</details>\r\n\r\n- Platform, version, and config files (Kubernetes, DC/OS, etc):\r\nKubernetes v1.9.6\r\nLinkerd 1.3.7\r\n\r\n- Cloud provider or hardware configuration:\r\nGoogle Container Engine\r\n\r\n- Related issues:\r\nhttps://github.com/linkerd/linkerd/issues/1730#issuecomment-386816067\r\nhttps://github.com/linkerd/linkerd/issues/1582"},"comment":{"url":"https://api.github.com/repos/linkerd/linkerd/issues/comments/394510592","html_url":"https://github.com/linkerd/linkerd/issues/1958#issuecomment-394510592","issue_url":"https://api.github.com/repos/linkerd/linkerd/issues/1958","id":394510592,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NDUxMDU5Mg==","user":{"login":"yoitsro","id":3006852,"node_id":"MDQ6VXNlcjMwMDY4NTI=","avatar_url":"https://avatars1.githubusercontent.com/u/3006852?v=4","gravatar_id":"","url":"https://api.github.com/users/yoitsro","html_url":"https://github.com/yoitsro","followers_url":"https://api.github.com/users/yoitsro/followers","following_url":"https://api.github.com/users/yoitsro/following{/other_user}","gists_url":"https://api.github.com/users/yoitsro/gists{/gist_id}","starred_url":"https://api.github.com/users/yoitsro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yoitsro/subscriptions","organizations_url":"https://api.github.com/users/yoitsro/orgs","repos_url":"https://api.github.com/users/yoitsro/repos","events_url":"https://api.github.com/users/yoitsro/events{/privacy}","received_events_url":"https://api.github.com/users/yoitsro/received_events","type":"User","site_admin":false},"created_at":"2018-06-04T21:46:49Z","updated_at":"2018-06-04T21:46:49Z","author_association":"NONE","body":"I'm not certain with these specific logs, but I've just been back to a time when I can remember linkerd failing, and there are none of the logs complaining about not being able to reach the API server."}}
{"actor":{"display_login":"yoitsro"}}